GoKawiilAustralia-based AI consultant and founder of Agentic Labs Jesse Davies woke up to an unpleasant surprise earlier this month: A Google Cloud bill of $25,672.86 AUD (approximately $18,391.78 USD) — even though there was a budget of $10 AUD (approximately $7 USD) on his account. And it happened overnight.
According to Davies' account on LinkedIn, he was well-versed with Google AI Studio and had followed practices such as per-project API keys, separate billing accounts, two-factor authentication, and Cloud audit logging. However, it only took a single weak link to nullify those precautions, as evidenced by the shockingly large overnight bill. On top of that, Davies found nine Google Cloud safety features that should have prevented this incident — but that were turned off by default.
"The attacker didn't steal my key. They found a Cloud Run service I'd published from AI Studio months earlier, hit the public URL, and Google's own proxy signed every request on their behalf using the API key stored as a plaintext environment variable in the container," Davies wrote in his LinkedIn post.
"Even though it was public, the link wasn't shared or indexed anywhere. By the time I got a budget alert the next morning, A$10,000 had already been charged to my credit card, now getting insufficient funds. I was still talking to Google support when A$15,000 more came through."
What’s worse was that Google automatically upgraded the tier of Davies' account without any notification. The account was initially at Tier 2, which had a $2,000 limit, but Google automatically upgraded it to the next level when the account crossed the $1,000 threshold during the incident. This increased the cap to between $20,000 and $100,000. While this is likely designed to make it easier for a service to scale, it also has the unwanted effect of costing the user more than intended, e.g. if they are the victim of an attack.
Their headaches did not end here, though. It took several days before Davies was able to get through to a real human customer support. Thankfully, it seems that the charge has been waived, while the transactions that actually pushed through were credited back by their bank. Still, the issue isn’t settled, and Davies has a meeting scheduled with Google managers to talk about the case.
Article continues below
... continue reading