5 ms·
Google Cloud customer wakes up to $18,000 bill despite $7 budget
- ReptileMan 7mo agoThat is quite hostile to their consumers, no matter how they spin it. If you put a budget on something it should be capped.
- dpoloncsak 7mo agoI am the last person to defend Google Cloud and it's awful UX. With that said, when you go to set a budget it warns you "Setting a budget does not cap resource or API consumption. Learn more." with a hyperlink to https://docs.cloud.google.com/billing/docs/how-to/budgets?_gl=1*pbeau7*_ga*NDE1OTYzNzUwLjE3NzQ4OTEwNjU.*_ga_WH2QY8WWF5*czE3NzY4ODEwNTYkbzIkZzEkdDE3NzY4ODExMzIkajQ0JGwwJGgw https://docs.cloud.google.com/billing/docs/how-to/budgets?_g...
- Telaneo 7mo agoThe only way I can read that is 'setting a cap does nothing' but reading that tells me that it only turns on email notifications. Not any better really. It's simply not a cap. It's an alarm.
- sunaookami 7mo agoYes, there is no way to set a budget for Google Cloud. And alarms are delayed up to two hours (!)
- dpoloncsak 7mo agoAgreed. I don't agree with the underlying design decisions either, but you literally set a "Budget Alert" on Google Cloud. It's designed to be an alarm, not a cap. I was just trying to point that out
- deleted 7mo ago[deleted]
- danny_codes 7mo agoIMO it’s deceptive to market it as a cap. It’s genuinely confusing. The service page should read “we can charge you an unlimited amount at any time if you make a technical mistake”
- 7bit 7mo agoClick here to let the puppy life* * By clicking here you agree to kill it And you're defending that?
- dpoloncsak 7mo agoNo, I think Google should provide easy tools to actually cap spending, instead of recommending you set quota limits on your APIs. The article, and the comment I was replying to, make it seem like an error in the Google Budget system. I'm simply trying to say this system is working as designed and documented.
- ReptileMan 7mo agoI am fairly sure that this is antipattern on purpose. If you ask a thousand people on the street what a budget means - they will coalesce on - the money I am willing to spend on something. The fact that google redefine what budget means and put a warning doesn't make it ok.
- torvoborvo 7mo agoEven if US clouds will make more money per capita in the US it has more total income and political advantage from the other 95% of people. US law will be as pro-consumer as Delaware law. (And it will be as restricted to the jurisdiction that decides it as IP law.)
- stevenicr 7mo agoWhoa! What?! I recently turned a couple things on in google cloud and I was under the impression that the budget was a hard limit, not an email notify system. If this is true I must seriously rethink my plan for marketing, and throttle on my own website somehow.
- perryizgr8 7mo agoI think I read somewhere that calculating and limiting cloud usage costs is a really hard problem. But I feel that if Google were motivated to do it, they can do it. It's hard, not impossible. They just don't care to solve this particular problem.
- deleted 7mo ago[deleted]
- AlotOfReading 7mo agoIt's the same fundamental problem as view counters, something Google is famously good at solving. Eventually consistent solutions are well-understood, and wouldn't have these kinds of massive cost-overruns.
- lazide 7mo agoDepends on latency. 24 hour delays on an eventually consistent counter used for billing absolutely would cause this problem.
- moring 7mo agoIt seems hard to believe that a one-hour delay on such a counter is impossible to achieve, and one hour would reduce the risk from "catastrophic" to "serious problem" in most cases. Also, if implementing a cap is a desired feature that justifies trade-offs to be made, then it is psosible to translate the budget cap (in terms of money) back into service-specific caps that are easier to keep consistent. Such as "autoscale this set of VMs" and "my budget cap is $1000/hour", with the VM type being priced at $10/hour, translated to "autoscale to at most 100 instances". That would need dev work (i.e. this feature being considered important) and would not respect the budget cap in a cross-service way automatically, but still it is another piece in the puzzle.
- lazide 7mo agoEh, suddenly turning off all services in your account because you hit your cap is just as much a DoS type event - just of your services, not your wallet.
- victor106 7mo agoWhy doesn’t GCP provide a way to say “shut down all my services if my cap is reached”?
- brazukadev 7mo agoThey have a $18,000 reason for that.
- caminanteblanco 7mo agoWell since they waived the fees, it sounds to me like they have an $18,000 reason to stop this kind of thing from happening in the first place. I understand that $18k is probably a drop in the bucket, but surely there's a middle ground here.
- pixl97 7mo agoThey can wave 18k pretty easily because they'll make it up in 18,000 other overcharges of varying amounts that aren't contested. Or that are corporate clients that just eat it. Things like this are the exact reason that companies end up having to comply with all kinds of regulations. It's just easier to screw the customer first.
- estimator7292 7mo agoSame reason that you can't cancel a gym membership by walking up to a person and saying "cancel my membership". Predation, pure and simple. Google doesn't care, no one is holding them responsible for predatory behavior. It's profitable to steal from your customers and there's no downside to doing so.
- rcbdev 7mo agoWhere do you live that consumer protection is this bad? Of course you can cancel any gym membership at their physical location, as long as you can also sign up at these physical locations. What idiot lawmaker would allow a system where you can sign up without a computer, phone or fax, but then need these media to cancel what you just signed up for?
- ddtaylor 7mo agoSounds like a great mistake for Google to find a way to repeat. Why innovate when you can abuse users and hide behind "complexity" as "plausible deniability"? You'll all keep using them either way.
- robotswantdata 7mo agoDo yourself a favour and automate unlinking the billing account. Set it to fire when your budget is hit or whatever your risk tolerance is. Yes nuclear option, but I’ll take an hour down time over a $100k unexpected bill
- 0cf8612b2e1e 7mo agoYou still owe a debt, even with no card on file.
- gonzalohm 7mo agoAnd Google owes honoring the budget limits no?
- rcbdev 7mo agoIf Google really is stupid enough to send collectors for this, they have every right to be sued for obviously deceptive business practices.
- danny_codes 7mo agoDangerous litigation for Google to get into. I assume they’d prefer to settle out of court or comp you
- robotswantdata 7mo agoWhen disconnected all services immediately stop. So no big bills.
- Havoc 7mo agoBudget alerts can be up to 24 hrs delayed. It’s the pull out method of billing protection
- fhn 7mo ago"Google automatically upgraded the tier" this is a google scam. don't pay and sue them.
- rcbdev 7mo agoGoogle is in dire need of more consumer protection lawsuits.
- symisc_devel 7mo agoWe got a $12,000 bill a few hours ago on a presumably leaked gemini API which I very much doubt, and we are trying to resolve the issue with a real support agent. I think they messed something internally and customers are getting these bills.
- prirun 7mo agoI never got a surprise bill myself, but reading a few cases like this motivated me to cancel my GCS account and remove my CC. Now if I try to use it it fails immediately with an error. As author of HashBackup, I know people are using it with GCS, and I'd like to be able to test against it, but not enough to swallow a large surprise Google bill.
- Havoc 7mo agoGCP is turning into a circus. Right now there are about 4 concurrent threads on the googlecloud subreddit about people getting hosed with life changing bills. Some no doubt through stupid mistakes (happens), but still bewildering that Google is insisting individuals like students are subject to the same scale to infinity bills as huge corporations and are unwilling to provide any mechanism to protect hobbyists. And then people tell you but there are quotas and then: >Google automatically upgraded it to the next level when the account crossed the $1,000 threshold during the incident.
- hak8or 7mo agoLooks like another instance, I am dealing with a very similar issue. I didn't even notice any sort of tiers, as I had 17 grand in costs accumulate at a similar pace. I (a hobbyist running a small side project for a dollar or two a month in normal usage, so my account is marked as "individual") got hit with a ~$17,000 bill from Google cloud because some combination of key got leaked or my homelab got compromised, and the attacker consumed tens of thousands in gemini usage in only a few hours. It wasn't even the same Google project as for my project, it was another that hasn't seen activity in a year+. Google refuses to apply any adjustments, their billing specialist even mixed up my account with someone else, refuses to provide further information for why adjustments are being rejected, refuses any escalation, etc. I already filed a complaint with the FTC and NYS attorney General but the rep couldn't care any less. My gripe is not that the key was potentially leaked or compromised or similar and then I have to pay as a very expensive "you messed up" mistake, it's that they let an api key rack up tens of thousands in maybe 4 hours or so with usage patterns (model selection, generating text vs image, volume of calls, likely different IP and user agent and whatnot). That's just predatory behavior on an account marked as individual/consumer (not a business).
- danny_codes 7mo agoRecommend switching to Hezner, they have hard billing limits. This current cloud paradigm of charging indefinitely is insane for normal consumers and many businesses.
- abrookewood 7mo agoThis problem exists to some degree for all cloud services. Is there a solution based on the credit card used? Can you get a CC that has a hard limit guarantee? EDIT: I guess that you'd still be responsible for the charges though.
- danny_codes 7mo agoHezner is the solution, since you can set hard billing limits there.
- 0xCE0 7mo agoThe job of hyperscalers is to scale and accelerate bills.