8 ms·

> We had a budget alert (€80) and a cost anomaly alert, both of which triggered with a delay of a few hours > By the time we reacted, costs were already around

> We had a budget alert (€80) and a cost anomaly alert, both of which triggered with a delay of a few hours > By the time we reacted, costs were already around €28,000 > The final amount settled at €54,000+ due to delayed cost reporting So much for the folks defending these three companies that refused to provide hard spending cap ("but you can set the budget", "you are doing it wrong if you worry about billing", "hard cap it's technically impossible" etc.)

villgax 6mo ago
Shirky’s principle at work is all
varispeed 6mo ago
This is clearly setup for VC backed companies where shareholders don't care about spend as long as they can brag about investing in this cool start up at dinner parties. Normal and true business should stay away.
Maxious 6mo ago
> The Gemini API supports monthly spend caps at both the billing account tier and project levels. These controls are designed to protect your account from unexpected overages, and the ecosystem to ensure service availability https://ai.google.dev/gemini-api/docs/billing#project-spend-caps https://ai.google.dev/gemini-api/docs/billing#project-spend-...
- rtkwe 6mo ago
  The problem is it's specific to that API and defaults to uncapped so people who aren't using it and haven't heard about the issues with the Firebase API keys probably won't have set them.
  - zozbot234 6mo ago
    Except that Google's own statements are extremely clear that "leaked" (i.e. public) API keys should not be able to access the Gemini API in the first place: "We have identified a vulnerability where some API keys may have been publicly exposed. To protect your data and prevent unauthorized access, we have proactively blocked these known leaked keys from accessing the Gemini API. ... We are defaulting to blocking API keys that are leaked and used with the Gemini API, helping prevent abuse of cost and your application data." https://ai.google.dev/gemini-api/docs/troubleshooting#googles_security_measures_for_leaked_keys https://ai.google.dev/gemini-api/docs/troubleshooting#google... For extra clarity on the exact so-called "vulnerability" that Google identified, see: https://news.ycombinator.com/item?id=47156925 https://news.ycombinator.com/item?id=47156925 This describes the very issue where some API keys were public by design (used for client-side web access), so the term "leaked" should be read in that unusually broad sense. Firebase keys are obviously covered, since they're also public by design. (As for "Firebase AI Logic", it is explicitly very different: it's supposed to be implemented via a proxy service so the Gemini API key is never seen by the client: https://firebase.google.com/docs/ai-logic https://firebase.google.com/docs/ai-logic Clearly, just casually "enabling" something - which is what OP says they did! - should never result in abuse of cost on the scale OP describes.)
    - sillysaurusx 6mo ago
      There are other vectors, e.g. a compromised GCP key leading to $13k in Gemini charges (posted 3 days ago) https://www.reddit.com/r/googlecloud/comments/1sjzat3/api_key_compromised_13428_fraudulent_charges/ https://www.reddit.com/r/googlecloud/comments/1sjzat3/api_ke...
  - isoldex 6mo ago
    Spend caps exist for Gemini (Maxious linked them) - they just default to OFF. For an API that can bill four figures per hour, opt-in safety by default isn't a UX choice, it's a billing strategy
- whywhywhywhy 6mo ago
  Why is the default uncapped then other than the hopes of billing people who screw up or get exploited.
  - drfloyd51 6mo ago
    See also: Why is the default cap so low? I lost €78bojillion because my API stopped working.
    - jamespo 6mo ago
      Monitoring could pick this up in minutes rather than how long this took to discover
      - drfloyd51 6mo ago
        Monitoring could have detected overspend as well. My point is either choice, caps or no caps, has its cons.
    - Barbing 6mo ago
      Demand on-call phone numbers, autodial the entire company when it looks like they’re about to lose their first bojillion. No, you don't really have to give Google a bunch of phone numbers. The input box will also accept entry of the following text: “I'm a big stupid idiot, and when my API stops working, which it will, it will be all my fault and not Google's.”
  - logankilpatrick 6mo ago
    We have a bunch of different protections in place, every account has a billing account cap by default (see: https://ai.google.dev/gemini-api/docs/billing#tier-spend-caps https://ai.google.dev/gemini-api/docs/billing#tier-spend-cap...), in the addition to the ability to set more granular developer spend caps.
janandonly 6mo ago
Yet another good reason to use a pre-paid service. There are many to choose from now, like Openrouter.com, PPQ.ai, and routstr.com.
- adriand 6mo ago
  You mean openrouter.ai. And yes, on reading this blog post, I immediately reviewed my API keys in OpenRouter to make sure that they were capped. My prod key was capped at $20/day (phew!) but my dev key had no cap, which I just updated. What a horrible story.
  - theanonymousone 6mo ago
    But isn't OpenRouter anyway prepaid, meaning the most you lose is your current credit?
    - adriand 6mo ago
      You can set it to auto top up if it drops below a certain amount. If you do that, then it would definitely be wise to add a cap. They let you add daily/weekly caps, which is convenient.
nurettin 6mo ago
I'd buy the technically impossible angle. Even if you manage to get your microservices to synch every penny spent to your payment account at realtime (impossible) you still have to waiver the excess, losing some money every time someone goes past their quota.
- bartread 6mo ago
  Sure, but 80 -> 28,000 -> 54,000 is a hell of a lot of slippage. Trading platforms can guarantee a maximum slippage on stops, and often even offer guaranteed stops (with an attached premium), so I don’t see why Google and Firebase can’t do similar. The way it works at present is ridiculous.
  - zbentley 6mo ago
    Yep. And cloud providers could eat any slippage cost (enforcing, say, every 5 minutes by stopping service) without even a rounding error on their balance sheets. The fact that they don’t indicates that there’s no market reason to support small spenders who get mad about runaway overages, not that it’s technically or financially hard to do so.
  - nurettin 6mo ago
    > Trading platforms can guarantee a maximum slippage on stops Yeah no, physically impossible. If nobody is selling at that price, there is no guarantee your sell stop will execute near that price. They can sweep the market, find the best seller price and execute. There might be a costly way to do it with microservices as I indicated, but your example easily falls apart.
    - bartread 6mo ago
      Not impossible to do: they can hedge and/or absorb the cost, hence the premium. They usually also specify a (fairly large) minimum distance for such stops.
      - nurettin 6mo ago
ch0wn 6mo ago
This should be illegal. If a contractor your hired to swap out a tile on your bathroom floor billed you for remodelling your back garden, you would obviously have the legal right to refuse that.
- jubilanti 6mo ago
  Not if your contractor had you first sign a 15 page contract that commits you to whatever costs they dream up and requires forced arbitration by a corporate friendly firm when any dispute arises. Because that's somehow normal in today's tech world.
  - sdevonoes 6mo ago
    So if their TOS say they can also rape my cat, then I cannot do anything about it, right? Ridiculous
    - ctdinjeu4 6mo ago
      [dead]
    - jubilanti 6mo ago
      In jurisdictions where beastiality is legal, then yes, from the libertarian perspective, that's all freedom of contract, baby. I'm not defending either beastiality or libertarianism, but the logic is that you don't want the government deciding what two private entities can and can't freely agree to. We're pretty far from the Lochner era in the US, where even minimum wage laws were held to be unconstitutional violations of a very broad view of freedom to contract. But it is still a principle in most legal system.
  - wing-_-nuts 6mo ago
    Slightly OT, but I've always taken a dim view of this sort of thing for consumers because the parties are never at equal parity, either in ability to understand the legalese they're agreeing to, or the ability to seek alternatives. Legal contracts for consumers should be written at whatever the prevailing reading level is, and the government should step in the more monopolistic position a company is in. It infuriates me to no end how preferential government is towards corporations vs individuals.
Leomuck 6mo ago
That's actually crazy. So I can build a project I love, that does good, but somehow get in a situation where I'm accidentally paying 30.000€ (or 50.000€) to a big tech company? How is that fair? I mean yes, as a software engineer, you ought to reflect on all possible weaknesses, but there was a time when overlooking something meant something completely different than being down 30/50k. That is actually life-altering.
- sdevonoes 6mo ago
  It’s not fair. Google, Amazon, Microsoft… they have never played fairly. They will never do.
  - saidnooneever 6mo ago
    you cannot earn billions a year and not be cheating your users out of their money. its that simple. they dont care for people, otherwise they wouldnt be putting so much effort in making them poor.
    - _DeadFred_ 6mo ago
      What about their behavior makes you think they are a company that doesn't care for people? https://nypost.com/2026/04/15/business/amazon-warehouse-employee-dies-after-collapsing-on-the-job-as-colleagues-kept-working-report/ https://nypost.com/2026/04/15/business/amazon-warehouse-empl...
      - Leomuck 6mo ago
        Wtf. Just wtf.
- benoau 6mo ago
  Your kid can do this in a smartphone game designated suitable for children, heavily optimized to exacerbate the possibility, and depending on where you live they can just choose not to refund you. When the FTC went investigating a decade-ish ago they found Facebook saying the quiet parts out loud: it was all extremely deliberate.
startages 6mo ago
Yeah, that the main reason I never use services like Google Cloud if I don't have to, it's impossible to have a hard cap, and anyone pretending to be an expert, is just off. Google says that they can't provide a hard cap because that would mean shutting down all your services..bla bla, but at least give users the option.
- logankilpatrick 6mo ago
  We have spend caps at the billing account level and the project level (developer set) in the Gemini API now. There is up to a 10 minute delay in processing everything but this should significantly mitigate the risk here: https://ai.google.dev/gemini-api/docs/billing#tier-spend-caps https://ai.google.dev/gemini-api/docs/billing#tier-spend-cap... By default, new Tier 1 paid accounts can only spend $250 in a given month.
  - flippyhead 6mo ago
    I'm sure it's me being an idiot, but once again I spent 20m trying to figure how to do a specific thing in google-land and still haven't figured it out. Even if I did set it somewhere, I see things like "Setting a budget does not cap resource or API consumption" with a link to a bunch of documentation I have to analyze.
    - bdashdash 6mo ago
      This is what working with cloud services is like, in my experience. Azure's UI feels like it was made as a joke flash game on Newgrounds.
  - not_your_vase 6mo ago
    I just find it extraordinary that the biggest tech company in the world can do cutting edge real time AI for millions of people, run Youtube and of course all the other google services with having literally the smartest people in the world and unlimited resources on board, but still can't keep real time track of the user's current billing and their spending limits, it's all best effort still. Somehow it doesn't add up. (Pun not intended, but I'm happy to have it)
reaperducer 6mo ago
hard cap it's technically impossible These companies can sell your personal information in a microsecond in an advertising auction, but somehow can't figure out how to give you timely alerts that stop their cash flow. Big shock.
TrackerFF 6mo ago
It's like a fire alarm system that goes off 30 mins after the it senses a fire. Good stuff.
- QuercusMax 6mo ago
  I was selling a house in a state I no longer lived in, and was under contract to close the sale, when I got an email from the water company. It told me they suspected based on my water usage that there was a leak on the properly. There had been a very cold February night (like -15F) and a pipe froze inside the walls, and it was just absolutely gushing out. They sent me the email after it had been leaking for a WEEK. I asked a friend to check it out and she said that the laminate floor went "squish" when she stepped in the front door. Fortunately I was covered by homeowner's insurance since I could prove that my heat had been on, but that was a very unpleasant "warning" to receive!
- pfortuny 6mo ago
  But sending an email, not blaring...
harrouet 6mo ago
As a manager I avoid Google Cloud for this kind of customer-service disasters; but as someone who has dealt with large-scale billing systems in the telecom world, probably similar to that of Google Cloud, I am not surprised that it takes 10 minutes to consolidate all the usage logs of a customer for billing. For telephony, it sometimes takes days when roaming is involved. You have to imagine TB/sec of data, if not more, coming from thousand of potential sources, and queuing for aggregation to the proper company account, all having to be auditable. This is not a small engineering feat and it can't be real-time. With that said, telcos usually include in their business model around 2-3% of bad debt (i.e. revenue that won't get paid), which accounts for frauds like this one. Given that the customer seems in good faith and has taken measures upon being notified, Google should manage this bill shock a bit more elegantly. Moreover, the fact that this happened immediately after this key opened the AI gates means that pirates permanently scan for the permissions of all the keys they could gathers. Google could and should detect that and act upon it.
sofixa 6mo ago
> So much for the folks defending these three companies that refused to provide hard spending cap ("but you can set the budget", "you are doing it wrong if you worry about billing", "hard cap it's technically impossible" etc.) Yes, it's technically+business impossible. To implement a hard cap, a bill never to go over, they'd have to cut your service, but also delete all your data in databases, object storage, data lake, etc. This is simply not an option, so they take the different option of authorising support to wave surprise surcharges / billing DDoSes.
- plorkyeran 6mo ago
  You can have a hard cap on compute spend while letting storage go over. Surprise huge bills are approximately never due to storage.
- benterix 6mo ago
  This argument simply doesn't hold water - their (smaller) competition solved this problem over a decade ago.
- Glemllksdf 6mo ago
  Srsly? The mighty cloud provider can't solve this issue? Google has second precision billing on compute. Its not hard to define a base layer of allowed billing increase and adding this type of context to resource allocation. You are not just suddenly creating a mlllion terabytes of data or a million db requests without supervision. It could even be as simple as basic level caps like 100 euro / month, 1000, 10.000 etc. And there is a difference between stoping everything before the spike happens vs. also deleting stuff.
aleksiy123 6mo ago
For everyone here, you should be able to use quotas in order to set actual limits, you can lower them yourself as a practical solution. https://docs.cloud.google.com/docs/quotas/view-manage https://docs.cloud.google.com/docs/quotas/view-manage Quotas are real time or near real time. Real time spend limits are probably never going to happen. Actual $ amounts are calculated by a centralized billing system offline in batch. It sounds easy but it’s bonkers complicated, because of things like discounts, free tiers, committed usage, currency conversions and having to support every payment and deal structure in GCP. Individual eng teams rarely actually think in dollar amounts, they think in the abstraction which is quotas.
nhuser2221 6mo ago
Happened to me to, luckily only was 40$, restricted the api the next day. They were using gemini 3 flash which I am not using.
- Moosdijk 6mo ago
  That’s users, for ya! They will always find ways to use your product that you didn’t intend to or even knew about!
  - nhuser2221 6mo ago
    The funny thing is that the website only has firebase auth, without any ai features. The default api key that was created (before the ai was even released a few years back), someone got it from the website and started using the gemini api with the key.
Moosdijk 6mo ago
Yet my phone service provider is able to cut off my internet access from the Kb I go over the limit…
fireant 6mo ago
"hard cap it's technically impossible" is really funny because every other provider manages it just fine. Even wrappers like OpenRouter enable you to set a hard cap on proxied Google resources, but Google themselves are unable to manage that inside gcloud.

villgax 6mo ago

Shirky’s principle at work is all

varispeed 6mo ago

This is clearly setup for VC backed companies where shareholders don't care about spend as long as they can brag about investing in this cool start up at dinner parties. Normal and true business should stay away.

Maxious 6mo ago

> The Gemini API supports monthly spend caps at both the billing account tier and project levels. These controls are designed to protect your account from unexpected overages, and the ecosystem to ensure service availability https://ai.google.dev/gemini-api/docs/billing#project-spend-caps https://ai.google.dev/gemini-api/docs/billing#project-spend-...

rtkwe 6mo ago
The problem is it's specific to that API and defaults to uncapped so people who aren't using it and haven't heard about the issues with the Firebase API keys probably won't have set them.
- zozbot234 6mo ago
  Except that Google's own statements are extremely clear that "leaked" (i.e. public) API keys should not be able to access the Gemini API in the first place: "We have identified a vulnerability where some API keys may have been publicly exposed. To protect your data and prevent unauthorized access, we have proactively blocked these known leaked keys from accessing the Gemini API. ... We are defaulting to blocking API keys that are leaked and used with the Gemini API, helping prevent abuse of cost and your application data." https://ai.google.dev/gemini-api/docs/troubleshooting#googles_security_measures_for_leaked_keys https://ai.google.dev/gemini-api/docs/troubleshooting#google... For extra clarity on the exact so-called "vulnerability" that Google identified, see: https://news.ycombinator.com/item?id=47156925 https://news.ycombinator.com/item?id=47156925 This describes the very issue where some API keys were public by design (used for client-side web access), so the term "leaked" should be read in that unusually broad sense. Firebase keys are obviously covered, since they're also public by design. (As for "Firebase AI Logic", it is explicitly very different: it's supposed to be implemented via a proxy service so the Gemini API key is never seen by the client: https://firebase.google.com/docs/ai-logic https://firebase.google.com/docs/ai-logic Clearly, just casually "enabling" something - which is what OP says they did! - should never result in abuse of cost on the scale OP describes.)
  - sillysaurusx 6mo ago
    There are other vectors, e.g. a compromised GCP key leading to $13k in Gemini charges (posted 3 days ago) https://www.reddit.com/r/googlecloud/comments/1sjzat3/api_key_compromised_13428_fraudulent_charges/ https://www.reddit.com/r/googlecloud/comments/1sjzat3/api_ke...
- isoldex 6mo ago
  Spend caps exist for Gemini (Maxious linked them) - they just default to OFF. For an API that can bill four figures per hour, opt-in safety by default isn't a UX choice, it's a billing strategy
whywhywhywhy 6mo ago
Why is the default uncapped then other than the hopes of billing people who screw up or get exploited.
- drfloyd51 6mo ago
  See also: Why is the default cap so low? I lost €78bojillion because my API stopped working.
  - jamespo 6mo ago
    Monitoring could pick this up in minutes rather than how long this took to discover
    - drfloyd51 6mo ago
      Monitoring could have detected overspend as well. My point is either choice, caps or no caps, has its cons.
  - Barbing 6mo ago
    Demand on-call phone numbers, autodial the entire company when it looks like they’re about to lose their first bojillion. No, you don't really have to give Google a bunch of phone numbers. The input box will also accept entry of the following text: “I'm a big stupid idiot, and when my API stops working, which it will, it will be all my fault and not Google's.”
- logankilpatrick 6mo ago
  We have a bunch of different protections in place, every account has a billing account cap by default (see: https://ai.google.dev/gemini-api/docs/billing#tier-spend-caps https://ai.google.dev/gemini-api/docs/billing#tier-spend-cap...), in the addition to the ability to set more granular developer spend caps.

janandonly 6mo ago

Yet another good reason to use a pre-paid service. There are many to choose from now, like Openrouter.com, PPQ.ai, and routstr.com.

adriand 6mo ago
You mean openrouter.ai. And yes, on reading this blog post, I immediately reviewed my API keys in OpenRouter to make sure that they were capped. My prod key was capped at $20/day (phew!) but my dev key had no cap, which I just updated. What a horrible story.
- theanonymousone 6mo ago
  But isn't OpenRouter anyway prepaid, meaning the most you lose is your current credit?
  - adriand 6mo ago
    You can set it to auto top up if it drops below a certain amount. If you do that, then it would definitely be wise to add a cap. They let you add daily/weekly caps, which is convenient.

nurettin 6mo ago

I'd buy the technically impossible angle. Even if you manage to get your microservices to synch every penny spent to your payment account at realtime (impossible) you still have to waiver the excess, losing some money every time someone goes past their quota.

bartread 6mo ago
Sure, but 80 -> 28,000 -> 54,000 is a hell of a lot of slippage. Trading platforms can guarantee a maximum slippage on stops, and often even offer guaranteed stops (with an attached premium), so I don’t see why Google and Firebase can’t do similar. The way it works at present is ridiculous.
- zbentley 6mo ago
  Yep. And cloud providers could eat any slippage cost (enforcing, say, every 5 minutes by stopping service) without even a rounding error on their balance sheets. The fact that they don’t indicates that there’s no market reason to support small spenders who get mad about runaway overages, not that it’s technically or financially hard to do so.
- nurettin 6mo ago
  > Trading platforms can guarantee a maximum slippage on stops Yeah no, physically impossible. If nobody is selling at that price, there is no guarantee your sell stop will execute near that price. They can sweep the market, find the best seller price and execute. There might be a costly way to do it with microservices as I indicated, but your example easily falls apart.
  - bartread 6mo ago
    Not impossible to do: they can hedge and/or absorb the cost, hence the premium. They usually also specify a (fairly large) minimum distance for such stops.
    - nurettin 6mo ago

ch0wn 6mo ago

This should be illegal. If a contractor your hired to swap out a tile on your bathroom floor billed you for remodelling your back garden, you would obviously have the legal right to refuse that.

jubilanti 6mo ago
Not if your contractor had you first sign a 15 page contract that commits you to whatever costs they dream up and requires forced arbitration by a corporate friendly firm when any dispute arises. Because that's somehow normal in today's tech world.
- sdevonoes 6mo ago
  So if their TOS say they can also rape my cat, then I cannot do anything about it, right? Ridiculous
  - ctdinjeu4 6mo ago
    [dead]
  - jubilanti 6mo ago
    In jurisdictions where beastiality is legal, then yes, from the libertarian perspective, that's all freedom of contract, baby. I'm not defending either beastiality or libertarianism, but the logic is that you don't want the government deciding what two private entities can and can't freely agree to. We're pretty far from the Lochner era in the US, where even minimum wage laws were held to be unconstitutional violations of a very broad view of freedom to contract. But it is still a principle in most legal system.
- wing-_-nuts 6mo ago
  Slightly OT, but I've always taken a dim view of this sort of thing for consumers because the parties are never at equal parity, either in ability to understand the legalese they're agreeing to, or the ability to seek alternatives. Legal contracts for consumers should be written at whatever the prevailing reading level is, and the government should step in the more monopolistic position a company is in. It infuriates me to no end how preferential government is towards corporations vs individuals.

Leomuck 6mo ago

That's actually crazy. So I can build a project I love, that does good, but somehow get in a situation where I'm accidentally paying 30.000€ (or 50.000€) to a big tech company? How is that fair? I mean yes, as a software engineer, you ought to reflect on all possible weaknesses, but there was a time when overlooking something meant something completely different than being down 30/50k. That is actually life-altering.

sdevonoes 6mo ago
It’s not fair. Google, Amazon, Microsoft… they have never played fairly. They will never do.
- saidnooneever 6mo ago
  you cannot earn billions a year and not be cheating your users out of their money. its that simple. they dont care for people, otherwise they wouldnt be putting so much effort in making them poor.
  - _DeadFred_ 6mo ago
    What about their behavior makes you think they are a company that doesn't care for people? https://nypost.com/2026/04/15/business/amazon-warehouse-employee-dies-after-collapsing-on-the-job-as-colleagues-kept-working-report/ https://nypost.com/2026/04/15/business/amazon-warehouse-empl...
    - Leomuck 6mo ago
      Wtf. Just wtf.
benoau 6mo ago
Your kid can do this in a smartphone game designated suitable for children, heavily optimized to exacerbate the possibility, and depending on where you live they can just choose not to refund you. When the FTC went investigating a decade-ish ago they found Facebook saying the quiet parts out loud: it was all extremely deliberate.

startages 6mo ago

Yeah, that the main reason I never use services like Google Cloud if I don't have to, it's impossible to have a hard cap, and anyone pretending to be an expert, is just off. Google says that they can't provide a hard cap because that would mean shutting down all your services..bla bla, but at least give users the option.

logankilpatrick 6mo ago
We have spend caps at the billing account level and the project level (developer set) in the Gemini API now. There is up to a 10 minute delay in processing everything but this should significantly mitigate the risk here: https://ai.google.dev/gemini-api/docs/billing#tier-spend-caps https://ai.google.dev/gemini-api/docs/billing#tier-spend-cap... By default, new Tier 1 paid accounts can only spend $250 in a given month.
- flippyhead 6mo ago
  I'm sure it's me being an idiot, but once again I spent 20m trying to figure how to do a specific thing in google-land and still haven't figured it out. Even if I did set it somewhere, I see things like "Setting a budget does not cap resource or API consumption" with a link to a bunch of documentation I have to analyze.
  - bdashdash 6mo ago
    This is what working with cloud services is like, in my experience. Azure's UI feels like it was made as a joke flash game on Newgrounds.
- not_your_vase 6mo ago
  I just find it extraordinary that the biggest tech company in the world can do cutting edge real time AI for millions of people, run Youtube and of course all the other google services with having literally the smartest people in the world and unlimited resources on board, but still can't keep real time track of the user's current billing and their spending limits, it's all best effort still. Somehow it doesn't add up. (Pun not intended, but I'm happy to have it)

reaperducer 6mo ago

hard cap it's technically impossible These companies can sell your personal information in a microsecond in an advertising auction, but somehow can't figure out how to give you timely alerts that stop their cash flow. Big shock.

TrackerFF 6mo ago

It's like a fire alarm system that goes off 30 mins after the it senses a fire. Good stuff.

QuercusMax 6mo ago
I was selling a house in a state I no longer lived in, and was under contract to close the sale, when I got an email from the water company. It told me they suspected based on my water usage that there was a leak on the properly. There had been a very cold February night (like -15F) and a pipe froze inside the walls, and it was just absolutely gushing out. They sent me the email after it had been leaking for a WEEK. I asked a friend to check it out and she said that the laminate floor went "squish" when she stepped in the front door. Fortunately I was covered by homeowner's insurance since I could prove that my heat had been on, but that was a very unpleasant "warning" to receive!
pfortuny 6mo ago
But sending an email, not blaring...

harrouet 6mo ago

As a manager I avoid Google Cloud for this kind of customer-service disasters; but as someone who has dealt with large-scale billing systems in the telecom world, probably similar to that of Google Cloud, I am not surprised that it takes 10 minutes to consolidate all the usage logs of a customer for billing. For telephony, it sometimes takes days when roaming is involved. You have to imagine TB/sec of data, if not more, coming from thousand of potential sources, and queuing for aggregation to the proper company account, all having to be auditable. This is not a small engineering feat and it can't be real-time. With that said, telcos usually include in their business model around 2-3% of bad debt (i.e. revenue that won't get paid), which accounts for frauds like this one. Given that the customer seems in good faith and has taken measures upon being notified, Google should manage this bill shock a bit more elegantly. Moreover, the fact that this happened immediately after this key opened the AI gates means that pirates permanently scan for the permissions of all the keys they could gathers. Google could and should detect that and act upon it.

sofixa 6mo ago

> So much for the folks defending these three companies that refused to provide hard spending cap ("but you can set the budget", "you are doing it wrong if you worry about billing", "hard cap it's technically impossible" etc.) Yes, it's technically+business impossible. To implement a hard cap, a bill never to go over, they'd have to cut your service, but also delete all your data in databases, object storage, data lake, etc. This is simply not an option, so they take the different option of authorising support to wave surprise surcharges / billing DDoSes.

plorkyeran 6mo ago
You can have a hard cap on compute spend while letting storage go over. Surprise huge bills are approximately never due to storage.
benterix 6mo ago
This argument simply doesn't hold water - their (smaller) competition solved this problem over a decade ago.
Glemllksdf 6mo ago
Srsly? The mighty cloud provider can't solve this issue? Google has second precision billing on compute. Its not hard to define a base layer of allowed billing increase and adding this type of context to resource allocation. You are not just suddenly creating a mlllion terabytes of data or a million db requests without supervision. It could even be as simple as basic level caps like 100 euro / month, 1000, 10.000 etc. And there is a difference between stoping everything before the spike happens vs. also deleting stuff.

aleksiy123 6mo ago

For everyone here, you should be able to use quotas in order to set actual limits, you can lower them yourself as a practical solution. https://docs.cloud.google.com/docs/quotas/view-manage https://docs.cloud.google.com/docs/quotas/view-manage Quotas are real time or near real time. Real time spend limits are probably never going to happen. Actual $ amounts are calculated by a centralized billing system offline in batch. It sounds easy but it’s bonkers complicated, because of things like discounts, free tiers, committed usage, currency conversions and having to support every payment and deal structure in GCP. Individual eng teams rarely actually think in dollar amounts, they think in the abstraction which is quotas.

nhuser2221 6mo ago

Happened to me to, luckily only was 40$, restricted the api the next day. They were using gemini 3 flash which I am not using.

Moosdijk 6mo ago
That’s users, for ya! They will always find ways to use your product that you didn’t intend to or even knew about!
- nhuser2221 6mo ago
  The funny thing is that the website only has firebase auth, without any ai features. The default api key that was created (before the ai was even released a few years back), someone got it from the website and started using the gemini api with the key.

Moosdijk 6mo ago

Yet my phone service provider is able to cut off my internet access from the Kb I go over the limit…

fireant 6mo ago

"hard cap it's technically impossible" is really funny because every other provider manages it just fine. Even wrappers like OpenRouter enable you to set a hard cap on proxied Google resources, but Google themselves are unable to manage that inside gcloud.