8 ms·
€54k spike in 13h from unrestricted Firebase browser key accessing Gemini APIs
- patcon 6mo agoThat's fucking bonkers that nothing in the system could see this as unusual and worthy of throttling. The embarrassment of this -- that a company LITERALLY SELLING machine learning services and expertise -- cannot spot such a thing... This should have led them to deal with this internally and refund it. Just... Wow Google.
- lukewarm707 6mo agothere is no way to cap your billing on gcp. you can get notifications but that's it. i don't want to get throttled below my quota but some type of spend limit would be good.
- rvnx 6mo agoand the notifications can be delayed because the spending system is not updated in real-time, so even if you have a Cloud Task triggering on spending to disable the project it may be too slow and several thousands may already be spent.
- bombcar 6mo agoIs there a cloud provider that does have hard unbreakable billing caps? Everything I've seen has always been notifications or soft caps. Not talking about fixed-access things like a Hetzer box.
- pwdisswordfishs 6mo agoBunny.net purports to have a pay-as-you-go prepaid credit system that sounds like it works the way people want, and with their description of the way it works probably being sufficient to be legally enforceable if it turns out that it actually works differently and you were to end up with a surprise bill from them. And evidently it really does work that way; see this post from a couple weeks ago: <https://news.ycombinator.com/item?id=47676416 https://news.ycombinator.com/item?id=47676416> The only other provider known to work that way is NearlyFreeSpeech.NET, which serves a completely different market segment (so much so that it might as well not even be considered the same kind of product/service).
- trick-or-treat 6mo agoThis is GCP's revenue model, lol. Let's provide a (semi) generous free tier and trick people into accidentally going over it.
- voidUpdate 6mo agoThe company selling machine learning services would probably love a €54k bonus
- owebmaster 6mo agoIt's so funny seeing people thinking this is not by design
- lukewarm707 6mo agoi have seen this so many times... i'm thinking it's time we replaced api keys. some type of real time crypto payment maybe?
- freedomben 6mo agoOh please no. And the "alternatives" to API keys aren't going to help much either, they'll just add friction to getting started (as reference: see the pain involved in writing a script that hits gmail or calendar API)
- trick-or-treat 6mo agoPrepaid only is a fantastic idea, especially for dumb-ass startups. Limiting your liability to $100 or so sound like a big-ass W.
- freedomben 6mo agoPrepaid only is a fantastic idea, until your site goes (desirably) viral and then gets shut off right as traffic is picking up, or you grow steadily and forget to increase your deposit amount and suddenly production is down. Billing alerts are a much better solution IMHO.
- dummydummy1234 6mo agoPrepaid/paid limits with shutoff is appropriate for this though. If you have per key limits, this is not possible, and even in a wild situation you should b able to expect that your firebase key will not use 50k.
- microtonal 6mo agoYou can also have both, a cap and one or more billing alert levels below it. Some providers do this (e.g. IIRC Backblaze B2).
- freedomben 6mo ago
- ajaystream 6mo ago[flagged]
- freedomben 6mo agoThere is some new stuff here, see https://news.ycombinator.com/item?id=47156925 https://news.ycombinator.com/item?id=47156925 for example.
- oezi 6mo ago> — billing alerts fire in hours, damage happens in minutes. And why do you need to use AI to tell us that. How much shorter could the prompt have been?
- jb1991 6mo agoYou are getting down voted but the first thing I thought when I read the above comment you replied to was that it was written by an LLM as well. It has all the stylings of it. Word choice, sentence structure, phrasing, metaphors, etc.
- deleted 6mo ago[deleted]
- theli0nheart 6mo agoStop using ChatGPT to write your comments please.
- hilariously 6mo agoThis reads like 100% an LLM comment. by design -- the enforcement Nothing new here - what is new is the A thing before anyone noticed - another thing, billing in hours, damage in minutes. has the signal, doesn't expose the control Every one of those "exposes the signal" to me.
- clapthewind 6mo agoDo not get hung up? Sounds like English ESL rewrote some insights for language. Content > Form.
- mdrzn 6mo agoRelated: https://news.ycombinator.com/item?id=47156925 https://news.ycombinator.com/item?id=47156925
- deleted 6mo ago[deleted]
- drtz 6mo ago> Are there recommended safeguards beyond ... moving calls server-side? This implies the API calls originated in the client, suggesting the client may have had they API key.
- embedding-shape 6mo agoYeah, the amount of people creating, running and maintaining websites yet don't understand how websites actually work in practice is very high and seems we haven't even come close to the ceiling yet.
- dpkirchner 6mo agoThat's standard for Firebase apps. It's also recommended by Google (they describe the keys as "public by design").
- Retr0id 6mo agoFeels like a confusing thing to name "key" if it's presumably more of an identifier.
- pwdisswordfishs 6mo agoIt's "implied" throughout the whole post (or more like assumed that the reader understands this, because it's the basic premise of the problem). It's why they link to a post that explains the basic concept after a remark that "This describes our issue in more detail". > tl;dr Google spent over a decade telling developers that Google API keys (like those used in Maps, Firebase, etc.) are not secrets. But that's no longer true: Gemini accepts the same keys to access your private data. We scanned millions of websites and found nearly 3,000 Google API keys, originally deployed for public services like Google Maps, that now also authenticate to Gemini even though they were never intended for it. With a valid key, an attacker can access uploaded files, cached data, and charge LLM-usage to your account. Even Google themselves had old public API keys, which they thought were non-sensitive, that we could use to access Google’s internal Gemini. From Google themselves, in the Firebase docs: > API keys for Firebase services are not secret. Firebase uses API keys only to identify your app's Firebase project to Firebase services, and not to control access to database or Cloud Storage data, which is done using Firebase Security Rules. For this reason, you do not need to treat API keys for Firebase services as secrets, and you can safely embed them in client code. <https://firebase.google.com/support/guides/security-checklist#api-keys-not-secret https://firebase.google.com/support/guides/security-checklis...> ... or at least that's what it used to say, until they quietly updated the docs to say this: > API keys for Firebase services are not secret. API keys for Firebase services only identify your Firebase project and app to those services. Authorization is handled through Google Cloud IAM permissions, Firebase Security Rules, and Firebase App Check. > All Firebase-provisioned API keys are automatically restricted to Firebase-related APIs. If your app's setup follows the guidelines in this page, then API keys restricted to Firebase services do not need to be treated as secrets, and it's safe to include them in your code or configuration files. Followed later by (in different section): > Use your Firebase-provisioned API keys only for Firebase-related APIs. If your app uses any other APIs (for example, the Places API for Maps or the Gemini Developer API), use a separate API key and restrict it to the applicable API.
- embedding-shape 6mo agoConsidering the amount of repositories on public GitHub with hard-coded Gemini API tokens inside the shared source code (https://github.com/search?q=gemini+%22AIza%22&type=code https://github.com/search?q=gemini+%22AIza%22&type=code), this hardly comes as a surprise. Google also has historically treated API keys as non-secrets, except with the introduction of the keys for LLM inference, then users are supposed to treat those secretly, but I'm not sure everyone got that memo yet. Considering that the author didn't share what website this is about, I'd wager they either leaked it accidentally themselves via their frontend, or they've shared their source code with credentials together with it.
- ckbkr10 6mo agotheres not a single real gemini api key in the results
- embedding-shape 6mo agoSetup a watcher and you'll come across live ones eventually :)
- dminik 6mo agoTry this one. Should remove most readme keys: Edit: self censor based on a request
- duskdozer 6mo agoOh, wow.
- ratsimihah 6mo agothis is such a wall of shame haha
- sillysaurusx 6mo agoI know you're well within your rights to post this, but would you consider replacing your comment with something like "It's easy to find working keys on github if you search the appropriate terms"? Think of it this way: although you're not to blame, HN drives a lot of traffic to your preconfigured github search. There are also bad actors who browse HN; I had a Firebase charge of $1k from someone who set up an automated script to hammer my endpoint as hard as possible, just to drive the price up. Point being, HN readers are motivated to exploit things like what you posted. It's true that the github search is a "wall of shame", and perhaps the users deserve to learn the hard way why it's a good idea to secure API keys. But there's also no benefit in doing that. The world before and after your comment will be exactly the same, except some random Gemini users are harmed. (It's very unlikely that Google or Github would see your comment and go "Oh, it's time we do something about this right now".) EDIT: I went through the search results and confirmed that the first several dozen keys don't work. They report as error code 403 "Your API key was reported as leaked. Please use another API key." or "Permission denied: Consumer 'api_key:xxx' has been suspended." So at least HN readers will need to work hard(er) to find a valid key. I wonder how you report a gemini API key as leaked... Searching "report gemini api key leaked" on Google only brings up similar horror stories (a $55k bill, waived https://www.reddit.com/r/googlecloud/comments/1noctxi/student_hit_with_a_5544478_google_cloud_bill/ https://www.reddit.com/r/googlecloud/comments/1noctxi/studen...) and (a $13k bill from 3d ago https://www.reddit.com/r/googlecloud/comments/1sjzat3/api_key_compromised_13428_fraudulent_charges/ https://www.reddit.com/r/googlecloud/comments/1sjzat3/api_ke...)
- luanmuniz 6mo agoUnfortunately, yet just another story like this. One of these unexpected usage charges in the thousands appears every month, and with the same automatic denied too. This is one of the reasons I just stopped using these kinds of pay-per-usage cloud services long ago. At best, I still use services that have hard-bounded usage limits, like EC2 from AWS, where one instance can never go beyond 24h/day usage and is always capped, with shutdowns when exceeded, and limited credit cards, too. It's super frustrating that this is the only option to realistically deal with this issue, since all stories end up the same way: The cloud company just saying "f* you, we don't care, pay up." and legal fees are always expensive :(
- embedding-shape 6mo ago> At best, I still use services that have hard-bounded usage limits, like EC2 from AWS, where one instance can never go beyond 24h/day usage and is always capped, with shutdowns when exceeded, and limited credit cards, too. Is this possible on AWS today? I'm the same way, if I cannot set a hard-limit for the billing so I can know for a fact how much it'll maximum cost in a month, I'm not interested in using that service for anything. Which is one of the top reasons I've stayed clear of AWS, they used to have only billing-alerts, but you couldn't actually set limits, guess one step forward that they've finally implemented that now.
- 827a 6mo agoI said this when this finding was originally posted and I'll say it again: This is by far the worst security incident Google has ever had, and that's why they aren't publicly or loudly responding to it. It's deeply embarrassing. They can't fix it without breaking customer workflows. They really, really want it to just go away and six months from now they'll complete their warning period to their enterprise contracts and then they can turn off this automated grant. Until then they want as few people to know about it as possible, and that means if you aren't on anyone's big & important customer list internally, and you missed the single 40px blurb they put on a buried developer documentation site, you're vulnerable and this will happen to you. Disgusting behavior.
- 100ms 6mo agoThis is only a little billing leakage, Operation Aurora in 2009 was 100x worse
- 827a 6mo agoIt's actually much more than a billing leak [1]; again, most people don't know how bad this is, because Google is trying to keep it hush-hush. These keys don't just grant access to Gemini completions; they grant access to any endpoint on the generative AI google cloud product. This includes: seeing all of the files that google cloud project has uploaded to gemini, and interacting with the gemini token cache. [1] https://trufflesecurity.com/blog/google-api-keys-werent-secrets-but-then-gemini-changed-the-rules https://trufflesecurity.com/blog/google-api-keys-werent-secr...
- zarzavat 6mo agoIt's not a security incident because it makes Google money. It's extra revenue. They are embarrassed all the way to the bank. At some point, when it appeared 2 months ago on HN and they still did nothing about it, intentionality can be assumed.
- bombcar 6mo agoThis is exactly it - and the normal "resolution" is a class-action lawsuit but no doubt their terms and conditions forbid that. However, anyone affected should probably pollute their docket with lawsuits anyway.
- p2detar 6mo agoI read the following [0] and immediately went to my firebase project to downgrade my plan. This is horrific. > Yes, I’m looking at a bill of $6,909 for calls to GenerativeLanguage.GenerateContent over about a month, none of which I made. I had quickly created an API key during a live Google training session. I never shared it with anyone and it’s not pushed to any public (or private) repo or website. 0 - https://discuss.ai.google.dev/t/unexpected-gemini-api-billing-spike/114095/7 https://discuss.ai.google.dev/t/unexpected-gemini-api-billin...
- jasonjmcghee 6mo agoSo someone took a picture of the key at the live training session or something? What's the suspected cause?
- deleted 6mo ago[deleted]
- mcccsm 6mo agoTwo things that should be default on any GCP project touching generative-AI APIs: 1 API-key restrictions by HTTP referrer AND by API (`generativelanguage.googleapis.com` only), 2 a billing budget with a Pub/Sub "cap" action, not just an email alert. Neither is on by default, and almost nobody sets them before shipping. 13 hours is actually fast for detection. most teams find out at end-of-month reconciliation.
- PunchyHamster 6mo agoI want API keys with monthly and hourly quotas and RATE LIMITING. like 50k requests per hour, above that 1/s/client up to 20 req/sec. I don't want to shotgun my service for every user if one user is misbehaving. I want to set rate of bleeding
- deleted 6mo ago[deleted]
- mcccsm 6mo agoyet another case of incentive misalignment between user and provider. and unfortunately it's a pattern that extends well beyond AI vendors. it's just a feature of the whole shitshow we live in
- danelliot 6mo ago[dead]
- dabedee 6mo agoAs others have said, this is a "feature" for Google, not a bug. There is no easy way to set a hard cap on billing on a project. I spent the better time of an hour trying to find it in the billing settings in GCP, only to land on reddit and figuring out that you could set a budget alert to trigger a Pub/Sub message, which triggers a Cloud Function to disable billing for the project. Insanity.
- onemoresoop 6mo agoCall it for what it is, an antifeature, a trap for the user.
- deleted 6mo ago[deleted]
- alasano 6mo agoMy favorite Google LLM benchmark is asking Gemini models to create a script that fetches API usage (just request counts) for a project from GCP. 100% failure rate.
- dpkirchner 6mo agoI've yet to receive an accurate response from Gemini about GCP services, beyond completely trivial topics. The most recent, I think, was Gemini advising me that I could attach an existing pd SSD PVC to a n4 or c4 VM. For whatever unknowable reason, Google doesn't allow this and doesn't offer a migration path, and Gemini doesn't "know" anything about it either. It's wild.
- alasano 6mo agoAgreed on the fact that it should know, it's their LLM. What bothers me is even having it do extensive research in documentation, it still can't figure it out. GCP must simply be so unintuitive that the LLM mind cannot comprehend it.
- intended 6mo ago
- thedangler 6mo agoAlso, can't you tie a key to a domain or IP address to help stop unauthorized usage?
- littlecranky67 6mo agoNot if its publicly called from Javascript, as your user's browser will make those requests. You neither know their IP addresses, nor is the referer or origin header a safe choice as it can be spoofed outside of a browser.
- lucavice 6mo agoIf it's called from Javascript in the browser, it's not a secret API key....
- shakna 6mo agoWhich is why Google calls it a public API key...
- littlecranky67 6mo agothere are plenty of API keys distributed like this by design. For example, google maps requires this, else your (anonymous) users can't use an embedded google map on your website. And a public firebase app needs some kind of API key, too.
- JohnScolaro 6mo ago> We had a budget alert (€80) and a cost anomaly alert, both of which triggered with a delay of a few hours. By the time we reacted, costs were already around €28,000. I had a similar experience with GCP where I set a budget of $100 and was only emailed 5 hours after exceeding the budget by which time I was well over it. It's mind boggling that features like this aren't prioritized. Sure it would probably make Google less money short term, but surely that's more preferable to providing devs with such a poor experience that they'd never recommend your platform to anyone else again.
- deleted 6mo ago[deleted]
- zanbezi 6mo agoExactly my thoughts, can not really understand how delayed alerts are acceptable... Have you managed to settle the cost with Google, what was the outcome?
- sillysaurusx 6mo agoBack in 2020 I had a similar situation. Ended up charging $500 due to an overnight TPU training run using egress bandwidth across zones. Google support was surprisingly understanding, after I explained the issue. They asked some clarifying questions. Then they said that they can offer a one time refund for this case. Since then I was paranoid not to accidentally do it again. I don't know whether GCP would refund a second time.
- genxy 6mo agoGCP charging for interzone traffic is an interesting financial choice. They own all the infra and in many cases this is literally moving from building to building.
- sillysaurusx 6mo agoThere's cross-region, and cross-zone. If both boxes are located within the same zone (e.g. us-east1) then the bandwidth is free, since it's intrazone traffic. Cross-zone egress traffic (e.g. us-east1 to us-central1) is billed at a certain rate, and cross-region egress traffic (e.g. us-east1 to europe-west8) is billed at a significantly higher rate. Amusingly enough, ingress traffic seems to always be free. So you can upload as much data as you want into their cloud, but good luck if you need to get it out.
- bcjdjsndon 6mo agoI thought the pricing model was meant to be a benefit of the cloud? All of a sudden, shock horror, paying by the minute turns out to be no cheaper and maybe even more expensive than just doing it yourself
- comrade1234 6mo agoCan you pre-load money into your account and have that be used until it's zero, at which time you have to load more? Deepseek does it this way.
- ok123456 6mo agoNo GCP is not prepay.
- Maxious 6mo agohttps://ai.google.dev/gemini-api/docs/billing#prepay https://ai.google.dev/gemini-api/docs/billing#prepay
- Bridged7756 6mo agoNo. I believe all major cloud providers are Pay As You Go. I think only Azure has a tier where you can run on free credits for a while.
- hypercube33 6mo agoThe only thing I've seen is in MECM (SCCM) the azure extension will hard shut down when you hit a limit. if you want.
- pwdisswordfishs 6mo agoThere's a brand-new, Gemini-specific feature for that (as new as March 23), but historically the answer has tended to be "no" from all the cloud providers. Most giants and indies alike have always been strongly opposed to implementing this feature for business reasons. (When you run across something that does let you do things that way, it's one of a handful of exceptions.) Their response is to tell you to set up budget alerts, which is not a solution, as described in this post. <https://en.wikipedia.org/wiki/The_purpose_of_a_system_is_what_it_does https://en.wikipedia.org/wiki/The_purpose_of_a_system_is_wha...>
- CWwdcdk7h 6mo agoGoogle doesn't allow disconnecting credit card from account unless you close it. That includes situation when you are just trying out free tier.
- Illniyar 6mo agoI think the logistics of calculating cost in real time is something that is extremely hard. I don't think there is one big cloud service provider that has hard limits instead of alerts. As long as they revert the charge when notified of scenarios like this , and they have historically done so for many cases, it's fine. It's an acceptable workaround for a hard problem and the cost of doing business ( just like Credit Cards accept a certain amount of loss to fraud as part of business)
- zulban 6mo agoRidiculous. They are clearly not trying at all. A hard wall preventing going over budget by 100x in a couple hours is not some devilishly complicated decentralized system problem. Don't tote the party line. Same reason why Azure AI only has easy rate limits by minute, not by day or week or month. Open source proxy projects do it easily tho. Think about the incentives. Going over a hard cap by 3% would be a reasonable failure to make, not by 30000%.
- EdwardDiego 6mo ago> I think the logistics of calculating cost in real time is something that is extremely hard. What makes you think that?
- wongarsu 6mo agoCutting off at the exact cent is difficult, but a hard limit that triggers within one dollar of the actual limit should really be possible If for some resources you can't sample measurements fast enough you could weaken it to "triggers within one dollar or five minutes after cost overrun, whichever comes later". But LLM APIs are one of those cases where time isn't a factor, your only issue is that if you only check quota before each inference a given query might bring you over
- Nathanba 6mo agoWhy would it be hard to calculate cost? Multiply a fixed price * requests/time ? It doesn't have to be exact in real time, it just has to report something approximately useful in realtime. It's absolutely not fine to be at the mercy of other people, that's what we buy cloud products or really any products for: So that we are not at the mercy of hardware faults, bad weather, bad teeth, hunger, thirst, [insert anything]
- bossyTeacher 6mo agoAs always, you will need to make lots of noise on here and similar channels visited by influential people so stuff can get actioned. Leading tech companies in 2026, folks.
- __natty__ 6mo agoIt's terrible that giant cloud providers such as Google or AWS doesn't allow for hard cap at project levels or prepaid. And that especially because alerts are delayed as author stated "We had a budget alert (€80) and a cost anomaly alert, both of which triggered with a delay of a few hours. By the time we reacted, costs were already around €28,000.".
- alibarber 6mo agoForgive my ignorance - but what's the payoff for fraudsters in getting access to a generative AI service for a short-ish period of time, before they get cut off? With EC2 / GCC credentials, I could understand going all out on bitcoin mining - but what are they asking the AI to do here that's worth setting up some kind of botnet or automation to sift the internet for compromised keys?
- varispeed 6mo agoIf they work for hostile state, the payoff is destruction of economy and social contract. Damage here, damage there. It all adds up.
- lxgr 6mo agoThere are plenty of services offering AI inference at a discount. Some of these will be using your data for future distillation; others might be making use of bulk discounts and passing these through to a number of individual users (while taking on billing, support etc. risk) – and maybe some are just selling tokens falling off the back of a truck?
- deleted 6mo ago[deleted]
- LelouBil 6mo agoTotally speculating here, but maybe they provide some sort of LLM as a service, and they rotate stolen API keys in the background so they don't have to pay anything ? Or they use the LLMs for criminal purposes (like automated social engineering) and so the API key can't be traced to their personal info (but they could also use a local model for this, so I don't know).
- Aurornis 6mo agoEarly Generative AI was popular with spammers before it became mainstream because it could be used to write infinite variations of spam messages. Making each message unique is more likely to bypass spam filters. There are also a lot of AI use cases that require a lot of token spend to brute force a problem. Someone might want to search for security exploits in a codebase but they don’t want to spend the $50,000 in tokens from their own money. Finding someone’s key and using it as hard as possible until getting locked out could move these projects forward.
- benterix 6mo ago> We had a budget alert (€80) and a cost anomaly alert, both of which triggered with a delay of a few hours > By the time we reacted, costs were already around €28,000 > The final amount settled at €54,000+ due to delayed cost reporting So much for the folks defending these three companies that refused to provide hard spending cap ("but you can set the budget", "you are doing it wrong if you worry about billing", "hard cap it's technically impossible" etc.)
- villgax 6mo agoShirky’s principle at work is all
- varispeed 6mo agoThis is clearly setup for VC backed companies where shareholders don't care about spend as long as they can brag about investing in this cool start up at dinner parties. Normal and true business should stay away.
- Maxious 6mo ago> The Gemini API supports monthly spend caps at both the billing account tier and project levels. These controls are designed to protect your account from unexpected overages, and the ecosystem to ensure service availability https://ai.google.dev/gemini-api/docs/billing#project-spend-caps https://ai.google.dev/gemini-api/docs/billing#project-spend-...
- rtkwe 6mo agoThe problem is it's specific to that API and defaults to uncapped so people who aren't using it and haven't heard about the issues with the Firebase API keys probably won't have set them.
- zozbot234 6mo agoExcept that Google's own statements are extremely clear that "leaked" (i.e. public) API keys should not be able to access the Gemini API in the first place: "We have identified a vulnerability where some API keys may have been publicly exposed. To protect your data and prevent unauthorized access, we have proactively blocked these known leaked keys from accessing the Gemini API. ... We are defaulting to blocking API keys that are leaked and used with the Gemini API, helping prevent abuse of cost and your application data." https://ai.google.dev/gemini-api/docs/troubleshooting#googles_security_measures_for_leaked_keys https://ai.google.dev/gemini-api/docs/troubleshooting#google... For extra clarity on the exact so-called "vulnerability" that Google identified, see: https://news.ycombinator.com/item?id=47156925 https://news.ycombinator.com/item?id=47156925 This describes the very issue where some API keys were public by design (used for client-side web access), so the term "leaked" should be read in that unusually broad sense. Firebase keys are obviously covered, since they're also public by design. (As for "Firebase AI Logic", it is explicitly very different: it's supposed to be implemented via a proxy service so the Gemini API key is never seen by the client: https://firebase.google.com/docs/ai-logic https://firebase.google.com/docs/ai-logic Clearly, just casually "enabling" something - which is what OP says they did! - should never result in abuse of cost on the scale OP describes.)
- time0ut 6mo agoIt is scary building on the public cloud as a solo dev or small team. No real safety net, possibly unbounded costs, etc. A large portion of each personal project I do is spent thinking about how to prevent unexpected costs, detect and limit them, and react to them. I used to just chuck everything onto a droplet or VPS, but a lot of the projects I am doing lately need services from Google or AWS. I tend to prefer GCP at this point because at least I can programmatically disconnect the billing account when they get around to tripping the alert.
- Bridged7756 6mo agoI wonder what happens if you just decide not to pay. Surely that would have some legal implications in the US, but what about elsewhere?
- spacebanana7 6mo agoThere are very few countries where consumer rights apply to B2B transactions, especially if it’s multiple people operating as a “small team”. A solo dev however might be able to present themselves as a retail consumer, and leverage some trading standards related rules for unclear pricing or something similar.
- wonchoi 6mo agogood
- deleted 6mo ago[deleted]
- ozlikethewizard 6mo agoThe top comment on the post physically hurt me. We've moved past the era of keep env files in code bases and are now actually serving them lol.
- chinathrow 6mo agoTake them to court.
- _nhh 6mo agoWith AI there is NO justification in NOT DOING IT BY YOURSELF. Why use firebase or <technology-x> if you can generate <the-thing> by yourself and deploy to hardware you own or rent.
- naturalauction 6mo agoWe had this exact same problem (the key initially wasn’t a secret but became a secret once we enabled Gemini API with no warnings). We managed to catch it somewhat early through alerting, so the damage was only $26k. We asked our Google cloud support rep for a refund - they initially came back with a no but now the case is under further consideration. I’d escalate this up the chain as much as possible.
- deleted 6mo ago[deleted]
- turtlebits 6mo agoSlightly off-topic, but Backblaze B2 has usage caps that actually work. I have $0 cap on API requests, and yesterday when litestream burned through the free tier (defaults to replicating every second), I got a notice and requests stopped working until I upped my cap.
- Havoc 6mo agoDon’t use GCP (and other big clouds) until they sort out their safeguards. All three of the big cloud subreddits have stories like this on a regular basis
- juancn 6mo agoThese are all poorly designed systems from a CX perspective (the billing systems). Billing is usually event driven. Each spending instance (e.g. API call) generates an event. Events go to queues/logs, aggregation is delayed. You get alerts when aggregation happens, which if the aggregation service has a hiccup, can be many hours later (the service SLA and the billing aggregator SLA are different). Even if you have hard limits, the limits trigger on the last known good aggregate, so a spike can make you overshoot the limit. All of these protect the company, but not the customer. If they really cared about customer experience, once a hard limit hits, that limit sets how much the customer pays until it is reset, period, regardless of any lags in billing event processing. That pushes the incentive to build a good billing system. Any delays in aggregation potentially cost the provider money, so they will make it good (it's in their own best interest).
- deleted 6mo ago[deleted]
- bux93 6mo agoIt's not typically a problem that usage is event driven. At least not for prepaid phone plans. Or debit cards. Or mailboxes. Or any myriad of prepaid or quota'd services. It's not rocket science, just a bad business practice on the part of Google.
- fhn 6mo agoon a more positive note, you saved a few bucks not running your own server or database.
- noplace1ikegone 6mo agoGoogle responded to your post so that’s good news. We all know the nature of APIs, but a secure transaction system is non-negotiable from Google and its peers for LLM API use. Right now LLM APIs are like unencrypted credit card numbers floating around.
- rmoriz 6mo agoCrude Oil Futures, Natural Gas Futures, Google Cloud API keys. the widow-maker list increases.
- RA_Fisher 6mo agoAnthropic and Claude are running circles around Google / Gemini for me these days. Anthropic was quite helpful for a while but strange limit issues started popping up. The final thread was a bug that essentially broke my ability to develop. I moved over to Claude Code full time and haven't looked back. Opus 4.6 is awesome for accelerating probabilistic programming!
- jcbb2000 6mo ago[dead]
- smcl 6mo agoIt's incredible that in 2026 your best bet for getting support from Google is still posting to HN and hoping a Product Owner at Google takes pity on you (or feels shamed...)
- bustah 6mo ago[flagged]
- jimmypk 6mo ago[flagged]
- croes 6mo agoA cap is a more fundamental fix. It’s easy to miss a setting especially if new features with opt-out are added
- DaedalusII 6mo agoon the one hand if you play with petrol you cant complain about burning down your garage on the other hand hetzner sell ipv4 instance with no security on by default, just raw ubuntu 24.x within 3-4 days of deploying one, it will be hacked and have crypto miners installed unless additional special config is added. i do wonder what % of hetzner vps instances are compromised
- perdomon 6mo agoDoes the blog post explain how this happened exactly? Did he leak his API key in frontend code somehow, or was his project itself vulnerable to misuse? I'm curious how someone racked up 30k in a few hours.
- sachinag 6mo agoHey folks, I just wanted to drop a quick note here that there’s a way to stop billing in an emergency that’s officially documented on the Google Cloud documentation site: https://docs.cloud.google.com/billing/docs/how-to/disable-billing-with-notifications https://docs.cloud.google.com/billing/docs/how-to/disable-bi... . You can see the big red warning that this could destroy resources that you can’t get back even if you reconnect a billing account, but this is a way to stop things before they get out of control. This billing account disconnect goes all the way to implement a full on “emergency hand brake” that “unplugs the thing from the wall” (or whatever analogy you prefer) without you having to affirmatively do it yourself. https://docs.cloud.google.com/billing/docs/how-to/modify-project#disable_billing_for_a_project https://docs.cloud.google.com/billing/docs/how-to/modify-pro... and https://docs.cloud.google.com/billing/docs/how-to/budgets-programmatic-notifications#cap_disable_billing_to_stop_usage https://docs.cloud.google.com/billing/docs/how-to/budgets-pr... are other documented alternatives to receive billing alerts without the billing account disconnect. The billing account disconnect obviously shouldn’t be used for any production apps or workloads you’re using to serve your own customers or users, since it could interrupt them without warning, but it’s a great option for internal workloads or test apps or proof of concept explorations. Hope this helps!
- dorgo 6mo ago>There's a delay between incurring costs and receiving budget notifications, so you might incur additional costs for usage that hasn't arrived at the time that all services are stopped. This delay may be hours or days. I managed to spend $400 in 5 minutes.
- riteshkew1001 6mo ago[flagged]
- lokimedes 6mo agoOh the days when these scenarios just lead to a slowdown of the database server, felt by everyone, not just accounting.
- doshiba 6mo ago[dead]
- arjie 6mo agoSurprised they don’t have usage limits. E.g. you can’t get many IPs from AWS for your region until you request a limit increase. The UX for these kinds of things seems like it should default to low and allow easy increasing.
- PufPufPuf 6mo agoFor personal projects, is there a cloud service that has actual working spend caps? I would perhaps try using a cloud service if I wasn't exposing myself to a risk of losing my yearly income by a small mistake. Or is renting a VPS the only sensible option?
- william0353 6mo agoHi I am just curious the reason behind it as I have a firebase app with firebase ai logic service as well. Is that the apiKey below which was used for web sdk init? const firebaseConfig = { apiKey: XXXX, authDomain: XXX Did Zanbezi enabled app check? This is kind worrying... };
- twoshotsai 6mo ago[dead]
- linkregister 6mo agoFrom the response from the Gemini product lead: > We are moving to disable the usage of unrestricted API keys in the Gemini API, should have more updates there soon. It's unacceptable the contract for client-side keys was broken in this manner, and doubly bad that it's taken so long for Google to remediate this issue. The Gemini team needs to publish a postmortem to explain what broke down in the engineering process to allow this to happen. context: https://news.ycombinator.com/item?id=47156925 https://news.ycombinator.com/item?id=47156925
- bwpw 6mo agoThe scarier version of this problem is coming. Imagine AI agents making API calls across multiple vendors. Each vendor tracks usage in isolation. But no one knows what the agent is spending across all vendors. Nobody building the cross-vendor spending cap today.
- ButlerianJihad 6mo agoBack in 2020–21, we were teaching our students how to stand up and configure cloud services, and I decided that a good extension of my homelab would be in AWS, so that I could learn basic cloud administration. I was using the Free Tier for starters, of course. I managed to start up a working MediaWiki server on a Linux machine in EC2. I began to explore some of the more esoteric options and always checked out the extra-secure methods, and IAM and so forth. My MediaWiki had a lot of spammers registering accounts. They weren't actually able to make edits, but I couldn't seem to stop them from creating user accounts. And I always felt rather... naked in terms of securing the Linux system itself. It seemed like the entire Internet had an Eye of Sauron focused on my open TCP ports and they were port-scanning it and running pentests 24/7. I honestly couldn't keep up! Ultimately, I did realize that I could never constrain the budget to an affordable $20 or $30. Signs seemed to indicate that any malicious traffic could crank up my network egress costs alone! There were some rudimentary controls but they would never permit a full-scale shutdown of all services that could actually cost money. So I shut down the cloud services and abandoned my Amazon AWS account. Migration to the cloud might seem like a good value proposition for any business that can't handle its own machine rooms or its own I.T. team to manage physical infrastructure. But it's an unconstrained cost nightmare waiting to happen for basically anyone of any scale. I would never recommend it, for personal or business use, until that aspect is somehow brought under control.
- aisaasbuilder 5mo ago[dead]