28 ms·

Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

282 points by bharatsb 6y ago

pettycashstash2 6y ago
Thank you for sharing. I was actually thinking of using fire base for my project. They make us so easy to sign up for free tier. Awaiting to see what happens in part 2
- bharatsb 6y ago
  Part 2: https://blog.tomilkieway.com/72k-2/ https://blog.tomilkieway.com/72k-2/
- spacemanmatt 6y ago
  I recently put the (soft) kibosh on a project in my stable trying to switch to FireBase at the last minute. It looks attractive but the business aspects are frankly frightening, and I'm not even talking about the risk of a large bill. Getting your metrics 24h late sounds like a deal killer for me. So much for observability!
  - asciimike 6y ago
    Minor nit: many non-billing metrics are near-real time, e.g. DB concurrents, cloud functions CPU/RAM usage; any metrics that require aggregation (storage, billing) are going to be batched less frequently. This is going to be true across all platforms of non-trivial scale (eventual consistency + batch jobs). Second note: the number of people who actually do this is very low (a few a year, of hundreds of thousands of developers). The blog posts are scary, but in my ~five years at Firebase, I'm pretty sure we refunded every one. As my boss (James Tamplin, CEO of Firebase) used to say, "There are lots of bad systems, but rarely are there bad people."
    - spacemanmatt 6y ago
      > I'm pretty sure we refunded every one. That's good to know. It seems like a particularly visible part of the cloud credibility puzzle to be able to monitor/limit dynamic resource costs.
onion2k 6y ago
The fact that cloud providers don't have a simple "This is how much I can afford, don't ever bill me more than that!" box on their platforms makes development a lot scarier than it really needs to be.
- jsiepkes 6y ago
  Not only development but also running in production. You can configure alerts but you can't configure a hard limit. Thats just insane. That makes working with GCP like playing with fire.
  - k__ 6y ago
    What about throttling?
    - herendin2 6y ago
      Nice to have, but people want a throttle that shuts off dead at a certain number of dollars
    - pwinnski 6y ago
      aka "Bankrupt me more slowly" Throttling doesn't stop the drain.
- serial_dev 6y ago
  It is baffling why cloud providers don't have that option. I might want to have an app because I don't mind spending 50 dollars on my pet project as a hobby, but I don't ever want to spend more than that. Not if I write a wrong query that's suddenly becomes very expensive, not when I got attacked, and not even when I have legit users. By the way, the same goes for some companies, too, just the threshold would be different.
  - greatgib 6y ago
    It's not complicated to add configurable hard limits for these companies but they don't allow it because the current situation is more interesting for them. They want to suck the maximum money from consumers before they realize. For one person that will complain wildly and having to do a gesture, there are hundreds other companies that will not notice or just pay without recourse.
bharatsb 6y ago
Part 2: https://blog.tomilkieway.com/72k-2/ https://blog.tomilkieway.com/72k-2/
- MaxBarraclough 6y ago
  > Google let go of our bill as a one time gesture! We've seen this happen with similar stories on AWS. Neither platform supports prepayment with a hard limit on costs, and this seems unlikely to change.
  - teekert 6y ago
    Yeah a friend of mine wanted a real cert, not letsencrypt (I don't understand how that is more real but ok), as a bit of a noob he clicked around on the AWS website and some days later had a bill op 1500 eur. They also nulled it. Still, this scares the hell out of me.
    - WrtCdEvrydy 6y ago
      AWS Certificate Manager gives you free non-extended SSL to your machine, it's pretty nifty.
  - coddle-hark 6y ago
    I can sympathise with some of these stories, like the ones where an overnight DDOS attack racks up a huge unexpected bill, but this one in particular is just a story of gross incompetence and negligence. The guy hacked together some code in a few days and deployed it to a service with unlimited billing without any kind of sanity checks and without even understanding what he was paying for. He’s an ex-Googler, it’s not like he hasn’t heard stories like this before. And the takeaway? “Oops don’t deploy buggy code” and “I shouldn’t have used the default settings”. OK, sure, let me know how that works out for you.
    - teekert 6y ago
      But "The Guy" was under the impression that he could do so because he set sane limits in both GCP and his Credit Card.
villgax 6y ago
I'd happily use a platform that allowed for an option for limiting billing on a daily/monthly/another metric.
- raphaelj 6y ago
  Some platforms do that: - Heroku costs are pretty predicable, and you can easily set a maximum scalability threshold to their auto-scalable dynos, so that they will never cost you more than a predefined amount of money; - BunnyCDN requires me to top-up their prepaid account, so that I'll never spend more than what I have on that account.
atian 6y ago
You can ask for a good faith billing adjustment. GCE or AWS is well aware that things happen, and collecting something is better than collecting nothing.
- ceejayoz 6y ago
  Part two of the article indicates they did that, and had the bill nulled out.
  - pettycashstash2 6y ago
    from part 2: "After going through our lengthy doc on this incident sharing our side of the story, various consults, talks, and internal discussions Google let go of our bill as a one time gesture!"
throwaway7281 6y ago
To put it into perspective: You give me $72K and I'll set you up a 1PB replicated storage infra with a total of 100+ available CPU cores and half a TB RAM. I saw people burning through cash in the cloud, which makes you wonder weather money is any concern at all.
- LeonM 6y ago
  You forget your own cost here. A full-time system administrator costs more than 72k a year.
  - walrus01 6y ago
    Once an organization reaches a certain size it will need one, who ideally should be a person that can wear the dual hats of linux/bsd sysadmin and also network engineer. If the person is already on payroll doing a number of other duties, the time/effort to set up such an environment as described in the post could be as short as a couple of days work.
  - nlitened 6y ago
    Yeah, but the absence of system administrator has just cost these guys 72k for several hours.
    - LeonM 6y ago
      I'm just trying to explain that server cost is more than just the hardware. In most cases cloud computing is actually still a very cost effective solution to infrastructure. But with infinite scalability also comes responsibility. In the case of the OP, had they had their own hardware they would have noticed that had written bad code (it would have crashed or become very slow at least), but the cloud just scaled up and processed their code. I'm not trying to defend Google in this case. Billing 72k when a 100 USD limit is set sounds like a scam.
  - gruturo 6y ago
    And for half the use cases you still need one. Unless you go full SaaS (which may or may not be an option depending what you are doing, and what's on offer in that field), you still get stuff which needs to be administered, updated, patched, etc. Maybe not the OS layer (or maybe even that), maybe not the DB (but then it might cost more), but you're not getting away from that. You only really start saving at some scale (get a small core of cloud-literate admins, and now you can have them run thousands of systems for effectively no incremental cost)
svrtknst 6y ago
cloud providers are scam artist, dont @ me
tlarkworthy 6y ago
And that's why you set the 'autoscaling.knative.dev/maxScale' for cloud run https://github.com/futurice/terraform-examples/blob/2ccb2fa3c4f7f9f23786630985156c66a286eb17/google_cloud/camunda-secure/camunda.tf#L75 https://github.com/futurice/terraform-examples/blob/2ccb2fa3...
- YawningAngel 6y ago
  A sane implementation would default this to a low value
  - steren 6y ago
    Cloud Run PM here: I'm sorry for the bad experience the customer shared in this article, we could certainly do better with bill management. We pick 1,000 as a default value for "maxScale", this can be considered high for some users, but low for users who expect infinite scaling from the service and start with a load test to evaluate it.
    - quesera 6y ago
      > We pick 1,000 as a default value for "maxScale", this can be considered high for some users, but low for users who expect infinite scaling from the service and start with a load test to evaluate it. That seems absurd to me. I think it makes much more sense to put the onus on the sophisticated customer to increase their maxScale to an unusual value. Users who "expect infinite scaling...and start with a load test" are sophisticated users. E.g. set maxScale low, like 2 or 4. The sophisticated customer would recognize their oversight quickly. Click-click, fixed, restart test. Effectively 100% of less-sophisticated customers will not need enormous scale on day 1. Customers with whom you do not have an existing billing relationship in the 10s of thousands of dollars per cycle will almost certainly not want it. I'd consider that level of overspecification to be a strong anti-pattern.
      - steren 6y ago
        My response at https://news.ycombinator.com/item?id=25379846 https://news.ycombinator.com/item?id=25379846
walrus01 6y ago
writing as somebody who runs a big collection of bare-metal hypervisors for ISP infrastructure purposes... this post quite honestly just makes me smirk. I have truly lost track of the numerous instances, and number of people who would be better served by buying a $1200 test/development 1U dual socket server with a few fast SSDs in it, and putting it in colocation somewhere for a few hundred dollars a month. The costs would be absolutely fixed and known. On a tight budget? You don't even need to go as far as $1200, I see totally fine test/development environment suitable, Dell 1U servers on eBay right now for under $500 with 128GB of RAM. Or that would be better off purchasing a fixed-configuration virtual machine (typically running on xen or kvm underneath) that has a certain specific amount of CPU, RAM and storage resources allocated to it which cannot balloon. For a fixed bill per month like $65 or $85. You want to deploy your weird app on some cloud platform? sure, go for it, once you've got the possible scaling-up cost issues and possible bugs worked out on your own platform.
- skrebbel 6y ago
  > The costs would be absolutely fixed and known. Until the server breaks and you have to drive over in the middle of the night and try to replace it but the only available server right now is a shitty one and oh shit only half the backups work cause the onsite backups are fried too etc etc etc. There's many good arguments against high-level BaaS such as Firebase but I'm not sure that "colo is cheaper" is one.
  - walrus01 6y ago
    fine, buy two identical ones. and set up proper backups. or even make the backup a hot-spare. if it's a test/development system that's meant to possibly break, you shouldn't be driving anywhere at 0300 in the morning anyways.
    - wrkronmiller 6y ago
      That implies you are going to be running prod in the cloud. Unless you're developing against purely synthetic data, the data transfer costs are potentially astronomical.
daneel_w 6y ago
It's "fantastic" how Google by the end of the article still come out as a good and friendly bunch...
- spacemanmatt 6y ago
  Author is an ex-Googler so he knew exactly how to speak to the system
  - bjarneh 6y ago
    I'm sure that was an important factor, but wouldn't this in any case just be a bill to a startup with no money on hand? It's hard to make companies (without money) economically responsible for anything I guess, it even seems hard to make companies with money responsible sometimes.
    - donmcronald 6y ago
      I can’t set a limit like that on my CC, so the first charge for $5k would have cleared meaning it would have run way longer and racked up way more usage. I’d bet you my computer I would have been out at least the $5k that cleared.
      - bjarneh 6y ago
        You're probably correct. It really makes you think twice about setting up some of those cloud services without a hard limit cutoff.
        Symbiote 6y ago
        I normally receive only my "part" of my corporate credit card statement, but earlier this year I was sent more of it. That's when I found out the card has a credit limit of over €50,000. Reading this, I wonder if we should contact the bank and ask for another card with a lower limit, to use with various cloud services. We are 99% on-premises, but have about €200/month in various GCS/AWS usage.
thrower123 6y ago
I'm not sure I want to know how much Azure and AWS revenue comes from people spinning up test VMs or a kubernetes cluster to work through a training, and then forgetting to turn it off. I've spent thousands extra this year because people stood up 4 MB SQL databases and let them default to charging by vCores instead of DTUs.
- beoberha 6y ago
  Much less than the amount from deals with strategic partners. The long tail of $5 a month from forgotten VMs is likely orders of magnitude less than the handshake deals you can publically read about.
akh 6y ago
> Had we chosen max-instances to be “2”, our costs would’ve been 500 times less. $72,000 bill would’ve been: $144. Had we chosen concurrency of “1” request, we probably wouldn’t have even noticed the bill. > If you count the number of pages in GCP documentation, it’s probably more than pages in few novels. Understanding Pricing, Usage, is not only time consuming, but requires a deep understanding of how Cloud services work. No wonder there are full time jobs for just this purpose! Great write-up - thanks for sharing @bharatsb! As you say, cloud pricing has become too complex for developers to understand quickly (they want to ship features, not calculate costs). Infra-as-code is great, but it has made it even harder to understand which code/config option costs what. `terraform apply` is like a checkout screen without prices. We're trying to solve this problem with infracost.io, initially looking at Terraform. It would be interesting to get your feedback on whether such an approach might have helped you? Probably not as it doesn't look like you were using Terraform?
- spoopyskelly 6y ago
  > Understanding Pricing, Usage, is not only time consuming, but requires a deep understanding of how Cloud services work Maybe you should brush up on that before launching another app that nobody needs?
FastQ 6y ago
I'm just a student but I've spent about 10 hours trying to figure out why Azure has been charging me >$5/day for their "basic" database @5DTUs, 2gb max storage. This morning I was so exasperated I sent a letter threatening to report them for fraud if nobody could tell me why I was being charged 30x the listed rate, which so far no one has. This is an extremely cathartic post to see that I'm not alone, thanks for sharing.
- deleted 6y ago
  [deleted]
- marktolson 6y ago
  Go to billing > cost analysis > filter by resource break down. Azure billing analysis is pretty amazing.
  - FastQ 6y ago
    Yeah, but it just shows my database cost which is higher than is listed as far as I can tell.
    - luser007 6y ago
      Could it be listed "hourly" and you're charged "daily"? Add in VAT (equal to 25% in some countries) and you match the 30 times higher than expected charge.
      - ylere 6y ago
        https://azure.microsoft.com/en-us/pricing/details/sql-database/single/ https://azure.microsoft.com/en-us/pricing/details/sql-databa... Basic tier, 5 DTUs, 2 GB is listed as ~$4.8971/month or $0.0068/hour on this page. Extra storage would cost more but is not available for the basic tier.
        mmcconnell1618 6y ago
        Do you have geo-replication turned on? More regions will be an additional $5/month (plus bandwidth between regions) if you replicate. You can serve everything out of a single region but it is pretty easy to add others if you're not paying attention during initial setup.
LordHeini 6y ago
That sort of crap is the reason we host all our stuff on root servers. Even trying to read the amazon pricing for their instances, hours and what not, drives me insane. Seems this is done on purpose. no wonder they make so much money with it. So i have never seen a reason to move any stuff to the cloud. Just grab a dedicated server for a few bucks and put a bunch of docker containers on those. Its way cheaper, usually not more complicated. Just use a CI with Gitlab runners or whatever and be done with it. Most apps don't need scaling anyway and if you do, just put that app on bare metal fitting your requirements.
- scrollaway 6y ago
  AWS pricing is not obscure, it's just not for you. So in that sense, you are correct to not see a reason to move to the cloud, but your advice does not apply to everyone. And I don't believe they make "more money" that way at all. AWS margins are either very low or very high, and the higher margins and prices tend to be the "simpler" ones: packaged, managed products such as Redshift that are billed on fewer tiers and flatter prices. When you design your application with AWS, pricing has to enter your design considerations. For example if you are designing something that will interact a lot with S3 you want to minimize PUTs. You want to minimize ram usage on lambda by streaming rather than buffering. Etc. AWS is not a suitable product for playground stuff. The only reason it gets used as such is because it's easier if you're already using AWS for other things (or it's you're already very familiar with it).
  - nojito 6y ago
    AWS's margin is currently 30+% which is massive. >AWS pricing is not obscure There is a massive secondary consulting market because of AWS's price obscurities.
    - scrollaway 6y ago
      The margin is absolutely not the same across all products. > There is a massive secondary consulting market because of AWS's price obscurities. Its. Not. For. You. AWS pricing is a part of your design. With some exceptions (that you aren't talking about), they charge you more for using more resources. You are forced to design systems that use less resources if you want to optimize your bill. That consulting market is an optimization market. It's economics at its best. If you are too small to have to take these things into account regardless, AWS is not for you. You're welcome to use it, but don't be surprised if you end up having to deal with these kinds of things which simply don't exist in the world of flat-price underprovisioned droplets.
greatgib 6y ago
At the end of page 2, there is a good ass licking bullshit sentence: << It’s also a great company to collaborate with. The tools provided by Google are very developer friendly, have a great documentation (for the most part), and are consistently expanding.>> He said that as an ex googler and as the beneficiary of a gesture, but this contradict the full history he told us. If the doc and tools were so great, why he felt into this situation?
- sudcha 6y ago
  OP here. Maybe it comes across like that, but as somebody building a product with very limited resources, Google's documentation is one of the best so far. The situation was our fault too. I just went with a test and fail fast attitude, just like with every things we do a dev environment.
  - greatgib 6y ago
    Let's be realistic. You are a team of a few persons, using a 'starting' free plan. How great tools and ux could lead you to not have clue on what they are doing and being able to incur a so high cost by surprise? Also, for example, in which world is it nice to have your service "auto upgraded" from "trial" mode to 72k "full billing" mode without your consent first?
MattyMc 6y ago
I know it's not cool these days, but I strongly prefer (and advise) fully-managed cloud services like Heroku. I can fix my database size and scale/resources (dynos) easily. It's simple, and controlled.
- qayxc 6y ago
  Or just test on a good old VM, which can be had for just a few cents per hour and doesn't even allow for storage or network traffic going out of hand. The first mistake is to deploy tests on completely opaque hyper-scalers. Pretty much any software infrastructure - from (SQL/No-SQL/In-Memory/etc.-) databases to entire web-frameworks can be found as ready-to-go VM images and containers these days. Sure, it's a bit more work to find and setup, but in the end you gain an understanding what system is actually doing, how it might behave and the ability to deploy into any environment - from local workstations to bare metal to (a fleet of) VMs to high-level hyper-scaler services.
jjk166 6y ago
> I jumped out of the bed, logged into Google Cloud Billing, and saw a bill for ~$5,000. Super stressed, and not sure what happened, I clicked around, trying to figure out what was happening. I also started thinking of what may have happened, and how we could “possibly” pay the $5K bill. > The problem was, every minute the bill kept going up. > After 5 minutes, the bill read $15,000, in 20 mins, it said $25,000. I wasn’t sure where it would stop. Perhaps it won’t stop? > After two hours, it settled at a little short of $72,000. > By this time, my team and I were on a call, I was in a state of complete shock and had absolutely no clue about what we would do next. We disabled billing, closed all services. 1) Why wouldn't you shut off the service as soon as you saw the $5000 bill? Really doesn't sound like a "hop on a call with the team for a few hours" kind of decision. 2) Why was the person taking a nap the only person who could get a usage limit alert? One of the great benefits of a team is that you can have multiple eyes looking out for problems. Someone could have raised a flag as soon as the first unexpected alert came in. 3) If going over the free tier limit was your chief concern, why not check the usage after a quick run before letting it go overnight and unsupervised? That the problem could get this bad is a UX failure, but the problem itself is easily seeable and avoidable.
- yawnxyz 6y ago
  I think the article said the Firebase dashboard data was at least 24 hours behind what they were getting billed
delduca 6y ago
happened to me something very similar, I was using cloud run to fetch some subreddit posts and ended with a "recursive" way, because of that, billions of invocations was made... luckly, I was at the front of the computer and stopped before, but the bill was around $4000! I contacted the google support and explained all, they "forgot" my debit because of the bug
- qayxc 6y ago
  So basically no one's testing their code anymore and just throws it into a paid service? Great that Google was so lenient and all, but I really don't get the appeal of using a hyper-scaler when a VM with docker support can be setup in literally seconds and on-demand pricing of less than 10 cents/hour for most quick-and-dirty tasks. Am I missing something here?
pwinnski 6y ago
This is most developers' worst nightmare when it comes to a completely new environment generally, and Cloud solutions specifically. It's easy and pointless to say they should have done things differently. Worse than pointless. Obviously they should have, and kudos to them about being open about the compounded mistakes. Still, this strikes at the fears that lie in the heart of any reasonable, honest developer doing something completely new. New developers should be cautious about cloud platforms, but they were! Not cautious enough, obviously, but they did set limits they thought would be honored. Platforms should have hard monetary limits at the account level, clearly, as well as an option to turn them off. Shame on all of them which don't.
mlboss 6y ago
I don't understand why developers use cloud for bootstrapping/side project. Digital Ocean is all you need $5 droplet + $15 Postgres or even better $7 dyno on Heroku.
- com2kid 6y ago
  I built my startup on a combo of DO and Firebase. If I knew something was going to take more than a couple dozen milliseconds to run, it was built on the DO droplet. Why would I pay by the CPU second for something that is taking a lot of CPU seconds? That billing model doesn't make sense. For my super quick REST endpoints, yeah, all on Firebase, the convenience of writing + deploying makes it an obvious win. (Unless something goes wrong, debugging Firebase functions is not fun...)
- WrtCdEvrydy 6y ago
  $5 droplet, 2GB swap (set it internally) and run Caprover. Deploy your Postgres (for DB), minio (for s3 storage) and your webapp from Caprover. Add nodes as you need to scale out.
rawgabbit 6y ago
Meh. I see this all the time with developers who want to abstract everything away and not worry about the impact their poorly performing code is having on the infrastructure or on the $$bottom line. Time out? No problem, we will just spin up more instances. I have heard that so many times. Maybe your code is just bad.
dilatedmind 6y ago
interesting, how did the spend breakdown between cloud run and firebase? did you have any limit to how many req/s you made to an individual site? It seems this would be difficult to implement with this architecture. how did you deal with following links in circles/ avoiding scraping the same page multiple times? I had built something similiar at a previous job, recursively scraping ecommerce sites. The first thing I noticed was some of the sites we were scaping couldn't handle more than a couple requests a second (in particular as we scaped uncached pages by sites running php). Other sites were quick to ip ban. I kept things simple, a few dozen micro instances on aws (think they were like $3 a day) running puppeteer. A single server acting as a controller, keeping a per site queue and allowing us to set per site request limits if necessary. All the state of which links were already seen just kept in memory. Of course everything was also persisted to a db, and if the controller process needed to be restarted, it could restore the queue/ seen state and resume.
kerng 6y ago
This is really scary. It's so unpredictable what one actually has to pay, especially for a small business moving to the cloud is much more challenging then it should be. When creating resources it's really unclear what one might be charged, then there are saving plans and pre-commitment options and so forth. Might be a good startup idea, basically just sell cloud resources via a simple, predictable payment model.
marcell 6y ago
If this happens you can usually reach out to Google to see if they will refund the charge. They don’t really benefit from making $72k of a solo developer’s buggy code. I’ve done it once and their team was very helpful and reversed the charge.
Alupis 6y ago
As J. Paul Getty once mused[1]: > If you owe the bank $100 that's your problem. If you owe the bank $100 million, that's the bank's problem. Crappy situation for OP and his startup, but I find the part about reading up on bankruptcy to be a bit premature. Perhaps not the most ethical choice, but what stops OP from just not paying the bill, and finding a different cloud provider? Obviously they'll want to not repeat the "experiment", but seriously... there's no mechanism at Google to stop a new client from running up a near-$100k bill in a single day? That's absurd, and should be a learning lesson for Google more than this startup. Some malicious actor could apparently consume hundreds of thousands of dollars of Google resources and "get away" with it. Wait and see what happens, then deal with it - would be sane advice. [1] https://www.brainyquote.com/quotes/j_paul_getty_129274 https://www.brainyquote.com/quotes/j_paul_getty_129274
- kabirgoel 6y ago
  One of my favorite quotes of all time. J. Paul Getty was quite the weirdo. His Wikipedia article is worth a look, especially the section on his frugality.
  - pontifier 6y ago
    Lol. I love it. I moved to a state I'd never considered because it had the largest, cheapest building in the US. It's 220,000 square feet, but I've lived in a tent out back for the last 6 months because I can't get an occupancy permit, it's not zoned residential, and I refuse to pay rent on an apartment.
    - sterlind 6y ago
      You live in a 220,000 square foot building?! Is this an abandoned missile silo or something? I want to know more.
      - pontifier 6y ago
        It's the old headquarters of Varco Pruden. They manufactured steel buildings, and there are long, wide manufacturing bays with overhead cranes. You can see much more on my YouTube channel. I've got a few videos of different areas. This is the first video I made there showing a bit of the size: https://youtu.be/qdkLzioUiAE https://youtu.be/qdkLzioUiAE
altdatathrow 6y ago
Do not use hosted cloud services where the implementation creates publicly accessible API keys and each HTTP request results in a charge to your account. A few specific examples are Firebase, Algolia, and AWS Lambda. All it takes is one programming mistake or one bad actor and you can find yourself in an equally precarious situation.
- mmcconnell1618 6y ago
  Yep, you should wrap then in something like Apigee or an API Server that can throttle requests and keep traffic to reasonable numbers for your service.
seanwilson 6y ago
https://blog.tomilkieway.com/72k-2/ https://blog.tomilkieway.com/72k-2/ > To overcome the timeout limitation, I suggested using POST requests (with URL as data) to send jobs to an instance, and use multiple instances in parallel instead of using one instance serially. Because each instance in Cloud Run would only be scraping one page, it would never time out, process all pages in parallel (scale), and also be highly optimized because Cloud Run usage is accurate to milliseconds. > If you look closely, the flow is missing few important pieces. > Exponential Recursion without Break: The instances wouldn’t know when to break, as there was no break statement. > The POST requests could be of the same URLs. If there’s a back link to the previous page, the Cloud Run service will be stuck in infinite recursion, but what’s worst is, that this recursion is multiplying exponentially (our max instances were set to 1000!) Did you not consider how to stop this blowing up before implementing? Having one cloud function trigger another like this with no way to control how many functions are running at the same time with no simple and quickly met termination condition (with uncapped billing) is playing with fire. It's not going to be optimal either if most of the time each function is waiting for the URL data to download. You need to be using something like a work queue, or just keep life simple and keep it on a single server if you can.
- stefan_ 6y ago
  We've all had a program crash from a stack overflow. The problem seems to be that instead of the "serverless panacea" they were promised, the code they built can now only run on one of many Google servers, none of which are theirs. No way to kick the tires at all. It honestly reminds me of debugging a Jenkins pipeline. Something that was designed to be super generic of a runtime but yet the tooling can inexplicably only live on computers that are not your local development machine, and all of it is maximally painful to stub or test or debug to seduce you into "just running it live". It's like the opposite of the "small agile team" thing they were talking about. If your program requires 7 API keys and some cloud environment to do a test run, I want no part of it.
  - seanwilson 6y ago
    > We've all had a program crash from a stack overflow. Launching a cloud function that recursively triggers the same cloud function, that doesn't have a simple safeguard for it looping or blowing up, and where billing scales with the number of cloud functions ticks the "very high risk" and "very high impact" boxes for me. A program running on a single server isn't similar here (you could accidentally create a DoS attack though). Typical cloud function use is some event gets triggered like a user sign up, the function executes, then it halts. The above isn't a standard use case and is so incredibly risky this approach shouldn't be attempted in my opinion.
sudcha 6y ago
OP here. Just found out that the post made it to HN. Thanks for sharing and I'll be replying to some comments.
salmonlogs 6y ago
As an ex-Googler working in a customer facing role in Cloud you did very well to get a $72k bill written off! It's definitely possible but requires a lot of approvals and pulling in a few favours. I went through the process to write off a ~$50k bill for one of my customers and it required action every day for 3 months of my life. Whoever helped you inside Google will have gone to a LOT of trouble, opened a bunch of tickets and attended many, many meetings to make this happen.
- cogman10 6y ago
  I know there's no reason for Google or AWS to do this, but man do I wish there was a way to put down a spending limit and simply disable anything that goes over that limit. It's a little bit nuts that there are no guardrails to prevent you from incurring such huge bills (especially as a solo developer that might just be trying out their services).
  - fweespeech 6y ago
    Tbh, its lack is why I don't use Google or AWS for projects.
  - brianwawok 6y ago
    There are guard rails in quotas. Like you can only spin up X servers without opening a ticket to ask for more. Now, think some of these quotas can still lead to some pretty crazy bills.. but that is the point of at least some of them....
    - dathinab 6y ago
      They are broken, unreliable, hard to correctly setup guard rails. I mean like the article mentioned they could have set the instances and concurrency settings to lower values. Which in this case would have worked. But finding the right settings to balance intentional auto-scaling and limiting auto-scaling to limit of how fast unexpected cost might rise is hard and prone to get wrong. Let's be honest it's in the end a very flawed workaround which maybe might help (if you know about it, and did it right).
tony 6y ago
Google eventually forgave the bill in Part 2: https://blog.tomilkieway.com/72k-2/ https://blog.tomilkieway.com/72k-2/ > Google let go of our bill as a one time gesture! Thank goodness. And it looks like it had to do with not understanding the API / system on the first order, IMO. This hit me hard a few months ago with CloudFront invalidations on AWS. I check billing and the things at 30usd in a single day, from a norm of <1usd per month, so it's showing a 13,000% increase (this is for documentation of open source projects). I'm writing their support and at the mercy so to speak - technically I ran up the bill. I ended up paying up, but I secretly hoped I'd get some AWS credits for the projects, heh Aside: Amazon has some nice features for rule-based alarms on accounts so when you spend more than X dollars, you get an email.
lewich 6y ago
So when in IT industry there will be responsibility for what we engineer?
awinter-py 6y ago
fwiw I've had cloud vendors be relatively willing to forgive bills when something went wrong with SLAs, bad bugs, or their internal dashes misrepresented usage. they know their systems aren't perfect, and if you velvet hammer them long enough, they'll do the right thing.
ljm 6y ago
I think I'll treat this as the latest in a several line of warnings about not going all-in on all these Cloud services until you seriously know what you're doing. So much of it is so unnecessary to begin with. You can do so much with a cheap VCS or two without thinking about lambdas or cloud functions or kubernetes or who knows what. But these days you'd be forgiven for thinking it's dark magic. You're not going to run up a 5 digit bill in a day by starting up on a few $10 VPSs. And you'll probably have an architecture that fits in your head to boot. Also: The article title should really be "Saved 72k and avoided bankruptcy by being an ex-Googler."
- tunesmith 6y ago
  Just don't go all-in on them at all unless you're spending someone else's money.
tedunangst 6y ago
What about the bill for the sites this request cannon was pointed at?
Twirrim 6y ago
Disclaimer I work for another cloud (not AWS), opinions are entirely my own. I try to avoid posting in a negative fashion about clouds, but holy crap this blog post... AWS has this principle of Customer Obsession that enters in to lots of discussions, design decisions etc. "What is the customer experience of $foo?". Along with asking the positive, you ask the negative too, and explore the customer impact of shit going wrong. What does the worst experience look like, what is the impact for the customer, how might you mitigate that or make it so you can at least make it up to customers quickly, if you really can't avoid it. I find it hard to fathom Sudeep's attitude here. So much of this article is ringing large alarm bells. These are not the things I'd want to see from a cloud provider as a customer. Is this Stockholm Syndrome? Too much drinking of Kool-Aid as an ex-googler? Unfamiliarity with how other cloud providers operate? (from part 1) > Automatic Upgrade of Firebase Account to Paid Account This is what I mean when I say look at negative vs positive use cases. I'm guessing some combination of customers having a lousy experience running in to Free Tier limits, and staff spending too long having to bump up accounts. So they implemented an automatic upgrade (What, then, is the point of a free tier? No room to experiment, no room to try it and see) This is precisely the sort of thing that customer obsession principle is supposed to aid in. Automated upgrade certainly solves the staffing time spent bumping up accounts, and it helps customers that used to have to request limits being increased, but it massively fails in the negative customer experience side of the equation here. Someone, somewhere, should have asked the question "What if the customer has made a mistake". Instead, make it easy and quick for anyone to click a button and get their account changed from Free to Paid, without staff engagement. Give customers easy agency to control their experience. > Billing “Limits” don’t exist. Budgets are at least a day late. That's insane. Clouds are about speed and dynamic scalability. Mistakes can ramp up the bill an crazy amount in a short period of time, as Sudeep found out. How is a 24 hour delay in billing sync and budget warnings even remotely acceptable to them / Sudeep / customers? Sure it's probably fine for the 90% cases, but that's crippling for the 10% and even if you decide you really don't give a crap about your customers, you don't want the bad press that 10% will likely give you. Picture what financial damage someone might do if they compromised some of your credentials somehow? You screwed up, credentials got leaked, and you won't necessarily know for a day that something has gone wrong, nor will your restrictions kick in?! Billing is the single highest TPS service in any cloud, with Identity often a close second (billing gets requests for every transaction, and internal requests related to ongoing charges). You need to handle a high rate of requests, with low latency both in request/response and processing data received. It's a hard engineering problem, and cloud platforms try to get some of the smartest engineers working on it. An organisation of Google's caliber has more than enough smart engineers to be working on these kinds of hard problems, even by temporary secondment. Quota / Limits in a fast changing cloud environment need to be dynamic and responsive. >I knew how to put the case for Google team when they would come back to work in 2 days. How is 2 days even remotely acceptable? Maybe it's just how it's written, but it reads like this is just accepted as the way things are. Why would you even have to carefully work to present your case? Where are the 24x7 response people with the ability to forgive bills? $72k is chump change for a cloud provider, and especially for a company of the scale of Google. Give your support agents the tools and authority they need to make reasonable decisions, with some appropriate kind of oversight process, and stick in feedback mechanisms so product managers know what problems customers are having. It's not like that would actually have cost them $72k in direct running costs either. That should have been a near instant no-brainer. Forgive, move on, and reap the benefits of good customer good will. That good will will earn you way more profit than forgiving it would have cost. You're investing in their continuing business. Sometimes those investments will fail, but most of the time they'll succeed. >In our case, it differed by 86,585,365.85 %, or 86 million percentage points. Even when the bill was notified to us, Firebase Console dashboard still said 42,000 read+writes for the month (below the daily limit). So it's just fake observability? What's the point? 24 hours delay here is nuts, almost to the point of being useless. It can be hard to calculate these figures out yourselves. A fast feedback cycle is critical. As Sudeep here found out, 24 hours is a great way to have zero clue what's going on until it's too late. Is there really no other way to get this information more up-to-date? Moving on to part 2: >I had a team of ~7 engineers/interns at this time, and it would take Google about 10 days to get back to us on this incident. Why is a 10 day response time from Google considered even remotely acceptable for a cloud provider? Your entire platform is down, you're working out ways around this situation, stressing about potential bankruptcy, and it's just cool with you that it took 10 days for them to make a business/life changing decision over what amounts to chump change? These kinds of mistakes happen with clouds, AWS is famous for waving these shock bills from mistakes and it never takes 10 days to get it done. Billing should be the easiest and most obvious thing. If your cloud provider is creating complicated billing structures, that's a problem the cloud provider should be solving, not expecting customers to unravel the mysteries. Companies being spun up to help people navigate your billing should be an alarm call, not something to celebrate or for customers to consider normal. > Fail fast, learn fast with Cloud is a bad idea It shouldn't be. With near immediate feedback you'd have known straight away that shit was bad, and cut the experiment out before it cost you an arm and a leg. > While creating a Cloud Run service, we chose default values in the service. The max-instances is preset to 1000, and concurrency set to 80 ...... Same goes with Cloud Run! With Concurrency == 60, max_containers == 1000 and each Request taking 400ms, number of requests Cloud Run can handle 9 million requests per minute! Why are the default values that high on a service? That seems like you're asking customers to shoot themselves in the foot. Where was the look at the negative customer experience side of the equation? Make it easy for customers to do the right thing. Then the bit that really bugs the crap out of me: > Thank you Google! He's thanking Google for having had an absolutely shitty experience on their platform: 10 days of stress from needless delays in forgiving a trivially small bill, dealings with multiple lawyers, investigating bankruptcy, risk of missing product launch date, working around the clock to dig themselves out of hell...
tunesmith 6y ago
Deeply dissatisfying to read. Ex-Googler uses connections to get his (understandable!) cloud mistake refunded. Every time I read one of these stories, I get more and more convinced I will just simply never use scalable cloud tech for my side projects. I'm not going to risk my family's retirement savings on the all-too-possible chance that a small deep-implication error will cause runaway charges.
- joshmanders 6y ago
  I'm absolutely astounded that cloud providers allow stuff to get this way and don't even go "ope, this looks out of the ordinary, we should look into it" Nor do they offer the ability to straight up kill all services if exceeding a certain price set by the customer. Cloud is still good though, I believe it's the future. I just don't believe in deceiving your customers to hopefully rack up a high bill with them. I view not having the ability to say "Shut everything down if I go over $100/mo" the same as pre-checked hidden cross sales that MasterCard/Visa cracked down on in the adult industry few years ago. Just money grabs. I will definitely be putting such measures in my cloud platform.
- sudcha 6y ago
  OP here. Your assumption is incorrect. I haven't been in touch with anyone in Google, and used 0 internal connections. Happy to make another post with my conversation + documentation to support this. I reached out to the GCP through their regular channels. This is not a paid post, and we are not sponsored by Google in anyway.
  - Exuma 6y ago
    You only wrote half the story and no technical details...... why even write a post at all, it's just clickbait and you give no other information how what went wrong, how it got fixed, what exact technical problem was...
    - dsymonds 6y ago
      Did you read Part 2, linked from the bottom? It has plenty of technical details.
steren 6y ago
(Cloud Run PM here) I am sorry for the experience described in the blog post, we could definitely be better at bill management. I am glad that it worked out in the end and the customer was not required to pay for the bill. Based on this experience, we decided to lower the default value of "max instances" to 100 for future deployments. We believe 100 is a better trade off between allowing customers to scale out and preventing big billing surprises. Of course, customers can always decrease it or increase it up to 1,000, or even above with a simple quota increase request.
- oblio 6y ago
  Well, the real question for all cloud providers, for which I expect crickets as an answer, is: Why don't cloud providers allow setting a budget which cannot be exceeded? A simple, 1-click way to say: this account should never go over $500 a month. Just stop creating resources or responding to requests if it does.
  - shaicoleman 6y ago
    AWS recently released AWS Budget Actions which should allow you to do that. https://aws.amazon.com/blogs/aws-cost-management/get-started-with-aws-budgets-actions/ https://aws.amazon.com/blogs/aws-cost-management/get-started...
    - oblio 6y ago
      It still looks rather complex but if it can actually enforce a budget, that would be great.
  - aidenn0 6y ago
    My guess is that the billing logic is separate from the application logic. There's probably a delay between the two and mostly one-way communication.
    - Retailer 6y ago
      1) But they supported this before on GAE. GAE had 'spending limits'. 2) Also if they are able to figure out when you've hit your daily free quota and cut you off almost immediately, how are they not able to figure this out?
asciimike 6y ago
Every time I see another post like this I always wonder how many people would be willing to buy "cloud insurance" where a premium would cover overages due to your mistakes in dev, outages, etc. I don't have an exact billing model worked out, and I assume the insurance provider would mandate certain practices (e.g. setting up billing alerts that go to them, allowing them to view/manage infra), but curious if people here would be willing to pay for such a thing. (My assumption is that most people who fall into this are too small to be willing to pay a reasonable % of their infra spend or change their infra practices to prevent this, but I'd be curious if this is something CIOs of companies who are thinking of "moving to the cloud but are leery due to cost concerns").
- meekrohprocess 6y ago
  You might be able to get an insurance company to sign on to that sort of thing; have you shopped for quotes? They'd probably want to look at a representative sample of "cloud overrun" events and see how much they cost, how likely they are, what reasonable measures could help prevent them, etc. But they aren't strangers to taking unusual bets. Apparently a lot of big moonshot / X-prize type competitions fund their prize money through insurance. You present your research stating why the problem is hard, the insurance company works out how likely it is for someone to manage the challenge, and they come up with a quote for footing the bill in the unlikely event that someone wins.
- sjg007 6y ago
  Does AWS implement spending limits? I would think that would be sufficient, if not, at least a start.
kordlessagain 6y ago
> The next morning on Saturday, March 28th, I called and emailed over a dozen law firms to book an appointment / have a chat with some attorney. OP is a dick.
- rangibaby 6y ago
  Why?
- sudcha 6y ago
  OP here. Ouch, that hurt.
- mwcampbell 6y ago
  More likely they were scared out of their mind and frantically trying to do anything and everything they could to handle the situation.
alasdair_ 6y ago
As a kind of reverse of this, I once had a Google App Engine site with a reasonable $300/day limit on it. We ended up getting a bunch of international news and usage spiked massively (hooray!) and I tried to update the limit manually only to find out I had to wait 24 hours before the new limit applied. This mean losing a TON of new users :(
maddyboo 6y ago
BRB, disabling all of my serverless functions.
infinitelooper 6y ago
The algorithm this ex-googles came up with is hilarious, hard to believe this guy came out of google. I thought they were big on algorithms in their interview process? Maybe I should consider putting in an application. If I was interviewing an applicant and they put down the marker after producing that in a whiteboard session they would have quite a bit of work to do in order to get to the next interview phase. Not only did ex-googler not catch those mistakes, his team of 7 actually sat down and coded it without spotting the problem.
segmondy 6y ago
If there's ever been a case against using GCP or AWS this is it. You better understand those systems or you will be in for the shock of your life when you get hit with this sort of crazy billing. I got $3,000 AWS credit and was terrified of running experiments in case I made a mistake. It should not be hard to set hard billing limits, these companies know how to set hard CPU limits, why can't they do that with billing?
Uptrenda 6y ago
It seems like you could easily build a fintech app that gives you a virtual pre-paid credit card to use with these services and gets funded only up until your budget amount. That might be a safer way to work with services like this. Could have a separate card where ever you expect a problem might occur. That would also avoid one service being able to take down everything else you need to bill Card and billing management? Probably a legit need tbh
tilolebo 6y ago
The main reason why cloud providers can't provide this killswitch feature everybody talk about here is probably that they just can't. I don't think any of them is capable of billing in real time. So the monitoring would have to happen on resource consumption. This is already enforced with resource limits in many cases.
- freemint 6y ago
  Why would this be an impossible technical challenge? I see that it would generate overhead however i know VPSPs which bill by the minute or second.
archi42 6y ago
And another day that I'm again totally surprised how some people do "engineering". Basic problems like recursion don't occur to them, they use a technology they don't understand, they are not careful with the number of instances and then burn ~16000 (or close to 2y) of cpu time. Why not test the algorithm locally, realize the problems and fix them? Why test on 1000 instances, not 1 or 2? I get the "move fast, fail fast" attitude and why it is deemed beneficial by some, but this was essentially "goto fail".
fancythat 6y ago
I don't understand why people keep doing this to themselves. OP is ex-Googler and we can surely state that he knows his way around tech. Why would you, as a startup, go on route of using convoluted mess of tech that every cloud provider is, instead of buying cheap dedicated server, setting your own k8s or whatever and just using that platform for your project. Cloud solutions are immensely and, for most cases, unnecessary complicated. In addition to that, they have the ability to kill any dream of yours with invoice far exceeding your darkest forecasts in less than an hour. The time OP spent solving this issue could be better spent learning about self-hosting and avoiding this trouble all together.
XCSme 6y ago
For anyone who wonders, the summary of the mistake: created a web crawler without any stop condition and without adding any checks for not visiting the same page twice. The crawler just kept running for a day, making around 9 million requests per minute.
nojvek 6y ago
This dives me mad. There needs to be a max bill per month setting. If you’re near the limit, bells and whistles go off, If you exceed then new things aren’t spun up. GCP refuses to write to storage or run compute. Or they absorb the cost. If SpaceX can send massive starship and bellyflop it, sure a company with 40,000 engineers can figure this out. This just says Google will not focus on a thing that gives shitty experience to a user but helps them make money.

282 points by bharatsb 6y ago

Thank you for sharing. I was actually thinking of using fire base for my project. They make us so easy to sign up for free tier. Awaiting to see what happens in part 2

bharatsb 6y ago
Part 2: https://blog.tomilkieway.com/72k-2/ https://blog.tomilkieway.com/72k-2/
spacemanmatt 6y ago
I recently put the (soft) kibosh on a project in my stable trying to switch to FireBase at the last minute. It looks attractive but the business aspects are frankly frightening, and I'm not even talking about the risk of a large bill. Getting your metrics 24h late sounds like a deal killer for me. So much for observability!
- asciimike 6y ago
  Minor nit: many non-billing metrics are near-real time, e.g. DB concurrents, cloud functions CPU/RAM usage; any metrics that require aggregation (storage, billing) are going to be batched less frequently. This is going to be true across all platforms of non-trivial scale (eventual consistency + batch jobs). Second note: the number of people who actually do this is very low (a few a year, of hundreds of thousands of developers). The blog posts are scary, but in my ~five years at Firebase, I'm pretty sure we refunded every one. As my boss (James Tamplin, CEO of Firebase) used to say, "There are lots of bad systems, but rarely are there bad people."
  - spacemanmatt 6y ago
    > I'm pretty sure we refunded every one. That's good to know. It seems like a particularly visible part of the cloud credibility puzzle to be able to monitor/limit dynamic resource costs.

onion2k 6y ago

The fact that cloud providers don't have a simple "This is how much I can afford, don't ever bill me more than that!" box on their platforms makes development a lot scarier than it really needs to be.

jsiepkes 6y ago
Not only development but also running in production. You can configure alerts but you can't configure a hard limit. Thats just insane. That makes working with GCP like playing with fire.
- k__ 6y ago
  What about throttling?
  - herendin2 6y ago
    Nice to have, but people want a throttle that shuts off dead at a certain number of dollars
  - pwinnski 6y ago
    aka "Bankrupt me more slowly" Throttling doesn't stop the drain.
serial_dev 6y ago
It is baffling why cloud providers don't have that option. I might want to have an app because I don't mind spending 50 dollars on my pet project as a hobby, but I don't ever want to spend more than that. Not if I write a wrong query that's suddenly becomes very expensive, not when I got attacked, and not even when I have legit users. By the way, the same goes for some companies, too, just the threshold would be different.
- greatgib 6y ago
  It's not complicated to add configurable hard limits for these companies but they don't allow it because the current situation is more interesting for them. They want to suck the maximum money from consumers before they realize. For one person that will complain wildly and having to do a gesture, there are hundreds other companies that will not notice or just pay without recourse.

bharatsb 6y ago

Part 2: https://blog.tomilkieway.com/72k-2/ https://blog.tomilkieway.com/72k-2/

MaxBarraclough 6y ago
> Google let go of our bill as a one time gesture! We've seen this happen with similar stories on AWS. Neither platform supports prepayment with a hard limit on costs, and this seems unlikely to change.
- teekert 6y ago
  Yeah a friend of mine wanted a real cert, not letsencrypt (I don't understand how that is more real but ok), as a bit of a noob he clicked around on the AWS website and some days later had a bill op 1500 eur. They also nulled it. Still, this scares the hell out of me.
  - WrtCdEvrydy 6y ago
    AWS Certificate Manager gives you free non-extended SSL to your machine, it's pretty nifty.
- coddle-hark 6y ago
  I can sympathise with some of these stories, like the ones where an overnight DDOS attack racks up a huge unexpected bill, but this one in particular is just a story of gross incompetence and negligence. The guy hacked together some code in a few days and deployed it to a service with unlimited billing without any kind of sanity checks and without even understanding what he was paying for. He’s an ex-Googler, it’s not like he hasn’t heard stories like this before. And the takeaway? “Oops don’t deploy buggy code” and “I shouldn’t have used the default settings”. OK, sure, let me know how that works out for you.
  - teekert 6y ago
    But "The Guy" was under the impression that he could do so because he set sane limits in both GCP and his Credit Card.

villgax 6y ago

I'd happily use a platform that allowed for an option for limiting billing on a daily/monthly/another metric.

raphaelj 6y ago
Some platforms do that: - Heroku costs are pretty predicable, and you can easily set a maximum scalability threshold to their auto-scalable dynos, so that they will never cost you more than a predefined amount of money; - BunnyCDN requires me to top-up their prepaid account, so that I'll never spend more than what I have on that account.

atian 6y ago

You can ask for a good faith billing adjustment. GCE or AWS is well aware that things happen, and collecting something is better than collecting nothing.

ceejayoz 6y ago
Part two of the article indicates they did that, and had the bill nulled out.
- pettycashstash2 6y ago
  from part 2: "After going through our lengthy doc on this incident sharing our side of the story, various consults, talks, and internal discussions Google let go of our bill as a one time gesture!"

throwaway7281 6y ago

To put it into perspective: You give me $72K and I'll set you up a 1PB replicated storage infra with a total of 100+ available CPU cores and half a TB RAM. I saw people burning through cash in the cloud, which makes you wonder weather money is any concern at all.

LeonM 6y ago
You forget your own cost here. A full-time system administrator costs more than 72k a year.
- walrus01 6y ago
  Once an organization reaches a certain size it will need one, who ideally should be a person that can wear the dual hats of linux/bsd sysadmin and also network engineer. If the person is already on payroll doing a number of other duties, the time/effort to set up such an environment as described in the post could be as short as a couple of days work.
- nlitened 6y ago
  Yeah, but the absence of system administrator has just cost these guys 72k for several hours.
  - LeonM 6y ago
    I'm just trying to explain that server cost is more than just the hardware. In most cases cloud computing is actually still a very cost effective solution to infrastructure. But with infinite scalability also comes responsibility. In the case of the OP, had they had their own hardware they would have noticed that had written bad code (it would have crashed or become very slow at least), but the cloud just scaled up and processed their code. I'm not trying to defend Google in this case. Billing 72k when a 100 USD limit is set sounds like a scam.
- gruturo 6y ago
  And for half the use cases you still need one. Unless you go full SaaS (which may or may not be an option depending what you are doing, and what's on offer in that field), you still get stuff which needs to be administered, updated, patched, etc. Maybe not the OS layer (or maybe even that), maybe not the DB (but then it might cost more), but you're not getting away from that. You only really start saving at some scale (get a small core of cloud-literate admins, and now you can have them run thousands of systems for effectively no incremental cost)

svrtknst 6y ago

cloud providers are scam artist, dont @ me

tlarkworthy 6y ago

And that's why you set the 'autoscaling.knative.dev/maxScale' for cloud run https://github.com/futurice/terraform-examples/blob/2ccb2fa3c4f7f9f23786630985156c66a286eb17/google_cloud/camunda-secure/camunda.tf#L75 https://github.com/futurice/terraform-examples/blob/2ccb2fa3...

YawningAngel 6y ago
A sane implementation would default this to a low value
- steren 6y ago
  Cloud Run PM here: I'm sorry for the bad experience the customer shared in this article, we could certainly do better with bill management. We pick 1,000 as a default value for "maxScale", this can be considered high for some users, but low for users who expect infinite scaling from the service and start with a load test to evaluate it.
  - quesera 6y ago
    > We pick 1,000 as a default value for "maxScale", this can be considered high for some users, but low for users who expect infinite scaling from the service and start with a load test to evaluate it. That seems absurd to me. I think it makes much more sense to put the onus on the sophisticated customer to increase their maxScale to an unusual value. Users who "expect infinite scaling...and start with a load test" are sophisticated users. E.g. set maxScale low, like 2 or 4. The sophisticated customer would recognize their oversight quickly. Click-click, fixed, restart test. Effectively 100% of less-sophisticated customers will not need enormous scale on day 1. Customers with whom you do not have an existing billing relationship in the 10s of thousands of dollars per cycle will almost certainly not want it. I'd consider that level of overspecification to be a strong anti-pattern.
    - steren 6y ago
      My response at https://news.ycombinator.com/item?id=25379846 https://news.ycombinator.com/item?id=25379846

walrus01 6y ago

writing as somebody who runs a big collection of bare-metal hypervisors for ISP infrastructure purposes... this post quite honestly just makes me smirk. I have truly lost track of the numerous instances, and number of people who would be better served by buying a $1200 test/development 1U dual socket server with a few fast SSDs in it, and putting it in colocation somewhere for a few hundred dollars a month. The costs would be absolutely fixed and known. On a tight budget? You don't even need to go as far as $1200, I see totally fine test/development environment suitable, Dell 1U servers on eBay right now for under $500 with 128GB of RAM. Or that would be better off purchasing a fixed-configuration virtual machine (typically running on xen or kvm underneath) that has a certain specific amount of CPU, RAM and storage resources allocated to it which cannot balloon. For a fixed bill per month like $65 or $85. You want to deploy your weird app on some cloud platform? sure, go for it, once you've got the possible scaling-up cost issues and possible bugs worked out on your own platform.

skrebbel 6y ago
> The costs would be absolutely fixed and known. Until the server breaks and you have to drive over in the middle of the night and try to replace it but the only available server right now is a shitty one and oh shit only half the backups work cause the onsite backups are fried too etc etc etc. There's many good arguments against high-level BaaS such as Firebase but I'm not sure that "colo is cheaper" is one.
- walrus01 6y ago
  fine, buy two identical ones. and set up proper backups. or even make the backup a hot-spare. if it's a test/development system that's meant to possibly break, you shouldn't be driving anywhere at 0300 in the morning anyways.
  - wrkronmiller 6y ago
    That implies you are going to be running prod in the cloud. Unless you're developing against purely synthetic data, the data transfer costs are potentially astronomical.

daneel_w 6y ago

It's "fantastic" how Google by the end of the article still come out as a good and friendly bunch...

spacemanmatt 6y ago
Author is an ex-Googler so he knew exactly how to speak to the system
- bjarneh 6y ago
  I'm sure that was an important factor, but wouldn't this in any case just be a bill to a startup with no money on hand? It's hard to make companies (without money) economically responsible for anything I guess, it even seems hard to make companies with money responsible sometimes.
  - donmcronald 6y ago
    I can’t set a limit like that on my CC, so the first charge for $5k would have cleared meaning it would have run way longer and racked up way more usage. I’d bet you my computer I would have been out at least the $5k that cleared.
    - bjarneh 6y ago
      You're probably correct. It really makes you think twice about setting up some of those cloud services without a hard limit cutoff.
      - Symbiote 6y ago
        I normally receive only my "part" of my corporate credit card statement, but earlier this year I was sent more of it. That's when I found out the card has a credit limit of over €50,000. Reading this, I wonder if we should contact the bank and ask for another card with a lower limit, to use with various cloud services. We are 99% on-premises, but have about €200/month in various GCS/AWS usage.

thrower123 6y ago

I'm not sure I want to know how much Azure and AWS revenue comes from people spinning up test VMs or a kubernetes cluster to work through a training, and then forgetting to turn it off. I've spent thousands extra this year because people stood up 4 MB SQL databases and let them default to charging by vCores instead of DTUs.

beoberha 6y ago
Much less than the amount from deals with strategic partners. The long tail of $5 a month from forgotten VMs is likely orders of magnitude less than the handshake deals you can publically read about.

akh 6y ago

> Had we chosen max-instances to be “2”, our costs would’ve been 500 times less. $72,000 bill would’ve been: $144. Had we chosen concurrency of “1” request, we probably wouldn’t have even noticed the bill. > If you count the number of pages in GCP documentation, it’s probably more than pages in few novels. Understanding Pricing, Usage, is not only time consuming, but requires a deep understanding of how Cloud services work. No wonder there are full time jobs for just this purpose! Great write-up - thanks for sharing @bharatsb! As you say, cloud pricing has become too complex for developers to understand quickly (they want to ship features, not calculate costs). Infra-as-code is great, but it has made it even harder to understand which code/config option costs what. `terraform apply` is like a checkout screen without prices. We're trying to solve this problem with infracost.io, initially looking at Terraform. It would be interesting to get your feedback on whether such an approach might have helped you? Probably not as it doesn't look like you were using Terraform?

spoopyskelly 6y ago
> Understanding Pricing, Usage, is not only time consuming, but requires a deep understanding of how Cloud services work Maybe you should brush up on that before launching another app that nobody needs?

FastQ 6y ago

I'm just a student but I've spent about 10 hours trying to figure out why Azure has been charging me >$5/day for their "basic" database @5DTUs, 2gb max storage. This morning I was so exasperated I sent a letter threatening to report them for fraud if nobody could tell me why I was being charged 30x the listed rate, which so far no one has. This is an extremely cathartic post to see that I'm not alone, thanks for sharing.

deleted 6y ago
[deleted]
marktolson 6y ago
Go to billing > cost analysis > filter by resource break down. Azure billing analysis is pretty amazing.
- FastQ 6y ago
  Yeah, but it just shows my database cost which is higher than is listed as far as I can tell.
  - luser007 6y ago
    Could it be listed "hourly" and you're charged "daily"? Add in VAT (equal to 25% in some countries) and you match the 30 times higher than expected charge.
    - ylere 6y ago
      https://azure.microsoft.com/en-us/pricing/details/sql-database/single/ https://azure.microsoft.com/en-us/pricing/details/sql-databa... Basic tier, 5 DTUs, 2 GB is listed as ~$4.8971/month or $0.0068/hour on this page. Extra storage would cost more but is not available for the basic tier.
      - mmcconnell1618 6y ago
        Do you have geo-replication turned on? More regions will be an additional $5/month (plus bandwidth between regions) if you replicate. You can serve everything out of a single region but it is pretty easy to add others if you're not paying attention during initial setup.

LordHeini 6y ago

That sort of crap is the reason we host all our stuff on root servers. Even trying to read the amazon pricing for their instances, hours and what not, drives me insane. Seems this is done on purpose. no wonder they make so much money with it. So i have never seen a reason to move any stuff to the cloud. Just grab a dedicated server for a few bucks and put a bunch of docker containers on those. Its way cheaper, usually not more complicated. Just use a CI with Gitlab runners or whatever and be done with it. Most apps don't need scaling anyway and if you do, just put that app on bare metal fitting your requirements.

scrollaway 6y ago
AWS pricing is not obscure, it's just not for you. So in that sense, you are correct to not see a reason to move to the cloud, but your advice does not apply to everyone. And I don't believe they make "more money" that way at all. AWS margins are either very low or very high, and the higher margins and prices tend to be the "simpler" ones: packaged, managed products such as Redshift that are billed on fewer tiers and flatter prices. When you design your application with AWS, pricing has to enter your design considerations. For example if you are designing something that will interact a lot with S3 you want to minimize PUTs. You want to minimize ram usage on lambda by streaming rather than buffering. Etc. AWS is not a suitable product for playground stuff. The only reason it gets used as such is because it's easier if you're already using AWS for other things (or it's you're already very familiar with it).
- nojito 6y ago
  AWS's margin is currently 30+% which is massive. >AWS pricing is not obscure There is a massive secondary consulting market because of AWS's price obscurities.
  - scrollaway 6y ago
    The margin is absolutely not the same across all products. > There is a massive secondary consulting market because of AWS's price obscurities. Its. Not. For. You. AWS pricing is a part of your design. With some exceptions (that you aren't talking about), they charge you more for using more resources. You are forced to design systems that use less resources if you want to optimize your bill. That consulting market is an optimization market. It's economics at its best. If you are too small to have to take these things into account regardless, AWS is not for you. You're welcome to use it, but don't be surprised if you end up having to deal with these kinds of things which simply don't exist in the world of flat-price underprovisioned droplets.

greatgib 6y ago

At the end of page 2, there is a good ass licking bullshit sentence: << It’s also a great company to collaborate with. The tools provided by Google are very developer friendly, have a great documentation (for the most part), and are consistently expanding.>> He said that as an ex googler and as the beneficiary of a gesture, but this contradict the full history he told us. If the doc and tools were so great, why he felt into this situation?

sudcha 6y ago
OP here. Maybe it comes across like that, but as somebody building a product with very limited resources, Google's documentation is one of the best so far. The situation was our fault too. I just went with a test and fail fast attitude, just like with every things we do a dev environment.
- greatgib 6y ago
  Let's be realistic. You are a team of a few persons, using a 'starting' free plan. How great tools and ux could lead you to not have clue on what they are doing and being able to incur a so high cost by surprise? Also, for example, in which world is it nice to have your service "auto upgraded" from "trial" mode to 72k "full billing" mode without your consent first?

MattyMc 6y ago

I know it's not cool these days, but I strongly prefer (and advise) fully-managed cloud services like Heroku. I can fix my database size and scale/resources (dynos) easily. It's simple, and controlled.

qayxc 6y ago
Or just test on a good old VM, which can be had for just a few cents per hour and doesn't even allow for storage or network traffic going out of hand. The first mistake is to deploy tests on completely opaque hyper-scalers. Pretty much any software infrastructure - from (SQL/No-SQL/In-Memory/etc.-) databases to entire web-frameworks can be found as ready-to-go VM images and containers these days. Sure, it's a bit more work to find and setup, but in the end you gain an understanding what system is actually doing, how it might behave and the ability to deploy into any environment - from local workstations to bare metal to (a fleet of) VMs to high-level hyper-scaler services.

jjk166 6y ago

> I jumped out of the bed, logged into Google Cloud Billing, and saw a bill for ~$5,000. Super stressed, and not sure what happened, I clicked around, trying to figure out what was happening. I also started thinking of what may have happened, and how we could “possibly” pay the $5K bill. > The problem was, every minute the bill kept going up. > After 5 minutes, the bill read $15,000, in 20 mins, it said $25,000. I wasn’t sure where it would stop. Perhaps it won’t stop? > After two hours, it settled at a little short of $72,000. > By this time, my team and I were on a call, I was in a state of complete shock and had absolutely no clue about what we would do next. We disabled billing, closed all services. 1) Why wouldn't you shut off the service as soon as you saw the $5000 bill? Really doesn't sound like a "hop on a call with the team for a few hours" kind of decision. 2) Why was the person taking a nap the only person who could get a usage limit alert? One of the great benefits of a team is that you can have multiple eyes looking out for problems. Someone could have raised a flag as soon as the first unexpected alert came in. 3) If going over the free tier limit was your chief concern, why not check the usage after a quick run before letting it go overnight and unsupervised? That the problem could get this bad is a UX failure, but the problem itself is easily seeable and avoidable.

yawnxyz 6y ago
I think the article said the Firebase dashboard data was at least 24 hours behind what they were getting billed

delduca 6y ago

happened to me something very similar, I was using cloud run to fetch some subreddit posts and ended with a "recursive" way, because of that, billions of invocations was made... luckly, I was at the front of the computer and stopped before, but the bill was around $4000! I contacted the google support and explained all, they "forgot" my debit because of the bug

qayxc 6y ago
So basically no one's testing their code anymore and just throws it into a paid service? Great that Google was so lenient and all, but I really don't get the appeal of using a hyper-scaler when a VM with docker support can be setup in literally seconds and on-demand pricing of less than 10 cents/hour for most quick-and-dirty tasks. Am I missing something here?

pwinnski 6y ago

This is most developers' worst nightmare when it comes to a completely new environment generally, and Cloud solutions specifically. It's easy and pointless to say they should have done things differently. Worse than pointless. Obviously they should have, and kudos to them about being open about the compounded mistakes. Still, this strikes at the fears that lie in the heart of any reasonable, honest developer doing something completely new. New developers should be cautious about cloud platforms, but they were! Not cautious enough, obviously, but they did set limits they thought would be honored. Platforms should have hard monetary limits at the account level, clearly, as well as an option to turn them off. Shame on all of them which don't.

mlboss 6y ago

I don't understand why developers use cloud for bootstrapping/side project. Digital Ocean is all you need $5 droplet + $15 Postgres or even better $7 dyno on Heroku.

com2kid 6y ago
I built my startup on a combo of DO and Firebase. If I knew something was going to take more than a couple dozen milliseconds to run, it was built on the DO droplet. Why would I pay by the CPU second for something that is taking a lot of CPU seconds? That billing model doesn't make sense. For my super quick REST endpoints, yeah, all on Firebase, the convenience of writing + deploying makes it an obvious win. (Unless something goes wrong, debugging Firebase functions is not fun...)
WrtCdEvrydy 6y ago
$5 droplet, 2GB swap (set it internally) and run Caprover. Deploy your Postgres (for DB), minio (for s3 storage) and your webapp from Caprover. Add nodes as you need to scale out.

rawgabbit 6y ago

Meh. I see this all the time with developers who want to abstract everything away and not worry about the impact their poorly performing code is having on the infrastructure or on the $$bottom line. Time out? No problem, we will just spin up more instances. I have heard that so many times. Maybe your code is just bad.

dilatedmind 6y ago

interesting, how did the spend breakdown between cloud run and firebase? did you have any limit to how many req/s you made to an individual site? It seems this would be difficult to implement with this architecture. how did you deal with following links in circles/ avoiding scraping the same page multiple times? I had built something similiar at a previous job, recursively scraping ecommerce sites. The first thing I noticed was some of the sites we were scaping couldn't handle more than a couple requests a second (in particular as we scaped uncached pages by sites running php). Other sites were quick to ip ban. I kept things simple, a few dozen micro instances on aws (think they were like $3 a day) running puppeteer. A single server acting as a controller, keeping a per site queue and allowing us to set per site request limits if necessary. All the state of which links were already seen just kept in memory. Of course everything was also persisted to a db, and if the controller process needed to be restarted, it could restore the queue/ seen state and resume.

kerng 6y ago

This is really scary. It's so unpredictable what one actually has to pay, especially for a small business moving to the cloud is much more challenging then it should be. When creating resources it's really unclear what one might be charged, then there are saving plans and pre-commitment options and so forth. Might be a good startup idea, basically just sell cloud resources via a simple, predictable payment model.

marcell 6y ago

If this happens you can usually reach out to Google to see if they will refund the charge. They don’t really benefit from making $72k of a solo developer’s buggy code. I’ve done it once and their team was very helpful and reversed the charge.

Alupis 6y ago

As J. Paul Getty once mused[1]: > If you owe the bank $100 that's your problem. If you owe the bank $100 million, that's the bank's problem. Crappy situation for OP and his startup, but I find the part about reading up on bankruptcy to be a bit premature. Perhaps not the most ethical choice, but what stops OP from just not paying the bill, and finding a different cloud provider? Obviously they'll want to not repeat the "experiment", but seriously... there's no mechanism at Google to stop a new client from running up a near-$100k bill in a single day? That's absurd, and should be a learning lesson for Google more than this startup. Some malicious actor could apparently consume hundreds of thousands of dollars of Google resources and "get away" with it. Wait and see what happens, then deal with it - would be sane advice. [1] https://www.brainyquote.com/quotes/j_paul_getty_129274 https://www.brainyquote.com/quotes/j_paul_getty_129274

kabirgoel 6y ago
One of my favorite quotes of all time. J. Paul Getty was quite the weirdo. His Wikipedia article is worth a look, especially the section on his frugality.
- pontifier 6y ago
  Lol. I love it. I moved to a state I'd never considered because it had the largest, cheapest building in the US. It's 220,000 square feet, but I've lived in a tent out back for the last 6 months because I can't get an occupancy permit, it's not zoned residential, and I refuse to pay rent on an apartment.
  - sterlind 6y ago
    You live in a 220,000 square foot building?! Is this an abandoned missile silo or something? I want to know more.
    - pontifier 6y ago
      It's the old headquarters of Varco Pruden. They manufactured steel buildings, and there are long, wide manufacturing bays with overhead cranes. You can see much more on my YouTube channel. I've got a few videos of different areas. This is the first video I made there showing a bit of the size: https://youtu.be/qdkLzioUiAE https://youtu.be/qdkLzioUiAE

altdatathrow 6y ago

Do not use hosted cloud services where the implementation creates publicly accessible API keys and each HTTP request results in a charge to your account. A few specific examples are Firebase, Algolia, and AWS Lambda. All it takes is one programming mistake or one bad actor and you can find yourself in an equally precarious situation.

mmcconnell1618 6y ago
Yep, you should wrap then in something like Apigee or an API Server that can throttle requests and keep traffic to reasonable numbers for your service.

seanwilson 6y ago

https://blog.tomilkieway.com/72k-2/ https://blog.tomilkieway.com/72k-2/ > To overcome the timeout limitation, I suggested using POST requests (with URL as data) to send jobs to an instance, and use multiple instances in parallel instead of using one instance serially. Because each instance in Cloud Run would only be scraping one page, it would never time out, process all pages in parallel (scale), and also be highly optimized because Cloud Run usage is accurate to milliseconds. > If you look closely, the flow is missing few important pieces. > Exponential Recursion without Break: The instances wouldn’t know when to break, as there was no break statement. > The POST requests could be of the same URLs. If there’s a back link to the previous page, the Cloud Run service will be stuck in infinite recursion, but what’s worst is, that this recursion is multiplying exponentially (our max instances were set to 1000!) Did you not consider how to stop this blowing up before implementing? Having one cloud function trigger another like this with no way to control how many functions are running at the same time with no simple and quickly met termination condition (with uncapped billing) is playing with fire. It's not going to be optimal either if most of the time each function is waiting for the URL data to download. You need to be using something like a work queue, or just keep life simple and keep it on a single server if you can.

stefan_ 6y ago
We've all had a program crash from a stack overflow. The problem seems to be that instead of the "serverless panacea" they were promised, the code they built can now only run on one of many Google servers, none of which are theirs. No way to kick the tires at all. It honestly reminds me of debugging a Jenkins pipeline. Something that was designed to be super generic of a runtime but yet the tooling can inexplicably only live on computers that are not your local development machine, and all of it is maximally painful to stub or test or debug to seduce you into "just running it live". It's like the opposite of the "small agile team" thing they were talking about. If your program requires 7 API keys and some cloud environment to do a test run, I want no part of it.
- seanwilson 6y ago
  > We've all had a program crash from a stack overflow. Launching a cloud function that recursively triggers the same cloud function, that doesn't have a simple safeguard for it looping or blowing up, and where billing scales with the number of cloud functions ticks the "very high risk" and "very high impact" boxes for me. A program running on a single server isn't similar here (you could accidentally create a DoS attack though). Typical cloud function use is some event gets triggered like a user sign up, the function executes, then it halts. The above isn't a standard use case and is so incredibly risky this approach shouldn't be attempted in my opinion.

sudcha 6y ago

OP here. Just found out that the post made it to HN. Thanks for sharing and I'll be replying to some comments.

salmonlogs 6y ago

As an ex-Googler working in a customer facing role in Cloud you did very well to get a $72k bill written off! It's definitely possible but requires a lot of approvals and pulling in a few favours. I went through the process to write off a ~$50k bill for one of my customers and it required action every day for 3 months of my life. Whoever helped you inside Google will have gone to a LOT of trouble, opened a bunch of tickets and attended many, many meetings to make this happen.

cogman10 6y ago
I know there's no reason for Google or AWS to do this, but man do I wish there was a way to put down a spending limit and simply disable anything that goes over that limit. It's a little bit nuts that there are no guardrails to prevent you from incurring such huge bills (especially as a solo developer that might just be trying out their services).
- fweespeech 6y ago
  Tbh, its lack is why I don't use Google or AWS for projects.
- brianwawok 6y ago
  There are guard rails in quotas. Like you can only spin up X servers without opening a ticket to ask for more. Now, think some of these quotas can still lead to some pretty crazy bills.. but that is the point of at least some of them....
  - dathinab 6y ago
    They are broken, unreliable, hard to correctly setup guard rails. I mean like the article mentioned they could have set the instances and concurrency settings to lower values. Which in this case would have worked. But finding the right settings to balance intentional auto-scaling and limiting auto-scaling to limit of how fast unexpected cost might rise is hard and prone to get wrong. Let's be honest it's in the end a very flawed workaround which maybe might help (if you know about it, and did it right).

tony 6y ago

Google eventually forgave the bill in Part 2: https://blog.tomilkieway.com/72k-2/ https://blog.tomilkieway.com/72k-2/ > Google let go of our bill as a one time gesture! Thank goodness. And it looks like it had to do with not understanding the API / system on the first order, IMO. This hit me hard a few months ago with CloudFront invalidations on AWS. I check billing and the things at 30usd in a single day, from a norm of <1usd per month, so it's showing a 13,000% increase (this is for documentation of open source projects). I'm writing their support and at the mercy so to speak - technically I ran up the bill. I ended up paying up, but I secretly hoped I'd get some AWS credits for the projects, heh Aside: Amazon has some nice features for rule-based alarms on accounts so when you spend more than X dollars, you get an email.

lewich 6y ago

So when in IT industry there will be responsibility for what we engineer?

awinter-py 6y ago

fwiw I've had cloud vendors be relatively willing to forgive bills when something went wrong with SLAs, bad bugs, or their internal dashes misrepresented usage. they know their systems aren't perfect, and if you velvet hammer them long enough, they'll do the right thing.

ljm 6y ago

I think I'll treat this as the latest in a several line of warnings about not going all-in on all these Cloud services until you seriously know what you're doing. So much of it is so unnecessary to begin with. You can do so much with a cheap VCS or two without thinking about lambdas or cloud functions or kubernetes or who knows what. But these days you'd be forgiven for thinking it's dark magic. You're not going to run up a 5 digit bill in a day by starting up on a few $10 VPSs. And you'll probably have an architecture that fits in your head to boot. Also: The article title should really be "Saved 72k and avoided bankruptcy by being an ex-Googler."

tunesmith 6y ago
Just don't go all-in on them at all unless you're spending someone else's money.

tedunangst 6y ago

What about the bill for the sites this request cannon was pointed at?

Twirrim 6y ago

Disclaimer I work for another cloud (not AWS), opinions are entirely my own. I try to avoid posting in a negative fashion about clouds, but holy crap this blog post... AWS has this principle of Customer Obsession that enters in to lots of discussions, design decisions etc. "What is the customer experience of $foo?". Along with asking the positive, you ask the negative too, and explore the customer impact of shit going wrong. What does the worst experience look like, what is the impact for the customer, how might you mitigate that or make it so you can at least make it up to customers quickly, if you really can't avoid it. I find it hard to fathom Sudeep's attitude here. So much of this article is ringing large alarm bells. These are not the things I'd want to see from a cloud provider as a customer. Is this Stockholm Syndrome? Too much drinking of Kool-Aid as an ex-googler? Unfamiliarity with how other cloud providers operate? (from part 1) > Automatic Upgrade of Firebase Account to Paid Account This is what I mean when I say look at negative vs positive use cases. I'm guessing some combination of customers having a lousy experience running in to Free Tier limits, and staff spending too long having to bump up accounts. So they implemented an automatic upgrade (What, then, is the point of a free tier? No room to experiment, no room to try it and see) This is precisely the sort of thing that customer obsession principle is supposed to aid in. Automated upgrade certainly solves the staffing time spent bumping up accounts, and it helps customers that used to have to request limits being increased, but it massively fails in the negative customer experience side of the equation here. Someone, somewhere, should have asked the question "What if the customer has made a mistake". Instead, make it easy and quick for anyone to click a button and get their account changed from Free to Paid, without staff engagement. Give customers easy agency to control their experience. > Billing “Limits” don’t exist. Budgets are at least a day late. That's insane. Clouds are about speed and dynamic scalability. Mistakes can ramp up the bill an crazy amount in a short period of time, as Sudeep found out. How is a 24 hour delay in billing sync and budget warnings even remotely acceptable to them / Sudeep / customers? Sure it's probably fine for the 90% cases, but that's crippling for the 10% and even if you decide you really don't give a crap about your customers, you don't want the bad press that 10% will likely give you. Picture what financial damage someone might do if they compromised some of your credentials somehow? You screwed up, credentials got leaked, and you won't necessarily know for a day that something has gone wrong, nor will your restrictions kick in?! Billing is the single highest TPS service in any cloud, with Identity often a close second (billing gets requests for every transaction, and internal requests related to ongoing charges). You need to handle a high rate of requests, with low latency both in request/response and processing data received. It's a hard engineering problem, and cloud platforms try to get some of the smartest engineers working on it. An organisation of Google's caliber has more than enough smart engineers to be working on these kinds of hard problems, even by temporary secondment. Quota / Limits in a fast changing cloud environment need to be dynamic and responsive. >I knew how to put the case for Google team when they would come back to work in 2 days. How is 2 days even remotely acceptable? Maybe it's just how it's written, but it reads like this is just accepted as the way things are. Why would you even have to carefully work to present your case? Where are the 24x7 response people with the ability to forgive bills? $72k is chump change for a cloud provider, and especially for a company of the scale of Google. Give your support agents the tools and authority they need to make reasonable decisions, with some appropriate kind of oversight process, and stick in feedback mechanisms so product managers know what problems customers are having. It's not like that would actually have cost them $72k in direct running costs either. That should have been a near instant no-brainer. Forgive, move on, and reap the benefits of good customer good will. That good will will earn you way more profit than forgiving it would have cost. You're investing in their continuing business. Sometimes those investments will fail, but most of the time they'll succeed. >In our case, it differed by 86,585,365.85 %, or 86 million percentage points. Even when the bill was notified to us, Firebase Console dashboard still said 42,000 read+writes for the month (below the daily limit). So it's just fake observability? What's the point? 24 hours delay here is nuts, almost to the point of being useless. It can be hard to calculate these figures out yourselves. A fast feedback cycle is critical. As Sudeep here found out, 24 hours is a great way to have zero clue what's going on until it's too late. Is there really no other way to get this information more up-to-date? Moving on to part 2: >I had a team of ~7 engineers/interns at this time, and it would take Google about 10 days to get back to us on this incident. Why is a 10 day response time from Google considered even remotely acceptable for a cloud provider? Your entire platform is down, you're working out ways around this situation, stressing about potential bankruptcy, and it's just cool with you that it took 10 days for them to make a business/life changing decision over what amounts to chump change? These kinds of mistakes happen with clouds, AWS is famous for waving these shock bills from mistakes and it never takes 10 days to get it done. Billing should be the easiest and most obvious thing. If your cloud provider is creating complicated billing structures, that's a problem the cloud provider should be solving, not expecting customers to unravel the mysteries. Companies being spun up to help people navigate your billing should be an alarm call, not something to celebrate or for customers to consider normal. > Fail fast, learn fast with Cloud is a bad idea It shouldn't be. With near immediate feedback you'd have known straight away that shit was bad, and cut the experiment out before it cost you an arm and a leg. > While creating a Cloud Run service, we chose default values in the service. The max-instances is preset to 1000, and concurrency set to 80 ...... Same goes with Cloud Run! With Concurrency == 60, max_containers == 1000 and each Request taking 400ms, number of requests Cloud Run can handle 9 million requests per minute! Why are the default values that high on a service? That seems like you're asking customers to shoot themselves in the foot. Where was the look at the negative customer experience side of the equation? Make it easy for customers to do the right thing. Then the bit that really bugs the crap out of me: > Thank you Google! He's thanking Google for having had an absolutely shitty experience on their platform: 10 days of stress from needless delays in forgiving a trivially small bill, dealings with multiple lawyers, investigating bankruptcy, risk of missing product launch date, working around the clock to dig themselves out of hell...

tunesmith 6y ago

Deeply dissatisfying to read. Ex-Googler uses connections to get his (understandable!) cloud mistake refunded. Every time I read one of these stories, I get more and more convinced I will just simply never use scalable cloud tech for my side projects. I'm not going to risk my family's retirement savings on the all-too-possible chance that a small deep-implication error will cause runaway charges.

joshmanders 6y ago
I'm absolutely astounded that cloud providers allow stuff to get this way and don't even go "ope, this looks out of the ordinary, we should look into it" Nor do they offer the ability to straight up kill all services if exceeding a certain price set by the customer. Cloud is still good though, I believe it's the future. I just don't believe in deceiving your customers to hopefully rack up a high bill with them. I view not having the ability to say "Shut everything down if I go over $100/mo" the same as pre-checked hidden cross sales that MasterCard/Visa cracked down on in the adult industry few years ago. Just money grabs. I will definitely be putting such measures in my cloud platform.
sudcha 6y ago
OP here. Your assumption is incorrect. I haven't been in touch with anyone in Google, and used 0 internal connections. Happy to make another post with my conversation + documentation to support this. I reached out to the GCP through their regular channels. This is not a paid post, and we are not sponsored by Google in anyway.
- Exuma 6y ago
  You only wrote half the story and no technical details...... why even write a post at all, it's just clickbait and you give no other information how what went wrong, how it got fixed, what exact technical problem was...
  - dsymonds 6y ago
    Did you read Part 2, linked from the bottom? It has plenty of technical details.

steren 6y ago

(Cloud Run PM here) I am sorry for the experience described in the blog post, we could definitely be better at bill management. I am glad that it worked out in the end and the customer was not required to pay for the bill. Based on this experience, we decided to lower the default value of "max instances" to 100 for future deployments. We believe 100 is a better trade off between allowing customers to scale out and preventing big billing surprises. Of course, customers can always decrease it or increase it up to 1,000, or even above with a simple quota increase request.

oblio 6y ago
Well, the real question for all cloud providers, for which I expect crickets as an answer, is: Why don't cloud providers allow setting a budget which cannot be exceeded? A simple, 1-click way to say: this account should never go over $500 a month. Just stop creating resources or responding to requests if it does.
- shaicoleman 6y ago
  AWS recently released AWS Budget Actions which should allow you to do that. https://aws.amazon.com/blogs/aws-cost-management/get-started-with-aws-budgets-actions/ https://aws.amazon.com/blogs/aws-cost-management/get-started...
  - oblio 6y ago
    It still looks rather complex but if it can actually enforce a budget, that would be great.
- aidenn0 6y ago
  My guess is that the billing logic is separate from the application logic. There's probably a delay between the two and mostly one-way communication.
  - Retailer 6y ago
    1) But they supported this before on GAE. GAE had 'spending limits'. 2) Also if they are able to figure out when you've hit your daily free quota and cut you off almost immediately, how are they not able to figure this out?

asciimike 6y ago

Every time I see another post like this I always wonder how many people would be willing to buy "cloud insurance" where a premium would cover overages due to your mistakes in dev, outages, etc. I don't have an exact billing model worked out, and I assume the insurance provider would mandate certain practices (e.g. setting up billing alerts that go to them, allowing them to view/manage infra), but curious if people here would be willing to pay for such a thing. (My assumption is that most people who fall into this are too small to be willing to pay a reasonable % of their infra spend or change their infra practices to prevent this, but I'd be curious if this is something CIOs of companies who are thinking of "moving to the cloud but are leery due to cost concerns").

meekrohprocess 6y ago
You might be able to get an insurance company to sign on to that sort of thing; have you shopped for quotes? They'd probably want to look at a representative sample of "cloud overrun" events and see how much they cost, how likely they are, what reasonable measures could help prevent them, etc. But they aren't strangers to taking unusual bets. Apparently a lot of big moonshot / X-prize type competitions fund their prize money through insurance. You present your research stating why the problem is hard, the insurance company works out how likely it is for someone to manage the challenge, and they come up with a quote for footing the bill in the unlikely event that someone wins.
sjg007 6y ago
Does AWS implement spending limits? I would think that would be sufficient, if not, at least a start.

kordlessagain 6y ago

> The next morning on Saturday, March 28th, I called and emailed over a dozen law firms to book an appointment / have a chat with some attorney. OP is a dick.

rangibaby 6y ago
Why?
sudcha 6y ago
OP here. Ouch, that hurt.
mwcampbell 6y ago
More likely they were scared out of their mind and frantically trying to do anything and everything they could to handle the situation.

alasdair_ 6y ago

As a kind of reverse of this, I once had a Google App Engine site with a reasonable $300/day limit on it. We ended up getting a bunch of international news and usage spiked massively (hooray!) and I tried to update the limit manually only to find out I had to wait 24 hours before the new limit applied. This mean losing a TON of new users :(

maddyboo 6y ago

BRB, disabling all of my serverless functions.

infinitelooper 6y ago

The algorithm this ex-googles came up with is hilarious, hard to believe this guy came out of google. I thought they were big on algorithms in their interview process? Maybe I should consider putting in an application. If I was interviewing an applicant and they put down the marker after producing that in a whiteboard session they would have quite a bit of work to do in order to get to the next interview phase. Not only did ex-googler not catch those mistakes, his team of 7 actually sat down and coded it without spotting the problem.

segmondy 6y ago

If there's ever been a case against using GCP or AWS this is it. You better understand those systems or you will be in for the shock of your life when you get hit with this sort of crazy billing. I got $3,000 AWS credit and was terrified of running experiments in case I made a mistake. It should not be hard to set hard billing limits, these companies know how to set hard CPU limits, why can't they do that with billing?

Uptrenda 6y ago

It seems like you could easily build a fintech app that gives you a virtual pre-paid credit card to use with these services and gets funded only up until your budget amount. That might be a safer way to work with services like this. Could have a separate card where ever you expect a problem might occur. That would also avoid one service being able to take down everything else you need to bill Card and billing management? Probably a legit need tbh

tilolebo 6y ago

The main reason why cloud providers can't provide this killswitch feature everybody talk about here is probably that they just can't. I don't think any of them is capable of billing in real time. So the monitoring would have to happen on resource consumption. This is already enforced with resource limits in many cases.

freemint 6y ago
Why would this be an impossible technical challenge? I see that it would generate overhead however i know VPSPs which bill by the minute or second.

archi42 6y ago

And another day that I'm again totally surprised how some people do "engineering". Basic problems like recursion don't occur to them, they use a technology they don't understand, they are not careful with the number of instances and then burn ~16000 (or close to 2y) of cpu time. Why not test the algorithm locally, realize the problems and fix them? Why test on 1000 instances, not 1 or 2? I get the "move fast, fail fast" attitude and why it is deemed beneficial by some, but this was essentially "goto fail".

fancythat 6y ago

I don't understand why people keep doing this to themselves. OP is ex-Googler and we can surely state that he knows his way around tech. Why would you, as a startup, go on route of using convoluted mess of tech that every cloud provider is, instead of buying cheap dedicated server, setting your own k8s or whatever and just using that platform for your project. Cloud solutions are immensely and, for most cases, unnecessary complicated. In addition to that, they have the ability to kill any dream of yours with invoice far exceeding your darkest forecasts in less than an hour. The time OP spent solving this issue could be better spent learning about self-hosting and avoiding this trouble all together.

XCSme 6y ago

For anyone who wonders, the summary of the mistake: created a web crawler without any stop condition and without adding any checks for not visiting the same page twice. The crawler just kept running for a day, making around 9 million requests per minute.

nojvek 6y ago

This dives me mad. There needs to be a max bill per month setting. If you’re near the limit, bells and whistles go off, If you exceed then new things aren’t spun up. GCP refuses to write to storage or run compute. Or they absorb the cost. If SpaceX can send massive starship and bellyflop it, sure a company with 40,000 engineers can figure this out. This just says Google will not focus on a thing that gives shitty experience to a user but helps them make money.