3 ms·
> I jumped out of the bed, logged into Google Cloud Billing, and saw a bill for ~$5,000. Super stressed, and not sure what happened, I clicked around, trying to
by jjk166 6y ago
> I jumped out of the bed, logged into Google Cloud Billing, and saw a bill for ~$5,000. Super stressed, and not sure what happened, I clicked around, trying to figure out what was happening. I also started thinking of what may have happened, and how we could “possibly” pay the $5K bill.
> The problem was, every minute the bill kept going up.
> After 5 minutes, the bill read $15,000, in 20 mins, it said $25,000. I wasn’t sure where it would stop. Perhaps it won’t stop?
> After two hours, it settled at a little short of $72,000.
> By this time, my team and I were on a call, I was in a state of complete shock and had absolutely no clue about what we would do next. We disabled billing, closed all services.
1) Why wouldn't you shut off the service as soon as you saw the $5000 bill? Really doesn't sound like a "hop on a call with the team for a few hours" kind of decision.
2) Why was the person taking a nap the only person who could get a usage limit alert? One of the great benefits of a team is that you can have multiple eyes looking out for problems. Someone could have raised a flag as soon as the first unexpected alert came in.
3) If going over the free tier limit was your chief concern, why not check the usage after a quick run before letting it go overnight and unsupervised?
That the problem could get this bad is a UX failure, but the problem itself is easily seeable and avoidable.
- yawnxyz 6y agoI think the article said the Firebase dashboard data was at least 24 hours behind what they were getting billed