4 ms·

Disclosure: I work on Google Cloud (but not Firestore or Firebase). For those that didn't read the article, it had a happy ending: > GOOGLE UNDERSTOOD AND POW

by boulos 8y ago

Disclosure: I work on Google Cloud (but not Firestore or Firebase). For those that didn't read the article, it had a happy ending: > GOOGLE UNDERSTOOD AND POWER US UP! > After we fixed this code mistake, and stopped the billing, we reached out to Google to let them know the case and to see if we could apply for the next grant they have for startups. We told them that we spent the full 25k grant we had just a few days ago and see the chance to apply for the 100k grant on Google Cloud Services. We contacted the team of Google Developers Latam, to tell them what had just happened. They allowed us to apply for the next grant, which google approved, and after some meetings with them, they let us pay our bill with the grant. > Now we could not be more grateful to Google, not only for having an awesome “Backend As A Service” like Firebase, but also for letting us have 2 million sessions, 60 supports per minute and billions of requests without letting our site go down. Besides they understood errors like ours can happen when a startup is growing and some expensive mistakes can jeopardize the future great companies.

nstj 8y ago
I don’t see why this is being downvoted. The title of the post should be changed to “We were given a $30k credit by GCP after not monitoring our account charges”
ocdtrekkie 8y ago
Perhaps somewhat optimistically, I assume most of the commenters read the whole article, but I don't think the happy ending abates the concern. It's great the Google Cloud team was able to bail them out afterwards, but the fact that they were able to rack up a $35,000 cost on a code mistake still highlights one of the major flaws with pay-as-you-go cloud computing. There's no guarantee if I made the same snafu next week that Google would necessarily be willing to help, but I can absolutely guarantee you that a VM sitting on a Dell PowerEdge I've got lying around would suddenly obligate me to a $35,000 bill, no matter how bad my code. Ideally, I guess I'd want to see rather than a hard cap, some sort of smart alert that would go "holy crud, this is an unusual spike in the rate of requests" when the delta changed unusually, rather than waiting until I say, hit a high static cost bar or a hard cap that kills the site.
- boulos 8y ago
  Right, I don’t deny that (and didn’t mean to imply this isn’t a problem). At the time of my comment it did seem like most people stopped before the end :). Budgets and quotas are really tricky as Dan pointed out elsewhere in this thread. App Engine has had default daily budgets (that you can change) forever, but then you run into people saying “What the hell, why did you take down my site?!”. In this case, they even intentionally pressed forward once they saw their bill was going up. If this had been say a static VM running MySQL with a “SELECT *” for every page view, the site would likely have just been effectively down. For some customers, that’s the wrong choice, even in the face of a crazy performance bug. That said, we (all) demonstrably need to do better at defaults as well as education (the monitoring exists!).
  - gcbirzan 8y ago
    That, or, as explained in the SRE book, the extra load should've been shed. The binary choice is not always the right one. Though, granted, in this case, it wouldn't have mattered...
    - ocdtrekkie 8y ago
      Yeah, I think "the web server ran out of money at 3 PM today" isn't the right choice, whereas "a lot of people had a hard time getting to the site when it got HN'd around 3 PM, but it was back to normal by 5 PM" is a better result. For smaller organizations, something being down during extreme load is a recoverable problem, but owing the cloud provider all of their money may not be. (Note that even in the case here where Google got them the grant to cover this bill, this is still probably 35K in grant money that could've gotten them further or been used better elsewhere.)

by boulos 8y ago

nstj 8y ago

I don’t see why this is being downvoted. The title of the post should be changed to “We were given a $30k credit by GCP after not monitoring our account charges”

ocdtrekkie 8y ago

Perhaps somewhat optimistically, I assume most of the commenters read the whole article, but I don't think the happy ending abates the concern. It's great the Google Cloud team was able to bail them out afterwards, but the fact that they were able to rack up a $35,000 cost on a code mistake still highlights one of the major flaws with pay-as-you-go cloud computing. There's no guarantee if I made the same snafu next week that Google would necessarily be willing to help, but I can absolutely guarantee you that a VM sitting on a Dell PowerEdge I've got lying around would suddenly obligate me to a $35,000 bill, no matter how bad my code. Ideally, I guess I'd want to see rather than a hard cap, some sort of smart alert that would go "holy crud, this is an unusual spike in the rate of requests" when the delta changed unusually, rather than waiting until I say, hit a high static cost bar or a hard cap that kills the site.

boulos 8y ago
Right, I don’t deny that (and didn’t mean to imply this isn’t a problem). At the time of my comment it did seem like most people stopped before the end :). Budgets and quotas are really tricky as Dan pointed out elsewhere in this thread. App Engine has had default daily budgets (that you can change) forever, but then you run into people saying “What the hell, why did you take down my site?!”. In this case, they even intentionally pressed forward once they saw their bill was going up. If this had been say a static VM running MySQL with a “SELECT *” for every page view, the site would likely have just been effectively down. For some customers, that’s the wrong choice, even in the face of a crazy performance bug. That said, we (all) demonstrably need to do better at defaults as well as education (the monitoring exists!).
- gcbirzan 8y ago
  That, or, as explained in the SRE book, the extra load should've been shed. The binary choice is not always the right one. Though, granted, in this case, it wouldn't have mattered...
  - ocdtrekkie 8y ago
    Yeah, I think "the web server ran out of money at 3 PM today" isn't the right choice, whereas "a lot of people had a hard time getting to the site when it got HN'd around 3 PM, but it was back to normal by 5 PM" is a better result. For smaller organizations, something being down during extreme load is a recoverable problem, but owing the cloud provider all of their money may not be. (Note that even in the case here where Google got them the grant to cover this bill, this is still probably 35K in grant money that could've gotten them further or been used better elsewhere.)