3 ms·
Right, I don’t deny that (and didn’t mean to imply this isn’t a problem). At the time of my comment it did seem like most people stopped before the end :). Bud
by boulos 8y ago
Right, I don’t deny that (and didn’t mean to imply this isn’t a problem). At the time of my comment it did seem like most people stopped before the end :).
Budgets and quotas are really tricky as Dan pointed out elsewhere in this thread. App Engine has had default daily budgets (that you can change) forever, but then you run into people saying “What the hell, why did you take down my site?!”.
In this case, they even intentionally pressed forward once they saw their bill was going up. If this had been say a static VM running MySQL with a “SELECT *” for every page view, the site would likely have just been effectively down. For some customers, that’s the wrong choice, even in the face of a crazy performance bug.
That said, we (all) demonstrably need to do better at defaults as well as education (the monitoring exists!).
- gcbirzan 8y agoThat, or, as explained in the SRE book, the extra load should've been shed. The binary choice is not always the right one. Though, granted, in this case, it wouldn't have mattered...
- ocdtrekkie 8y agoYeah, I think "the web server ran out of money at 3 PM today" isn't the right choice, whereas "a lot of people had a hard time getting to the site when it got HN'd around 3 PM, but it was back to normal by 5 PM" is a better result. For smaller organizations, something being down during extreme load is a recoverable problem, but owing the cloud provider all of their money may not be. (Note that even in the case here where Google got them the grant to cover this bill, this is still probably 35K in grant money that could've gotten them further or been used better elsewhere.)