3 ms·
Several years back when I was working at Google I made a mistake that caused some of the special results in the knowledge cards to become unclickable for a smal
by zeven7 6y ago
Several years back when I was working at Google I made a mistake that caused some of the special results in the knowledge cards to become unclickable for a small subset of queries for about an hour. As part of the postmortem I had to calculate how many people likely tried to interact with it while it was broken. It was a lot and really made me realize the magnitude of an otherwise seemingly small production failure. My boss didn't give me a hard time, just pointed me toward documentation about how to write the report. And crunching the numbers is what really made me feel the weight of it. It was a good process.
I feel for the engineer who has to calculate the cost of this bug.
- zymhan 6y agoPresumably there were more failures than a single engineer could've been responsible for here.
- fjcp 6y agoIts absolutely possible, the worst AWS outage was caused by one engineer running the wrong command [0]. "This past Tuesday morning Pacific Time an Amazon Web Services engineer was debugging an issue with the billing system for the company’s popular cloud storage service S3 and accidentally mistyped a command. What followed was a several hours’ long cloud outage that wreaked havoc across the internet and resulted in hundreds of millions of dollars in losses for AWS customers and others who rely on third-party services hosted by AWS." [0] - https://www.datacenterknowledge.com/archives/2017/03/02/aws-outage-that-broke-the-internet-caused-by-mistyped-command https://www.datacenterknowledge.com/archives/2017/03/02/aws-...
- robotnikman 6y agoThis sounds like a good practice and hopefully something they still do. Calculating the exact numbers would definitely help cement the experience and its consequences into your mind.