3 ms·
Seconded. Fast recovery of the problem, fast to publish a postmortem, and a very thurough postmortem. Outages suck, but are inevitable even for Google. With a
by nzoschke 11y ago
Seconded. Fast recovery of the problem, fast to publish a postmortem, and a very thurough postmortem.
Outages suck, but are inevitable even for Google. With a response like this Google has gained even more trust from me.
- kyrra 11y agoGoogle's take on postmortems is really nice. As the SRE book points out, they are seen as a learning tool for others. Most internal postmortems are available for anyone within the company to see and learn from. As well, they are always blameless. No fingers are pointed at the person who caused the issue in the postmortem. They explain the issue, what happened, and how it can be prevented in the future. Pair this with the outage tracking tools and you can find all the outages that have happened across Google and what caused them. Then there is DiRT[0] testing to try and catch problems in a controlled manner. Having things break randomly through Google's infrastructure and you have to see if your service's setup and oncall people handle it properly is a really awesome exercise. [0] http://queue.acm.org/detail.cfm?id=2371516 http://queue.acm.org/detail.cfm?id=2371516 The opinions stated here are my own, not necessarily those of Google. Edit: Changed from saying "all" to "most" postmortems being available to Googlers to see.