5 ms·
> With all services restored to normal operation, Google’s engineering teams are now conducting a thorough post-mortem ... This is not the post-mortem, that is
by mrunkel 7y ago
> With all services restored to normal operation, Google’s engineering teams are now conducting a thorough post-mortem ...
This is not the post-mortem, that is still to come.
- tbodt 7y agoThe postmortem is currently only visible to Googlers.
- echelon 7y agoWhat happened?
- tylerhou 7y agoTBH the linked update is a pretty accurate summary of the postmortem in my opinion.
- MrBuddyCasino 7y ago> linked update Where?
- tedunangst 7y agohttps://cloud.google.com/blog/topics/inside-google-cloud/an-update-on-sundays-service-disruption https://cloud.google.com/blog/topics/inside-google-cloud/an-...
- kyrra 7y ago+1 (I'm a Googler, opinions my own) As someone who has been on oncall for 3 year and done a decent amount of production support, this public doc better explains the cause than the internal one if you aren't well versed in the underlying infra. A config change reduced used network capacity by half, and then things started falling over. And pushing the fix took a while due to the now overloaded network.
- mcny 7y agoWhat can prevent this from happening in the future? Why was the config change required?
- geofft 7y agoI think that's likely to be answered in the post-mortem, which is still to come.
- mlthoughts2018 7y agoObviously to prevent things like this, Google needs more binary search tree whiteboard trivia problems in the interview process.
- lallysingh 7y agoPrioritize configuration traffic so they can fix the problem quicker.
- jacquesm 7y agoTest changes on smaller parts of the network before pushing them to critical parts would be a first guess. Config changes tend to be nasty in that their implications are often hard to oversee until they have been made, and if the effects preclude you from making another config change then you've just cut off the branch that you were sitting on. Google is best-in-class when it comes to this stuff, the thing you should take away from this is that if they can mess up everybody does. And that pretty much correlates with my experience to date. This stuff is hard, maybe needlessly so but that does not change the fact that it is hard and that accidents can and will happen. So you plan for things to go wrong when you design your systems. Failure is not only an option, it is the default.
- rst 7y agoWhich seems to have been what they were trying to do; according to this update, the config change which caused the problem was intended to apply to a smaller portion of the network than it actually hit. But automated enforcement of procedures like this can also be tricky; how's a machine supposed to know that "this change was already tried on a smaller part of the network"?
- JMTQp8lwXL 7y agoThe article leaves it to the reader to decide whether or not we will be updated with the results of that post-mortem.
- theevilsharpie 7y agoWhile this article leaves it ambiguous, the incident status report[1] makes it clear that customers can expect an actual postmortem: "We will conduct an internal investigation of this issue and make appropriate improvements to our systems to help prevent or minimize future recurrence. We will provide a detailed report of this incident once we have completed our internal investigation. This detailed report will contain information regarding SLA credits." [1] https://status.cloud.google.com/incident/cloud-networking/19009 https://status.cloud.google.com/incident/cloud-networking/19...
- phs318u 7y agoIts the pre-post-mortem.
- laurentl 7y agoThe mortem ?
- awalton 7y agoPerimortem?
- killjoywashere 7y agoIn pathology we call issue a "preliminary autopsy report" within the first 48 hours and a "final autopsy report" within 30-60 days. A final report may be followed by additional addenda (new information, e.g. toxicology reports) and amendments (significant changes to the original report, e.g. "I was wrong")
- OJFord 7y agoYeah, the issue really is that post-mortem's been adopted as a noun. This is a de facto post-mortem <thing> (allowing the adoption of 'mortem' to mean problem, but not the phrase as noun) just perhaps not the full analysis it's been taken to imply.
- dragon_greens 7y agoPre-post-erous mortem?
- thom 7y agoThe purgatorium.