7 ms·

This is a surprisingly vague postmortem. No timeline, no specific identification of affected regions. And no explanation of why a configuration change that was

This is a surprisingly vague postmortem. No timeline, no specific identification of affected regions. And no explanation of why a configuration change that was (apparently) made with a single command required so much effort to undo, or why repair efforts were hampered when (again, apparently) the network was successfully prioritizing high-priority traffic. Even for a public postmortem, this seems pretty weak.

paxys 7y ago
It's definitely not a postmortem. Seems like just a quick update.
mrunkel 7y ago
> With all services restored to normal operation, Google’s engineering teams are now conducting a thorough post-mortem ... This is not the post-mortem, that is still to come.
- tbodt 7y ago
  The postmortem is currently only visible to Googlers.
  - echelon 7y ago
    What happened?
    - tylerhou 7y ago
      TBH the linked update is a pretty accurate summary of the postmortem in my opinion.
      - MrBuddyCasino 7y ago
        > linked update Where?
        tedunangst 7y ago
        https://cloud.google.com/blog/topics/inside-google-cloud/an-update-on-sundays-service-disruption https://cloud.google.com/blog/topics/inside-google-cloud/an-...
      - kyrra 7y ago
        +1 (I'm a Googler, opinions my own) As someone who has been on oncall for 3 year and done a decent amount of production support, this public doc better explains the cause than the internal one if you aren't well versed in the underlying infra. A config change reduced used network capacity by half, and then things started falling over. And pushing the fix took a while due to the now overloaded network.
        mcny 7y ago
        What can prevent this from happening in the future? Why was the config change required?
        geofft 7y ago
        I think that's likely to be answered in the post-mortem, which is still to come.
        mlthoughts2018 7y ago
        Obviously to prevent things like this, Google needs more binary search tree whiteboard trivia problems in the interview process.
        lallysingh 7y ago
        Prioritize configuration traffic so they can fix the problem quicker.
        jacquesm 7y ago
        Test changes on smaller parts of the network before pushing them to critical parts would be a first guess. Config changes tend to be nasty in that their implications are often hard to oversee until they have been made, and if the effects preclude you from making another config change then you've just cut off the branch that you were sitting on. Google is best-in-class when it comes to this stuff, the thing you should take away from this is that if they can mess up everybody does. And that pretty much correlates with my experience to date. This stuff is hard, maybe needlessly so but that does not change the fact that it is hard and that accidents can and will happen. So you plan for things to go wrong when you design your systems. Failure is not only an option, it is the default.
- JMTQp8lwXL 7y ago
  The article leaves it to the reader to decide whether or not we will be updated with the results of that post-mortem.
  - theevilsharpie 7y ago
    While this article leaves it ambiguous, the incident status report[1] makes it clear that customers can expect an actual postmortem: "We will conduct an internal investigation of this issue and make appropriate improvements to our systems to help prevent or minimize future recurrence. We will provide a detailed report of this incident once we have completed our internal investigation. This detailed report will contain information regarding SLA credits." [1] https://status.cloud.google.com/incident/cloud-networking/19009 https://status.cloud.google.com/incident/cloud-networking/19...
- phs318u 7y ago
  Its the pre-post-mortem.
  - laurentl 7y ago
    The mortem ?
    - awalton 7y ago
      Perimortem?
      - killjoywashere 7y ago
        In pathology we call issue a "preliminary autopsy report" within the first 48 hours and a "final autopsy report" within 30-60 days. A final report may be followed by additional addenda (new information, e.g. toxicology reports) and amendments (significant changes to the original report, e.g. "I was wrong")
        OJFord 7y ago
        Yeah, the issue really is that post-mortem's been adopted as a noun. This is a de facto post-mortem <thing> (allowing the adoption of 'mortem' to mean problem, but not the phrase as noun) just perhaps not the full analysis it's been taken to imply.
  - dragon_greens 7y ago
    Pre-post-erous mortem?
  - thom 7y ago
    The purgatorium.
JMTQp8lwXL 7y ago
The lack of transparency makes me want to consider other cloud providers. All providers will have outages -- that's a reality I can live with -- but I will prioritize the ones who are the most forthcoming in their statuses and explanations into failures.
- theevilsharpie 7y ago
  Google kept their status page up to date as the outage was progressing, and now (the day after the outage), they've provided an apology and a preliminary explanation of what happened. If that's not sufficient, what more are you looking for, and what other large cloud providers consistently meet that standard?
  - JMTQp8lwXL 7y ago
    The status page wasn't up to date, but I don't have any way of backing up that claim. It's certainly isn't tied to any automated system failure reporting -- the status page seems to require manual updates. When minutes turn into hours, and no updates are available on the status page, it certainly doesn't leave me at ease.
    - theevilsharpie 7y ago
      We began noticing erratic behavior with some of our GCE instances at around 11:50 Pacific on the day of the outage, and Google posted a notice on their on their status page that GCE was having an outage about 30 minutes later. They also updated the status page every hour (or when they had new information), which is what they said they would do in their status updates. While it sucks that multiple regions malfunctioned simultaneously for several hours, I can't really fault them for their communication about the issue.
      - mh- 7y ago
        I wasn't able to load a google doc from drive, or my calendar (both g suite), around 11:45 PDT.
deleted 7y ago
[deleted]
deadmutex 7y ago
But it's not a post mortem? Since the article states: > Next Steps > With all services restored to normal operation, Google’s engineering teams are now conducting a thorough post-mortem to ensure we understand all the contributing factors
zoobab 7y ago
What do you expect from a blackbox setup?
brown9-2 7y ago
they typically post a full post-mortem in the status page of the original incident: https://status.cloud.google.com/incident/compute/19003 https://status.cloud.google.com/incident/compute/19003
snewman 7y ago
Replying to my own comment: As many have pointed out, this was not actually the postmortem, just an "update". I still find it pretty weak. There has been plenty of time to assemble basic details, such as a rough timeline. Yes they are busy with the actual response and cleanup, but this is a big professional team. The update feels less like "we're on it, here's what we have so far, more to come", and more like a bland PR minimization exercise. I'll note that they've found time to determine that "For most Google users there was little or no visible change to their services", "YouTube measured a 10% drop in global views during the incident", and "approximately 1% of active Gmail users had problems", and yet they don't mention the time at which the incident was fully resolved or even when it started! Did the impact last for a few minutes, an hour, or multiple hours? Reading this, I have no idea. But I do learn that "the Google teams were keenly aware that every minute which passed represented another minute of user impact", they "brought on additional help to parallelize restoration efforts", "networking systems correctly triaged the traffic overload", and "low-bandwidth services like Google Search recorded only a short-lived increase in latency". In other words, this update on what seems to have been a very major incident consists mostly of vague, positive statements. It does say "we take it very seriously", but it doesn't make me believe it. (I would add that I worked at Google from 2006 to 2010, and based on that experience I'm sure they are taking this very seriously and that there will be an excellent internal postmortem. But man, reading this sure makes it hard to remember that.)

paxys 7y ago

It's definitely not a postmortem. Seems like just a quick update.

mrunkel 7y ago

> With all services restored to normal operation, Google’s engineering teams are now conducting a thorough post-mortem ... This is not the post-mortem, that is still to come.

tbodt 7y ago
The postmortem is currently only visible to Googlers.
- echelon 7y ago
  What happened?
  - tylerhou 7y ago
    TBH the linked update is a pretty accurate summary of the postmortem in my opinion.
    - MrBuddyCasino 7y ago
      > linked update Where?
      - tedunangst 7y ago
        https://cloud.google.com/blog/topics/inside-google-cloud/an-update-on-sundays-service-disruption https://cloud.google.com/blog/topics/inside-google-cloud/an-...
    - kyrra 7y ago
      +1 (I'm a Googler, opinions my own) As someone who has been on oncall for 3 year and done a decent amount of production support, this public doc better explains the cause than the internal one if you aren't well versed in the underlying infra. A config change reduced used network capacity by half, and then things started falling over. And pushing the fix took a while due to the now overloaded network.
      - mcny 7y ago
        What can prevent this from happening in the future? Why was the config change required?
        geofft 7y ago
        I think that's likely to be answered in the post-mortem, which is still to come.
        mlthoughts2018 7y ago
        Obviously to prevent things like this, Google needs more binary search tree whiteboard trivia problems in the interview process.
        lallysingh 7y ago
        Prioritize configuration traffic so they can fix the problem quicker.
        jacquesm 7y ago
        Test changes on smaller parts of the network before pushing them to critical parts would be a first guess. Config changes tend to be nasty in that their implications are often hard to oversee until they have been made, and if the effects preclude you from making another config change then you've just cut off the branch that you were sitting on. Google is best-in-class when it comes to this stuff, the thing you should take away from this is that if they can mess up everybody does. And that pretty much correlates with my experience to date. This stuff is hard, maybe needlessly so but that does not change the fact that it is hard and that accidents can and will happen. So you plan for things to go wrong when you design your systems. Failure is not only an option, it is the default.
JMTQp8lwXL 7y ago
The article leaves it to the reader to decide whether or not we will be updated with the results of that post-mortem.
- theevilsharpie 7y ago
  While this article leaves it ambiguous, the incident status report[1] makes it clear that customers can expect an actual postmortem: "We will conduct an internal investigation of this issue and make appropriate improvements to our systems to help prevent or minimize future recurrence. We will provide a detailed report of this incident once we have completed our internal investigation. This detailed report will contain information regarding SLA credits." [1] https://status.cloud.google.com/incident/cloud-networking/19009 https://status.cloud.google.com/incident/cloud-networking/19...
phs318u 7y ago
Its the pre-post-mortem.
- laurentl 7y ago
  The mortem ?
  - awalton 7y ago
    Perimortem?
    - killjoywashere 7y ago
      In pathology we call issue a "preliminary autopsy report" within the first 48 hours and a "final autopsy report" within 30-60 days. A final report may be followed by additional addenda (new information, e.g. toxicology reports) and amendments (significant changes to the original report, e.g. "I was wrong")
      - OJFord 7y ago
        Yeah, the issue really is that post-mortem's been adopted as a noun. This is a de facto post-mortem <thing> (allowing the adoption of 'mortem' to mean problem, but not the phrase as noun) just perhaps not the full analysis it's been taken to imply.
- dragon_greens 7y ago
  Pre-post-erous mortem?
- thom 7y ago
  The purgatorium.

JMTQp8lwXL 7y ago

The lack of transparency makes me want to consider other cloud providers. All providers will have outages -- that's a reality I can live with -- but I will prioritize the ones who are the most forthcoming in their statuses and explanations into failures.

theevilsharpie 7y ago
Google kept their status page up to date as the outage was progressing, and now (the day after the outage), they've provided an apology and a preliminary explanation of what happened. If that's not sufficient, what more are you looking for, and what other large cloud providers consistently meet that standard?
- JMTQp8lwXL 7y ago
  The status page wasn't up to date, but I don't have any way of backing up that claim. It's certainly isn't tied to any automated system failure reporting -- the status page seems to require manual updates. When minutes turn into hours, and no updates are available on the status page, it certainly doesn't leave me at ease.
  - theevilsharpie 7y ago
    We began noticing erratic behavior with some of our GCE instances at around 11:50 Pacific on the day of the outage, and Google posted a notice on their on their status page that GCE was having an outage about 30 minutes later. They also updated the status page every hour (or when they had new information), which is what they said they would do in their status updates. While it sucks that multiple regions malfunctioned simultaneously for several hours, I can't really fault them for their communication about the issue.
    - mh- 7y ago
      I wasn't able to load a google doc from drive, or my calendar (both g suite), around 11:45 PDT.

deleted 7y ago

[deleted]

deadmutex 7y ago

But it's not a post mortem? Since the article states: > Next Steps > With all services restored to normal operation, Google’s engineering teams are now conducting a thorough post-mortem to ensure we understand all the contributing factors

zoobab 7y ago

What do you expect from a blackbox setup?

brown9-2 7y ago

they typically post a full post-mortem in the status page of the original incident: https://status.cloud.google.com/incident/compute/19003 https://status.cloud.google.com/incident/compute/19003

snewman 7y ago

Replying to my own comment: As many have pointed out, this was not actually the postmortem, just an "update". I still find it pretty weak. There has been plenty of time to assemble basic details, such as a rough timeline. Yes they are busy with the actual response and cleanup, but this is a big professional team. The update feels less like "we're on it, here's what we have so far, more to come", and more like a bland PR minimization exercise. I'll note that they've found time to determine that "For most Google users there was little or no visible change to their services", "YouTube measured a 10% drop in global views during the incident", and "approximately 1% of active Gmail users had problems", and yet they don't mention the time at which the incident was fully resolved or even when it started! Did the impact last for a few minutes, an hour, or multiple hours? Reading this, I have no idea. But I do learn that "the Google teams were keenly aware that every minute which passed represented another minute of user impact", they "brought on additional help to parallelize restoration efforts", "networking systems correctly triaged the traffic overload", and "low-bandwidth services like Google Search recorded only a short-lived increase in latency". In other words, this update on what seems to have been a very major incident consists mostly of vague, positive statements. It does say "we take it very seriously", but it doesn't make me believe it. (I would add that I worked at Google from 2006 to 2010, and based on that experience I'm sure they are taking this very seriously and that there will be an excellent internal postmortem. But man, reading this sure makes it hard to remember that.)