13 ms·

> One of the features contained a bug which would cause the GFE to restart; this bug had not been detected in either of testing and initial rollout. I wish the

> One of the features contained a bug which would cause the GFE to restart; this bug had not been detected in either of testing and initial rollout. I wish they elaborated more on what type of bug was that, which was not caught by testing / initial rollout. Either tests must be poorly written or the bug must be very subtle. On an unrelated note: kudos to Google for publishing this postmortem and hope that this becomes an industrywide practice. I also wish they publish (a belated) one about Google+ and their throng of messaging apps over the years.

actuator 8y ago
> On an unrelated note: kudos to Google for publishing this postmortem and hope that this becomes an industrywide practice. I think most already do this. I have seen AWS also publish detailed postmortems for outages like this. Ex: https://aws.amazon.com/message/41926/ https://aws.amazon.com/message/41926/
- macintux 8y ago
  I started tracking them a while back, didn't stick with it for very long. https://github.com/macintux/Service-postmortems https://github.com/macintux/Service-postmortems
puzzle 8y ago
Sometimes bugs are triggered by very specific data or queries (aka "queries of death"). GFEs have many tests, so I'd go for the subtle bug.
- nine_k 8y ago
  I suppose every detected "query of death" should be included in the next version of the service's test suite.
  - tweenagedream 8y ago
    I bet there is already a regression test for this one.
morrbo 8y ago
With all due respect, it's not a postmortem, it's an advert. It doesn't really say anything other than "We had a problem, we fixed it.". There are virtually no technical details in there other than "something would restart spontaneously, which shifted the load somewhere else". Maybe i'm a bit jaded by cloudflare and aws writeups, but this really isn't anything special or worthwhile reading.
- tgtweak 8y ago
  Very little detail. I'm sure they're worried about disclosing something about their stack that would open them up to more vulnerability which is understandable, but not explaining why they didn't see it in their testing... That's something they could disclose without any threats. It's easier to disclose details of a physical infrastructure outage or a bug in someone else's code/product than a bug in your own code.
- phyzome 8y ago
  In this case, they might be hard-pressed to go into details, since it was related to an unreleased feature: « These features had been introduced into the second layer GFE code base but not yet put into service. One of the features contained a bug which would cause the GFE to restart; this bug had not been detected in either of testing and initial rollout. » ...but yeah, they definitely could have done better.
  - puzzle 8y ago
    Another reason postmortems can be vague is that, for each detail you add, you might need to add even more background details. Then it becomes a recursive problem. In this case, the "configuration change" could be a feature flag in the L2 GFEs, something in the L1 GFEs that changed L1->L2 requests in minor ways, or maybe something else entirely (since it's partially security-related: a dynamic LOAS handshake change to use different cyphers? So many possibilities). At the end of the day, though, it's still a specific permutation of all possible features and flags that hadn't been vetted before. Given how large Google's monorepo is, it's not impossible for one of your many dependencies, even indirect ones, which might be configured by another team entirely, to have subtle time bombs that only trigger well after you have built and deployed the code. Having been on the other side, I know that, for every detail added, a bunch more questions come up.
ebikelaw 8y ago
It reads to me like the bug required a particular feature to be enabled, and the feature wasn't enable when the software was rolling out, then the feature was globally knife-switched to the "on" position without an orderly staged rollout. I actually think it looks pretty bad that one of the action items in this report is to make a feature dashboard for GFEs. They've been saying they will do that for years, and the team that operates GFEs is considered the most elite of all SRE teams. Most famous outages of Google products have been caused by bogus configurations pushed to GFEs or network devices in front of them.
- mjevans 8y ago
  It's sort of like that extra disk in your raid 1 setup at home. Compared to a single SSD the performance improvement doesn't really show (for desktop loads)... However when things come to a rare (but somewhat inevitable) and screeching halt and that mirror has one of the copies shatter beyond recognition... that's when the doubled price proves that it was worth it. Such a dashboard would invariably also add load and complexity (both failure points) to the system, but outwardly most users would be unaware of their existence.
readams 8y ago
As it says in the note, the bug was in a feature that was latent but not yet being used. Then a configuration change started hitting the feature. With highly redundant systems such as this, you generally need multiple layers of things going wrong all at once to notice an issue. This was the case here as well.
kerng 8y ago
This is super high level summary not a postmortem, in fact it doesn't say anything about the bug and why there was no test for it, what repair items are done, etc.. Amazon and Microsoft's post mortems are much more to the point, one can actually learn from them to not make the same or similar mistake.
- skj 8y ago
  I will note that the word "postmortem" appears nowhere on the page, which is dedicated to status updates about the incident in question.
emmelaich 8y ago
> ... this bug had not been detected in either of testing and initial rollout. It makes me so happy[0] that the big smart corps have the same problems that us plebs have. I tell people that testing can never precisely duplicate the production environment but do they believe me? Also, this is an argument for feature switches vs staging environments. [0] But not in a schadenfreude sense.
michaelt 8y ago
Either tests must be poorly written or the bug must be very subtle. The easiest way to make a web service work in testing but fail in production is a problem with the settings that are necessarily different between environments. For example, you must have different database credentials between test and production, and you must limit who can read the production credentials. If the production credentials are malformed, a service that worked in test will fail in production. And the same applies to your SSL certificates, your settings for enabled/disabled features, your flashy markers that stop people mistaking production for test....
ams6110 8y ago
Another possibility is that the change was not tested at all or very minimally tested -- and they would not want to admit to that publicly.

actuator 8y ago

> On an unrelated note: kudos to Google for publishing this postmortem and hope that this becomes an industrywide practice. I think most already do this. I have seen AWS also publish detailed postmortems for outages like this. Ex: https://aws.amazon.com/message/41926/ https://aws.amazon.com/message/41926/

macintux 8y ago
I started tracking them a while back, didn't stick with it for very long. https://github.com/macintux/Service-postmortems https://github.com/macintux/Service-postmortems

puzzle 8y ago

Sometimes bugs are triggered by very specific data or queries (aka "queries of death"). GFEs have many tests, so I'd go for the subtle bug.

nine_k 8y ago
I suppose every detected "query of death" should be included in the next version of the service's test suite.
- tweenagedream 8y ago
  I bet there is already a regression test for this one.

morrbo 8y ago

With all due respect, it's not a postmortem, it's an advert. It doesn't really say anything other than "We had a problem, we fixed it.". There are virtually no technical details in there other than "something would restart spontaneously, which shifted the load somewhere else". Maybe i'm a bit jaded by cloudflare and aws writeups, but this really isn't anything special or worthwhile reading.

tgtweak 8y ago
Very little detail. I'm sure they're worried about disclosing something about their stack that would open them up to more vulnerability which is understandable, but not explaining why they didn't see it in their testing... That's something they could disclose without any threats. It's easier to disclose details of a physical infrastructure outage or a bug in someone else's code/product than a bug in your own code.
phyzome 8y ago
In this case, they might be hard-pressed to go into details, since it was related to an unreleased feature: « These features had been introduced into the second layer GFE code base but not yet put into service. One of the features contained a bug which would cause the GFE to restart; this bug had not been detected in either of testing and initial rollout. » ...but yeah, they definitely could have done better.
- puzzle 8y ago
  Another reason postmortems can be vague is that, for each detail you add, you might need to add even more background details. Then it becomes a recursive problem. In this case, the "configuration change" could be a feature flag in the L2 GFEs, something in the L1 GFEs that changed L1->L2 requests in minor ways, or maybe something else entirely (since it's partially security-related: a dynamic LOAS handshake change to use different cyphers? So many possibilities). At the end of the day, though, it's still a specific permutation of all possible features and flags that hadn't been vetted before. Given how large Google's monorepo is, it's not impossible for one of your many dependencies, even indirect ones, which might be configured by another team entirely, to have subtle time bombs that only trigger well after you have built and deployed the code. Having been on the other side, I know that, for every detail added, a bunch more questions come up.

ebikelaw 8y ago

It reads to me like the bug required a particular feature to be enabled, and the feature wasn't enable when the software was rolling out, then the feature was globally knife-switched to the "on" position without an orderly staged rollout. I actually think it looks pretty bad that one of the action items in this report is to make a feature dashboard for GFEs. They've been saying they will do that for years, and the team that operates GFEs is considered the most elite of all SRE teams. Most famous outages of Google products have been caused by bogus configurations pushed to GFEs or network devices in front of them.

mjevans 8y ago
It's sort of like that extra disk in your raid 1 setup at home. Compared to a single SSD the performance improvement doesn't really show (for desktop loads)... However when things come to a rare (but somewhat inevitable) and screeching halt and that mirror has one of the copies shatter beyond recognition... that's when the doubled price proves that it was worth it. Such a dashboard would invariably also add load and complexity (both failure points) to the system, but outwardly most users would be unaware of their existence.

readams 8y ago

As it says in the note, the bug was in a feature that was latent but not yet being used. Then a configuration change started hitting the feature. With highly redundant systems such as this, you generally need multiple layers of things going wrong all at once to notice an issue. This was the case here as well.

kerng 8y ago

This is super high level summary not a postmortem, in fact it doesn't say anything about the bug and why there was no test for it, what repair items are done, etc.. Amazon and Microsoft's post mortems are much more to the point, one can actually learn from them to not make the same or similar mistake.

skj 8y ago
I will note that the word "postmortem" appears nowhere on the page, which is dedicated to status updates about the incident in question.

emmelaich 8y ago

> ... this bug had not been detected in either of testing and initial rollout. It makes me so happy[0] that the big smart corps have the same problems that us plebs have. I tell people that testing can never precisely duplicate the production environment but do they believe me? Also, this is an argument for feature switches vs staging environments. [0] But not in a schadenfreude sense.

michaelt 8y ago

Either tests must be poorly written or the bug must be very subtle. The easiest way to make a web service work in testing but fail in production is a problem with the settings that are necessarily different between environments. For example, you must have different database credentials between test and production, and you must limit who can read the production credentials. If the production credentials are malformed, a service that worked in test will fail in production. And the same applies to your SSL certificates, your settings for enabled/disabled features, your flashy markers that stop people mistaking production for test....

ams6110 8y ago

Another possibility is that the change was not tested at all or very minimally tested -- and they would not want to admit to that publicly.