12 ms·

Post-mortem of March 12 Google outage

204 points by antoncohen 8y ago

odorousrex 8y ago
Whoever had "Configuration Change" in the betting pool in the other thread wins all the upvotes! (I believe it was the most upvoted response too, although now I of course can't find the thread)
- rconti 8y ago
  IIRC that commenter was pushing a line about quarterly bonuses which struck me as odd.
- emmelaich 8y ago
  "configuration" is what we call code that has the greatest potential impact and the least amount of static checking.
defen 8y ago
In Firefox I get a blank page with 2 "PARSER.process false" messages in the javascript console...
turdnagel 8y ago
The FB outage yesterday was also apparently caused by a configuration change: https://twitter.com/facebook/status/1106229690069442560 https://twitter.com/facebook/status/1106229690069442560
- jedberg 8y ago
  LOL at the first reply: "I was hoping that @FBI was raiding your campus and arresting your leadership."
  - JeremyBanks 8y ago
    Keeping my fingers crossed.
- Xorlev 8y ago
  Configuration changes are often riskier than code changes. They're faster and often have outsized effects on the characteristics of your service and downstream services (even assuming they don't turn on new features.)
m0zg 8y ago
99 cases out of a 100 it's a "configuration change" in a very large system that's so large and complex nobody can comprehend it anymore. Decentralize, folks.
- gambler 8y ago
  Not sure why this gets downvoted. This is an entirely valid comments. It's getting increasingly obvious that despite being highly distributed, cloud systems run by Google, Amazon and other big cloud companies are inherently centralized. If nothing else, they have centralized controls. A decentralized system can, of course, fail, but it would never have "configuration change" as the root cause.
  - Twirrim 8y ago
    S3 is (for want of a better term) a legacy system in AWS, having been around since pretty much the beginning. A series of choices made when the idea that there would ever be more than two regions, leave them unfortunately stuck having to tie regions together to a limited degree (bucket names are global). Beyond those bucket mutations, everything else is purely regional scoped. I'm _really_ surprised to see that GCP has made the same mistakes though and had things tied together. I'm not convinced there's a good excuse. AWS/S3 is understandable. No one knew what things would look like. Not one engineer would have made the same decisions then if they'd known what the future was for the platform. Google came to the cloud market much later. Their engineers should have seen the issues with AWS and the problems that interdependence cause, and specifically architected isolated infrastructure (which from this post-mortem it seems like they didn't). That's both remarkable and disappointing to see.
    - tybit 8y ago
      GCP may have come later than AWS but googles internal blob storage could of predated AWS for all we know.
    - nerpderp82 8y ago
      The worlds infrastructure is inherently fragile. It should be difficult to take down these large distributed systems, but they are over-coupled.
Areading314 8y ago
Google down March 12 Facebook down March 13 Apple(iCloud) down March 14
- JustSomeNobody 8y ago
  If I were a betting man, I'd say Microsoft (Azure) down March 15 and AWS down March 16.
  - ttul 8y ago
    And Pizza Hut's online ordering system is going to fail on April 20th.
    - morgosmaci 8y ago
      (°ʖ°)
  - cmrx64 8y ago
    your hypothetical bet missed; AWS happened today.
- wlesieutre 8y ago
  Is iCloud a prediction or is there an actual outage today?
  - garren 8y ago
    It looks like it’s resolved now, but according to the iCloud status page some services were down or degraded earlier today. https://www.apple.com/support/systemstatus/ https://www.apple.com/support/systemstatus/
- dgacmu 8y ago
  The ides of March are (almost) come. :) [1] https://en.m.wikipedia.org/wiki/Ides_of_March https://en.m.wikipedia.org/wiki/Ides_of_March
dap 8y ago
> On Monday 11 March 2019, Google SREs were alerted to a significant increase in storage resources for metadata used by the internal blob service. On Tuesday 12 March, to reduce resource usage, SREs made a configuration change which had a side effect of overloading a key part of the system for looking up the location of blob data. The increased load eventually lead to a cascading failure. Any idea what this means? What kind of configuration change would reduce storage but overload the system? I imagine turning on compression could do that, but I'd expect it would be easy to reverse. (Edit: Maybe it was the cascading failure that made it more serious?)
- bostik 8y ago
  > What kind of configuration change would reduce storage but overload the system? Maybe the increased storage usage was due to wrong kind of load rebalancing? I can easily see that a change to reduce _object_ hit rate in any given location, might just increase the _metadata_ hit rate. If storage is balanced by spreading the load to workers that keep their data nearby (for reduced latency), then an unexpected increase in these workers could also cause an increase in required storage. If you want to balance the load on these properly, you'll have to do more location data lookups. Oops. Now your metadata's metadata is the chokepoint. Massive read contention on blob store address lookups --> cascading failure.
- rhizome 8y ago
  At a high level it sounds to me like they're paying Complexity Debt.
- puzzle 8y ago
  Two straightforward changes in Bigtable or Spanner are changing tablet compression settings or tuning compaction more aggressively. Edit: given the mention of a job that got stopped, perhaps they had a tool that manually forced compaction of data in order to adopt new compression settings or purge deleted rows. Normally, compactions would happen on the tablet servers on a cycle of N days. Another possible explanation is that they tweaked the garbage collector, which runs on the master.
  - 8y ago
isomorphic 8y ago
> We will make software measures to prevent any configuration changes that cause overloading of key parts of the system. If it was possible to know these a priori, why allow them in the first place? Or are they saying they will be writing software to roll back configuration changes that cause disruption? ("disallowd")
- mjevans 8y ago
  I read that as either rate/resource consumption limits, or a hard time-limit for a task to complete before it's aborted (rolled back).
ignoramous 8y ago
> https://landing.google.com/sre/sre-book/chapters/reliable-product-launches/ https://landing.google.com/sre/sre-book/chapters/reliable-pr... > Almost all updates to Google’s services proceed gradually, according to a defined process, with appropriate verification steps interspersed. [..] The first stages of a rollout are usually called "canaries” — an allusion to canaries carried by miners into a coal mine to detect dangerous gases. Our canary servers detect dangerous effects from the behavior of the new software under real user traffic. > Canary testing is a concept embedded into many of Google’s internal tools used to make automated changes, as well as for systems that change configuration files. [..] If the change doesn’t pass the validation period, it’s automatically rolled back. Any Googlers around to explain why Canaries didn't catch configuration causing this side effect and stopping the deployment before the issue cascaded globally? The outage here reminds me of issues CloudFront faced wrt configurations in their earlier years. Here's master presenter Harvo Jones going into a bit more detail: https://youtube.com/watch?v=n8qQGLJeUYA https://youtube.com/watch?v=n8qQGLJeUYA Kind of vindicates AWS' decision (after major outages to DynamoDB and S3) to not just isolate regions or zones but resources within those regions/zones further into what they call "cells" https://youtube.com/watch?v=swQbA4zub20 https://youtube.com/watch?v=swQbA4zub20 Tumblr spoke abt "cells" too in the context of achieving isolation at scale, but in a quite different way: http://highscalability.com/blog/2012/2/13/tumblr-architecture-15-billion-page-views-a-month-and-harder.html http://highscalability.com/blog/2012/2/13/tumblr-architectur... As these known-knowns (and the Google sre-book is teeming with these) get designed around, it'd be interesting to see why/how future outages would happen.
- quicklime 8y ago
  > Any Googlers around to explain why Canaries didn't catch configuration causing this side effect and stopping the deployment before the issue cascaded globally? I'm not (any longer) a Googler, but according to the postmortem this was a load-related incident. Canary changes only have a small amount of traffic directed at them (by definition) and the problem probably didn't become apparent until after the change was promoted from canary.
  - j_coder 8y ago
cced 8y ago
Why are all these high profile services going down in such a short window? Anyone else thing this is odd? Rarely do we see such incidents occur, let alone to pretty much all these companies.
- dboreham 8y ago
  Because load increased across all of them at the same time such that they all hit some (different) submerged iceberg scaling wall?
- masto 8y ago
  https://en.m.wikipedia.org/wiki/Poisson_clumping https://en.m.wikipedia.org/wiki/Poisson_clumping
  - anowlcalledjosh 8y ago
    Non-mobile link: https://en.wikipedia.org/wiki/Poisson_clumping https://en.wikipedia.org/wiki/Poisson_clumping
- shric 8y ago
  Apple does use Google's storage (among others) for iCloud, but it could be unrelated.
- powera 8y ago
  I think it's very odd. They say odd things come in threes.
Pirate-of-SV 8y ago
Nice post-mortem. Being in a smaller engineering org. these write ups feels really premium and something to strive for to provide to our customers when things hit the fan.
shanemhansen 8y ago
> On Monday 11 March 2019, Google SREs were alerted to a significant increase in storage resources for metadata used by the internal blob service. On Tuesday 12 March, to reduce resource usage, SREs made a configuration change which had a side effect of overloading a key part of the system for looking up the location of blob data. The increased load eventually lead to a cascading failure. Just some guesses (not affiliated): I wonder if there was something related to Tombstones and deletion of extra data? That can often be a failure of replicated sharded big data systems using sstables and tombstones for deletion. Another guess might be that some sort of internal shard count was increased resulting in some lookup process needing to check n more shards per call.
- londons_explore 8y ago
  The root cause of a cascading failure can be really minor - just something tiny that tips the system over the point of no return. The real question is why don't us systems-designers have better tests for cascading failure? Even Google has very few tests which deliberately overload a system (to cause it to fail), and then verify that it can recover on it's own promptly under typical loads. Eg. consider a database which serves a web application. If the application times out querying the database, it retries up to 5 times. That is a system with an unrecoverable cascading failure, since if one day we get a small spike in users, the database becomes slow, and a few timeouts occur, some requests will get retried, putting more load on the database, making it slower, causing more timeouts, and the problem gets rapidly worse until a few minutes later all requests are failing. Even when the spike in users is over, the database will remain effectively dead, because it is still failing under load from all those retries. A load test that didn't push the system over the edge and then test for recovery wouldn't be able to detect that. Too many people loadtest things to the point of failure and then stop. All it takes is to continue the test for a few minutes longer with a slightly lower load to verify recovery!
  - azurezyq 8y ago
    Because there are unlimited number of failure modes for cascading failures. Going through all postmortems, you'll find it hard to find same kind of failures. Every time it's something unique. Also it's difficult to get a good coverage by just pushing things to its limit. Let's say you have a system which has this RPC chain: A -> B -> C -> D. And not all requests cost the same and not all requests make way to D. In reality it's interesting that different composition of traffic may overload any of them. For example, some queries may hit expensive ML models while some other queries have a large number of candidates. In this case the former hits predictor service hard while the latter may hit index serving systems. I've actually read an outage happened during superbowl because sports-related queries are particularly expensive for some part of the system.
CzarnyZiutek 8y ago
"destroy the skynet" john connor
mlthoughts2018 8y ago
I’ve worked in three separate companies that had each hired a ton of ex Google engineers in their early years, and the systems they have are all identical to each other and seem like a poorly recalled-from-memory version of these types of patterns. It’s like a strange simulacrum of some bazel-like build tools but not as mature as bazel but also can’t be swapped with real bazel due to tech debt issues, and same with canary deployments, monorepo layouts, etc. It’s become something that I actively seek out information about when I interview for new jobs so I can actively avoid all these Google-but-not-quite-with-enough-resources-to-pull-it-off antipatterns. Worse yet is that in many of these shops, those early employees who came from Google just stayed long enough to mutate the whole monorepo / homebrewed bazel clone / canary deploys mess into a pile of unmaintainable crap that couldn’t be migrated away from without huge business risks, and thenthey mostly all quit and a lot of them even went back to Google! I call it the Google Borg Syndrome. Same crap monorepo ideas, same crap canary ideas, same crap bazel-like build system ideas just spread like kudzu. It really shows how bad the ideas themselves are, and how if it wasn’t for a giant money and labor-hours faucet that Google can shoot at these systems, they would be unmasked as deeply poor ways of solving engineering problems.
- londons_explore 8y ago
  >they would be unmasked as deeply poor ways of solving engineering problems. The best technical solution to problems can depend a lot on company size. These people are using a big-company solution in a small-company. And it nearly works, but not quite. Big companies reinvent pretty much everything internally - You won't find Chef, Jenkins, Slack, Apache, Nginx, or any of that stuff at Google. Yet a small startup would be stupid not to use tooling developed and maintained by someone else almost everywhere. In turn, the best solution to a problem depends on which tools you have at your disposal.
  - mlthoughts2018 8y ago
    I agree to a large extent. But specifically in the case of - using bazel instead of make - using a monorepo ever - using Google-style canary deployment ever I think they are just nearly totally objectively bad ways to solve problems whether at Google scale or otherwise, with obvious alternatives that are strictly dominating in the sense of being unilaterally better in all use cases. These techniques have to be kept afloat with the money faucet. I guess I can agree that if you have enough money that you can offer pay packages that cause people to be willing to endure the hardship of propping bad systems up, then you don’t have to care about what systems are effective. For example, you could shoot lots of holes in a boat and then hire a full time staff on $500k / year to scoop water out fast enough that the boat seems to operate just fine. Then get the ACM to write a big publicity piece about how hole-filled boats are the best design and anyone not shooting holes in their boat is Doing It Wrong. This is not even exaggeration to offer as an analogy to Google’s combo use of monorepos + bazel.
crb002 8y ago
When SREs are the reliably problem...
peter_d_sherman 8y ago
Dear Google: I don't want to sound negative (that's not the purpose of this message), but my belief is that this incident is a heart palpitation on the way to a major coronary, stroke, or heart attack... Let me explain to you why. You see, as your company has grown, so has the complexity of your codebase and so has the number of people who maintain that code. The complexity of your code has increased, perhaps asymptotically (since you seem to like Big-O notation and related subject matter, from what I hear, at your interviews), yet I'd be willing to bet that the overall knowledge of the codebase that the average worker there has, has actually diminished (although perhaps not asymptotically, perhaps linearly) over time. In other words, you've got too many chefs, and a soup with too many ingredients where every chef knows some of them, but not all of them. Maybe Jeff Dean is the exception to that, but that's what you've got going on, basically. When something like this happens, your responsibility is to a) Educate your rank-and-file, even at the expense of new product releases, b) Re-delegate, insuring that there aren't too many chefs to soup, c) Reduce the ingredient count (refactor), and perhaps most important d) Make sure there's someone, anyone, who understands ALL of the codebase from top to bottom (difficult when you've got multiple source-code contributors making changes, and multiple layers of people only responsible for some aspects of the system...). (Oh, and if it were me, I'd go as far as to buffer the entire incoming datastream so as to be able to replicate the problem, and if you can't do that (run a buffered datastream from a point in time against a point-in-time snapshot of all data at that point in time, and see exactly where in code the problem occurs), then your first goal is to be able to do exactly that. But I'm a little extreme in what I feel qualifies as acceptable testing...)
- jhgg 8y ago
  >Make sure there's someone, anyone, who understands ALL of the codebase from top to bottom Either you seriously over estimate the limits of ones understanding, or you severely under estimate the size of Google’s codebase/monorepo. At this point, I’m pretty sure it’s impossible to even check it out on a single developers workstation due to its size.
  - peter_d_sherman 8y ago
    https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-abstractions/ https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-a... ...Also, compare that to the Bible's "Tower Of Babel" story: https://en.wikipedia.org/wiki/Tower_of_Babel https://en.wikipedia.org/wiki/Tower_of_Babel The Bible's story is about human language, specifically about how human languages encompass greater degrees of abstraction over time (languages built on top of languages, abstractions built on top of abstractions). But, when you get to a certain level of abstractions, the abstractions start to "leak". That is why there are communication difficulties in societies (people who are more educated typically speak a different language than people who are less educated; academia and lawyers speak their own language). This is also when/where/why/how software starts to fail. Too many levels of abstraction, and those abstractions will "leak". Don't take my word for it, use your own logic, your own common sense and think about it... Software problem A exists. Management Team: "We have a problem, can you fix it?" Programming Team: "Sure." (fixes it, but this causes problem B, which is not recognized until several weeks later) Management Team: "Now we have this different problem, can you fix it?" Programming Team: "Sure." (fixes it, but this cause problem A to resurface, which is not detected for several weeks...) Management Team: "We have a problem, can you fix it?" And the cycle repeats... Was it the programming team's fault? No, it was the nature of the beast called "complexity". The higher you go in abstractions (and this is especially true for AI), the more you must exercise ENGINEERING DISCIPLINE. You must be able to go BACKWARDS IN TIME (this is accomplished in programming by being able to go back to simpler programs and fully understand / test them before advancing to more complex ones, and PRESERVING THAT CHAIN OF UNDERSTANDING...). This is probably why advanced societies destroy themselves, too much abstraction, too much "magic" (without understanding of all of the sublevels), and no one has the exact knowledge necessary to pinpoint exactly when/where/why the exact problem occurred... But again, don't take my word for it. Use your own common sense to determine if what I'm saying has any merit...

204 points by antoncohen 8y ago

Whoever had "Configuration Change" in the betting pool in the other thread wins all the upvotes! (I believe it was the most upvoted response too, although now I of course can't find the thread)

rconti 8y ago
IIRC that commenter was pushing a line about quarterly bonuses which struck me as odd.
emmelaich 8y ago
"configuration" is what we call code that has the greatest potential impact and the least amount of static checking.

defen 8y ago

In Firefox I get a blank page with 2 "PARSER.process false" messages in the javascript console...

turdnagel 8y ago

The FB outage yesterday was also apparently caused by a configuration change: https://twitter.com/facebook/status/1106229690069442560 https://twitter.com/facebook/status/1106229690069442560

jedberg 8y ago
LOL at the first reply: "I was hoping that @FBI was raiding your campus and arresting your leadership."
- JeremyBanks 8y ago
  Keeping my fingers crossed.
Xorlev 8y ago
Configuration changes are often riskier than code changes. They're faster and often have outsized effects on the characteristics of your service and downstream services (even assuming they don't turn on new features.)

m0zg 8y ago

99 cases out of a 100 it's a "configuration change" in a very large system that's so large and complex nobody can comprehend it anymore. Decentralize, folks.

gambler 8y ago
Not sure why this gets downvoted. This is an entirely valid comments. It's getting increasingly obvious that despite being highly distributed, cloud systems run by Google, Amazon and other big cloud companies are inherently centralized. If nothing else, they have centralized controls. A decentralized system can, of course, fail, but it would never have "configuration change" as the root cause.
- Twirrim 8y ago
  S3 is (for want of a better term) a legacy system in AWS, having been around since pretty much the beginning. A series of choices made when the idea that there would ever be more than two regions, leave them unfortunately stuck having to tie regions together to a limited degree (bucket names are global). Beyond those bucket mutations, everything else is purely regional scoped. I'm _really_ surprised to see that GCP has made the same mistakes though and had things tied together. I'm not convinced there's a good excuse. AWS/S3 is understandable. No one knew what things would look like. Not one engineer would have made the same decisions then if they'd known what the future was for the platform. Google came to the cloud market much later. Their engineers should have seen the issues with AWS and the problems that interdependence cause, and specifically architected isolated infrastructure (which from this post-mortem it seems like they didn't). That's both remarkable and disappointing to see.
  - tybit 8y ago
    GCP may have come later than AWS but googles internal blob storage could of predated AWS for all we know.
  - nerpderp82 8y ago
    The worlds infrastructure is inherently fragile. It should be difficult to take down these large distributed systems, but they are over-coupled.

Areading314 8y ago

Google down March 12 Facebook down March 13 Apple(iCloud) down March 14

JustSomeNobody 8y ago
If I were a betting man, I'd say Microsoft (Azure) down March 15 and AWS down March 16.
- ttul 8y ago
  And Pizza Hut's online ordering system is going to fail on April 20th.
  - morgosmaci 8y ago
    (°ʖ°)
- cmrx64 8y ago
  your hypothetical bet missed; AWS happened today.
wlesieutre 8y ago
Is iCloud a prediction or is there an actual outage today?
- garren 8y ago
  It looks like it’s resolved now, but according to the iCloud status page some services were down or degraded earlier today. https://www.apple.com/support/systemstatus/ https://www.apple.com/support/systemstatus/
dgacmu 8y ago
The ides of March are (almost) come. :) [1] https://en.m.wikipedia.org/wiki/Ides_of_March https://en.m.wikipedia.org/wiki/Ides_of_March

dap 8y ago

> On Monday 11 March 2019, Google SREs were alerted to a significant increase in storage resources for metadata used by the internal blob service. On Tuesday 12 March, to reduce resource usage, SREs made a configuration change which had a side effect of overloading a key part of the system for looking up the location of blob data. The increased load eventually lead to a cascading failure. Any idea what this means? What kind of configuration change would reduce storage but overload the system? I imagine turning on compression could do that, but I'd expect it would be easy to reverse. (Edit: Maybe it was the cascading failure that made it more serious?)

bostik 8y ago
> What kind of configuration change would reduce storage but overload the system? Maybe the increased storage usage was due to wrong kind of load rebalancing? I can easily see that a change to reduce _object_ hit rate in any given location, might just increase the _metadata_ hit rate. If storage is balanced by spreading the load to workers that keep their data nearby (for reduced latency), then an unexpected increase in these workers could also cause an increase in required storage. If you want to balance the load on these properly, you'll have to do more location data lookups. Oops. Now your metadata's metadata is the chokepoint. Massive read contention on blob store address lookups --> cascading failure.
rhizome 8y ago
At a high level it sounds to me like they're paying Complexity Debt.
puzzle 8y ago
Two straightforward changes in Bigtable or Spanner are changing tablet compression settings or tuning compaction more aggressively. Edit: given the mention of a job that got stopped, perhaps they had a tool that manually forced compaction of data in order to adopt new compression settings or purge deleted rows. Normally, compactions would happen on the tablet servers on a cycle of N days. Another possible explanation is that they tweaked the garbage collector, which runs on the master.
- 8y ago

isomorphic 8y ago

> We will make software measures to prevent any configuration changes that cause overloading of key parts of the system. If it was possible to know these a priori, why allow them in the first place? Or are they saying they will be writing software to roll back configuration changes that cause disruption? ("disallowd")

mjevans 8y ago
I read that as either rate/resource consumption limits, or a hard time-limit for a task to complete before it's aborted (rolled back).

ignoramous 8y ago

> https://landing.google.com/sre/sre-book/chapters/reliable-product-launches/ https://landing.google.com/sre/sre-book/chapters/reliable-pr... > Almost all updates to Google’s services proceed gradually, according to a defined process, with appropriate verification steps interspersed. [..] The first stages of a rollout are usually called "canaries” — an allusion to canaries carried by miners into a coal mine to detect dangerous gases. Our canary servers detect dangerous effects from the behavior of the new software under real user traffic. > Canary testing is a concept embedded into many of Google’s internal tools used to make automated changes, as well as for systems that change configuration files. [..] If the change doesn’t pass the validation period, it’s automatically rolled back. Any Googlers around to explain why Canaries didn't catch configuration causing this side effect and stopping the deployment before the issue cascaded globally? The outage here reminds me of issues CloudFront faced wrt configurations in their earlier years. Here's master presenter Harvo Jones going into a bit more detail: https://youtube.com/watch?v=n8qQGLJeUYA https://youtube.com/watch?v=n8qQGLJeUYA Kind of vindicates AWS' decision (after major outages to DynamoDB and S3) to not just isolate regions or zones but resources within those regions/zones further into what they call "cells" https://youtube.com/watch?v=swQbA4zub20 https://youtube.com/watch?v=swQbA4zub20 Tumblr spoke abt "cells" too in the context of achieving isolation at scale, but in a quite different way: http://highscalability.com/blog/2012/2/13/tumblr-architecture-15-billion-page-views-a-month-and-harder.html http://highscalability.com/blog/2012/2/13/tumblr-architectur... As these known-knowns (and the Google sre-book is teeming with these) get designed around, it'd be interesting to see why/how future outages would happen.

quicklime 8y ago
> Any Googlers around to explain why Canaries didn't catch configuration causing this side effect and stopping the deployment before the issue cascaded globally? I'm not (any longer) a Googler, but according to the postmortem this was a load-related incident. Canary changes only have a small amount of traffic directed at them (by definition) and the problem probably didn't become apparent until after the change was promoted from canary.
- j_coder 8y ago

cced 8y ago

Why are all these high profile services going down in such a short window? Anyone else thing this is odd? Rarely do we see such incidents occur, let alone to pretty much all these companies.

dboreham 8y ago
Because load increased across all of them at the same time such that they all hit some (different) submerged iceberg scaling wall?
masto 8y ago
https://en.m.wikipedia.org/wiki/Poisson_clumping https://en.m.wikipedia.org/wiki/Poisson_clumping
- anowlcalledjosh 8y ago
  Non-mobile link: https://en.wikipedia.org/wiki/Poisson_clumping https://en.wikipedia.org/wiki/Poisson_clumping
shric 8y ago
Apple does use Google's storage (among others) for iCloud, but it could be unrelated.
powera 8y ago
I think it's very odd. They say odd things come in threes.

Pirate-of-SV 8y ago

Nice post-mortem. Being in a smaller engineering org. these write ups feels really premium and something to strive for to provide to our customers when things hit the fan.

shanemhansen 8y ago

> On Monday 11 March 2019, Google SREs were alerted to a significant increase in storage resources for metadata used by the internal blob service. On Tuesday 12 March, to reduce resource usage, SREs made a configuration change which had a side effect of overloading a key part of the system for looking up the location of blob data. The increased load eventually lead to a cascading failure. Just some guesses (not affiliated): I wonder if there was something related to Tombstones and deletion of extra data? That can often be a failure of replicated sharded big data systems using sstables and tombstones for deletion. Another guess might be that some sort of internal shard count was increased resulting in some lookup process needing to check n more shards per call.

londons_explore 8y ago
The root cause of a cascading failure can be really minor - just something tiny that tips the system over the point of no return. The real question is why don't us systems-designers have better tests for cascading failure? Even Google has very few tests which deliberately overload a system (to cause it to fail), and then verify that it can recover on it's own promptly under typical loads. Eg. consider a database which serves a web application. If the application times out querying the database, it retries up to 5 times. That is a system with an unrecoverable cascading failure, since if one day we get a small spike in users, the database becomes slow, and a few timeouts occur, some requests will get retried, putting more load on the database, making it slower, causing more timeouts, and the problem gets rapidly worse until a few minutes later all requests are failing. Even when the spike in users is over, the database will remain effectively dead, because it is still failing under load from all those retries. A load test that didn't push the system over the edge and then test for recovery wouldn't be able to detect that. Too many people loadtest things to the point of failure and then stop. All it takes is to continue the test for a few minutes longer with a slightly lower load to verify recovery!
- azurezyq 8y ago
  Because there are unlimited number of failure modes for cascading failures. Going through all postmortems, you'll find it hard to find same kind of failures. Every time it's something unique. Also it's difficult to get a good coverage by just pushing things to its limit. Let's say you have a system which has this RPC chain: A -> B -> C -> D. And not all requests cost the same and not all requests make way to D. In reality it's interesting that different composition of traffic may overload any of them. For example, some queries may hit expensive ML models while some other queries have a large number of candidates. In this case the former hits predictor service hard while the latter may hit index serving systems. I've actually read an outage happened during superbowl because sports-related queries are particularly expensive for some part of the system.

CzarnyZiutek 8y ago

"destroy the skynet" john connor

mlthoughts2018 8y ago

I’ve worked in three separate companies that had each hired a ton of ex Google engineers in their early years, and the systems they have are all identical to each other and seem like a poorly recalled-from-memory version of these types of patterns. It’s like a strange simulacrum of some bazel-like build tools but not as mature as bazel but also can’t be swapped with real bazel due to tech debt issues, and same with canary deployments, monorepo layouts, etc. It’s become something that I actively seek out information about when I interview for new jobs so I can actively avoid all these Google-but-not-quite-with-enough-resources-to-pull-it-off antipatterns. Worse yet is that in many of these shops, those early employees who came from Google just stayed long enough to mutate the whole monorepo / homebrewed bazel clone / canary deploys mess into a pile of unmaintainable crap that couldn’t be migrated away from without huge business risks, and thenthey mostly all quit and a lot of them even went back to Google! I call it the Google Borg Syndrome. Same crap monorepo ideas, same crap canary ideas, same crap bazel-like build system ideas just spread like kudzu. It really shows how bad the ideas themselves are, and how if it wasn’t for a giant money and labor-hours faucet that Google can shoot at these systems, they would be unmasked as deeply poor ways of solving engineering problems.

londons_explore 8y ago
>they would be unmasked as deeply poor ways of solving engineering problems. The best technical solution to problems can depend a lot on company size. These people are using a big-company solution in a small-company. And it nearly works, but not quite. Big companies reinvent pretty much everything internally - You won't find Chef, Jenkins, Slack, Apache, Nginx, or any of that stuff at Google. Yet a small startup would be stupid not to use tooling developed and maintained by someone else almost everywhere. In turn, the best solution to a problem depends on which tools you have at your disposal.
- mlthoughts2018 8y ago
  I agree to a large extent. But specifically in the case of - using bazel instead of make - using a monorepo ever - using Google-style canary deployment ever I think they are just nearly totally objectively bad ways to solve problems whether at Google scale or otherwise, with obvious alternatives that are strictly dominating in the sense of being unilaterally better in all use cases. These techniques have to be kept afloat with the money faucet. I guess I can agree that if you have enough money that you can offer pay packages that cause people to be willing to endure the hardship of propping bad systems up, then you don’t have to care about what systems are effective. For example, you could shoot lots of holes in a boat and then hire a full time staff on $500k / year to scoop water out fast enough that the boat seems to operate just fine. Then get the ACM to write a big publicity piece about how hole-filled boats are the best design and anyone not shooting holes in their boat is Doing It Wrong. This is not even exaggeration to offer as an analogy to Google’s combo use of monorepos + bazel.

crb002 8y ago

When SREs are the reliably problem...

peter_d_sherman 8y ago

Dear Google: I don't want to sound negative (that's not the purpose of this message), but my belief is that this incident is a heart palpitation on the way to a major coronary, stroke, or heart attack... Let me explain to you why. You see, as your company has grown, so has the complexity of your codebase and so has the number of people who maintain that code. The complexity of your code has increased, perhaps asymptotically (since you seem to like Big-O notation and related subject matter, from what I hear, at your interviews), yet I'd be willing to bet that the overall knowledge of the codebase that the average worker there has, has actually diminished (although perhaps not asymptotically, perhaps linearly) over time. In other words, you've got too many chefs, and a soup with too many ingredients where every chef knows some of them, but not all of them. Maybe Jeff Dean is the exception to that, but that's what you've got going on, basically. When something like this happens, your responsibility is to a) Educate your rank-and-file, even at the expense of new product releases, b) Re-delegate, insuring that there aren't too many chefs to soup, c) Reduce the ingredient count (refactor), and perhaps most important d) Make sure there's someone, anyone, who understands ALL of the codebase from top to bottom (difficult when you've got multiple source-code contributors making changes, and multiple layers of people only responsible for some aspects of the system...). (Oh, and if it were me, I'd go as far as to buffer the entire incoming datastream so as to be able to replicate the problem, and if you can't do that (run a buffered datastream from a point in time against a point-in-time snapshot of all data at that point in time, and see exactly where in code the problem occurs), then your first goal is to be able to do exactly that. But I'm a little extreme in what I feel qualifies as acceptable testing...)

jhgg 8y ago
>Make sure there's someone, anyone, who understands ALL of the codebase from top to bottom Either you seriously over estimate the limits of ones understanding, or you severely under estimate the size of Google’s codebase/monorepo. At this point, I’m pretty sure it’s impossible to even check it out on a single developers workstation due to its size.
- peter_d_sherman 8y ago
  https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-abstractions/ https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-a... ...Also, compare that to the Bible's "Tower Of Babel" story: https://en.wikipedia.org/wiki/Tower_of_Babel https://en.wikipedia.org/wiki/Tower_of_Babel The Bible's story is about human language, specifically about how human languages encompass greater degrees of abstraction over time (languages built on top of languages, abstractions built on top of abstractions). But, when you get to a certain level of abstractions, the abstractions start to "leak". That is why there are communication difficulties in societies (people who are more educated typically speak a different language than people who are less educated; academia and lawyers speak their own language). This is also when/where/why/how software starts to fail. Too many levels of abstraction, and those abstractions will "leak". Don't take my word for it, use your own logic, your own common sense and think about it... Software problem A exists. Management Team: "We have a problem, can you fix it?" Programming Team: "Sure." (fixes it, but this causes problem B, which is not recognized until several weeks later) Management Team: "Now we have this different problem, can you fix it?" Programming Team: "Sure." (fixes it, but this cause problem A to resurface, which is not detected for several weeks...) Management Team: "We have a problem, can you fix it?" And the cycle repeats... Was it the programming team's fault? No, it was the nature of the beast called "complexity". The higher you go in abstractions (and this is especially true for AI), the more you must exercise ENGINEERING DISCIPLINE. You must be able to go BACKWARDS IN TIME (this is accomplished in programming by being able to go back to simpler programs and fully understand / test them before advancing to more complex ones, and PRESERVING THAT CHAIN OF UNDERSTANDING...). This is probably why advanced societies destroy themselves, too much abstraction, too much "magic" (without understanding of all of the sublevels), and no one has the exact knowledge necessary to pinpoint exactly when/where/why the exact problem occurred... But again, don't take my word for it. Use your own common sense to determine if what I'm saying has any merit...