4 ms·

> There are a number of lessons to be learned from this event -- for example, that the safeguard of a progressive rollout can be undone by a system designed to

> There are a number of lessons to be learned from this event -- for example, that the safeguard of a progressive rollout can be undone by a system designed to mask partial failures -- ... This is a really important point that should be more generally known. To quote Google's own "Paxos Made Live" paper, from 2007: > In closing we point out a challenge that we faced in testing our system for which we have no systematic solution. By their very nature, fault-tolerant systems try to mask problems. Thus they can mask bugs or configuration problems while insidiously lowering their own fault-tolerance. As developers we can try to bear this principle in mind, but as Monday's incident demonstrated, mistakes can still happen. So, has anyone managed to make progress toward a "systematic solution" in the last 9 years?

atomic77 11y ago
This is an interesting question, and seems to get to the core of Nassim Taleb's ideas [1] about fragility and the limits of what we can understand, and how many of our attempts to create artificial stability ultimately bring about the opposite. That said, based on this post-mortem, I think Google, and our industry as a whole, is doing a pretty good job. Periodic failures like this are inevitable, and if they serve to make it less likely that a similar failure occurs in the future, then that is a system as a whole that could be described as "anti-fragile". [1] At least my interpretation of them
woodman 11y ago
> So, has anyone managed to make progress toward a "systematic solution" in the last 9 years? That depends on how you define "solution". If development time isn't a concern, then formal verification is a pretty solid solution. AWS has used TLA+ on a subset of its systems. [0] [0] https://en.wikipedia.org/wiki/TLA%2B https://en.wikipedia.org/wiki/TLA%2B
deleted 11y ago
[deleted]
strictfp 11y ago
Degraded modes of operation is one example of how to visualize masked errors. Another is to trigger an alarm on fallbacks. As a general reflection, many distributed system leave out the cause of their changes and only log actions. Instead of logging "new membership, new members are b,c,d" you are better of logging "node a has not responded to heartbeat in the last 30 seconds, considering it faulty". Following such a principle makes it much easier to spot masked bugs, since you can reason about the behaviour much better. Aggregating logs to a central location and being able to analyze global behaviour in retrospect is also a great feature.
- senderista 11y ago
  Great point! More precisely, each state transition in a system should report the old state, the new state, and the triggering event (cause) to a monitoring system (possibly just a log).
nostrademons 11y ago
Google actually does have a systematic solution: fault injection. Google's systems are designed so that you can (manually, if you have the right privileges) tell an RPC to fail regardless of whether it would otherwise have succeeded, and then test the response of the system as a whole. The problem is that these failure cases are exercised much less frequently than the "normal execution" code paths are. For example, every year Google does DiRT [1] exercises which test system responses to a large calamity, eg. a California earthquake that kills everyone in Mountain View and SF including the senior leadership, and also knocks out all west coast datacenters. The half-life of code at Google (in my observation) is roughly 1 year, which means that half of all code has never gone through a DiRT exercise. The same applies to other, less serious fault injection mechanisms: they may get executed once every year or two, and serious bugs can crop up in the meantime. Automated testing of fault injection isn't really feasible, because the number of potential faults grows combinatorially with the number of independent RPCs in the system. I'd be willing to bet that the two bugs that caused this outage were less than 6 months old. In my tenure at Google, the vast majority of bugs that showed up in postmortems were introduced < 3 months before the outage. [1] http://everythingsysadmin.com/2012/09/devops-google-reveals-their-di.html http://everythingsysadmin.com/2012/09/devops-google-reveals-...
- LaFolle 11y ago
  Direct link to DiRT => http://queue.acm.org/detail.cfm?id=2371516 http://queue.acm.org/detail.cfm?id=2371516 I remember my founder (ex Googler) telling us about fault injection at Google. We were pretty amazed by the idea. Thanks for the link @nostrademons.
- peterwwillis 11y ago
  Testing doesn't detect failure, it only detects the failure of a test. Real failures happen more often than test failures, for the same test on the same code with the same input and output. The best systematic solution would detect real failures, not see what happens when you fail a test.
  - nostrademons 11y ago
brandmeyer 11y ago
The standard solution in realtime safety-critical systems is to perform health monitoring in addition to robust fallbacks, such that when the system is falling back, it is reported as unhealthy. For example, the CAN bus normally has an automatic retry feature on a variety of errors. A properly functioning CAN bus should have a bit error rate that is nearly zero. Lightly loaded, it can tolerate a very high error rate (say, due to noise, poor termination, etc). In that situation, the product would report a specific warning message to higher-level SCADA systems, such that it gets bubbled up all the way to the operators.
- nostrademons 11y ago
  The approach at Google is to report the actual error rate up to the monitoring system, and then let the monitoring system decide at what threshold to alert with a warning message. This lets you catch a wide variety of errors, eg. if a single replica has a high error rate, that's probably a wildly different problem from if a whole rack of machines has a high error rate, which is different from every machine in the service having a high error rate, which is different from only the set of machines that were fed a specific query having a high error rate. One of the bugs in this postmortem was that the process in question didn't do this, instead masking the error. Somewhat understandable, as I found the whole "execute a fallback, report the failure, and let the monitoring rules deal with it" philosophy one of the most confusing parts of being a Noogler. If you've never worked on distributed systems before, the idea that there is a monitoring system is a strange concept.
- dfox 11y ago
  CAN and it's automatic retransmit is actually pretty good example of how simple transient problems can quickly overgrow into global system failures. On typical CAN the bandwidth headroom is small enough that when all colliding/failed telegrams would be blindly retransmitted the collision rate would skyrocket and thus only high-priority traffic will make any progress, and as on CAN priority and purpose is intrinsically linked together, from the global point of view nothing will make progress. That's why most CAN controllers have configurable retransmit behavior per packet (drop/retry/raise error and application deal with that) and partially why today's cars have multiple CAN buses.
pmarreck 11y ago
> So, has anyone managed to make progress toward a "systematic solution" in the last 9 years? Allow me to introduce you to the fantastic and battle-tested http://learnyousomeerlang.com/what-is-otp http://learnyousomeerlang.com/what-is-otp , preferably utilized (IMHO) via http://elixir-lang.org/ http://elixir-lang.org/
nkristoffersen 11y ago
The best example so far regarding fault tolerance. The Netflix Simian Army. Introduce failures constantly! http://techblog.netflix.com/2011/07/netflix-simian-army.html http://techblog.netflix.com/2011/07/netflix-simian-army.html

atomic77 11y ago

This is an interesting question, and seems to get to the core of Nassim Taleb's ideas [1] about fragility and the limits of what we can understand, and how many of our attempts to create artificial stability ultimately bring about the opposite. That said, based on this post-mortem, I think Google, and our industry as a whole, is doing a pretty good job. Periodic failures like this are inevitable, and if they serve to make it less likely that a similar failure occurs in the future, then that is a system as a whole that could be described as "anti-fragile". [1] At least my interpretation of them

woodman 11y ago

> So, has anyone managed to make progress toward a "systematic solution" in the last 9 years? That depends on how you define "solution". If development time isn't a concern, then formal verification is a pretty solid solution. AWS has used TLA+ on a subset of its systems. [0] [0] https://en.wikipedia.org/wiki/TLA%2B https://en.wikipedia.org/wiki/TLA%2B

deleted 11y ago

[deleted]

strictfp 11y ago

Degraded modes of operation is one example of how to visualize masked errors. Another is to trigger an alarm on fallbacks. As a general reflection, many distributed system leave out the cause of their changes and only log actions. Instead of logging "new membership, new members are b,c,d" you are better of logging "node a has not responded to heartbeat in the last 30 seconds, considering it faulty". Following such a principle makes it much easier to spot masked bugs, since you can reason about the behaviour much better. Aggregating logs to a central location and being able to analyze global behaviour in retrospect is also a great feature.

senderista 11y ago
Great point! More precisely, each state transition in a system should report the old state, the new state, and the triggering event (cause) to a monitoring system (possibly just a log).

nostrademons 11y ago

Google actually does have a systematic solution: fault injection. Google's systems are designed so that you can (manually, if you have the right privileges) tell an RPC to fail regardless of whether it would otherwise have succeeded, and then test the response of the system as a whole. The problem is that these failure cases are exercised much less frequently than the "normal execution" code paths are. For example, every year Google does DiRT [1] exercises which test system responses to a large calamity, eg. a California earthquake that kills everyone in Mountain View and SF including the senior leadership, and also knocks out all west coast datacenters. The half-life of code at Google (in my observation) is roughly 1 year, which means that half of all code has never gone through a DiRT exercise. The same applies to other, less serious fault injection mechanisms: they may get executed once every year or two, and serious bugs can crop up in the meantime. Automated testing of fault injection isn't really feasible, because the number of potential faults grows combinatorially with the number of independent RPCs in the system. I'd be willing to bet that the two bugs that caused this outage were less than 6 months old. In my tenure at Google, the vast majority of bugs that showed up in postmortems were introduced < 3 months before the outage. [1] http://everythingsysadmin.com/2012/09/devops-google-reveals-their-di.html http://everythingsysadmin.com/2012/09/devops-google-reveals-...

LaFolle 11y ago
Direct link to DiRT => http://queue.acm.org/detail.cfm?id=2371516 http://queue.acm.org/detail.cfm?id=2371516 I remember my founder (ex Googler) telling us about fault injection at Google. We were pretty amazed by the idea. Thanks for the link @nostrademons.
peterwwillis 11y ago
Testing doesn't detect failure, it only detects the failure of a test. Real failures happen more often than test failures, for the same test on the same code with the same input and output. The best systematic solution would detect real failures, not see what happens when you fail a test.
- nostrademons 11y ago

brandmeyer 11y ago

The standard solution in realtime safety-critical systems is to perform health monitoring in addition to robust fallbacks, such that when the system is falling back, it is reported as unhealthy. For example, the CAN bus normally has an automatic retry feature on a variety of errors. A properly functioning CAN bus should have a bit error rate that is nearly zero. Lightly loaded, it can tolerate a very high error rate (say, due to noise, poor termination, etc). In that situation, the product would report a specific warning message to higher-level SCADA systems, such that it gets bubbled up all the way to the operators.

nostrademons 11y ago
The approach at Google is to report the actual error rate up to the monitoring system, and then let the monitoring system decide at what threshold to alert with a warning message. This lets you catch a wide variety of errors, eg. if a single replica has a high error rate, that's probably a wildly different problem from if a whole rack of machines has a high error rate, which is different from every machine in the service having a high error rate, which is different from only the set of machines that were fed a specific query having a high error rate. One of the bugs in this postmortem was that the process in question didn't do this, instead masking the error. Somewhat understandable, as I found the whole "execute a fallback, report the failure, and let the monitoring rules deal with it" philosophy one of the most confusing parts of being a Noogler. If you've never worked on distributed systems before, the idea that there is a monitoring system is a strange concept.
dfox 11y ago
CAN and it's automatic retransmit is actually pretty good example of how simple transient problems can quickly overgrow into global system failures. On typical CAN the bandwidth headroom is small enough that when all colliding/failed telegrams would be blindly retransmitted the collision rate would skyrocket and thus only high-priority traffic will make any progress, and as on CAN priority and purpose is intrinsically linked together, from the global point of view nothing will make progress. That's why most CAN controllers have configurable retransmit behavior per packet (drop/retry/raise error and application deal with that) and partially why today's cars have multiple CAN buses.

pmarreck 11y ago

> So, has anyone managed to make progress toward a "systematic solution" in the last 9 years? Allow me to introduce you to the fantastic and battle-tested http://learnyousomeerlang.com/what-is-otp http://learnyousomeerlang.com/what-is-otp , preferably utilized (IMHO) via http://elixir-lang.org/ http://elixir-lang.org/

nkristoffersen 11y ago

The best example so far regarding fault tolerance. The Netflix Simian Army. Introduce failures constantly! http://techblog.netflix.com/2011/07/netflix-simian-army.html http://techblog.netflix.com/2011/07/netflix-simian-army.html