3 ms·

The standard solution in realtime safety-critical systems is to perform health monitoring in addition to robust fallbacks, such that when the system is falling

The standard solution in realtime safety-critical systems is to perform health monitoring in addition to robust fallbacks, such that when the system is falling back, it is reported as unhealthy. For example, the CAN bus normally has an automatic retry feature on a variety of errors. A properly functioning CAN bus should have a bit error rate that is nearly zero. Lightly loaded, it can tolerate a very high error rate (say, due to noise, poor termination, etc). In that situation, the product would report a specific warning message to higher-level SCADA systems, such that it gets bubbled up all the way to the operators.

nostrademons 11y ago
The approach at Google is to report the actual error rate up to the monitoring system, and then let the monitoring system decide at what threshold to alert with a warning message. This lets you catch a wide variety of errors, eg. if a single replica has a high error rate, that's probably a wildly different problem from if a whole rack of machines has a high error rate, which is different from every machine in the service having a high error rate, which is different from only the set of machines that were fed a specific query having a high error rate. One of the bugs in this postmortem was that the process in question didn't do this, instead masking the error. Somewhat understandable, as I found the whole "execute a fallback, report the failure, and let the monitoring rules deal with it" philosophy one of the most confusing parts of being a Noogler. If you've never worked on distributed systems before, the idea that there is a monitoring system is a strange concept.
dfox 11y ago
CAN and it's automatic retransmit is actually pretty good example of how simple transient problems can quickly overgrow into global system failures. On typical CAN the bandwidth headroom is small enough that when all colliding/failed telegrams would be blindly retransmitted the collision rate would skyrocket and thus only high-priority traffic will make any progress, and as on CAN priority and purpose is intrinsically linked together, from the global point of view nothing will make progress. That's why most CAN controllers have configurable retransmit behavior per packet (drop/retry/raise error and application deal with that) and partially why today's cars have multiple CAN buses.

nostrademons 11y ago

The approach at Google is to report the actual error rate up to the monitoring system, and then let the monitoring system decide at what threshold to alert with a warning message. This lets you catch a wide variety of errors, eg. if a single replica has a high error rate, that's probably a wildly different problem from if a whole rack of machines has a high error rate, which is different from every machine in the service having a high error rate, which is different from only the set of machines that were fed a specific query having a high error rate. One of the bugs in this postmortem was that the process in question didn't do this, instead masking the error. Somewhat understandable, as I found the whole "execute a fallback, report the failure, and let the monitoring rules deal with it" philosophy one of the most confusing parts of being a Noogler. If you've never worked on distributed systems before, the idea that there is a monitoring system is a strange concept.

dfox 11y ago

CAN and it's automatic retransmit is actually pretty good example of how simple transient problems can quickly overgrow into global system failures. On typical CAN the bandwidth headroom is small enough that when all colliding/failed telegrams would be blindly retransmitted the collision rate would skyrocket and thus only high-priority traffic will make any progress, and as on CAN priority and purpose is intrinsically linked together, from the global point of view nothing will make progress. That's why most CAN controllers have configurable retransmit behavior per packet (drop/retry/raise error and application deal with that) and partially why today's cars have multiple CAN buses.