3 ms·
As I was reading the paper, it dawned on me that persistent large-scale systemwide problems in many other domains can be understood as metastable failures -- in
by cs702 5y ago
As I was reading the paper, it dawned on me that persistent large-scale systemwide problems in many other domains can be understood as metastable failures -- including, for example, persistent supply chain problems, persistent waves of disinformation on social networks, persistent government support/rescue of financial firms, persistent bear markets, and deep economic recessions. Quoting from the paper:
"A system starts in a stable state. Once the load rises above a certain threshold -- implicit and invisible -- the system enters a vulnerable state. The vulnerable system is healthy, but may fall into an unrecoverable metastable state due to a trigger. The vulnerable state is not an overloaded state; a system can run for months or years in the vulnerable state and then get stuck in a metastable state without any increase in load. In fact, many ... systems choose to run in the vulnerable state all the time because it has much higher efficiency than the stable state.
When one of many potential triggers causes the system to enter the metastable state, a feedback loop sustains the failure, causing the system to remain in the failure state until a big enough corrective action is applied. In the most severe outages, the feedback loop is contagious, causing portions of the system that weren’t exposed to the trigger to enter the failure state as well. It is common for an outage that involves a metastable failure to be initially blamed on the trigger, but the true root cause is the sustaining effect.
Metastable failures have a disproportionate impact on hyperscale distributed systems. ... The strength of many feedback loops is proportional to the scale, so they can slip past even a robust testing and deployment regime. The difference between the trigger and the sustaining effect makes it hard to discover the correct response, increasing the time to recovery. Shedding load as a corrective action can be a further source of disruption for users."