3 ms·

Metastability and Distributed Systems

42 points by kiyanwang 4y ago

smcameron 4y ago
> There's no more time-honored way to get things working again, from toasters to global-scale distributed systems, than turning them off and on again. Apart from the reasons discussed in the article, another reason to turn things off and then on again is because if you never do that, there's no guarantee that you can do that successfully. It's easy in this day of distributed systems to accidentally build an "always on" system that involves mutually dependent systems that cannot recover when everything goes down. It's good to exercise the "everything is down/recover from zero" path now and then.
- hinkley 4y ago
  If you can't turn things on, you can't turn them off (well, you can, but will you survive?). If you can't turn things on, that means you also can't spin up microcosms of your system and reproduce tricky bugs, or try out crazy theories for making things much better, instead of a tiny bit better. Every project where I've gotten that has gone much, much better for everyone afterward, including the clever people. Half the ones where I didn't, we all suffered together. For the other half people seemed to think I was scapegoating, covering for my own inadequacies. But I've never quite figured out what sort of masochist/sadist thinks those two statements mean the same thing. Half the point of engineering and the entire point of human factors is exactly that: covering for your own inadequacies. It's not scapegoating, it's planning your way out of the problems you can see, making space for yourself for the ones you can't. Systems that cannot be replicated on cold hardware have very little space in them.
- mjb 4y ago
  > It's good to exercise the "everything is down/recover from zero" path now and then. Yes! The more often the better. Over the years I've been at AWS, we've focused more and more on building systems with relatively small internal partitions ("cells"). There are several reasons for this---the most important being reducing the "blast radius" of outages. One often-overlooked reason is that building out of smaller independent units allows units small enough to be economically turned off and on at "full scale". You can't do that with a region-scale deployment of any big AWS service, but may be able to with a "cell".
mhh__ 4y ago
There's probably a Dunning-Kruger shaped curve for "probability of turning the widget on and off" versus "widget knowledge".
vajrabum 4y ago
To get a real handle on this you need to look at metastability in electronic circuits. Essentially any asynchronous circuit with feedback even one as simple as a flip flop can fall into a stuck state such that it may never converge to a valid state (i.e. 1 or 0). The fix in electronics is to clock everything such that the setup and hold times are satisfied. The price for the clocking is a reduction in speed. In more general distributed systems, the fixes are going to be analogous. You'll need some mechanism for admission control and you may need it internally and externally. The article discusses not meeting the response time budget for the web page or app. With admission control you can tell if you're not going to meet that budget and implement a circuit breaker, rate limit and/or polling mechanism on the clients. That increases the complexity of the system but provides increased reliability. Building glitchy systems or circuits is an engineering failure. See here for the discussion of metastable circuits. https://en.wikipedia.org/wiki/Metastability_(electronics) https://en.wikipedia.org/wiki/Metastability_(electronics)
vitus 4y ago
> allows us to start applying the mathematical techniques of control theory and dynamical systems theory. I'm skeptical that this approach will bear much fruit beyond, say, implementing a basic PID (more realistically IMO, proportional) controller (e.g. number of active processing threads is high? reject a greater fraction of incoming requests), for a number of reasons: - In order to apply state-space systems theory, you need to be able to model your system. Not only is your input/output relationship typically highly nonlinear, you also have a heavy mix of discrete and continuous components. - Oh, and you have stochastic processes at play here too, whether it's modeling arrival times / sizes of HTTP requests or performance characteristics in the underlying hardware, or even that of RPCs / database queries. - One classic result of systems theory is that input latency leads to oscillation. One constant of distributed systems is message latency (whether due to propagation delay, or due to retransmissions to combat loss). Most classical systems theory you'll encounter (short of advanced graduate material) applies specifically to linear time-invariant systems. Guess what, we don't write many of those. And even if you have a nonlinear time-invariant system, there are cases where the linearized model isn't guaranteed to behave the same even locally. I can't even begin to imagine how you'd accurately model anything beyond the simplest systems.
- mjb 4y ago
  I think that's a reasonable criticism, but I'm more optimistic that there's value here for two reasons. One is that even a "basic proportional" level of control is under-used, and most practitioners don't even go that far into analyzing the dynamic behavior of their systems. The other is that I think that fixing "local" behaviors where systems are linear enough and time invariant enough is likely to have a lot of value. Many of the "hold down" loops are relatively local effects. You're right that a lot of the techniques of classical control theory (both continuous and discrete-time) are hard to directly apply in a distributed systems context today. I do think there's a lot of value in the patterns of thinking, and that future research along those lines is likely to bear fruit (or, at least, be the genesis of better-suited techniques).
cb321 4y ago
The link imbalance section of the HotOS paper discussed in the blog post reminds me of issues with regular old TCP discussed in, e.g. https://dl.acm.org/doi/pdf/10.1145/633025.633035 https://dl.acm.org/doi/pdf/10.1145/633025.633035 (as does the mention of control theory).

42 points by kiyanwang 4y ago

> There's no more time-honored way to get things working again, from toasters to global-scale distributed systems, than turning them off and on again. Apart from the reasons discussed in the article, another reason to turn things off and then on again is because if you never do that, there's no guarantee that you can do that successfully. It's easy in this day of distributed systems to accidentally build an "always on" system that involves mutually dependent systems that cannot recover when everything goes down. It's good to exercise the "everything is down/recover from zero" path now and then.

hinkley 4y ago
If you can't turn things on, you can't turn them off (well, you can, but will you survive?). If you can't turn things on, that means you also can't spin up microcosms of your system and reproduce tricky bugs, or try out crazy theories for making things much better, instead of a tiny bit better. Every project where I've gotten that has gone much, much better for everyone afterward, including the clever people. Half the ones where I didn't, we all suffered together. For the other half people seemed to think I was scapegoating, covering for my own inadequacies. But I've never quite figured out what sort of masochist/sadist thinks those two statements mean the same thing. Half the point of engineering and the entire point of human factors is exactly that: covering for your own inadequacies. It's not scapegoating, it's planning your way out of the problems you can see, making space for yourself for the ones you can't. Systems that cannot be replicated on cold hardware have very little space in them.
mjb 4y ago
> It's good to exercise the "everything is down/recover from zero" path now and then. Yes! The more often the better. Over the years I've been at AWS, we've focused more and more on building systems with relatively small internal partitions ("cells"). There are several reasons for this---the most important being reducing the "blast radius" of outages. One often-overlooked reason is that building out of smaller independent units allows units small enough to be economically turned off and on at "full scale". You can't do that with a region-scale deployment of any big AWS service, but may be able to with a "cell".

mhh__ 4y ago

There's probably a Dunning-Kruger shaped curve for "probability of turning the widget on and off" versus "widget knowledge".

vajrabum 4y ago

To get a real handle on this you need to look at metastability in electronic circuits. Essentially any asynchronous circuit with feedback even one as simple as a flip flop can fall into a stuck state such that it may never converge to a valid state (i.e. 1 or 0). The fix in electronics is to clock everything such that the setup and hold times are satisfied. The price for the clocking is a reduction in speed. In more general distributed systems, the fixes are going to be analogous. You'll need some mechanism for admission control and you may need it internally and externally. The article discusses not meeting the response time budget for the web page or app. With admission control you can tell if you're not going to meet that budget and implement a circuit breaker, rate limit and/or polling mechanism on the clients. That increases the complexity of the system but provides increased reliability. Building glitchy systems or circuits is an engineering failure. See here for the discussion of metastable circuits. https://en.wikipedia.org/wiki/Metastability_(electronics) https://en.wikipedia.org/wiki/Metastability_(electronics)

vitus 4y ago

> allows us to start applying the mathematical techniques of control theory and dynamical systems theory. I'm skeptical that this approach will bear much fruit beyond, say, implementing a basic PID (more realistically IMO, proportional) controller (e.g. number of active processing threads is high? reject a greater fraction of incoming requests), for a number of reasons: - In order to apply state-space systems theory, you need to be able to model your system. Not only is your input/output relationship typically highly nonlinear, you also have a heavy mix of discrete and continuous components. - Oh, and you have stochastic processes at play here too, whether it's modeling arrival times / sizes of HTTP requests or performance characteristics in the underlying hardware, or even that of RPCs / database queries. - One classic result of systems theory is that input latency leads to oscillation. One constant of distributed systems is message latency (whether due to propagation delay, or due to retransmissions to combat loss). Most classical systems theory you'll encounter (short of advanced graduate material) applies specifically to linear time-invariant systems. Guess what, we don't write many of those. And even if you have a nonlinear time-invariant system, there are cases where the linearized model isn't guaranteed to behave the same even locally. I can't even begin to imagine how you'd accurately model anything beyond the simplest systems.

mjb 4y ago
I think that's a reasonable criticism, but I'm more optimistic that there's value here for two reasons. One is that even a "basic proportional" level of control is under-used, and most practitioners don't even go that far into analyzing the dynamic behavior of their systems. The other is that I think that fixing "local" behaviors where systems are linear enough and time invariant enough is likely to have a lot of value. Many of the "hold down" loops are relatively local effects. You're right that a lot of the techniques of classical control theory (both continuous and discrete-time) are hard to directly apply in a distributed systems context today. I do think there's a lot of value in the patterns of thinking, and that future research along those lines is likely to bear fruit (or, at least, be the genesis of better-suited techniques).

cb321 4y ago

The link imbalance section of the HotOS paper discussed in the blog post reminds me of issues with regular old TCP discussed in, e.g. https://dl.acm.org/doi/pdf/10.1145/633025.633035 https://dl.acm.org/doi/pdf/10.1145/633025.633035 (as does the mention of control theory).