3 ms·

> There's no more time-honored way to get things working again, from toasters to global-scale distributed systems, than turning them off and on again. Apart fr

> There's no more time-honored way to get things working again, from toasters to global-scale distributed systems, than turning them off and on again. Apart from the reasons discussed in the article, another reason to turn things off and then on again is because if you never do that, there's no guarantee that you can do that successfully. It's easy in this day of distributed systems to accidentally build an "always on" system that involves mutually dependent systems that cannot recover when everything goes down. It's good to exercise the "everything is down/recover from zero" path now and then.

hinkley 4y ago
If you can't turn things on, you can't turn them off (well, you can, but will you survive?). If you can't turn things on, that means you also can't spin up microcosms of your system and reproduce tricky bugs, or try out crazy theories for making things much better, instead of a tiny bit better. Every project where I've gotten that has gone much, much better for everyone afterward, including the clever people. Half the ones where I didn't, we all suffered together. For the other half people seemed to think I was scapegoating, covering for my own inadequacies. But I've never quite figured out what sort of masochist/sadist thinks those two statements mean the same thing. Half the point of engineering and the entire point of human factors is exactly that: covering for your own inadequacies. It's not scapegoating, it's planning your way out of the problems you can see, making space for yourself for the ones you can't. Systems that cannot be replicated on cold hardware have very little space in them.
mjb 4y ago
> It's good to exercise the "everything is down/recover from zero" path now and then. Yes! The more often the better. Over the years I've been at AWS, we've focused more and more on building systems with relatively small internal partitions ("cells"). There are several reasons for this---the most important being reducing the "blast radius" of outages. One often-overlooked reason is that building out of smaller independent units allows units small enough to be economically turned off and on at "full scale". You can't do that with a region-scale deployment of any big AWS service, but may be able to with a "cell".

hinkley 4y ago

If you can't turn things on, you can't turn them off (well, you can, but will you survive?). If you can't turn things on, that means you also can't spin up microcosms of your system and reproduce tricky bugs, or try out crazy theories for making things much better, instead of a tiny bit better. Every project where I've gotten that has gone much, much better for everyone afterward, including the clever people. Half the ones where I didn't, we all suffered together. For the other half people seemed to think I was scapegoating, covering for my own inadequacies. But I've never quite figured out what sort of masochist/sadist thinks those two statements mean the same thing. Half the point of engineering and the entire point of human factors is exactly that: covering for your own inadequacies. It's not scapegoating, it's planning your way out of the problems you can see, making space for yourself for the ones you can't. Systems that cannot be replicated on cold hardware have very little space in them.

mjb 4y ago

> It's good to exercise the "everything is down/recover from zero" path now and then. Yes! The more often the better. Over the years I've been at AWS, we've focused more and more on building systems with relatively small internal partitions ("cells"). There are several reasons for this---the most important being reducing the "blast radius" of outages. One often-overlooked reason is that building out of smaller independent units allows units small enough to be economically turned off and on at "full scale". You can't do that with a region-scale deployment of any big AWS service, but may be able to with a "cell".