3 ms·
weird. they had a problem caused by a series of bugs, yet the word "test" doesn't appear anywhere in that page.
by andrewcooke 14y ago
weird. they had a problem caused by a series of bugs, yet the word "test" doesn't appear anywhere in that page.
- pja 14y agoTesting distributed systems is much, much harder than doing so on a monolithic codebase. The number of failure modes goes up very rapidly with the number of nodes in the system & your code has to (in principle) cope with every possible one.
- andrewcooke 14y agotrue, but it sounds like they (and perhaps you) have never even heard of the chaos monkey.
- pja 14y agoRandomly killing instances wouldn't have detected this particular failure mode as far as I can see, since the error lay in the inability to resurrect a failed process under certain circumstances.