4 ms·
To add to that: "uptime" is a sort of misleading statistic. A single server can have an "uptime" of 20 years sitting in the closet of an office in a corporate p
by throwaway984393 5y ago
To add to that: "uptime" is a sort of misleading statistic. A single server can have an "uptime" of 20 years sitting in the closet of an office in a corporate park somewhere in suburban Maryland. To the point that nobody can find the server, but they can ssh to it, and yup... it's still up.
On the other hand, the more components (servers, disks, switches, etc) you have, the higher the probability of failure. The more traffic, more data, more changes more... anything, tends to lead to more failure. So paradoxically, small systems can have better "uptime" than large robust well-designed ones.
In my experience managing large distributed decentralized systems, the more buggy the hardware, the larger the cascades of software failures, the harder it was to fix. Industrial-grade hardware that nobody monkeyed with (or was replaced as soon as any failures were detected) led to the most stable systems. Reliable, dedicated private network paths led to more stability. At a certain point we didn't really need leader election, because they'd never need a new leader, because things hardly ever failed.
- nine_k 5y agoEvery device has a small probability of failure. If you just take one device, it can either be completely functional and chug along for 20 years, or completely fail, which happens rarely but is painful. In either case, your software can ignore the failure state, there's nothing it can do at the case of a failure. If you take 1000 such devices, then the vast majority will be fine, but a small number of them will fail. No matter how you fix them and replace them, you will keep seeing some degraded nodes, or temporarily disconnected nodes because the network has glitches, too. So your software has to always be ready to face a failure of a remote node, and always be ready to gracefully handle it. This completely changes the way you think.