3 ms·
How do companies deal with this in practice? Running 3 or more redundant servers which vote on the correct result of an operation?
by FartyMcFarter 5y ago
How do companies deal with this in practice? Running 3 or more redundant servers which vote on the correct result of an operation?
- H8crilA 5y agoYou don't deal with this, other than fixing the damage and removing the bad machines. The only way forward is expanded testing, that's what the paper from Google is (also) about. I think this issue will always be with us, to a larger or smaller extent. There's probably a ton of data corruption out there that happens to be in places that doesn't really cause big problems.
- secondcoming 5y agoCould cloud providers not run some tests on each machine core at startup and have the hypervisor isolate any bad cores from the guest?
- H8crilA 5y agoThey already do that, not exactly on each reboot but trust me they really test the hardware. The problem is that nobody knows how to write tests that catch all errors.
- stubish 5y agoYes, but generally only when human life is on the line, such as aerospace. For just about everything else, the cost of redundancy in dollars and speed is too high. After all, problems are extremely rare, and when the extremely rare problems occur, it is extremely unlikely it matters. Versus a guaranteed cost of more than tripling your hardware, power, data center etc. costs. Maybe you do it in part of your stack, such as redundant mainframes or database nodes. But the most you will see on your appservers is ECC memory in an attempt to avoid cosmic ray bit flips (a similar problem we have been aware of for decades).