3 ms·

> By definition, if a failure occurs, it's because it passed multiple safeguards that were already in place. Having worked on a good bunch of critical system,

> By definition, if a failure occurs, it's because it passed multiple safeguards that were already in place. Having worked on a good bunch of critical system, there aren't multiple safeguards in most hardware. E.g. a multiplication error in a core will not be detected by an external device. Or a bit flip when reading cache, or from a storage device. Very often the only real safeguard is to do the whole computation twice on two different hosts. I would rather have many low-reliability hosts and do computation twice than few high-reliability and very expensive host. Unfortunately the software side is really lagging behind when it comes to reproducible computing. Reproducible builds are a good step in that direction and it took many decades to get there.

xvector 5y ago
> Very often the only real safeguard is to do the whole computation twice on two different hosts. Three different hosts, for quorum, right?
- ajross 5y ago
  Depends on the system. In this case it seems like retries are possible after a failure, so two is sufficient to detect bad data. You need three in real time situations where you don't have the capability to go back and figure it out.
  - goodpoint 5y ago
    Spot on. Very often doing an occasional (very rare) retry is acceptable. Sometimes doing the same processing twice can be also a way to implement safe(r) rolling updates.
- sanctified384 5y ago
  Two hosts is efficient. Do it twice on two different hosts and then compare the results. If there is a mismatch, throw it away and redo it again on 2 hosts. A total of 4 computations are needed. But only if the difference really was due to bit flips, the chance of which are exceedingly rare. In all the rest of the cases, you get away with two instead of three computations.
aseipp 5y ago
Safeguards do not just exist at the level of technology but also politics, social structures, policies, design decisions, human interactions, and so on and so forth. "Criticality" in particular is something defined by humans, not a vacuum, and humans are components of all complex systems, as much as any multiplier or hardware unit is. The fact a multiplier can return an error is exactly in line with this: it can only happen after an array of other things allow it to, some of them not computational or computerized at all. And not every failure will also result in catastrophe as it did here. More generally such failures cannot be eliminated, because latent failures exist everywhere even when you have TMR or whatever it is people do these days. Thinking there is any "only real safeguard" like quorums or TMR is exactly part of the problem with this line of thought. The quote I made is actually very specifically is in reference to this paper, in particular point 2, which is mandatory reading for any systems engineer, IMO, though perhaps the word "safeguard" is too strong for the taste of some here. But focusing on definitions of words is besides the point and falls into the same traps this paper mentions: https://how.complexsystems.fail/ https://how.complexsystems.fail/ Back to the original point: is ECC the single solution to this catastrophe? No, probably not. Systems are constantly changing and failure is impossible to eliminate. Design decisions and a number of other decisions could have mitigated it and caused this failure to not be catastrophic. Another thing might also cause it to topple. But let's not pretend like we don't know what we're dealing with, either, when we've already built these tools and know they work. We've studied ECC plenty! You don't need to carry water for a corporation trying to keep its purse filled to the brim (by cutting costs) to proclaim that failure is inevitable and most things chug on, regardless. We already know that much.
twistedpair 5y ago
Failing to account for bit flips and HW failure is too common in web/service coding. Lookup how Google dropped a massive Big Table instance in prod, and traced it back to a cosmic ray bit flip that made a WRITE instruction into a DROP TABLE instruction. I laugh when I compare my day to day coding to that of an avionics programmer in the aero industry.
- xpasky 5y ago
  I couldn't find it. Do you have a reference?
  - twistedpair 5y ago
    There are some examples here [1]. Dig as I may, I cannot locate the original Google source I read about this ~3y ago. 1. https://www.usenix.org/system/files/conference/atc12/atc12-final190.pdf https://www.usenix.org/system/files/conference/atc12/atc12-f...
- goodpoint 5y ago
  The "web" coding words falls for things like leftpad, imagine talking about bit flips. It's sad how immature the software industry can be. It's been around for "only" 60/70 years after all.

xvector 5y ago

> Very often the only real safeguard is to do the whole computation twice on two different hosts. Three different hosts, for quorum, right?

ajross 5y ago
Depends on the system. In this case it seems like retries are possible after a failure, so two is sufficient to detect bad data. You need three in real time situations where you don't have the capability to go back and figure it out.
- goodpoint 5y ago
  Spot on. Very often doing an occasional (very rare) retry is acceptable. Sometimes doing the same processing twice can be also a way to implement safe(r) rolling updates.
sanctified384 5y ago
Two hosts is efficient. Do it twice on two different hosts and then compare the results. If there is a mismatch, throw it away and redo it again on 2 hosts. A total of 4 computations are needed. But only if the difference really was due to bit flips, the chance of which are exceedingly rare. In all the rest of the cases, you get away with two instead of three computations.

aseipp 5y ago

Safeguards do not just exist at the level of technology but also politics, social structures, policies, design decisions, human interactions, and so on and so forth. "Criticality" in particular is something defined by humans, not a vacuum, and humans are components of all complex systems, as much as any multiplier or hardware unit is. The fact a multiplier can return an error is exactly in line with this: it can only happen after an array of other things allow it to, some of them not computational or computerized at all. And not every failure will also result in catastrophe as it did here. More generally such failures cannot be eliminated, because latent failures exist everywhere even when you have TMR or whatever it is people do these days. Thinking there is any "only real safeguard" like quorums or TMR is exactly part of the problem with this line of thought. The quote I made is actually very specifically is in reference to this paper, in particular point 2, which is mandatory reading for any systems engineer, IMO, though perhaps the word "safeguard" is too strong for the taste of some here. But focusing on definitions of words is besides the point and falls into the same traps this paper mentions: https://how.complexsystems.fail/ https://how.complexsystems.fail/ Back to the original point: is ECC the single solution to this catastrophe? No, probably not. Systems are constantly changing and failure is impossible to eliminate. Design decisions and a number of other decisions could have mitigated it and caused this failure to not be catastrophic. Another thing might also cause it to topple. But let's not pretend like we don't know what we're dealing with, either, when we've already built these tools and know they work. We've studied ECC plenty! You don't need to carry water for a corporation trying to keep its purse filled to the brim (by cutting costs) to proclaim that failure is inevitable and most things chug on, regardless. We already know that much.

twistedpair 5y ago

Failing to account for bit flips and HW failure is too common in web/service coding. Lookup how Google dropped a massive Big Table instance in prod, and traced it back to a cosmic ray bit flip that made a WRITE instruction into a DROP TABLE instruction. I laugh when I compare my day to day coding to that of an avionics programmer in the aero industry.

xpasky 5y ago
I couldn't find it. Do you have a reference?
- twistedpair 5y ago
  There are some examples here [1]. Dig as I may, I cannot locate the original Google source I read about this ~3y ago. 1. https://www.usenix.org/system/files/conference/atc12/atc12-final190.pdf https://www.usenix.org/system/files/conference/atc12/atc12-f...
goodpoint 5y ago
The "web" coding words falls for things like leftpad, imagine talking about bit flips. It's sad how immature the software industry can be. It's been around for "only" 60/70 years after all.