5 ms·

Is silicon cheap enough that we can do what we do for spacecraft and have multiple processors (where the majority "wins") for fault tolerance?

maweki 5y ago
Cheap is probably not the question. The synchronization and communication overhead is enormous. Would you check that every register assignment matches? Or every page write?
- mirker 5y ago
  You can have a log of N register “transactions”, where N is large enough to hide core-to-core communication. If any of N transactions fail due to mismatch between cores, you roll back and throw exceptions. A lot of this logic is already in out of order execution (e.g., Tomasulo algorithm). Memory has ECC and is probably a different problem.
jeffbee 5y ago
The economics will never favor this approach. Customers will not choose to pay double to avoid the 1-in-a-million chance of occasionally getting a slightly wrong answer.
- theevilsharpie 5y ago
  > Customers will not choose to pay double to avoid the 1-in-a-million chance of occasionally getting a slightly wrong answer. With today's high-speed multi-core processors, a 1-in-a-million chance of a computation error would mean tens to hundreds of thousands of errors per second.
  - eric__cartman 5y ago
    I can imagine most consumers that do any sort of work with their computer would appreciate close to 100% stability when they need to get work done. That's usually why no one that depends on their computers to work day in and day out overclocks their components. The marginal performance gains aren't worth the added unreliability and added power/heat/noise footprint.
    - sokoloff 5y ago
      The lack of adoption/demand for just ECC RAM by consumers would seem to be an argument in the opposing direction. (Yes, it’s not widely available currently, but I think it’s safe to safe that availability is driven by predictions about adoption given past market behavior.)
      - temac 5y ago
        Who decided there is a lack of consumer demand? There is a lack of OEM demand for sure which is driven by the fact that most companies are willing to sell crap if it can save 1 cent per product. The average consumer does not even know this pb exists. Adding to that the artificial market seg by Intel which is absolutely stupid and the consummer actually can not buy a consummer CPU that supports ECC. The situation is then locked into a vicious circle where all the components have ridiculous premiums and lower volumes.
        sokoloff 5y ago
        Do you think the OEMs and Intel generally ignore what consumers demand (and are willing to pay for)? I don’t.
        ncmncm 5y ago
        Consumers are not the customer. System integrators are the customer. They are motivated to minimize the number of distinct manufacturing targets. Consumers have no choice but to take what is offered.
        a1369209993 5y ago
        Yes. For example, Intel Management Engine.
        google234123 5y ago
        Consumers don't care about it. Just like they don't care about the internal software running in their microwave oven.
- mikepurvis 5y ago
  Does it have to be double? I know it's not a direct analogue, but parity schemes like RAID 6 or ECC RAM don't double the cost. So the question is, how do you check these results without actually doing them twice? Is there a role here for frameworks or OS to impose sanity checks? Obviously we already have assertions, but something gentler than a panic, where it says "this is suspect, let's go back and confirm."
icegreentea2 5y ago
I think it honestly all depends on what the dominant causal factors and how this scales with node size. Effectively, if unreliability increases at the same rate or faster than the performance increase as node size decreases, and 'high reliability' compute can be easily and generally segregated from other compute, then it would probably be easier just to not decrease node size rather than parallelize at the chip/core level. Certainly, the software cost would be much easier.
colechristensen 5y ago
You could probably do something like using idle cores (or idle hyperthreads) to duplicate instructions on an opportunistic basis to verify outcomes on a less than complete basis. There would be thermal and power consequences but some situations care about that less than others.
- haneefmubarak 5y ago
  Unless the idle hyperthreads are on different cores, you'd most likely have the same execution results. Using idle cores could be interesting, but your thermal and power budget would be shared so your overall performance would still decrease. This is probably difficult to do at a fine grained level, but I imagine that coarser synchronization and checks (both in software) could provide the necessary assurances that code executing on a single core is consistent with that of other cores.
mywittyname 5y ago
If data integrity issues become a problem, it might be cheaper to mark certain cores as fault-tolerant and provide the OS with mechanisms to denote certain threads as requiring fault-tolerance.
- yummypaint 5y ago
  This is a good idea. In a sense this is somewhat available now when choosing whether to run certain things on gpu vs cpu. My understanding is gpus tend to play faster and looser since people don't tend to notice a few occasional weird pixels in a single frame. What if it could be made finer grained by the instruction?
  - mywittyname 5y ago
    This is one of the reasons people using GPUs for Important Calculations (CAD) spend big bucks on Quadro cards rather than the ostensibly equivalent GeForce. Quadro cards have ECC memory and allow higher precision floats to be used for certain calculations.
toast0 5y ago
I think there would be a pretty large hardware cost to ensure the input signals come to both processors at the same clock everytime on the many high speed interfaces a modern CPU is using. And you'd need to eliminate use of non-deterministic CPU local data, like RDRAND and on die temperature, power, etc sensors. Most likely, you'd want to run the CPUs at fixed clock speed to avoid any differences in settling time when switching speeds. This could probably effectively fine broken CPUs (although you wouldn't know which of the pair was broken), but you could still have other broken devices resulting in bad computations. It might be better to run calculations on two separate nodes and compare; possibly only for important calculations.
- mirker 5y ago
  It’s not necessary to serialize the full execution to detect errors. On an out of order processor, there is already buffering of results happening that is eventually serialized to visible state. To check errors, you could just have one buffer per processor and compare results before serialization, raising an error on mismatch between processors. Serialization is merely indicating that both _visible_ executions have agreed up to that instruction but it still allows for some local/internal disagreements. For example, two instructions can finish in opposite orders across cores and that is fine as long as they are serialized in order. As for settling times, those are random anyway. Processors are binned according to how good the settling times ended up being. It’s unlikely to have two homogeneous chips.
einpoklum 5y ago
> Is silicon cheap enough etc. No, it is not. You can always trade off performance for reliability by repeating your computations several times, preferably with some variation in the distribution/timing of work to avoid the same potential hardware failure pattern.

maweki 5y ago

Cheap is probably not the question. The synchronization and communication overhead is enormous. Would you check that every register assignment matches? Or every page write?

mirker 5y ago
You can have a log of N register “transactions”, where N is large enough to hide core-to-core communication. If any of N transactions fail due to mismatch between cores, you roll back and throw exceptions. A lot of this logic is already in out of order execution (e.g., Tomasulo algorithm). Memory has ECC and is probably a different problem.

jeffbee 5y ago

The economics will never favor this approach. Customers will not choose to pay double to avoid the 1-in-a-million chance of occasionally getting a slightly wrong answer.

theevilsharpie 5y ago
> Customers will not choose to pay double to avoid the 1-in-a-million chance of occasionally getting a slightly wrong answer. With today's high-speed multi-core processors, a 1-in-a-million chance of a computation error would mean tens to hundreds of thousands of errors per second.
- eric__cartman 5y ago
  I can imagine most consumers that do any sort of work with their computer would appreciate close to 100% stability when they need to get work done. That's usually why no one that depends on their computers to work day in and day out overclocks their components. The marginal performance gains aren't worth the added unreliability and added power/heat/noise footprint.
  - sokoloff 5y ago
    The lack of adoption/demand for just ECC RAM by consumers would seem to be an argument in the opposing direction. (Yes, it’s not widely available currently, but I think it’s safe to safe that availability is driven by predictions about adoption given past market behavior.)
    - temac 5y ago
      Who decided there is a lack of consumer demand? There is a lack of OEM demand for sure which is driven by the fact that most companies are willing to sell crap if it can save 1 cent per product. The average consumer does not even know this pb exists. Adding to that the artificial market seg by Intel which is absolutely stupid and the consummer actually can not buy a consummer CPU that supports ECC. The situation is then locked into a vicious circle where all the components have ridiculous premiums and lower volumes.
      - sokoloff 5y ago
        Do you think the OEMs and Intel generally ignore what consumers demand (and are willing to pay for)? I don’t.
        ncmncm 5y ago
        Consumers are not the customer. System integrators are the customer. They are motivated to minimize the number of distinct manufacturing targets. Consumers have no choice but to take what is offered.
        a1369209993 5y ago
        Yes. For example, Intel Management Engine.
        google234123 5y ago
        Consumers don't care about it. Just like they don't care about the internal software running in their microwave oven.
mikepurvis 5y ago
Does it have to be double? I know it's not a direct analogue, but parity schemes like RAID 6 or ECC RAM don't double the cost. So the question is, how do you check these results without actually doing them twice? Is there a role here for frameworks or OS to impose sanity checks? Obviously we already have assertions, but something gentler than a panic, where it says "this is suspect, let's go back and confirm."

icegreentea2 5y ago

I think it honestly all depends on what the dominant causal factors and how this scales with node size. Effectively, if unreliability increases at the same rate or faster than the performance increase as node size decreases, and 'high reliability' compute can be easily and generally segregated from other compute, then it would probably be easier just to not decrease node size rather than parallelize at the chip/core level. Certainly, the software cost would be much easier.

colechristensen 5y ago

You could probably do something like using idle cores (or idle hyperthreads) to duplicate instructions on an opportunistic basis to verify outcomes on a less than complete basis. There would be thermal and power consequences but some situations care about that less than others.

haneefmubarak 5y ago
Unless the idle hyperthreads are on different cores, you'd most likely have the same execution results. Using idle cores could be interesting, but your thermal and power budget would be shared so your overall performance would still decrease. This is probably difficult to do at a fine grained level, but I imagine that coarser synchronization and checks (both in software) could provide the necessary assurances that code executing on a single core is consistent with that of other cores.

mywittyname 5y ago

If data integrity issues become a problem, it might be cheaper to mark certain cores as fault-tolerant and provide the OS with mechanisms to denote certain threads as requiring fault-tolerance.

yummypaint 5y ago
This is a good idea. In a sense this is somewhat available now when choosing whether to run certain things on gpu vs cpu. My understanding is gpus tend to play faster and looser since people don't tend to notice a few occasional weird pixels in a single frame. What if it could be made finer grained by the instruction?
- mywittyname 5y ago
  This is one of the reasons people using GPUs for Important Calculations (CAD) spend big bucks on Quadro cards rather than the ostensibly equivalent GeForce. Quadro cards have ECC memory and allow higher precision floats to be used for certain calculations.

toast0 5y ago

I think there would be a pretty large hardware cost to ensure the input signals come to both processors at the same clock everytime on the many high speed interfaces a modern CPU is using. And you'd need to eliminate use of non-deterministic CPU local data, like RDRAND and on die temperature, power, etc sensors. Most likely, you'd want to run the CPUs at fixed clock speed to avoid any differences in settling time when switching speeds. This could probably effectively fine broken CPUs (although you wouldn't know which of the pair was broken), but you could still have other broken devices resulting in bad computations. It might be better to run calculations on two separate nodes and compare; possibly only for important calculations.

mirker 5y ago
It’s not necessary to serialize the full execution to detect errors. On an out of order processor, there is already buffering of results happening that is eventually serialized to visible state. To check errors, you could just have one buffer per processor and compare results before serialization, raising an error on mismatch between processors. Serialization is merely indicating that both _visible_ executions have agreed up to that instruction but it still allows for some local/internal disagreements. For example, two instructions can finish in opposite orders across cores and that is fine as long as they are serialized in order. As for settling times, those are random anyway. Processors are binned according to how good the settling times ended up being. It’s unlikely to have two homogeneous chips.

einpoklum 5y ago

> Is silicon cheap enough etc. No, it is not. You can always trade off performance for reliability by repeating your computations several times, preferably with some variation in the distribution/timing of work to avoid the same potential hardware failure pattern.