3 ms·

If this is process node dependent and has started turning up relatively frequently in places like Facebook and Google, I wonder if it's more common in current-g

by makomk 5y ago

If this is process node dependent and has started turning up relatively frequently in places like Facebook and Google, I wonder if it's more common in current-gen AMD CPUs than others... the timing would seem to fit, and they're on the smallest process node of anything currently in datacenter use.

H8crilA 5y ago
I've seen this for years, it is not new. You have to be running at the scale of tens/hundreds of petabytes of data to experience really quirky anomalies. The failure rate and failure modes vary from model to model and even from batch to batch. The biggest issue is not that failures exist, but that they're hard to trigger in testing and when they trigger they do not cause a crash. The CPUs that any actually used computation is running on have already been through a lot of testing (manufacturer, cloud provider), the failures have to slip through all of them. And the failures can be really nasty: imagine some distributed DB server damaging large % of database metadata records because 1 instruction malfunctioned, but didn't crash the process. By the time you know something's wrong you cannot recover 70% of the data because you have no idea which data blocks correspond to which logical rows (remember, the metadata is garbled), except through a manual process that can take weeks or months. Or something like a v-table "miss" where instead of something like Table.Info() the CPU calls Table.Drop() because that function happens to be exactly 64 bytes lower in the v-table and has similar enough signature for the call to succeed. Those are two real examples.
- FartyMcFarter 5y ago
  How do companies deal with this in practice? Running 3 or more redundant servers which vote on the correct result of an operation?
  - H8crilA 5y ago
    You don't deal with this, other than fixing the damage and removing the bad machines. The only way forward is expanded testing, that's what the paper from Google is (also) about. I think this issue will always be with us, to a larger or smaller extent. There's probably a ton of data corruption out there that happens to be in places that doesn't really cause big problems.
    - secondcoming 5y ago
      Could cloud providers not run some tests on each machine core at startup and have the hypervisor isolate any bad cores from the guest?
      - H8crilA 5y ago
        They already do that, not exactly on each reboot but trust me they really test the hardware. The problem is that nobody knows how to write tests that catch all errors.
  - stubish 5y ago
    Yes, but generally only when human life is on the line, such as aerospace. For just about everything else, the cost of redundancy in dollars and speed is too high. After all, problems are extremely rare, and when the extremely rare problems occur, it is extremely unlikely it matters. Versus a guaranteed cost of more than tripling your hardware, power, data center etc. costs. Maybe you do it in part of your stack, such as redundant mainframes or database nodes. But the most you will see on your appservers is ECC memory in an attempt to avoid cosmic ray bit flips (a similar problem we have been aware of for decades).
- wejick 5y ago
  The scary part is detecting and reproducing the error reliably is difficult. Haven't read the paper, but I expect tooling on this field will be growing following more awareness.

by makomk 5y ago

H8crilA 5y ago

I've seen this for years, it is not new. You have to be running at the scale of tens/hundreds of petabytes of data to experience really quirky anomalies. The failure rate and failure modes vary from model to model and even from batch to batch. The biggest issue is not that failures exist, but that they're hard to trigger in testing and when they trigger they do not cause a crash. The CPUs that any actually used computation is running on have already been through a lot of testing (manufacturer, cloud provider), the failures have to slip through all of them. And the failures can be really nasty: imagine some distributed DB server damaging large % of database metadata records because 1 instruction malfunctioned, but didn't crash the process. By the time you know something's wrong you cannot recover 70% of the data because you have no idea which data blocks correspond to which logical rows (remember, the metadata is garbled), except through a manual process that can take weeks or months. Or something like a v-table "miss" where instead of something like Table.Info() the CPU calls Table.Drop() because that function happens to be exactly 64 bytes lower in the v-table and has similar enough signature for the call to succeed. Those are two real examples.

FartyMcFarter 5y ago
How do companies deal with this in practice? Running 3 or more redundant servers which vote on the correct result of an operation?
- H8crilA 5y ago
  You don't deal with this, other than fixing the damage and removing the bad machines. The only way forward is expanded testing, that's what the paper from Google is (also) about. I think this issue will always be with us, to a larger or smaller extent. There's probably a ton of data corruption out there that happens to be in places that doesn't really cause big problems.
  - secondcoming 5y ago
    Could cloud providers not run some tests on each machine core at startup and have the hypervisor isolate any bad cores from the guest?
    - H8crilA 5y ago
      They already do that, not exactly on each reboot but trust me they really test the hardware. The problem is that nobody knows how to write tests that catch all errors.
- stubish 5y ago
  Yes, but generally only when human life is on the line, such as aerospace. For just about everything else, the cost of redundancy in dollars and speed is too high. After all, problems are extremely rare, and when the extremely rare problems occur, it is extremely unlikely it matters. Versus a guaranteed cost of more than tripling your hardware, power, data center etc. costs. Maybe you do it in part of your stack, such as redundant mainframes or database nodes. But the most you will see on your appservers is ECC memory in an attempt to avoid cosmic ray bit flips (a similar problem we have been aware of for decades).
wejick 5y ago
The scary part is detecting and reproducing the error reliably is difficult. Haven't read the paper, but I expect tooling on this field will be growing following more awareness.