9 ms·

Using time travel to remotely debug faulty DRAM

161 points by KenoFischer 6y ago

cesaref 6y ago
I was just thinking, it's odd that there are still processors being made that don't support ECC memory, and that this isn't a thing that is expected given the (relatively) large amounts of RAM that are included in modern machines.
- KMnO4 6y ago
  I agree. The only processors that seem to support it are designed for servers (eg Xeon). Other processors say they're compatible but don't actually use the ECC part of the memory. I built a ML workstation for my work team and was disappointed that my options were: expensive, low clock speed server CPU + ECC, or inexpensive, very fast, desktop CPU without ECC. Even if I were willing to pay more money, it was really hard to get the same performance if I needed ECC.
  - Leo_Verto 6y ago
    That's just Intel though. All AMD Ryzen CPUs without integrated graphics do support ECC although YMMV depending on the motherboard you use. [1] [1]: https://hardwarecanucks.com/cpu-motherboard/ecc-memory-amds-ryzen-deep-dive/5/ https://hardwarecanucks.com/cpu-motherboard/ecc-memory-amds-...
  - CodesInChaos 6y ago
    Supposedly AMD Ryzen and Threadripper support unbuffered ECC with the right mainboard (Asrock was mentioned several times).
    - account42 6y ago
      I can confirm that Threadripper does check for errors with ECC ram. (MSI motherboard in my case but it should work with all of them)
  - segfaultbuserr 6y ago
    It's intentional. Intel has a tradition to cripple features on consumer and enthusiast-grade CPUs in order to sell more Xeon CPUs. The way Intel crippled virtualization/IOMMU in the past was also particularly annoying - on mainstream CPUs it was allowed, but on overclockable enthusiast CPUs, it was disabled. On the other hand, AMD doesn't do that.
    - omgwtfbyobbq 6y ago
      This bugs me so much. DW's i7-6700HQ laptop only supports 16gb, but my old 1007U supports 32gb. :/
- toast0 6y ago
  ECC adds significant cost, and the benefits are stastically meager. I ran a fleet of a few thousand servers with all having 64GB and some having up to 768GB ECC RAM. I'd estimate we had no more than one swap a month. Most systems never had any (reported) ECC corrections. I still think it's important, but it makes sense that people don't want to pay more for something to detect errors that are quite unlikely.
  - dmm 6y ago
    What do you mean by ECC is more expensive? Are you talking about ECC ram being more expensive? That's probably true but why not give people the option whether to use ECC? ECC being optional is a good idea anyway because ECC memory is generally slower. I don't think the actual ECC implementation in the processor is very expensive to implement. Consumer CPUs not having ECC is pure product segmentation, without technical justification, IMO.
    - toast0 6y ago
      ECC ram is more expensive, yes. But supporting ECC ram also adds more expense. To properly support it, BIOS engineers have to support and test it; motherboard makers have to support and test it. I appreciate that AMD offers it for their consumer oriented processors, but because it's a best effort feature, and it's hard for an end user (or reviewer) to test, you never really know if you're going to get full support, or if the support is really just that you can use ECC ram, if you want to spend a little more for your ram, and get none of the benefits of ECC. It certainly adds something to the cost (die space) of the memory controller, but I agree it's probably not much.
      - cesaref 6y ago
        Well those costs are mainly related to adding support for another memory standard, but i'm suggesting ECC become the standard with non-ECC being dropped. The BIOS/testing cost would be unaffected, they would just be testing only ECC memory. Realistically the mainstream CPU manufacturers have ECC solutions, so including them in the mainstream processors shouldn't be a huge issue, it's just a market segmentation ploy to exclude the feature from the consumer processor designs.
jstrong 6y ago
hugely impressed at the skills displayed.
guenthert 6y ago
My Dell M6800 RAM has 2 bad bits out of 32GiB (easily found with the wonderful 'stressapptest' tool). I'm not exchanging the DIMMs because of that (besides, it's of course in the DIMMs which are more difficult to access). When running Linux I mark a MiB around those memory locations to not be used (memmap option). I haven't bothered to do the same when running MS Windows (it seems a bit more complex). Every once in a while that memory is accessed in a way which causes some corruption leading to a fault. I feel a bit guilty about all those senseless bug reports automatically generated and sent ...
- popotamonga 6y ago
  Is that tool the same as windows memtest? My brand new pc randomly blue screens twice a month with 'corrupted structure' or something like that, i suspect of ram but windows tool said no errors found after a 3 hour test. Cant even put it on warranty because it cant be reproduced easily. Cant afford to just start replacing components. 5000€ lemon.
  - Nextgrid 6y ago
    Memtest86+ (http://www.memtest.org http://www.memtest.org) is what hardware enthusiasts use to troubleshoot memory issues despite the built-in Windows memory test being preinstalled and easier to access (no need to boot from a USB, etc) so I'm assuming it's much better than the Windows one.
    - MawKKe 6y ago
      FYI Last I tried memtest86+ (in June?), it had been outdated for quite some time and downright crashed on newer hardware. The non-free alternative called Memtest86 (by Passmark) works just as well if needed.
    - fomine3 6y ago
      Nowadays I use memtest86 that supports UEFI. On Windows, "RAM Test" paidware works well.
  - 6y ago
ViralBShah 6y ago
In general, in the Julia community, we are all used to see Keno do impressive stuff routinely, but this one tops everything so far. What is particularly impressive is that this lowest of the low-level debugging work is done completely in Julia - all the way.
yjftsjthsd-h 6y ago
Well that's remarkable; this is one of those demos that makes me think maybe I should go learn this tech stack just so I could do this. (Another is org mode in emacs.) Looks like rr is flexible enough that I can use it without having to pick up Julia, which is an easier on-ramp even if this post makes me think that maybe I should learn Julia...
Scaevolus 6y ago
When you collect lots of coredumps you'll get a long tail of bizarre, one-off crashes. Some of these are subtle memory corruption bugs that cause random crashes, but others are caused by bitflips-- either faulty DRAM or cosmic rays. Usually you throw your hands up and only look at crashes that happen multiple times because they're more likely to be reproducible. The unique factor here is that rr provides _reproducible_ crash recordings, so when it fails to reproduce a crash, you've found some nondeterminism-- either a bug in rr where it didn't replay syscalls correctly or nail down thread behavior accurately, or a hardware issue like this.
hinkley 6y ago
There are some things in software development that are obviously wrong to most people. There are some things in software development that are obviously wrong to a few people. And there are some things people have a hunch we are doing wrong but nobody can crystallize it. Removing redundancy in code is great most of the time, but it's not a panacea. NASA had to contend with physical failures of memory, and catastrophic costs of failures in 'production'. They solved this problem by consensus pools of three, on physically separate hardware and in some cases using multiple manufacturers. Inability to reach consensus would invoke failsafes. I have a vague suspicion about how we condense the very most critical bits of our software down to the fewest bits of data and instructions. This may ultimately be a policy we reject. One bad bit and you can end up taking the opposite action of the one you should have performed.
- vosper 6y ago
  > One bad bit and you can end up taking the opposite action of the one you should have performed. And bit flips do happen https://dropbox.tech/infrastructure/-broccoli--syncing-faster-by-syncing-less#-compressed-uploads https://dropbox.tech/infrastructure/-broccoli--syncing-faste... https://lobste.rs/s/310xjb/broccoli_syncing_faster_by_syncing_less#c_2jddne https://lobste.rs/s/310xjb/broccoli_syncing_faster_by_syncin...
  - contravariant 6y ago
    Even more so in space where there's no shielding from cosmic rays whatsoever.
- schoen 6y ago
  One thing I've often wondered about NASA's triple redundancy: what system calculates or determines the consensus? Is it also a programmable computer, just smaller?
  - akira2501 6y ago
    Usually the driven element itself. From [0]: "One reason why the redundancy management software was able to be kept to a minimum is that NASA decided to move voting to the actuators, rather than to do it before commands are sent on buses. Each actuator is quadruple redundant. If a single computer fails, it continues to send commands to an actuator until the crew takes it out of the redundant set. Since the Shuttle's other three computers are sending apparently correct commands to their actuators, the failed computer's commands are physically out-voted79. Theoretically, the only serious possibility is that three computers would fail simultaneously, thus negating the effects of the voting. If that occurs, and if the proper warnings are given, the crew can then engage the backup system simply by pressing a button located on each of the forward rotational hand controllers." [0]: https://history.nasa.gov/computers/Ch4-4.html#:~:text=Its%20five%20general%2Dpurpose%20computers,such%20as%20ascent%20and%20descent https://history.nasa.gov/computers/Ch4-4.html#:~:text=Its%20....
andresramon60 6y ago
What’s a traveler to do during a global pandemic? There are many ways to answer that question. “Despair,” though, is not one of them. Expeditions are led by National Geographic experts or expedition teams, and feature world-class accommodations and experiences, including special access to sites and field researchers. Most activities and meals are included in the expedition cost, which ranges from $1,500 to $4,500 per person. Email natgeotraveln@gmail.com When you book a National Geographic Journey, you’ll receive all pre-trip information electronically, including destination-specific stories and information prepared by National Geographic.

161 points by KenoFischer 6y ago

I was just thinking, it's odd that there are still processors being made that don't support ECC memory, and that this isn't a thing that is expected given the (relatively) large amounts of RAM that are included in modern machines.

KMnO4 6y ago
I agree. The only processors that seem to support it are designed for servers (eg Xeon). Other processors say they're compatible but don't actually use the ECC part of the memory. I built a ML workstation for my work team and was disappointed that my options were: expensive, low clock speed server CPU + ECC, or inexpensive, very fast, desktop CPU without ECC. Even if I were willing to pay more money, it was really hard to get the same performance if I needed ECC.
- Leo_Verto 6y ago
  That's just Intel though. All AMD Ryzen CPUs without integrated graphics do support ECC although YMMV depending on the motherboard you use. [1] [1]: https://hardwarecanucks.com/cpu-motherboard/ecc-memory-amds-ryzen-deep-dive/5/ https://hardwarecanucks.com/cpu-motherboard/ecc-memory-amds-...
- CodesInChaos 6y ago
  Supposedly AMD Ryzen and Threadripper support unbuffered ECC with the right mainboard (Asrock was mentioned several times).
  - account42 6y ago
    I can confirm that Threadripper does check for errors with ECC ram. (MSI motherboard in my case but it should work with all of them)
- segfaultbuserr 6y ago
  It's intentional. Intel has a tradition to cripple features on consumer and enthusiast-grade CPUs in order to sell more Xeon CPUs. The way Intel crippled virtualization/IOMMU in the past was also particularly annoying - on mainstream CPUs it was allowed, but on overclockable enthusiast CPUs, it was disabled. On the other hand, AMD doesn't do that.
  - omgwtfbyobbq 6y ago
    This bugs me so much. DW's i7-6700HQ laptop only supports 16gb, but my old 1007U supports 32gb. :/
toast0 6y ago
ECC adds significant cost, and the benefits are stastically meager. I ran a fleet of a few thousand servers with all having 64GB and some having up to 768GB ECC RAM. I'd estimate we had no more than one swap a month. Most systems never had any (reported) ECC corrections. I still think it's important, but it makes sense that people don't want to pay more for something to detect errors that are quite unlikely.
- dmm 6y ago
  What do you mean by ECC is more expensive? Are you talking about ECC ram being more expensive? That's probably true but why not give people the option whether to use ECC? ECC being optional is a good idea anyway because ECC memory is generally slower. I don't think the actual ECC implementation in the processor is very expensive to implement. Consumer CPUs not having ECC is pure product segmentation, without technical justification, IMO.
  - toast0 6y ago
    ECC ram is more expensive, yes. But supporting ECC ram also adds more expense. To properly support it, BIOS engineers have to support and test it; motherboard makers have to support and test it. I appreciate that AMD offers it for their consumer oriented processors, but because it's a best effort feature, and it's hard for an end user (or reviewer) to test, you never really know if you're going to get full support, or if the support is really just that you can use ECC ram, if you want to spend a little more for your ram, and get none of the benefits of ECC. It certainly adds something to the cost (die space) of the memory controller, but I agree it's probably not much.
    - cesaref 6y ago
      Well those costs are mainly related to adding support for another memory standard, but i'm suggesting ECC become the standard with non-ECC being dropped. The BIOS/testing cost would be unaffected, they would just be testing only ECC memory. Realistically the mainstream CPU manufacturers have ECC solutions, so including them in the mainstream processors shouldn't be a huge issue, it's just a market segmentation ploy to exclude the feature from the consumer processor designs.

jstrong 6y ago

hugely impressed at the skills displayed.

guenthert 6y ago

My Dell M6800 RAM has 2 bad bits out of 32GiB (easily found with the wonderful 'stressapptest' tool). I'm not exchanging the DIMMs because of that (besides, it's of course in the DIMMs which are more difficult to access). When running Linux I mark a MiB around those memory locations to not be used (memmap option). I haven't bothered to do the same when running MS Windows (it seems a bit more complex). Every once in a while that memory is accessed in a way which causes some corruption leading to a fault. I feel a bit guilty about all those senseless bug reports automatically generated and sent ...

popotamonga 6y ago
Is that tool the same as windows memtest? My brand new pc randomly blue screens twice a month with 'corrupted structure' or something like that, i suspect of ram but windows tool said no errors found after a 3 hour test. Cant even put it on warranty because it cant be reproduced easily. Cant afford to just start replacing components. 5000€ lemon.
- Nextgrid 6y ago
  Memtest86+ (http://www.memtest.org http://www.memtest.org) is what hardware enthusiasts use to troubleshoot memory issues despite the built-in Windows memory test being preinstalled and easier to access (no need to boot from a USB, etc) so I'm assuming it's much better than the Windows one.
  - MawKKe 6y ago
    FYI Last I tried memtest86+ (in June?), it had been outdated for quite some time and downright crashed on newer hardware. The non-free alternative called Memtest86 (by Passmark) works just as well if needed.
  - fomine3 6y ago
    Nowadays I use memtest86 that supports UEFI. On Windows, "RAM Test" paidware works well.
- 6y ago

ViralBShah 6y ago

In general, in the Julia community, we are all used to see Keno do impressive stuff routinely, but this one tops everything so far. What is particularly impressive is that this lowest of the low-level debugging work is done completely in Julia - all the way.

yjftsjthsd-h 6y ago

Well that's remarkable; this is one of those demos that makes me think maybe I should go learn this tech stack just so I could do this. (Another is org mode in emacs.) Looks like rr is flexible enough that I can use it without having to pick up Julia, which is an easier on-ramp even if this post makes me think that maybe I should learn Julia...

Scaevolus 6y ago

When you collect lots of coredumps you'll get a long tail of bizarre, one-off crashes. Some of these are subtle memory corruption bugs that cause random crashes, but others are caused by bitflips-- either faulty DRAM or cosmic rays. Usually you throw your hands up and only look at crashes that happen multiple times because they're more likely to be reproducible. The unique factor here is that rr provides _reproducible_ crash recordings, so when it fails to reproduce a crash, you've found some nondeterminism-- either a bug in rr where it didn't replay syscalls correctly or nail down thread behavior accurately, or a hardware issue like this.

hinkley 6y ago

There are some things in software development that are obviously wrong to most people. There are some things in software development that are obviously wrong to a few people. And there are some things people have a hunch we are doing wrong but nobody can crystallize it. Removing redundancy in code is great most of the time, but it's not a panacea. NASA had to contend with physical failures of memory, and catastrophic costs of failures in 'production'. They solved this problem by consensus pools of three, on physically separate hardware and in some cases using multiple manufacturers. Inability to reach consensus would invoke failsafes. I have a vague suspicion about how we condense the very most critical bits of our software down to the fewest bits of data and instructions. This may ultimately be a policy we reject. One bad bit and you can end up taking the opposite action of the one you should have performed.

vosper 6y ago
> One bad bit and you can end up taking the opposite action of the one you should have performed. And bit flips do happen https://dropbox.tech/infrastructure/-broccoli--syncing-faster-by-syncing-less#-compressed-uploads https://dropbox.tech/infrastructure/-broccoli--syncing-faste... https://lobste.rs/s/310xjb/broccoli_syncing_faster_by_syncing_less#c_2jddne https://lobste.rs/s/310xjb/broccoli_syncing_faster_by_syncin...
- contravariant 6y ago
  Even more so in space where there's no shielding from cosmic rays whatsoever.
schoen 6y ago
One thing I've often wondered about NASA's triple redundancy: what system calculates or determines the consensus? Is it also a programmable computer, just smaller?
- akira2501 6y ago
  Usually the driven element itself. From [0]: "One reason why the redundancy management software was able to be kept to a minimum is that NASA decided to move voting to the actuators, rather than to do it before commands are sent on buses. Each actuator is quadruple redundant. If a single computer fails, it continues to send commands to an actuator until the crew takes it out of the redundant set. Since the Shuttle's other three computers are sending apparently correct commands to their actuators, the failed computer's commands are physically out-voted79. Theoretically, the only serious possibility is that three computers would fail simultaneously, thus negating the effects of the voting. If that occurs, and if the proper warnings are given, the crew can then engage the backup system simply by pressing a button located on each of the forward rotational hand controllers." [0]: https://history.nasa.gov/computers/Ch4-4.html#:~:text=Its%20five%20general%2Dpurpose%20computers,such%20as%20ascent%20and%20descent https://history.nasa.gov/computers/Ch4-4.html#:~:text=Its%20....

andresramon60 6y ago

What’s a traveler to do during a global pandemic? There are many ways to answer that question. “Despair,” though, is not one of them. Expeditions are led by National Geographic experts or expedition teams, and feature world-class accommodations and experiences, including special access to sites and field researchers. Most activities and meals are included in the expedition cost, which ranges from $1,500 to $4,500 per person. Email natgeotraveln@gmail.com When you book a National Geographic Journey, you’ll receive all pre-trip information electronically, including destination-specific stories and information prepared by National Geographic.