2 ms·

This stack overflow link is interesting but most of the concern is over very theoretical issues. In practice a significant portion of the humanity uses multiple

by simias 9y ago

This stack overflow link is interesting but most of the concern is over very theoretical issues. In practice a significant portion of the humanity uses multiple non-ECC RAM devices every day and yet most of us don't seem to experience widespread memory issues. I can't even remember the last time my desktop experienced a hard crash (well actually I can, it was because of a faulty... graphic card). I wish my phone fared that well, but I'm not sure RAM would be the first suspect for my general Android stability issues...

muro 9y ago
It's not just about crashes, but also about (silent) data corruption.
sho 9y ago
> most of the concern is over very theoretical issues I've seen photo and other binary files become corrupted that were sitting on RAID drives. The RAID swears they're fine, the filesystem swears they're fine, both are checksummed so I believe them. The only possibility that I can see is that they were corrupted while being modified or transferred on non-ECC desktops connected to the RAID. I'm not afraid of my computer crashing. I'm afraid of data I take great pains to preserve being silently, indeed undetectably, corrupted while in flight or in use. So that's why ECC is worth it to me.
- wila 9y ago
  Exactly! In the past I've had a flimsy RAM module in a macbook Pro and it was a real pain. Everything appeared to work just fine, but on stressing the RAM with a lot of virtual machines the host would crash. That was not the main issue, but took some time to diagnose as I was also running beta virtualisation software and was tempted to blame the change instead of the hardware. Copying virtual machines from one disk to another did end up corrupting the data. That was painful to find out.
  - sho 9y ago
    Almost exactly the same thing happened to me - marginal RAM in an old style macbook pro. Drove me absolutely crazy. People always talk about cosmic rays as what ECC is guarding against - not at all! It's shitty RAM, especially in laptops. When it's under stress.. for example buffering files in a large copy.. and you find out months later when it's too late to fix it. Not "theoretical" at all.
- semi-extrinsic 9y ago
  I'm curious: if storing lots of photos as .dng, .png or .jpg on ZFS without ECC, one presumably gets bit flips eventually. How does this affect the files? Do you just get artifacts in the photo? Or does the file become unreadable? If so, can you recover the file (with artifacts)? I guess the answer boils down to how much non-recoverable but essential-for-reconstruction metadata there is in these file formats.
  - sosuke 9y ago
    I had bit flips on a few JPGs and it renders them useless. Luckily I had a backup of a backup that had them uncorrupted. I'm still trying to find a complete solution to this problem. Presumably the TIFF or BMP file formats are more stable against bit flips. I'd been reading so much about it over the past year or so I got to wondering just how many times cosmic rays affect our brains and what kind of protections we're running up in our skulls.
    - ianai 9y ago
      Our brains evolved through a chaotic, organic process. We're all the time storing new data and even losing data (selective memory). I'm thinking there's no mitigation process. If anything the random environmental noise might play some role in consciousness.
    - semi-extrinsic 9y ago
      Have you tried (ab)using AFL-fuzz for this? It can create JPGs out of literally thin air, so I wouldn't be surprised. https://lcamtuf.blogspot.com/2014/11/pulling-jpegs-out-of-thin-air.html https://lcamtuf.blogspot.com/2014/11/pulling-jpegs-out-of-th...
  - acchow 9y ago
    Depends which part of the file gets corrupted, and the issues between PNG and JPG are dramatically different. There are key bytes in the file (like size of segment, start of segment, etc) that if corrupted would dramatically mess up your image. If it's just in the compressed image data you'll just see some artifacts, and a JPEG already has plenty of compression artifacts anyhow...
nabla9 9y ago
> experience widespread memory issues. Because they rarely spread wide. If you edit images or videos, maybe you detect small corruption in the image. If you use databases or do data analysis, there may be one number that is wrong, or some string has one byte of garbage. Sometimes, application may crash. All this is very rare. It only matters if you need data integrity and do work where data has value.
mshook 9y ago
People don't realize how frequent it could be because most of the time you don't see any of the consequences. Here's a "funny" consequence of bit flips: bit squatting. It's about exploiting a bit flip before a DNS query: you register the proper DNS domain and you wait for machines to wrongly contact you because they got the name wrong. http://dinaburg.org/bitsquatting.html http://dinaburg.org/bitsquatting.html http://dinaburg.org/data/DC19_Dinaburg_Presentation.pdf http://dinaburg.org/data/DC19_Dinaburg_Presentation.pdf http://www.youtube.com/watch?v=lZ8s1JwtNas http://www.youtube.com/watch?v=lZ8s1JwtNas Just saw this was already posted below...
deleted 9y ago
[deleted]

by simias 9y ago

muro 9y ago

It's not just about crashes, but also about (silent) data corruption.

sho 9y ago

> most of the concern is over very theoretical issues I've seen photo and other binary files become corrupted that were sitting on RAID drives. The RAID swears they're fine, the filesystem swears they're fine, both are checksummed so I believe them. The only possibility that I can see is that they were corrupted while being modified or transferred on non-ECC desktops connected to the RAID. I'm not afraid of my computer crashing. I'm afraid of data I take great pains to preserve being silently, indeed undetectably, corrupted while in flight or in use. So that's why ECC is worth it to me.

wila 9y ago
Exactly! In the past I've had a flimsy RAM module in a macbook Pro and it was a real pain. Everything appeared to work just fine, but on stressing the RAM with a lot of virtual machines the host would crash. That was not the main issue, but took some time to diagnose as I was also running beta virtualisation software and was tempted to blame the change instead of the hardware. Copying virtual machines from one disk to another did end up corrupting the data. That was painful to find out.
- sho 9y ago
  Almost exactly the same thing happened to me - marginal RAM in an old style macbook pro. Drove me absolutely crazy. People always talk about cosmic rays as what ECC is guarding against - not at all! It's shitty RAM, especially in laptops. When it's under stress.. for example buffering files in a large copy.. and you find out months later when it's too late to fix it. Not "theoretical" at all.
semi-extrinsic 9y ago
I'm curious: if storing lots of photos as .dng, .png or .jpg on ZFS without ECC, one presumably gets bit flips eventually. How does this affect the files? Do you just get artifacts in the photo? Or does the file become unreadable? If so, can you recover the file (with artifacts)? I guess the answer boils down to how much non-recoverable but essential-for-reconstruction metadata there is in these file formats.
- sosuke 9y ago
  I had bit flips on a few JPGs and it renders them useless. Luckily I had a backup of a backup that had them uncorrupted. I'm still trying to find a complete solution to this problem. Presumably the TIFF or BMP file formats are more stable against bit flips. I'd been reading so much about it over the past year or so I got to wondering just how many times cosmic rays affect our brains and what kind of protections we're running up in our skulls.
  - ianai 9y ago
    Our brains evolved through a chaotic, organic process. We're all the time storing new data and even losing data (selective memory). I'm thinking there's no mitigation process. If anything the random environmental noise might play some role in consciousness.
  - semi-extrinsic 9y ago
    Have you tried (ab)using AFL-fuzz for this? It can create JPGs out of literally thin air, so I wouldn't be surprised. https://lcamtuf.blogspot.com/2014/11/pulling-jpegs-out-of-thin-air.html https://lcamtuf.blogspot.com/2014/11/pulling-jpegs-out-of-th...
- acchow 9y ago
  Depends which part of the file gets corrupted, and the issues between PNG and JPG are dramatically different. There are key bytes in the file (like size of segment, start of segment, etc) that if corrupted would dramatically mess up your image. If it's just in the compressed image data you'll just see some artifacts, and a JPEG already has plenty of compression artifacts anyhow...

nabla9 9y ago

> experience widespread memory issues. Because they rarely spread wide. If you edit images or videos, maybe you detect small corruption in the image. If you use databases or do data analysis, there may be one number that is wrong, or some string has one byte of garbage. Sometimes, application may crash. All this is very rare. It only matters if you need data integrity and do work where data has value.

mshook 9y ago

People don't realize how frequent it could be because most of the time you don't see any of the consequences. Here's a "funny" consequence of bit flips: bit squatting. It's about exploiting a bit flip before a DNS query: you register the proper DNS domain and you wait for machines to wrongly contact you because they got the name wrong. http://dinaburg.org/bitsquatting.html http://dinaburg.org/bitsquatting.html http://dinaburg.org/data/DC19_Dinaburg_Presentation.pdf http://dinaburg.org/data/DC19_Dinaburg_Presentation.pdf http://www.youtube.com/watch?v=lZ8s1JwtNas http://www.youtube.com/watch?v=lZ8s1JwtNas Just saw this was already posted below...

deleted 9y ago

[deleted]