6 ms·

I think irradiating the system would be a better test since it would induce random bitflips anywhere in the system while it is keeps running and reading/writing

I think irradiating the system would be a better test since it would induce random bitflips anywhere in the system while it is keeps running and reading/writing data instead of inducing a massive fault that will almost immediately stop any IO operations. Overclocking some components might work too.

blackflame7000 7y ago
It wouldn’t matter. An error would have to occur in such a way as to also corrupt the data and the checksum in such a way that the corrupted checksum is valid for the errant block.
- blackflame7000 7y ago
  Not sure why the downvotes, this is in fact how it works and why ZFS doesn't need ECC Ram. It doesn't matter how much corruption you experience it always comes down to does the checksum of the block of data read from disk match the checksum stored when that block was written. If either mismatch, then the block is rebuilt from parity drives. If you're going to downvote, at least say why.
  - beatgammit 7y ago
    You still want ECC to make sure the data doesn't get corrupted on the way in or on the way out. ZFS will store what you send it, and it isn't responsible for your data once in hands it to you. The best part about ECC RAM, IMO, isn't the correction, but the checking. I want to know when my RAM goes bad, and I'd prefer to know before ZFS detects a problem.
    - blackflame7000 7y ago
      I'm so sick of this argument every time a ZFS article comes up. This is by the inventor, Matthew Ahrens: “There’s nothing special about ZFS that requires/encourages the use of ECC RAM more so than any other filesystem.” https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/ https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-y... Now give me my karma back.
      - dfeojm-zlib 7y ago
        The thing about ECC vs. non-ECC. If you don't have ECC, you don't know silent bit corruption is happening, and when it's finally (if it's ever) discovered, there's nothing that can be done. ECC (SECDED) keeps data safer than it would be otherwise. ECC is necessary for anything real or important based on the financial cost of losing data. For everything else, it can be optional. Plus, don't forget the security issues of bitsquatting and other attacks that are the real results of silent bitflips. DEF CON 19 - Artem Dinaburg - Bit-squatting: DNS Hijacking Without Exploitation https://youtu.be/9WcHsT97suU https://youtu.be/9WcHsT97suU
        blackflame7000 7y ago
        All computers do error checking. Here is a summary: Non-ECC - Checksum: parity bit BlockSize: 1 byte ECC - Checksum: 7 bits BlockSize: 8 bytes ZFS - Checksum: 256 bits BlockSize: ~128kb
        devonkim 7y ago
        It’s a lot further than this though with unbuffered RAM, registered RAM in the mix affecting performance and such. Error detection and error correction are different and require different levels of parity codes. Standard RAM at one point (SIMMs in the 90s) detected errors but won’t ever correct them. ECC can detect more but error patterns and now correct error patterns up to so many bits. The issues are more subtle beyond hardware though - your OS kernel has to understand the detection from the BIOS / EFI and if the motherboard manufacturer decided not to opt for wiring in the checksum fail signal you will run blind to data corruption that’s potentially really subtle on typical consumer hardware. With some BIOSes RAM failures result in a hard panic and will hard reboot the machine (the idea being a crash is better than continuing with an error). For me, ECC (both UDIMM and RDIMM) today are so inexpensive it’s a no-brainer for builds for business.
      - tasty_freeze 7y ago
        The comment from the inventor isn't supporting your position. ZFS doens't need it any more, but if you care about data integrity, then you want ECC memory in your system. Yes, ZFS can take care of errors while it inside the ZFS API boundary, but the data on the outside of that is at risk, unless you have some higher level error correction protocol going on.
myrandomcomment 7y ago
Worked at a major switch vendor as one of the 50 original staff. We put our gear through a ton of test - 4 corners - low power high heat, high power low heat, etc. One of the most interesting was taking the device to a government lab here in NorCal in order to shoot radioactive particle into the board while passing packets. It was pretty cool. The goal was to test TCAM and switch OS RAM to see how it handle bit flips. It passed. The board itself, well we could not have it back for a few weeks to allow it to "cool". The other thing of note was winning a US gov. contact was to enable commands to override any heat related auto shutdown. We were told "look, if the device is in a situation where this is an issue, burn it up but keep it working as long as you can". This was for navy ships.
- WrtCdEvrydy 7y ago
  > burn it up but keep it working as long as you can I mean, if the ship is on fire, you want the missile control systems to shut down after all missiles have been fired.
- shabble 7y ago
  Sometimes referred to as "Battleshort"[1] mode [1] https://en.wikipedia.org/wiki/Battleshort https://en.wikipedia.org/wiki/Battleshort

blackflame7000 7y ago

It wouldn’t matter. An error would have to occur in such a way as to also corrupt the data and the checksum in such a way that the corrupted checksum is valid for the errant block.

blackflame7000 7y ago
Not sure why the downvotes, this is in fact how it works and why ZFS doesn't need ECC Ram. It doesn't matter how much corruption you experience it always comes down to does the checksum of the block of data read from disk match the checksum stored when that block was written. If either mismatch, then the block is rebuilt from parity drives. If you're going to downvote, at least say why.
- beatgammit 7y ago
  You still want ECC to make sure the data doesn't get corrupted on the way in or on the way out. ZFS will store what you send it, and it isn't responsible for your data once in hands it to you. The best part about ECC RAM, IMO, isn't the correction, but the checking. I want to know when my RAM goes bad, and I'd prefer to know before ZFS detects a problem.
  - blackflame7000 7y ago
    I'm so sick of this argument every time a ZFS article comes up. This is by the inventor, Matthew Ahrens: “There’s nothing special about ZFS that requires/encourages the use of ECC RAM more so than any other filesystem.” https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/ https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-y... Now give me my karma back.
    - dfeojm-zlib 7y ago
      The thing about ECC vs. non-ECC. If you don't have ECC, you don't know silent bit corruption is happening, and when it's finally (if it's ever) discovered, there's nothing that can be done. ECC (SECDED) keeps data safer than it would be otherwise. ECC is necessary for anything real or important based on the financial cost of losing data. For everything else, it can be optional. Plus, don't forget the security issues of bitsquatting and other attacks that are the real results of silent bitflips. DEF CON 19 - Artem Dinaburg - Bit-squatting: DNS Hijacking Without Exploitation https://youtu.be/9WcHsT97suU https://youtu.be/9WcHsT97suU
      - blackflame7000 7y ago
        All computers do error checking. Here is a summary: Non-ECC - Checksum: parity bit BlockSize: 1 byte ECC - Checksum: 7 bits BlockSize: 8 bytes ZFS - Checksum: 256 bits BlockSize: ~128kb
        devonkim 7y ago
        It’s a lot further than this though with unbuffered RAM, registered RAM in the mix affecting performance and such. Error detection and error correction are different and require different levels of parity codes. Standard RAM at one point (SIMMs in the 90s) detected errors but won’t ever correct them. ECC can detect more but error patterns and now correct error patterns up to so many bits. The issues are more subtle beyond hardware though - your OS kernel has to understand the detection from the BIOS / EFI and if the motherboard manufacturer decided not to opt for wiring in the checksum fail signal you will run blind to data corruption that’s potentially really subtle on typical consumer hardware. With some BIOSes RAM failures result in a hard panic and will hard reboot the machine (the idea being a crash is better than continuing with an error). For me, ECC (both UDIMM and RDIMM) today are so inexpensive it’s a no-brainer for builds for business.
    - tasty_freeze 7y ago
      The comment from the inventor isn't supporting your position. ZFS doens't need it any more, but if you care about data integrity, then you want ECC memory in your system. Yes, ZFS can take care of errors while it inside the ZFS API boundary, but the data on the outside of that is at risk, unless you have some higher level error correction protocol going on.

myrandomcomment 7y ago

Worked at a major switch vendor as one of the 50 original staff. We put our gear through a ton of test - 4 corners - low power high heat, high power low heat, etc. One of the most interesting was taking the device to a government lab here in NorCal in order to shoot radioactive particle into the board while passing packets. It was pretty cool. The goal was to test TCAM and switch OS RAM to see how it handle bit flips. It passed. The board itself, well we could not have it back for a few weeks to allow it to "cool". The other thing of note was winning a US gov. contact was to enable commands to override any heat related auto shutdown. We were told "look, if the device is in a situation where this is an issue, burn it up but keep it working as long as you can". This was for navy ships.

WrtCdEvrydy 7y ago
> burn it up but keep it working as long as you can I mean, if the ship is on fire, you want the missile control systems to shut down after all missiles have been fired.
shabble 7y ago
Sometimes referred to as "Battleshort"[1] mode [1] https://en.wikipedia.org/wiki/Battleshort https://en.wikipedia.org/wiki/Battleshort