6 ms·

Last year when I wanted to build a 10 disk ZFS server in RAIDZ2 and researched about the data-integrity and fault-tolerance aspects, I found this video of guys

Last year when I wanted to build a 10 disk ZFS server in RAIDZ2 and researched about the data-integrity and fault-tolerance aspects, I found this video of guys literally making hardware failure by conducting electricity into the motherboard attached to a RAIDZ2 array: https://www.youtube.com/watch?v=vxFNBZIAClc https://www.youtube.com/watch?v=vxFNBZIAClc and they could not make any errors. This was pretty brutal. When I saw this video, I decided I never want to use any other filesystem than ZFS ever.

mirceal 7y ago
this is cool, but I have to wonder how much of this is ZFS and how much is the hardware. don’t get me wrong: it’s impressive, but to make the claims about ZFS you need to have a scientific approach that would involve a control group, various types of motherboards, memories, power supplies, etc + thorough reproducibility of the conditions.
- mrighele 7y ago
  Anecdote, but I had once a broken SATA controller that would send borked data to the disks. That was tricky because the disks were correctly writing the data sent to them and from the OS side all the operations were completed successfully. I realized that only because there was a continuous stream of checksum errors on all the disks of the pool.
  - blackflame7000 7y ago
    Had this same problem too. Compared two sata controllers and noticed one was missing an inductor from the board. Amazing it worked at all
- viraptor 7y ago
  Same. Specifically, I the fault is inserted in a way which may never even manifest at zfs level. Any relevant failures could be more reliably replicated by injecting read/write failures and by writing random bytes to the disk. What else would you expect zfs to correct? Whether it worked out not can be read back from the error counters.
the8472 7y ago
I think irradiating the system would be a better test since it would induce random bitflips anywhere in the system while it is keeps running and reading/writing data instead of inducing a massive fault that will almost immediately stop any IO operations. Overclocking some components might work too.
- blackflame7000 7y ago
  It wouldn’t matter. An error would have to occur in such a way as to also corrupt the data and the checksum in such a way that the corrupted checksum is valid for the errant block.
  - blackflame7000 7y ago
    Not sure why the downvotes, this is in fact how it works and why ZFS doesn't need ECC Ram. It doesn't matter how much corruption you experience it always comes down to does the checksum of the block of data read from disk match the checksum stored when that block was written. If either mismatch, then the block is rebuilt from parity drives. If you're going to downvote, at least say why.
    - beatgammit 7y ago
      You still want ECC to make sure the data doesn't get corrupted on the way in or on the way out. ZFS will store what you send it, and it isn't responsible for your data once in hands it to you. The best part about ECC RAM, IMO, isn't the correction, but the checking. I want to know when my RAM goes bad, and I'd prefer to know before ZFS detects a problem.
      - blackflame7000 7y ago
        I'm so sick of this argument every time a ZFS article comes up. This is by the inventor, Matthew Ahrens: “There’s nothing special about ZFS that requires/encourages the use of ECC RAM more so than any other filesystem.” https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/ https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-y... Now give me my karma back.
thinkingkong 7y ago
Ha! That reminds me of the "Yelling at the Fileserver" video. https://www.youtube.com/watch?v=tDacjrSCeq4 https://www.youtube.com/watch?v=tDacjrSCeq4
techslave 7y ago
that is just a sensational science video. there’s no testimony given that normal raid5 wouldn’t have survived exactly as well. he needed to have mentioned what happened on reboot fatter running a scrub. well not exactly true. we know that the raid5 write hole won’t tolerate random power off / memory removal etc. as someone else said, x ray test would have been better.
tjoff 7y ago
Did they even perform any file (write)operations when testing? If not that was a waste of time... FAT32 would have performed just as well.

mirceal 7y ago

this is cool, but I have to wonder how much of this is ZFS and how much is the hardware. don’t get me wrong: it’s impressive, but to make the claims about ZFS you need to have a scientific approach that would involve a control group, various types of motherboards, memories, power supplies, etc + thorough reproducibility of the conditions.

mrighele 7y ago
Anecdote, but I had once a broken SATA controller that would send borked data to the disks. That was tricky because the disks were correctly writing the data sent to them and from the OS side all the operations were completed successfully. I realized that only because there was a continuous stream of checksum errors on all the disks of the pool.
- blackflame7000 7y ago
  Had this same problem too. Compared two sata controllers and noticed one was missing an inductor from the board. Amazing it worked at all
viraptor 7y ago
Same. Specifically, I the fault is inserted in a way which may never even manifest at zfs level. Any relevant failures could be more reliably replicated by injecting read/write failures and by writing random bytes to the disk. What else would you expect zfs to correct? Whether it worked out not can be read back from the error counters.

the8472 7y ago

I think irradiating the system would be a better test since it would induce random bitflips anywhere in the system while it is keeps running and reading/writing data instead of inducing a massive fault that will almost immediately stop any IO operations. Overclocking some components might work too.

blackflame7000 7y ago
It wouldn’t matter. An error would have to occur in such a way as to also corrupt the data and the checksum in such a way that the corrupted checksum is valid for the errant block.
- blackflame7000 7y ago
  Not sure why the downvotes, this is in fact how it works and why ZFS doesn't need ECC Ram. It doesn't matter how much corruption you experience it always comes down to does the checksum of the block of data read from disk match the checksum stored when that block was written. If either mismatch, then the block is rebuilt from parity drives. If you're going to downvote, at least say why.
  - beatgammit 7y ago
    You still want ECC to make sure the data doesn't get corrupted on the way in or on the way out. ZFS will store what you send it, and it isn't responsible for your data once in hands it to you. The best part about ECC RAM, IMO, isn't the correction, but the checking. I want to know when my RAM goes bad, and I'd prefer to know before ZFS detects a problem.
    - blackflame7000 7y ago
      I'm so sick of this argument every time a ZFS article comes up. This is by the inventor, Matthew Ahrens: “There’s nothing special about ZFS that requires/encourages the use of ECC RAM more so than any other filesystem.” https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/ https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-y... Now give me my karma back.

thinkingkong 7y ago

Ha! That reminds me of the "Yelling at the Fileserver" video. https://www.youtube.com/watch?v=tDacjrSCeq4 https://www.youtube.com/watch?v=tDacjrSCeq4

techslave 7y ago

that is just a sensational science video. there’s no testimony given that normal raid5 wouldn’t have survived exactly as well. he needed to have mentioned what happened on reboot fatter running a scrub. well not exactly true. we know that the raid5 write hole won’t tolerate random power off / memory removal etc. as someone else said, x ray test would have been better.

tjoff 7y ago

Did they even perform any file (write)operations when testing? If not that was a waste of time... FAT32 would have performed just as well.