6 ms·

Out of curiosity what RAID setup would you use for replaceable media (e.g. blu ray movies)? Is there a synology solution for checksums to pevent bitrot which Z

by nugget 11y ago

Out of curiosity what RAID setup would you use for replaceable media (e.g. blu ray movies)? Is there a synology solution for checksums to pevent bitrot which ZFS advocates talk about a lot?

pfarnsworth 11y ago
You can run periodic scans to look for bitrot. I think this is the standard way of handling this, my ReadyNAS before this had the same functionality.
- usefulcat 11y ago
  How can it look for bitrot without any checksum info?
  - rakoo 11y ago
    Instead of running some homemade script, you better use something like SnapRaid (http://www.snapraid.it/ http://www.snapraid.it/)
  - pfarnsworth 11y ago
    RAID has redundant information so they use that.
    - Moru 11y ago
      Depending on what RAID you are running, you will only know you have bitrot, you can't fix it since you don't know what harddrive has the correct information.
    - cmurf 11y ago
      This doesn't unambiguously detect mismatches though. If it's a mirror, you have two copies of data, which one is bad? All RAID 1 knows is that they're different. If it's RAID456, again, all it knows is that there's a mismatch, it doesn't know if the data strips are wrong, or if the parity strips are wrong. The way ZFS and Btrfs deal with this is data is checksummed, and the fs metadata which includes the parity strips, are checksummed. So there's a completely independent way of knowing what's incorrect, rather than merely that there's a difference.
      - baruch 11y ago
        In RAID6 you can find which of the different combinations shows a mismatch and which combination doesn't. Run through all combinations and find the one that shows no mismatch, the dropped drive is the one with the bad data, rewrite it and go on with your life. This can only detect one bad drive, if you have two you are toasted.
        Ded7xSEoPKYNsDd 11y ago
        FWIW Linux software RAID doesn't do that. I think the argument was that differences like this were mostly related to power loss where some disks have the new data and others the old data. And at the block layer, it's impossible to tell which is which and so the code just picks a winner basically at random. I'm not 100% convinced myself that a 'majority wins' strategy like you described wouldn't be superior, but I can see why they decided otherwise.
        SirMaster 11y ago
        Except Synology uses MDRAID (Linux Kernel RAID) and even in RAWID 6 mode it doesn't do this. It just overwrites the corrupt sector with a new value to make the parity data consistent. It doesn't know which is right or wrong even though technically with RAID6 it would be possible to determine.
    - usefulcat 11y ago
      Theoretically for RAID6, but not for anything less than that. And in any case I'm pretty sure Linux RAID doesn't implement that. Though, as the owner of a Synology NAS I'd love to be wrong about that.
- ksec 11y ago
  I am not aware Synology offer this at the moment.
jimmcslim 11y ago
Synology are adding Btrfs support in the next version of their OS, for what that's worth.
- nugget 11y ago
  Can existing synology boxes upgrade for that or would it only be in newer models?
  - jimmcslim 11y ago
    "Btrfs is a modern file system developed by multiple parties and now supported by select Synology NAS models." The higher performance models I would expect. DS716+ and various rack-mount models presently listed... I think you would be needing an Intel-based model at least to support this when DSM 6 is released.
- SirMaster 11y ago
  Except they are using MDADM and LVM on top of it so it can only detect corruption, it cannot self heal like BTRFS can when it's in it's own redundant configuration. https://www.reddit.com/r/synology/comments/3qpezu/btrfs_and_lvm/ https://www.reddit.com/r/synology/comments/3qpezu/btrfs_and_... https://www.reddit.com/r/synology/comments/402m8d/so_i_was_going_to_return_my_ds716_and_go_with/cyr4s4o https://www.reddit.com/r/synology/comments/402m8d/so_i_was_g...
puzzlingcaptcha 11y ago
For bluray movies I simply use JBOD and a worldwide distributed network of peers who keep hashed pieces of the files...

by nugget 11y ago

Out of curiosity what RAID setup would you use for replaceable media (e.g. blu ray movies)? Is there a synology solution for checksums to pevent bitrot which ZFS advocates talk about a lot?

pfarnsworth 11y ago

You can run periodic scans to look for bitrot. I think this is the standard way of handling this, my ReadyNAS before this had the same functionality.

usefulcat 11y ago
How can it look for bitrot without any checksum info?
- rakoo 11y ago
  Instead of running some homemade script, you better use something like SnapRaid (http://www.snapraid.it/ http://www.snapraid.it/)
- pfarnsworth 11y ago
  RAID has redundant information so they use that.
  - Moru 11y ago
    Depending on what RAID you are running, you will only know you have bitrot, you can't fix it since you don't know what harddrive has the correct information.
  - cmurf 11y ago
    This doesn't unambiguously detect mismatches though. If it's a mirror, you have two copies of data, which one is bad? All RAID 1 knows is that they're different. If it's RAID456, again, all it knows is that there's a mismatch, it doesn't know if the data strips are wrong, or if the parity strips are wrong. The way ZFS and Btrfs deal with this is data is checksummed, and the fs metadata which includes the parity strips, are checksummed. So there's a completely independent way of knowing what's incorrect, rather than merely that there's a difference.
    - baruch 11y ago
      In RAID6 you can find which of the different combinations shows a mismatch and which combination doesn't. Run through all combinations and find the one that shows no mismatch, the dropped drive is the one with the bad data, rewrite it and go on with your life. This can only detect one bad drive, if you have two you are toasted.
      - Ded7xSEoPKYNsDd 11y ago
        FWIW Linux software RAID doesn't do that. I think the argument was that differences like this were mostly related to power loss where some disks have the new data and others the old data. And at the block layer, it's impossible to tell which is which and so the code just picks a winner basically at random. I'm not 100% convinced myself that a 'majority wins' strategy like you described wouldn't be superior, but I can see why they decided otherwise.
      - SirMaster 11y ago
        Except Synology uses MDRAID (Linux Kernel RAID) and even in RAWID 6 mode it doesn't do this. It just overwrites the corrupt sector with a new value to make the parity data consistent. It doesn't know which is right or wrong even though technically with RAID6 it would be possible to determine.
  - usefulcat 11y ago
    Theoretically for RAID6, but not for anything less than that. And in any case I'm pretty sure Linux RAID doesn't implement that. Though, as the owner of a Synology NAS I'd love to be wrong about that.
ksec 11y ago
I am not aware Synology offer this at the moment.

jimmcslim 11y ago

Synology are adding Btrfs support in the next version of their OS, for what that's worth.

nugget 11y ago
Can existing synology boxes upgrade for that or would it only be in newer models?
- jimmcslim 11y ago
  "Btrfs is a modern file system developed by multiple parties and now supported by select Synology NAS models." The higher performance models I would expect. DS716+ and various rack-mount models presently listed... I think you would be needing an Intel-based model at least to support this when DSM 6 is released.
SirMaster 11y ago
Except they are using MDADM and LVM on top of it so it can only detect corruption, it cannot self heal like BTRFS can when it's in it's own redundant configuration. https://www.reddit.com/r/synology/comments/3qpezu/btrfs_and_lvm/ https://www.reddit.com/r/synology/comments/3qpezu/btrfs_and_... https://www.reddit.com/r/synology/comments/402m8d/so_i_was_going_to_return_my_ds716_and_go_with/cyr4s4o https://www.reddit.com/r/synology/comments/402m8d/so_i_was_g...

puzzlingcaptcha 11y ago

For bluray movies I simply use JBOD and a worldwide distributed network of peers who keep hashed pieces of the files...