3 ms·

I can afford it, but have a hard time justifying the costs, not to mention scrapped (working) hardware and inconvenience (of swapping to a whole new array). I

I can afford it, but have a hard time justifying the costs, not to mention scrapped (working) hardware and inconvenience (of swapping to a whole new array). I started using snapraid [1] several years ago, after finding zfs couldn't expand. Often when I went to add space the "sweet spot" disk size (best $/TB) was 2-3x the size of the previous biggest disk I ran. This was very economical compared to replacing the whole array every couple years. It works by having "data" and "parity" drives. Data drives are totally normal filesystems, and joined with unionfs. In fact you can mount them independently and access whatever files are on it. Parity drives are just a big file that snapraid updates nightly. The big downside is it's not realtime redundant: you can lose a day's worth of data from a (data) drive failure. For my use case this is acceptable. A huge upside is rebuilds are fairly painless. Rebuilding a parity drive has zero downtime, just degraded performance. Rebuilding a data drive leaves it offline, but the rest work fine (I think the individual files are actually accessible as they're restored though). In the worst case you can mount each data drive independently on any system and recover its contents. I've been running the "same" array for a decade, but at this point every disk has been swapped out at least once (for a larger one), and it's been in at least two different host systems. [1] https://www.snapraid.it/ https://www.snapraid.it/

RulerOf 3y ago
I gave snapraid a serious look a few months back and decided it might not be for me because the act of balancing writes out to the "array" member disks appeared to be manual. I didn't want to point applications at 100 T of "free" space only for attires to start blocking after 8. Am I mistaken about that?
- gregmac 3y ago
  Sorry I said unionfs but it's actually handled by mergerfs [1], and it's all automatic. There are a whole boatload of policies [2] to control writes. I use "existing path, least free space". Once a path is created, it keeps using it for new files in that path. If it runs out of space, it creates that same path on another drive. If the path exists on both drives for some reason, my rationale is this keeps most of the related files (same path) together on the same drive. I see there's some newer "most shared path" options I don't remember that might even make more sense for me, so maybe that's something I'll change next time I need to touch it. [1] https://github.com/trapexit/mergerfs https://github.com/trapexit/mergerfs [2] https://github.com/trapexit/mergerfs#policy-descriptions https://github.com/trapexit/mergerfs#policy-descriptions
  - _trapexit 3y ago
    > If it runs out of space, it creates that same path on another drive. That's not how it works. The policy picks what branch to use and then once selected mergerfs will clone the relative path as needed. With "ep" policies it will never select a branch that doesn't have the full relative path. "msp" will always rerun the check one level up in the hierarchy if nothing is found at the current level.
    - RulerOf 3y ago
      This is immensely helpful, thanks. Will MergerFS mount points behave the same as on the host inside of a docker container if passed as a bind mount?
didntcheck 3y ago
I like Snapraid, but there are a few downsides worth mentioning * Rebuilds are semi-offline, as you said. Almost every other solution, even Unraid, will immediately emulate the data from a failed drive. On Snapraid you have to wait for each file to be restored, and depending on your union setup this may mean you have directories with half the files missing * Whenever you modify or delete a file between syncs, your parity is now out of sync. This means some other files may fail to restore if you have a failure now. However using more than single parity will make this far less likely to happen. Another way to fix this completely the "snapraid-btrfs" tool, which runs Snapraid on snapshots of independent btrfs disks (somewhat like Synology?), meaning the old data is still available * It saves almost no file metadata, not even owner and mode. Restored files just use the umask of the user running the restore command. A minor one, but surprisingly annoying However one big advantage over Unraid it has is how transactional and rigorous it is. A power failure can cause Unraid parity to desync, with no clear way to know which disk is right. Snapraid OTOH is designed to survive interruptions gracefully, and checksums all files so should never accidentally restore corrupt data. And can detect silent drive failure ("bitrot") as a bonus And for a typical home NAS storing movies and family photos (mostly append-only), those downsides are probably no big deal anyway

RulerOf 3y ago

I gave snapraid a serious look a few months back and decided it might not be for me because the act of balancing writes out to the "array" member disks appeared to be manual. I didn't want to point applications at 100 T of "free" space only for attires to start blocking after 8. Am I mistaken about that?

gregmac 3y ago
Sorry I said unionfs but it's actually handled by mergerfs [1], and it's all automatic. There are a whole boatload of policies [2] to control writes. I use "existing path, least free space". Once a path is created, it keeps using it for new files in that path. If it runs out of space, it creates that same path on another drive. If the path exists on both drives for some reason, my rationale is this keeps most of the related files (same path) together on the same drive. I see there's some newer "most shared path" options I don't remember that might even make more sense for me, so maybe that's something I'll change next time I need to touch it. [1] https://github.com/trapexit/mergerfs https://github.com/trapexit/mergerfs [2] https://github.com/trapexit/mergerfs#policy-descriptions https://github.com/trapexit/mergerfs#policy-descriptions
- _trapexit 3y ago
  > If it runs out of space, it creates that same path on another drive. That's not how it works. The policy picks what branch to use and then once selected mergerfs will clone the relative path as needed. With "ep" policies it will never select a branch that doesn't have the full relative path. "msp" will always rerun the check one level up in the hierarchy if nothing is found at the current level.
  - RulerOf 3y ago
    This is immensely helpful, thanks. Will MergerFS mount points behave the same as on the host inside of a docker container if passed as a bind mount?

didntcheck 3y ago

I like Snapraid, but there are a few downsides worth mentioning * Rebuilds are semi-offline, as you said. Almost every other solution, even Unraid, will immediately emulate the data from a failed drive. On Snapraid you have to wait for each file to be restored, and depending on your union setup this may mean you have directories with half the files missing * Whenever you modify or delete a file between syncs, your parity is now out of sync. This means some other files may fail to restore if you have a failure now. However using more than single parity will make this far less likely to happen. Another way to fix this completely the "snapraid-btrfs" tool, which runs Snapraid on snapshots of independent btrfs disks (somewhat like Synology?), meaning the old data is still available * It saves almost no file metadata, not even owner and mode. Restored files just use the umask of the user running the restore command. A minor one, but surprisingly annoying However one big advantage over Unraid it has is how transactional and rigorous it is. A power failure can cause Unraid parity to desync, with no clear way to know which disk is right. Snapraid OTOH is designed to survive interruptions gracefully, and checksums all files so should never accidentally restore corrupt data. And can detect silent drive failure ("bitrot") as a bonus And for a typical home NAS storing movies and family photos (mostly append-only), those downsides are probably no big deal anyway