3 ms·

I've been managing a raid6 ext4 array with mdadm for 10 years. Started with 4 x 4TB disks and kept adding, up to 11 disks now. It works reliably and as designed

I've been managing a raid6 ext4 array with mdadm for 10 years. Started with 4 x 4TB disks and kept adding, up to 11 disks now. It works reliably and as designed. Had a few disk failures and replaced them without issues. That's one of the nice things about mdadm vs ZFS: you can add and remove disks from the array as you see fit, rather than being forced to upgrade all disks if you want to increase the size of your array.

oarsinsync 4y ago
I did that for a long time. I reached 15 disks. Then something went wrong, and I lost everything. I now run multiple 6 disk raidz2s, in part because I wanted to remove the option that led me down the bad path. I hope your journey ends better than mine did.
- imiric 4y ago
  Reading these horror stories just makes me not want to rely on smart filesystems at all. If a bug can take out my whole array, or if the system is as inflexible and magical as ZFS, I'd rather not rely on it at all. I've been managing a home NAS for years with SnapRAID and plain old ext4. Sure, it doesn't have the bells and whistles of something like ZFS, but it's simple to use and understand, scales with anything you throw at it, and there's no way to lose the whole array. I'm currently transitioning to a multi-node setup, and will probably move to Ceph, otherwise I would stick with SnapRAID. Every so often I look into btrfs/ZFS/mdadm, but keep reaching the same conclusion that it's just not worth the risk.
  - michaelmrose 4y ago
    Cars require regular maintenance and have known failure modes if you don't bother to do it its neither magical nor dangerous. Snapraid + ext4 doesn't particularly simpler than zfs just different. The fundamental tradeoff of space vs reliability seems to be mathematically tied. It's impossible to do better in one dimension without sacrificing the other. Making it possible to lose SOME of your data in fact seems to be the worst choice. ZFS makes it trivial to lose nothing. Use enough hardware and do cheap replication regularly and scrub periodically. If anything the fact that snapraid by not live replicating every change to every disk in the array makes it impossible to achieve the reliability offered by ZFS. It's forever the inferior cousin and in fact more complicated for layering a dissimilar technology on top of the other.
    - imiric 4y ago
      SnapRAID has its drawbacks, sure. As with any technology, deciding to use one solution over another is a balancing act of choosing the set of drawbacks that are acceptable for a particular use case. In this case, _for my simple needs_ of running a home NAS, the fact SnapRAID doesn't run in real-time is not an issue. In fact, I prefer being in control of when it runs, and what it's doing exactly. Having a short time window where some new data is not replicated is a negligible drawback to me. OTOH, while ZFS solves this particular issue, its drawbacks of being difficult to scale, and a huge black box that claims to "just work", when in fact my _entire_ array relies on it working perfectly, 100% of the time, are a tough pill to swallow. I'm sure that with the years of dedicated improvements and stability fixes, it's a battle-tested system where the likelihood of it failing is close to 0, even on Linux. But the fact that it's theoretically possible is a deal-breaker to me. In this sense, I much prefer SnapRAID's approach that makes this literally impossible. SnapRAID could stop working entirely or disappear tomorrow, and all my data is perfectly safe. So, yes, SnapRAID + ext4 is radically simpler than ZFS, IMO. Would I recommend this setup in a corporate environment, where company resources are on the line? Probably not. But I would still advocate against ZFS and mdadm, and probably suggest something like Ceph instead.
  - oarsinsync 4y ago
    > will probably move to Ceph ... Every so often I look into btrfs/ZFS/mdadm, but keep reaching the same conclusion that it's just not worth the risk. The worst thing about my experiences with mdadm and ZFS is that I've had exactly one catastrophic failure in 15 years. The best thing about my experiences with Ceph is that it's really reinforced the importance of a good backup strategy. The combination of the two means I now run ZFS, and I have a comprehensive backup strategy that is regularly tested, but has never needed to be invoked. I strongly recommend developing a strong backup story before you go down your Ceph journey.
- herpderperator 4y ago
  Sorry that you lost everything. Keep in mind that RAID is not backup. I also don't understand the correlation between reaching a certain number of disks vs any other scenario. Disaster can always strike, so it's best to be prepared.
WastingMyTime89 4y ago
Why is this comment downvoted? ZFS limited disk management forcing you to use the size of the smaller disk and making it difficult to resize array is not a feature. It’s a serious limitation when using it as a normal user and not in a professional environment. It’s extremely annoying that most of the ZFS zealots are in denial.
- deleted 4y ago
  [deleted]

oarsinsync 4y ago

I did that for a long time. I reached 15 disks. Then something went wrong, and I lost everything. I now run multiple 6 disk raidz2s, in part because I wanted to remove the option that led me down the bad path. I hope your journey ends better than mine did.

imiric 4y ago
Reading these horror stories just makes me not want to rely on smart filesystems at all. If a bug can take out my whole array, or if the system is as inflexible and magical as ZFS, I'd rather not rely on it at all. I've been managing a home NAS for years with SnapRAID and plain old ext4. Sure, it doesn't have the bells and whistles of something like ZFS, but it's simple to use and understand, scales with anything you throw at it, and there's no way to lose the whole array. I'm currently transitioning to a multi-node setup, and will probably move to Ceph, otherwise I would stick with SnapRAID. Every so often I look into btrfs/ZFS/mdadm, but keep reaching the same conclusion that it's just not worth the risk.
- michaelmrose 4y ago
  Cars require regular maintenance and have known failure modes if you don't bother to do it its neither magical nor dangerous. Snapraid + ext4 doesn't particularly simpler than zfs just different. The fundamental tradeoff of space vs reliability seems to be mathematically tied. It's impossible to do better in one dimension without sacrificing the other. Making it possible to lose SOME of your data in fact seems to be the worst choice. ZFS makes it trivial to lose nothing. Use enough hardware and do cheap replication regularly and scrub periodically. If anything the fact that snapraid by not live replicating every change to every disk in the array makes it impossible to achieve the reliability offered by ZFS. It's forever the inferior cousin and in fact more complicated for layering a dissimilar technology on top of the other.
  - imiric 4y ago
    SnapRAID has its drawbacks, sure. As with any technology, deciding to use one solution over another is a balancing act of choosing the set of drawbacks that are acceptable for a particular use case. In this case, _for my simple needs_ of running a home NAS, the fact SnapRAID doesn't run in real-time is not an issue. In fact, I prefer being in control of when it runs, and what it's doing exactly. Having a short time window where some new data is not replicated is a negligible drawback to me. OTOH, while ZFS solves this particular issue, its drawbacks of being difficult to scale, and a huge black box that claims to "just work", when in fact my _entire_ array relies on it working perfectly, 100% of the time, are a tough pill to swallow. I'm sure that with the years of dedicated improvements and stability fixes, it's a battle-tested system where the likelihood of it failing is close to 0, even on Linux. But the fact that it's theoretically possible is a deal-breaker to me. In this sense, I much prefer SnapRAID's approach that makes this literally impossible. SnapRAID could stop working entirely or disappear tomorrow, and all my data is perfectly safe. So, yes, SnapRAID + ext4 is radically simpler than ZFS, IMO. Would I recommend this setup in a corporate environment, where company resources are on the line? Probably not. But I would still advocate against ZFS and mdadm, and probably suggest something like Ceph instead.
- oarsinsync 4y ago
  > will probably move to Ceph ... Every so often I look into btrfs/ZFS/mdadm, but keep reaching the same conclusion that it's just not worth the risk. The worst thing about my experiences with mdadm and ZFS is that I've had exactly one catastrophic failure in 15 years. The best thing about my experiences with Ceph is that it's really reinforced the importance of a good backup strategy. The combination of the two means I now run ZFS, and I have a comprehensive backup strategy that is regularly tested, but has never needed to be invoked. I strongly recommend developing a strong backup story before you go down your Ceph journey.
herpderperator 4y ago
Sorry that you lost everything. Keep in mind that RAID is not backup. I also don't understand the correlation between reaching a certain number of disks vs any other scenario. Disaster can always strike, so it's best to be prepared.

WastingMyTime89 4y ago

Why is this comment downvoted? ZFS limited disk management forcing you to use the size of the smaller disk and making it difficult to resize array is not a feature. It’s a serious limitation when using it as a normal user and not in a professional environment. It’s extremely annoying that most of the ZFS zealots are in denial.

deleted 4y ago
[deleted]