5 ms·

I might be misguided here (please correct me), but I feel like ZFS and hardware RAID had their place pre-SSD, and now they don't make much sense. If you want a

I might be misguided here (please correct me), but I feel like ZFS and hardware RAID had their place pre-SSD, and now they don't make much sense. If you want an array of disks with a mutable configuration, I think the right choice is mdadm and LVM. They're reliable, relatively easy to use, let you move disks between machines, don't rely on hardware RAID, and support all file systems. If you need something faster and more optimized like ZFS or btrfs, you're probably better off spending on SSDs anyway. Tell me why I'm wrong, HN :)

the8472 7y ago
mdadm and lvm don't provide data checksumming.
- zozbot234 7y ago
  DM does, via dm-integrity.
  - the8472 7y ago
    does that talk to the raid part so it can restore the correct sector when two raid1 mirrors disagree? And it seems to come with its own journal, so stacking several journaled systems could lead to write amplification, no?
    - cthalupa 7y ago
      You would use dm-integrity on each individual disk below the RAID, and when it detected the corrupted file it would return an error, which would then be used as a signal to the RAID system to read from the other disk. You have to manually set up dm-integrity on the disks, however - it is not used by default when creating an mdadm/lvm setup.
      - the8472 7y ago
        That setup would mean more overhead. Separate journals for each disk, one checksum copy on each disk instead of only up to your redundancy level, more disk writes.
        cthalupa 7y ago
        For sure. I'm a ZFS kind of guy myself. But you can start hacking together a lot of the functionality the more dm modules you start using.
        zozbot234 7y ago
        It's not "more" overhead, just a different distribution of it. You can actually use erasure coding instead of conventional RAID once the lower-level blocks are protected by their own checksums.
        the8472 7y ago
        It is certainly more overhead when you have to keep a journal for each disk and a checksum for each cluster on each disk instead of one checksum per stripe or extent and a per-filesystem journal. In other words zfs and btrfs incur less overhead by having the raid, checksumming and journalling layers talk to each other. Implementing them separately in as in device-mapper comes with inefficiencies.
lunixbochs 7y ago
Price per gigabyte. Far more economical to use ZFS on spinning disks with a huge NVME + RAM cache than it is to buy 80TB of NVME. As long as you don’t need fast random access to the whole dataset all-SSD is wasted.
Wowfunhappy 7y ago
I think most people use ZFS for data integrity, not performance. I make backups regularly (manually, because I want them to be offline), but I'd rather not have to use them if one of my drives die. Unless I literally just finished making the backup, I'm going to lose some amount of work. My mirrored ZFS pool means that (A) a drive can die and I won't lose _any_ data, no matter how recent, and (B) I'm confident none of the data on either drive is corrupted, because ZFS checks it regularly (every time it's accessed, and during weekly scrubs).
- kyuudou 7y ago
  Yep I'd use ZFS for archival and tier-2 storage and use expensive SSD for I/O that needs to be fast and nimble.
- stiray 7y ago
  Exactly. Anyway, my home server is having 6Tb redundant storage (3xToshiba 3Tb) and 10Tb Hitachi drive, non redundant. What would the cost of SSDs be? Over $1000 not to mention the controlle 4 SATA port vs 19+ port controller is a huge difference in price. I do use SSDs, where I need speed - L2ARC ;), far better use than buying 19Tb of SSDs, not to mention the price. Anyway, I do use ZFS on laptop for linux root. Reason? Snapshots. Kvm + ZFS, perfect match. /.zfs/snapshots? Fabulous.
  - nickik 7y ago
    There is this fun backup tool, that have yet to set up. https://www.znapzend.org/ https://www.znapzend.org/
- deleted 7y ago
  [deleted]
rbanffy 7y ago
RAID that knows about which blocks are occupied and which are not will take a shorter time to rebuild because it'll need to read and write fewer blocks. For `mdadm` to do that, it'd need to know a lot about the internals of the filesystems that are mapped to the block device. Filesystems like BtrFS and ZFS know what is being used and what is not and won't waste time reading or writing to blocks that don't hold any currently used data.
- zozbot234 7y ago
  > For `mdadm` to do that, it'd need to know a lot about the internals of the filesystems that are mapped to the block device. That's taken care of by TRIM/DISCARD, which all modern filesystems support.
  - wang_li 7y ago
    Does LVM maintain a bitmap of used blocks that is updated by upper level filesystems sending TRIM/DISCARD?
    - wtallis 7y ago
      It does if you're using thin provisioning.
      - rbanffy 7y ago
        That's pretty cool. Is that used to inform RAID rebuilds?
greenshackle2 7y ago
ZFS is optimized first for data integrity, not speed. It's like you're saying, why do we need ECC when there is fast DDR4 memory? ZFS is a filesystem not just a layer like LVM. You have to compare it to LVM + some filesystem. Which filesystem would you suggest using with LVM to get ZFS-level integrity? (Technically you can use ZFS on top of LVM but that would be a weird thing to do.)
- anon9001 7y ago
  I use ECC and mdadm+LVM+ext4. I haven't had any issues yet, but this thread is making me wonder if there's some data integrity problem that I'm setting myself up for. With RAID6 I can have 2 drives fail without data loss. It does take several days to rebuild when a drive dies, so maybe that's the ZFS feature I'm missing? If the concerns about "data integrity" are really about time until a disk is replaced, then it makes sense. Otherwise, what's the data integrity concern? Ext4 is a journaled file system, which I thought was the equivalent of ECC for filesystems.
  - cthalupa 7y ago
    >I use ECC and mdadm+LVM+ext4. I haven't had any issues yet, but this thread is making me wonder if there's some data integrity problem that I'm setting myself up for. Nothing in your setup there protects you from bitrot. >With RAID6 I can have 2 drives fail without data loss. It does take several days to rebuild when a drive dies, so maybe that's the ZFS feature I'm missing? Drive rebuild time is primarily a function of a variety of things - speed of the disks, processing power involved, etc. There's nothing magic about any solution here. ZFS does get a leg up in that it knows what blocks are in use, but hardware raid and mdadm are quite fast when they find empty blocks too, so it's not some overwhelmingly massive improvement. >Otherwise, what's the data integrity concern? Ext4 is a journaled file system, which I thought was the equivalent of ECC for filesystems. I think you're fundamentally confused about what journaling provides a filesystem. A journal helps you in situations where your system crashes, loses power, etc. and you had not finished committing your changes to the filesystem. It does not protect you from data integrity issues that are unrelated to this sort of situation - a bit flipping in an HDD from electromagnetic radiation or an SSD from electron leakage will not be caught by a journal.
briffle 7y ago
ssd's still can become corrupt (bitrot) and can still die. File systems like zfs and btrfs checksum every block to ensure that if they do become corrupt, they can get repaired. (otherwise, if you had a mirror of disks, and one disk says the value is 0010 and the other says its 0001, wich one is right?) Additionally, the 'raid' concept is not just about speed. Its also about capacity, and how many disks can fail. I might have 48 drives in a single server. I certainly don't want an LVM stripe across all drives and lose data. I also don't want raid1, losing the capacity of 24 of the drives. I want it so that any 3 drives can fail in that system, and I can still recover the data.
- thedance 7y ago
  SSDs are always corrupt. Their raw bit error rate can be 90%+ in normal operation. The only reason SSDs work at all is because of awesome math.
- throwaway-9320 7y ago
  You might be interested in this article which does mention valid points for choosing mirrored data instead of any RAIDZ levels in ZFS: https://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-not-raidz/ https://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs... It certainly made me reconsider my ZFS practices.
rhinoceraptor 7y ago
ZFS might not have the single disk, raw synthetic benchmark performance of a stupid file system with no correctness/integrity built in, but that use case isn't really what anyone wants or needs. ZFS isn't really a filesystem. It's more like a storage backend, that happens to support the POSIX interface. The original intent of ZFS was to completely rethink the volume management/filesystem layering nonsense, and to provide for storage what the MMU/VM system provides for memory. So it can do much more interesting things like memory backed read caching (via ARC), SSD backed read caching (via L2ARC), and SSD backed write buffering (via ZIL/SLOG), as well as the normal RAID stuff.
ses1984 7y ago
I have lost data with mdadm. Never lost data with ZFS yet. Granted I was a much nooblier noob when using mdadm.
matheusmoreira 7y ago
LVM software RAID has the same problems as ZFS's implementation and hardware RAID. When drives fail, high capacity arrays will take over 24 hours to rebuild themselves and are likely to run into additional read errors due to the heavy load. Expanding the capacity of the array is an incredibly complex process: https://raid.wiki.kernel.org/index.php/Growing https://raid.wiki.kernel.org/index.php/Growing It should be possible to buy disks of any size, add them to the computer and tell the file system to use them. Drive failures shouldn't increase the risk of additional failures.
rodgerd 7y ago
If you think LVM snapshots are an adequate replacement for ZFS snapshots, I can only assume you've not used either in anger.
- cmurf 7y ago
  There are two kinds of LVM snapshots: thick and thin. You're probably thinking of conventional (thick) snapshots, rather than the newer thinly provisioned snapshots. Snapshotting an LV will FIFREEZE the file system, then snapshot, then FITHAW the file system. So depending on how dirty things are, it might take a while for the snapshot to happen, but otherwise they are nearly instant to create and destroy.
  - rodgerd 7y ago
    I am aware of the difference - and even the much-touted thins are very poor by comparison, as anyone who has had the displeasure of trying to use them in anger can attest.
    - cmurf 7y ago
      I have but only small scale: single 2T drive and data, with maybe a dozen snapshots. I use Btrfs for most everything these days, with ZFS archive. The biggest issue I had, a while back now, with thinp is if the metadata volume gets full...kaboom. And it's non-trivial to repair.

the8472 7y ago

mdadm and lvm don't provide data checksumming.

zozbot234 7y ago
DM does, via dm-integrity.
- the8472 7y ago
  does that talk to the raid part so it can restore the correct sector when two raid1 mirrors disagree? And it seems to come with its own journal, so stacking several journaled systems could lead to write amplification, no?
  - cthalupa 7y ago
    You would use dm-integrity on each individual disk below the RAID, and when it detected the corrupted file it would return an error, which would then be used as a signal to the RAID system to read from the other disk. You have to manually set up dm-integrity on the disks, however - it is not used by default when creating an mdadm/lvm setup.
    - the8472 7y ago
      That setup would mean more overhead. Separate journals for each disk, one checksum copy on each disk instead of only up to your redundancy level, more disk writes.
      - cthalupa 7y ago
        For sure. I'm a ZFS kind of guy myself. But you can start hacking together a lot of the functionality the more dm modules you start using.
      - zozbot234 7y ago
        It's not "more" overhead, just a different distribution of it. You can actually use erasure coding instead of conventional RAID once the lower-level blocks are protected by their own checksums.
        the8472 7y ago
        It is certainly more overhead when you have to keep a journal for each disk and a checksum for each cluster on each disk instead of one checksum per stripe or extent and a per-filesystem journal. In other words zfs and btrfs incur less overhead by having the raid, checksumming and journalling layers talk to each other. Implementing them separately in as in device-mapper comes with inefficiencies.

lunixbochs 7y ago

Price per gigabyte. Far more economical to use ZFS on spinning disks with a huge NVME + RAM cache than it is to buy 80TB of NVME. As long as you don’t need fast random access to the whole dataset all-SSD is wasted.

Wowfunhappy 7y ago

I think most people use ZFS for data integrity, not performance. I make backups regularly (manually, because I want them to be offline), but I'd rather not have to use them if one of my drives die. Unless I literally just finished making the backup, I'm going to lose some amount of work. My mirrored ZFS pool means that (A) a drive can die and I won't lose _any_ data, no matter how recent, and (B) I'm confident none of the data on either drive is corrupted, because ZFS checks it regularly (every time it's accessed, and during weekly scrubs).

kyuudou 7y ago
Yep I'd use ZFS for archival and tier-2 storage and use expensive SSD for I/O that needs to be fast and nimble.
stiray 7y ago
Exactly. Anyway, my home server is having 6Tb redundant storage (3xToshiba 3Tb) and 10Tb Hitachi drive, non redundant. What would the cost of SSDs be? Over $1000 not to mention the controlle 4 SATA port vs 19+ port controller is a huge difference in price. I do use SSDs, where I need speed - L2ARC ;), far better use than buying 19Tb of SSDs, not to mention the price. Anyway, I do use ZFS on laptop for linux root. Reason? Snapshots. Kvm + ZFS, perfect match. /.zfs/snapshots? Fabulous.
- nickik 7y ago
  There is this fun backup tool, that have yet to set up. https://www.znapzend.org/ https://www.znapzend.org/
deleted 7y ago
[deleted]

rbanffy 7y ago

RAID that knows about which blocks are occupied and which are not will take a shorter time to rebuild because it'll need to read and write fewer blocks. For `mdadm` to do that, it'd need to know a lot about the internals of the filesystems that are mapped to the block device. Filesystems like BtrFS and ZFS know what is being used and what is not and won't waste time reading or writing to blocks that don't hold any currently used data.

zozbot234 7y ago
> For `mdadm` to do that, it'd need to know a lot about the internals of the filesystems that are mapped to the block device. That's taken care of by TRIM/DISCARD, which all modern filesystems support.
- wang_li 7y ago
  Does LVM maintain a bitmap of used blocks that is updated by upper level filesystems sending TRIM/DISCARD?
  - wtallis 7y ago
    It does if you're using thin provisioning.
    - rbanffy 7y ago
      That's pretty cool. Is that used to inform RAID rebuilds?

greenshackle2 7y ago

ZFS is optimized first for data integrity, not speed. It's like you're saying, why do we need ECC when there is fast DDR4 memory? ZFS is a filesystem not just a layer like LVM. You have to compare it to LVM + some filesystem. Which filesystem would you suggest using with LVM to get ZFS-level integrity? (Technically you can use ZFS on top of LVM but that would be a weird thing to do.)

anon9001 7y ago
I use ECC and mdadm+LVM+ext4. I haven't had any issues yet, but this thread is making me wonder if there's some data integrity problem that I'm setting myself up for. With RAID6 I can have 2 drives fail without data loss. It does take several days to rebuild when a drive dies, so maybe that's the ZFS feature I'm missing? If the concerns about "data integrity" are really about time until a disk is replaced, then it makes sense. Otherwise, what's the data integrity concern? Ext4 is a journaled file system, which I thought was the equivalent of ECC for filesystems.
- cthalupa 7y ago
  >I use ECC and mdadm+LVM+ext4. I haven't had any issues yet, but this thread is making me wonder if there's some data integrity problem that I'm setting myself up for. Nothing in your setup there protects you from bitrot. >With RAID6 I can have 2 drives fail without data loss. It does take several days to rebuild when a drive dies, so maybe that's the ZFS feature I'm missing? Drive rebuild time is primarily a function of a variety of things - speed of the disks, processing power involved, etc. There's nothing magic about any solution here. ZFS does get a leg up in that it knows what blocks are in use, but hardware raid and mdadm are quite fast when they find empty blocks too, so it's not some overwhelmingly massive improvement. >Otherwise, what's the data integrity concern? Ext4 is a journaled file system, which I thought was the equivalent of ECC for filesystems. I think you're fundamentally confused about what journaling provides a filesystem. A journal helps you in situations where your system crashes, loses power, etc. and you had not finished committing your changes to the filesystem. It does not protect you from data integrity issues that are unrelated to this sort of situation - a bit flipping in an HDD from electromagnetic radiation or an SSD from electron leakage will not be caught by a journal.

briffle 7y ago

ssd's still can become corrupt (bitrot) and can still die. File systems like zfs and btrfs checksum every block to ensure that if they do become corrupt, they can get repaired. (otherwise, if you had a mirror of disks, and one disk says the value is 0010 and the other says its 0001, wich one is right?) Additionally, the 'raid' concept is not just about speed. Its also about capacity, and how many disks can fail. I might have 48 drives in a single server. I certainly don't want an LVM stripe across all drives and lose data. I also don't want raid1, losing the capacity of 24 of the drives. I want it so that any 3 drives can fail in that system, and I can still recover the data.

thedance 7y ago
SSDs are always corrupt. Their raw bit error rate can be 90%+ in normal operation. The only reason SSDs work at all is because of awesome math.
throwaway-9320 7y ago
You might be interested in this article which does mention valid points for choosing mirrored data instead of any RAIDZ levels in ZFS: https://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-not-raidz/ https://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs... It certainly made me reconsider my ZFS practices.

rhinoceraptor 7y ago

ZFS might not have the single disk, raw synthetic benchmark performance of a stupid file system with no correctness/integrity built in, but that use case isn't really what anyone wants or needs. ZFS isn't really a filesystem. It's more like a storage backend, that happens to support the POSIX interface. The original intent of ZFS was to completely rethink the volume management/filesystem layering nonsense, and to provide for storage what the MMU/VM system provides for memory. So it can do much more interesting things like memory backed read caching (via ARC), SSD backed read caching (via L2ARC), and SSD backed write buffering (via ZIL/SLOG), as well as the normal RAID stuff.

ses1984 7y ago

I have lost data with mdadm. Never lost data with ZFS yet. Granted I was a much nooblier noob when using mdadm.

matheusmoreira 7y ago

LVM software RAID has the same problems as ZFS's implementation and hardware RAID. When drives fail, high capacity arrays will take over 24 hours to rebuild themselves and are likely to run into additional read errors due to the heavy load. Expanding the capacity of the array is an incredibly complex process: https://raid.wiki.kernel.org/index.php/Growing https://raid.wiki.kernel.org/index.php/Growing It should be possible to buy disks of any size, add them to the computer and tell the file system to use them. Drive failures shouldn't increase the risk of additional failures.

rodgerd 7y ago

If you think LVM snapshots are an adequate replacement for ZFS snapshots, I can only assume you've not used either in anger.

cmurf 7y ago
There are two kinds of LVM snapshots: thick and thin. You're probably thinking of conventional (thick) snapshots, rather than the newer thinly provisioned snapshots. Snapshotting an LV will FIFREEZE the file system, then snapshot, then FITHAW the file system. So depending on how dirty things are, it might take a while for the snapshot to happen, but otherwise they are nearly instant to create and destroy.
- rodgerd 7y ago
  I am aware of the difference - and even the much-touted thins are very poor by comparison, as anyone who has had the displeasure of trying to use them in anger can attest.
  - cmurf 7y ago
    I have but only small scale: single 2T drive and data, with maybe a dozen snapshots. I use Btrfs for most everything these days, with ZFS archive. The biggest issue I had, a while back now, with thinp is if the metadata volume gets full...kaboom. And it's non-trivial to repair.