6 ms·

ZFS is nice but as far as I understand the Linux version does not yet have support for copy-on-write clones using e.g. "cp --reflink=always", which to me was re

ZFS is nice but as far as I understand the Linux version does not yet have support for copy-on-write clones using e.g. "cp --reflink=always", which to me was reason enough to choose BTRFS instead. Apart from this the two systems seem to be quite comparable (from my limited user perspective), with BTRFS having quite good Linux support as well. Maybe someone more experienced with the COW functionality could comment on that as it would be very interesting to hear how other people deal with this.

aidenn0 11y ago
I've used both ZFS on both BSD and OpenIndiana, and I've used btrfs on linux. as recently as 4.0.5 As of 4.0.5 btrfs was IMO completely unusable as a daily file system. Some examples of issues I ran into: 1) System became unbootable with the version of btrfs I had installed and I had to use either an older or newer kernel to recover 2) I have a periodic backup of my mailbox that runs, and when it runs my system becomes completely unusable until it completes. The same script running on zfs on bsd and with ext4 or reiser3 on linux would show I/O slowdowns, but I could still use my machine. 3) In general I would run into other minor issues and the consensus in #btrfs was that since my kernel was more than 3 months old, it was probably fixed in the latest version, and why would somebody using an experimental filesystem not be tracking mainline more closely? [edit] To be fair, here's some issues with ZFS: 1) Do not ever accidentally enable dedupe on commodity hardware; it will slowly consume all your RAM if you aren't on a sun server (where 64GB of RAM is a resource constrained environment), and there aren't effective ways to undo dedupe, other than copying all the data onto a different pool. 2) You can't shrink a pool. Hugely annoying, apparently non-trivial to solve.
- rsync 11y ago
  "To be fair, here's some issues with ZFS:" Let me add: 3) Do not allow a pool to exceed 90% capacity ... and probably don't let it exceed 85%. ZFS does not have a defrag utility and it badly needs one. You can permanently wreck zpool performance by running it up past 90% capacity - even if you later reduce capacity back down to 75-80%. You can sort of fix it if you add additional top level vdevs to the zpool, thus farming out some IO to the new set of disks, but it's still going to be performance constrained forever. The only solution is to create a new pool and export the data to it. This is unacceptable, by the way. It is not at all reasonable to require a filesystem to stay below 80% capacity (our target "full" number at rsync.net) nor is it acceptable that hitting 90% is a (performance) death sentence. When you consider that you might have already sacrificed 25% of your physical disk just for the raidz2/raidz3 overhead, being constrained to 80% means you're only using 60% or so of your physical disks that you bought. ZFS needs defrag. Badly.
  - ryao 11y ago
    If gang blocks are generated, you get more I/Os than necessary and an extra level of indirection, but the ZFS code base tries very hard to avoid that situation by switching to the allocator behavior at the metaslab level to best fit, such that most data written to the pool would not have gang blocks used at all. This is understandably terrible for IOPS, but it is likely the source of the performance degradation that you saw. You can probe for the zio_gang_* functions during a scrub to see if any gang blocks exist. On my pool that has exceeded 90% on multiple ocasioks, there are zero gang blocks and consequently, no permanent degradation from them. The only other problem that you might have (which tends to be caused by the order of writes rather than the fullness of the pool) is lower sequential performance from nonlinear block placement (one ZoL user measured this as cutting sequential reads in half on a pool filled with files made by bit torrent), but that is a much less severe problem, especially on solid state drives. If you want to fix placement, you can do a file copy or send/recv. The new locations should have blocks picked in sequence whenever possible.
  - l1ambda 11y ago
    The Reiser guys were talking about this too. I think they needed defrag and were trying to get DARPA money to implement it, but it never happened.
  - cthalupa 11y ago
    For ZFS to get defrag, someone with enough ability and knowledge would need to bite the bullet and go through the fairly massive and difficult task of adding in block pointer rewrite, which doesn't seem to be something anyone has been willing to do, and I've seen a lot of concerns about the actual feasibility of it from some very smart people that are knowledgeable with the codebase.
    - XorNot 11y ago
      Which also incidentally would solve the pool shrinking problem as well.
      - voltagex_ 11y ago
- riquito 11y ago
  I've been using btrfs since 2 years (Fedora) and the only problem I've ever run into was "no space left on device", solvable with a rebalance. btrfs also survived many hard poweroff.
  - feld 11y ago
    I tried to use the mirror functionality when it was new. I tested booting with one disk missing. Errors all the way. I went into IRC and chatted with btrfs folk about the "bug". Their response? "Booting without all members of your mirror is unsupported."
    - dsr_ 11y ago
      It's certainly supported now. mount -o degraded. It ought to be the default, though.
      - feld 11y ago
        if it's not default yet it's a serious bug. a server should still come up completely with one disk in a mirror.
  - zanny 11y ago
    I'll pitch in with a neutral position. I've been using btrfs for four years, and in that time I've had unrecoverable fs corruption probably three times. This is on Arch, on bleeding edge kernels, where new releases are prone to regressions that break the filesystem. But there was a tangible progression from instability towards increasing stability. I haven't had one lick of issue with btrfs in about a dozen kernel releases. I'm close to saying I'd trust in a production environment, since I use it everywhere else as a daily driver, I would just use an LTS release version just to be safe. It is not all sunshine and roses, though. While Facebook employs several major btrfs developers, a lot of features that have been talked about for years still have not seen the light of day or any development whatsoever. lz4 compression, better checksum algorithms, per-subvolume encryption, online filesystem checking, and the Raid 5/6 support is still kind of garbage a year later. I worry that btrfs is suffering from a lack of interest in actually making the last legitimate pushes it needs, and code audits, and integration testing, to make it truly trustworthy. But at the end of the day checksum integrity and COW are basically a game changer for me in terms of data integrity.
nisa 11y ago
If you use ZFS you only have snapshots for this. cp --reflink also has some gotchas on btrfs - if you do a balance it's not preserved and there are some odd problems with snaphots (I have no link at hand, take a look at the mailing list) ZFS is not comparable to btrfs at the moment. Everything device related is missing on btrfs. No detection of missing or broken devices in btrfs at the moment, no hotspare functionality, btrfs RAID1 uses the pid to decide with disk to read from. RAID5/6 is still experimental and there are some odd behaviours. Using btrfs for production is a risky bet and may very well bite you. The tooling is terrible at the moment (IMHO) and benchmarks favour ZFS most of the time.
barrkel 11y ago
Running hourly, daily, weekly and monthly snapshots is reason enough to choose ZFS on Linux for me. And ironically, it runs better on Linux than it did for me on Solaris - I used to get occasional pauses every few minutes when streaming media. Memory utilization for cache purposes isn't fantastic since it doesn't interact well with the rest of the kernel's logic, but everything else is pretty good.
ryao 11y ago
No ZFS version has support for that. It has been discussed and it might be implemented at some point. That said, I would like to point out that ZFS' dataset level operations are more powerful than reflinks. ZFS' dataset level operations give separate independent snapshot and clone capabilities. They also provide the ability to rollback without killing things on top (which is useful in some cases). You cannot do that with reflinks. I suppose the immutable bit could be used to fix a reflink so that it retains the state at creation, but that is racy. In the case of virtual machines which seems to be a major application of reflinks, zvols are lower overhead and support incremental send/recv. One benefit of reflinks would be that regular users can use it, but regular users should be able to snapshot, clone and rollback when delegation support is implemented.

aidenn0 11y ago

I've used both ZFS on both BSD and OpenIndiana, and I've used btrfs on linux. as recently as 4.0.5 As of 4.0.5 btrfs was IMO completely unusable as a daily file system. Some examples of issues I ran into: 1) System became unbootable with the version of btrfs I had installed and I had to use either an older or newer kernel to recover 2) I have a periodic backup of my mailbox that runs, and when it runs my system becomes completely unusable until it completes. The same script running on zfs on bsd and with ext4 or reiser3 on linux would show I/O slowdowns, but I could still use my machine. 3) In general I would run into other minor issues and the consensus in #btrfs was that since my kernel was more than 3 months old, it was probably fixed in the latest version, and why would somebody using an experimental filesystem not be tracking mainline more closely? [edit] To be fair, here's some issues with ZFS: 1) Do not ever accidentally enable dedupe on commodity hardware; it will slowly consume all your RAM if you aren't on a sun server (where 64GB of RAM is a resource constrained environment), and there aren't effective ways to undo dedupe, other than copying all the data onto a different pool. 2) You can't shrink a pool. Hugely annoying, apparently non-trivial to solve.

rsync 11y ago
"To be fair, here's some issues with ZFS:" Let me add: 3) Do not allow a pool to exceed 90% capacity ... and probably don't let it exceed 85%. ZFS does not have a defrag utility and it badly needs one. You can permanently wreck zpool performance by running it up past 90% capacity - even if you later reduce capacity back down to 75-80%. You can sort of fix it if you add additional top level vdevs to the zpool, thus farming out some IO to the new set of disks, but it's still going to be performance constrained forever. The only solution is to create a new pool and export the data to it. This is unacceptable, by the way. It is not at all reasonable to require a filesystem to stay below 80% capacity (our target "full" number at rsync.net) nor is it acceptable that hitting 90% is a (performance) death sentence. When you consider that you might have already sacrificed 25% of your physical disk just for the raidz2/raidz3 overhead, being constrained to 80% means you're only using 60% or so of your physical disks that you bought. ZFS needs defrag. Badly.
- ryao 11y ago
  If gang blocks are generated, you get more I/Os than necessary and an extra level of indirection, but the ZFS code base tries very hard to avoid that situation by switching to the allocator behavior at the metaslab level to best fit, such that most data written to the pool would not have gang blocks used at all. This is understandably terrible for IOPS, but it is likely the source of the performance degradation that you saw. You can probe for the zio_gang_* functions during a scrub to see if any gang blocks exist. On my pool that has exceeded 90% on multiple ocasioks, there are zero gang blocks and consequently, no permanent degradation from them. The only other problem that you might have (which tends to be caused by the order of writes rather than the fullness of the pool) is lower sequential performance from nonlinear block placement (one ZoL user measured this as cutting sequential reads in half on a pool filled with files made by bit torrent), but that is a much less severe problem, especially on solid state drives. If you want to fix placement, you can do a file copy or send/recv. The new locations should have blocks picked in sequence whenever possible.
- l1ambda 11y ago
  The Reiser guys were talking about this too. I think they needed defrag and were trying to get DARPA money to implement it, but it never happened.
- cthalupa 11y ago
  For ZFS to get defrag, someone with enough ability and knowledge would need to bite the bullet and go through the fairly massive and difficult task of adding in block pointer rewrite, which doesn't seem to be something anyone has been willing to do, and I've seen a lot of concerns about the actual feasibility of it from some very smart people that are knowledgeable with the codebase.
  - XorNot 11y ago
    Which also incidentally would solve the pool shrinking problem as well.
    - voltagex_ 11y ago
riquito 11y ago
I've been using btrfs since 2 years (Fedora) and the only problem I've ever run into was "no space left on device", solvable with a rebalance. btrfs also survived many hard poweroff.
- feld 11y ago
  I tried to use the mirror functionality when it was new. I tested booting with one disk missing. Errors all the way. I went into IRC and chatted with btrfs folk about the "bug". Their response? "Booting without all members of your mirror is unsupported."
  - dsr_ 11y ago
    It's certainly supported now. mount -o degraded. It ought to be the default, though.
    - feld 11y ago
      if it's not default yet it's a serious bug. a server should still come up completely with one disk in a mirror.
- zanny 11y ago
  I'll pitch in with a neutral position. I've been using btrfs for four years, and in that time I've had unrecoverable fs corruption probably three times. This is on Arch, on bleeding edge kernels, where new releases are prone to regressions that break the filesystem. But there was a tangible progression from instability towards increasing stability. I haven't had one lick of issue with btrfs in about a dozen kernel releases. I'm close to saying I'd trust in a production environment, since I use it everywhere else as a daily driver, I would just use an LTS release version just to be safe. It is not all sunshine and roses, though. While Facebook employs several major btrfs developers, a lot of features that have been talked about for years still have not seen the light of day or any development whatsoever. lz4 compression, better checksum algorithms, per-subvolume encryption, online filesystem checking, and the Raid 5/6 support is still kind of garbage a year later. I worry that btrfs is suffering from a lack of interest in actually making the last legitimate pushes it needs, and code audits, and integration testing, to make it truly trustworthy. But at the end of the day checksum integrity and COW are basically a game changer for me in terms of data integrity.

nisa 11y ago

If you use ZFS you only have snapshots for this. cp --reflink also has some gotchas on btrfs - if you do a balance it's not preserved and there are some odd problems with snaphots (I have no link at hand, take a look at the mailing list) ZFS is not comparable to btrfs at the moment. Everything device related is missing on btrfs. No detection of missing or broken devices in btrfs at the moment, no hotspare functionality, btrfs RAID1 uses the pid to decide with disk to read from. RAID5/6 is still experimental and there are some odd behaviours. Using btrfs for production is a risky bet and may very well bite you. The tooling is terrible at the moment (IMHO) and benchmarks favour ZFS most of the time.

barrkel 11y ago

Running hourly, daily, weekly and monthly snapshots is reason enough to choose ZFS on Linux for me. And ironically, it runs better on Linux than it did for me on Solaris - I used to get occasional pauses every few minutes when streaming media. Memory utilization for cache purposes isn't fantastic since it doesn't interact well with the rest of the kernel's logic, but everything else is pretty good.

ryao 11y ago

No ZFS version has support for that. It has been discussed and it might be implemented at some point. That said, I would like to point out that ZFS' dataset level operations are more powerful than reflinks. ZFS' dataset level operations give separate independent snapshot and clone capabilities. They also provide the ability to rollback without killing things on top (which is useful in some cases). You cannot do that with reflinks. I suppose the immutable bit could be used to fix a reflink so that it retains the state at creation, but that is racy. In the case of virtual machines which seems to be a major application of reflinks, zvols are lower overhead and support incremental send/recv. One benefit of reflinks would be that regular users can use it, but regular users should be able to snapshot, clone and rollback when delegation support is implemented.