5 ms·

Question: I currently run ZFS via Ubuntu on my home devices. I've had a very positive experience with it. My only (minor) complaint is the extra 30 seconds it

by bmurphy1976 3y ago

Question: I currently run ZFS via Ubuntu on my home devices. I've had a very positive experience with it. My only (minor) complaint is the extra 30 seconds it adds to my desktop's boot time. It's been fantastic for all my other devices, particularly the snapshotting features and ease of which I can ship the snapshots between the devices. I've read some comments on various forums that seem to hint Ubuntu is moving on from ZFS (see https://discourse.ubuntu.com/t/future-of-zfs-on-ubuntu-desktop/33001 https://discourse.ubuntu.com/t/future-of-zfs-on-ubuntu-deskt... for one example). This is obviously concerning and the lack of support across the whole Linux ecosystem continues to be a nuisance. Realistically how well could bcachefs serve as a long term ZFS replacement? I know very little about it. I've used btrfs in the past, and between my negative experiences and the repeated anecdotes posted all over the internet to this day I'm hesitant to give it another go.

Filligree 3y ago
I tried precisely that, and had it corrupt my data… twice. Bcachefs isn’t ready for prime time. I’m reasonably sure it’s better designed than Btrfs, and it’ll get there, but don’t do it yet. If Ubuntu becomes a problem, NixOS has 100% support for ZFS.
- koverstreet 3y ago
  Were you able to report this? If you can get me enough information (e.g. via bcachefs dump), I'd like to track this down.
  - Filligree 3y ago
    We talked on IRC. :-) I'm planning to do that, but it'll have to be after my vacation; don't want to be running bcachefs on a remote server I can't fix. For those following along, this sort of response is why I'm optimistic overall. And bcachefs was dramatically higher performance than ZFS, although the lack of an ARC is a particular pain point. === It's going to be an interesting bug, I'm pretty sure. It affected two sets of files: The Steam Linux runtime (Solder), and everything in /nix/store, but absolutely nothing else. Actually, I think it's probably a bug in the fsck routine; everything worked perfectly until I rebooted.
    - vladvasiliu 3y ago
      > And bcachefs was dramatically higher performance than ZFS Would you care to go into some specifics? On my middle of the road machines, the limiting factor has always been the drive. Encrypted ZFS has the same transfer speeds as unencrypted ext4, as tested with fio. My best machine is a zen 3 u laptop, so pcie 3 with a samsung 990 nvme drive. It reads and writes at over 2 GB/s.
      - bcrl 3y ago
        2GB/s is pretty low performance for storage these days. =-)
      - Filligree 3y ago
        Caching, mostly. My desktop has two HDDs and one NVMe. The NVMe isn't nearly that fast -- it's a 970 Evo which tops out at 1.6 GB/s -- but it's still much faster than the HDDs. When I write data to the setup, bcachefs and ZFS had very different behaviour. ZFS: - Writes to the HDDs first. - Blocks further writes until the first ones are on disk, unless I use sync=disabled. (Which I do; it's safe for my use-case.) - Is in any case limited by how much can be buffered in RAM. - Doesn't write to the SSD (which is configured as cache) unless it first reads data into RAM and then that data gets dropped from the in-core ARC. BcacheFS: - Writes to the SSD first. - Writes to the HDDs at its leisure. - Can be told the NVMe is more reliable than the HDDs. - Can be configured to leave data uncompressed on the SSD (because even LZ4 is a notable drain at 1.6 GB/s), but use zstd for the HDDs (because it's async so why not, and decompressing zstd at 300 MB/s is no issue). - Uses LRU instead of ARC, so if your working set is larger than your cache ZFS will have much better behaviour... once it's warmed up. The bug I hit is apparently fixed now, so I'm likely to switch back to bcachefs more or less ASAP. My working set is not in fact smaller than my SSD... but I'll just get a second SSD, I think.
        koverstreet 3y ago
        I don't think it's fixed yet? biggest bugfix in the pipeline is that bfoster just figured out the 'missing backpointer' bug, but that's just in the btree write buffer, it wouldn't cause data corruption - and the fix for that isn't in yet
BaseballPhysics 3y ago
> My only (minor) complaint is the extra 30 seconds it adds to my desktop's boot time. Wait. What? You'll forgive me, I have no experience with ZFS: why??
- dsr_ 3y ago
  I have a fair amount of experience with ZFS, and have the same question.
- tremon 3y ago
  Metadata scanning, I presume. It's not quite 30 seconds, but mounting my 4-drive 12TB btrfs volume (spinning rust) also feels like it takes upwards of ten seconds (I've never bothered to actually measure the mount time, so can't say with certainty).
  - dTP90pN 3y ago
    Btrfs mounting of larger arrays became near-instant for me with the new block group tree enabled; see btrfstune: https://btrfs.readthedocs.io/en/latest/btrfstune.html https://btrfs.readthedocs.io/en/latest/btrfstune.html
- aidenn0 3y ago
  Importing ZFS pools from spinning disks can be slow. I'm guessing there's some serious random access going on because it's nearly instantaneous on SSDs
- someplaceguy 3y ago
  When importing/opening a ZFS pool, ZFS traverses the last few transaction groups written to the ZFS pool as well as the intent logs as a verification step to make sure that the ZFS pool is in a safe/consistent state before continuing (which could otherwise corrupt a ZFS pool even more and make the pool unrecoverable if something were to be wrong). I'm not sure if this is the main bottleneck when opening a ZFS pool during boot, but I wouldn't be surprised if it were. And if it is, personally I'm willing to sacrifice some boot time for some assurances that my data will be safer, but of course, not everyone's priorities are the same as mine :) (Also, for the record, for me the pool import/opening process is nearly instantaneous on SSDs/NVMe and takes 2.4 seconds on a 12-year-old Intel Atom machine with a single HDD pool).
dilliholla77 3y ago
> I've used btrfs in the past, and between my negative experiences and the repeated anecdotes posted all over the internet to this day I'm hesitant to give it another go. What are those issues? I've not followed the conversation, but I've been running btrfs for over a decade on multiple machines with different usage patterns without any issues. I'm genuinely interested. Are those substantiated claims? Or just the general noisy rants you can find about essentially anything people have opinions about?
- Filligree 3y ago
  In my own case: Btrfs performance got absolutely terrible once I was up to a few thousand snapshots. Never had it lose data, but I also never ran it long enough to lose a disk.
  - unqueued 3y ago
    This is an example of how btrfs is poorly handled by many userland tools by default. You should never ever need to have thousands of snapshots. I use btrbk to manage my snapshots. It keeps my snapshots pruned to 300, pruning intermediate snapshots, so I can still easily step back 52 weeks, or I can purge them if I want to improve performance and reclaim space. While at the same time, every single snapshot is sent to two compressed encrypted backup images over ssh. The diff algorithm is pretty good, but for some types of files that fragment frequently, it is best to put them on their own subvolumes. My places.sqlite files from my browser testing profiles were adding tens or hundreds of MiB to snapshots, so they went in a seperate subvolume. I don't think the zfs would perform any better. I don't blame people for having bad experiences with btrfs, it is very inconsistently documented. I think it is especially annoying how many userland tools keep every single snapshot as a subvolume of the root, so that you have tens of millions of read-only files stored in /.snapshots. Awful.
    - seanw444 3y ago
      My setup is that I only have subvolumes for root, and (for now) my Doom Emacs config. Whenever I do a system upgrade, or a Doom Emacs upgrade, I manually take a snapshot of the appropriate subvolume first, just in case anything goes wrong. And then I only keep a max of 7 or so snapshots before I go through and delete the old ones. Has worked well for me so far and saved my butt a few times, and since I don't have a bunch of subvolumes to manage, I'm not overwhelmed. It's nice and simple, and I need nothing other than the default `btrfs` CLI utility.
ericbarrett 3y ago
I've been a storage engineer since 2000. The industry rule of thumb for file systems is to give them ten years after mainline release to bake. And the longer I've done this, the more I trust it.
- ilc 3y ago
  Mainline release in the kernel or actually be used by distros? 10 seems a bit short ;)
  - ugjka 3y ago
    i switched back from btrfs to ext4 because i still had problems with it
kevincox 3y ago
I think it could be a very good replacement. In my opinion ZFS is still very static and disk/block based with some traditional RAID modes on top. It can provide the same redundancy and safety but with tons more flexibility. For example having different redundancy levels using the same pool of disks and changing redundancy levels on the fly. It does feel like a huge step in in terms of "here are my disks" and then "please store this data this way". bcachefs then just does it in a reasonable and efficient way. Way better than having to set up all of your disk RAID configs upfront. That being said it definitely still needs stability work. I tried it and it corrupted itself. So I love the idea, the design seems good overall. But it is going to need lots of testing and tweaking to become reliable. However it does seem like a solid addition to the filesystem space and I think being upstream will allow more people to work on it, avoid extra maintenance burden by staying in sync with changes and get the attention it deserves. I wrote more about why I like the design and how it corrupted for me here: https://kevincox.ca/2023/06/10/bcachefs-attempt/ https://kevincox.ca/2023/06/10/bcachefs-attempt/
KolenCh 3y ago
ZFS on Linux will not goes away. It is just that Ubuntu dropping it from the installer. You can still install it: https://launchpad.net/ubuntu/+source/zfs-linux/ https://launchpad.net/ubuntu/+source/zfs-linux/ And you can install it on a lot of distro too: https://openzfs.github.io/openzfs-docs/Getting%20Started/index.html https://openzfs.github.io/openzfs-docs/Getting%20Started/ind... My current favorite is to install kABI-tracking kmod on RHEL. (But their kABI-tracking is not a guarantee as they uses more.) In fact I move away from Ubuntu knowing that they knowingly ship a ZFS version with file corruption bug, which is introduced by their own patch of OpenZFS. So if OpenZFS is serving you well, don't worry about the Ubuntu situation because there's plenty better options.

Filligree 3y ago

I tried precisely that, and had it corrupt my data… twice. Bcachefs isn’t ready for prime time. I’m reasonably sure it’s better designed than Btrfs, and it’ll get there, but don’t do it yet. If Ubuntu becomes a problem, NixOS has 100% support for ZFS.

koverstreet 3y ago
Were you able to report this? If you can get me enough information (e.g. via bcachefs dump), I'd like to track this down.
- Filligree 3y ago
  We talked on IRC. :-) I'm planning to do that, but it'll have to be after my vacation; don't want to be running bcachefs on a remote server I can't fix. For those following along, this sort of response is why I'm optimistic overall. And bcachefs was dramatically higher performance than ZFS, although the lack of an ARC is a particular pain point. === It's going to be an interesting bug, I'm pretty sure. It affected two sets of files: The Steam Linux runtime (Solder), and everything in /nix/store, but absolutely nothing else. Actually, I think it's probably a bug in the fsck routine; everything worked perfectly until I rebooted.
  - vladvasiliu 3y ago
    > And bcachefs was dramatically higher performance than ZFS Would you care to go into some specifics? On my middle of the road machines, the limiting factor has always been the drive. Encrypted ZFS has the same transfer speeds as unencrypted ext4, as tested with fio. My best machine is a zen 3 u laptop, so pcie 3 with a samsung 990 nvme drive. It reads and writes at over 2 GB/s.
    - bcrl 3y ago
      2GB/s is pretty low performance for storage these days. =-)
    - Filligree 3y ago
      Caching, mostly. My desktop has two HDDs and one NVMe. The NVMe isn't nearly that fast -- it's a 970 Evo which tops out at 1.6 GB/s -- but it's still much faster than the HDDs. When I write data to the setup, bcachefs and ZFS had very different behaviour. ZFS: - Writes to the HDDs first. - Blocks further writes until the first ones are on disk, unless I use sync=disabled. (Which I do; it's safe for my use-case.) - Is in any case limited by how much can be buffered in RAM. - Doesn't write to the SSD (which is configured as cache) unless it first reads data into RAM and then that data gets dropped from the in-core ARC. BcacheFS: - Writes to the SSD first. - Writes to the HDDs at its leisure. - Can be told the NVMe is more reliable than the HDDs. - Can be configured to leave data uncompressed on the SSD (because even LZ4 is a notable drain at 1.6 GB/s), but use zstd for the HDDs (because it's async so why not, and decompressing zstd at 300 MB/s is no issue). - Uses LRU instead of ARC, so if your working set is larger than your cache ZFS will have much better behaviour... once it's warmed up. The bug I hit is apparently fixed now, so I'm likely to switch back to bcachefs more or less ASAP. My working set is not in fact smaller than my SSD... but I'll just get a second SSD, I think.
      - koverstreet 3y ago
        I don't think it's fixed yet? biggest bugfix in the pipeline is that bfoster just figured out the 'missing backpointer' bug, but that's just in the btree write buffer, it wouldn't cause data corruption - and the fix for that isn't in yet

BaseballPhysics 3y ago

> My only (minor) complaint is the extra 30 seconds it adds to my desktop's boot time. Wait. What? You'll forgive me, I have no experience with ZFS: why??

dsr_ 3y ago
I have a fair amount of experience with ZFS, and have the same question.
tremon 3y ago
Metadata scanning, I presume. It's not quite 30 seconds, but mounting my 4-drive 12TB btrfs volume (spinning rust) also feels like it takes upwards of ten seconds (I've never bothered to actually measure the mount time, so can't say with certainty).
- dTP90pN 3y ago
  Btrfs mounting of larger arrays became near-instant for me with the new block group tree enabled; see btrfstune: https://btrfs.readthedocs.io/en/latest/btrfstune.html https://btrfs.readthedocs.io/en/latest/btrfstune.html
aidenn0 3y ago
Importing ZFS pools from spinning disks can be slow. I'm guessing there's some serious random access going on because it's nearly instantaneous on SSDs
someplaceguy 3y ago
When importing/opening a ZFS pool, ZFS traverses the last few transaction groups written to the ZFS pool as well as the intent logs as a verification step to make sure that the ZFS pool is in a safe/consistent state before continuing (which could otherwise corrupt a ZFS pool even more and make the pool unrecoverable if something were to be wrong). I'm not sure if this is the main bottleneck when opening a ZFS pool during boot, but I wouldn't be surprised if it were. And if it is, personally I'm willing to sacrifice some boot time for some assurances that my data will be safer, but of course, not everyone's priorities are the same as mine :) (Also, for the record, for me the pool import/opening process is nearly instantaneous on SSDs/NVMe and takes 2.4 seconds on a 12-year-old Intel Atom machine with a single HDD pool).

dilliholla77 3y ago

> I've used btrfs in the past, and between my negative experiences and the repeated anecdotes posted all over the internet to this day I'm hesitant to give it another go. What are those issues? I've not followed the conversation, but I've been running btrfs for over a decade on multiple machines with different usage patterns without any issues. I'm genuinely interested. Are those substantiated claims? Or just the general noisy rants you can find about essentially anything people have opinions about?

Filligree 3y ago
In my own case: Btrfs performance got absolutely terrible once I was up to a few thousand snapshots. Never had it lose data, but I also never ran it long enough to lose a disk.
- unqueued 3y ago
  This is an example of how btrfs is poorly handled by many userland tools by default. You should never ever need to have thousands of snapshots. I use btrbk to manage my snapshots. It keeps my snapshots pruned to 300, pruning intermediate snapshots, so I can still easily step back 52 weeks, or I can purge them if I want to improve performance and reclaim space. While at the same time, every single snapshot is sent to two compressed encrypted backup images over ssh. The diff algorithm is pretty good, but for some types of files that fragment frequently, it is best to put them on their own subvolumes. My places.sqlite files from my browser testing profiles were adding tens or hundreds of MiB to snapshots, so they went in a seperate subvolume. I don't think the zfs would perform any better. I don't blame people for having bad experiences with btrfs, it is very inconsistently documented. I think it is especially annoying how many userland tools keep every single snapshot as a subvolume of the root, so that you have tens of millions of read-only files stored in /.snapshots. Awful.
  - seanw444 3y ago
    My setup is that I only have subvolumes for root, and (for now) my Doom Emacs config. Whenever I do a system upgrade, or a Doom Emacs upgrade, I manually take a snapshot of the appropriate subvolume first, just in case anything goes wrong. And then I only keep a max of 7 or so snapshots before I go through and delete the old ones. Has worked well for me so far and saved my butt a few times, and since I don't have a bunch of subvolumes to manage, I'm not overwhelmed. It's nice and simple, and I need nothing other than the default `btrfs` CLI utility.

ericbarrett 3y ago

I've been a storage engineer since 2000. The industry rule of thumb for file systems is to give them ten years after mainline release to bake. And the longer I've done this, the more I trust it.

ilc 3y ago
Mainline release in the kernel or actually be used by distros? 10 seems a bit short ;)
- ugjka 3y ago
  i switched back from btrfs to ext4 because i still had problems with it

kevincox 3y ago

I think it could be a very good replacement. In my opinion ZFS is still very static and disk/block based with some traditional RAID modes on top. It can provide the same redundancy and safety but with tons more flexibility. For example having different redundancy levels using the same pool of disks and changing redundancy levels on the fly. It does feel like a huge step in in terms of "here are my disks" and then "please store this data this way". bcachefs then just does it in a reasonable and efficient way. Way better than having to set up all of your disk RAID configs upfront. That being said it definitely still needs stability work. I tried it and it corrupted itself. So I love the idea, the design seems good overall. But it is going to need lots of testing and tweaking to become reliable. However it does seem like a solid addition to the filesystem space and I think being upstream will allow more people to work on it, avoid extra maintenance burden by staying in sync with changes and get the attention it deserves. I wrote more about why I like the design and how it corrupted for me here: https://kevincox.ca/2023/06/10/bcachefs-attempt/ https://kevincox.ca/2023/06/10/bcachefs-attempt/

KolenCh 3y ago

ZFS on Linux will not goes away. It is just that Ubuntu dropping it from the installer. You can still install it: https://launchpad.net/ubuntu/+source/zfs-linux/ https://launchpad.net/ubuntu/+source/zfs-linux/ And you can install it on a lot of distro too: https://openzfs.github.io/openzfs-docs/Getting%20Started/index.html https://openzfs.github.io/openzfs-docs/Getting%20Started/ind... My current favorite is to install kABI-tracking kmod on RHEL. (But their kABI-tracking is not a guarantee as they uses more.) In fact I move away from Ubuntu knowing that they knowingly ship a ZFS version with file corruption bug, which is introduced by their own patch of OpenZFS. So if OpenZFS is serving you well, don't worry about the Ubuntu situation because there's plenty better options.