3 ms·

> And bcachefs was dramatically higher performance than ZFS Would you care to go into some specifics? On my middle of the road machines, the limiting factor h

> And bcachefs was dramatically higher performance than ZFS Would you care to go into some specifics? On my middle of the road machines, the limiting factor has always been the drive. Encrypted ZFS has the same transfer speeds as unencrypted ext4, as tested with fio. My best machine is a zen 3 u laptop, so pcie 3 with a samsung 990 nvme drive. It reads and writes at over 2 GB/s.

bcrl 3y ago
2GB/s is pretty low performance for storage these days. =-)
Filligree 3y ago
Caching, mostly. My desktop has two HDDs and one NVMe. The NVMe isn't nearly that fast -- it's a 970 Evo which tops out at 1.6 GB/s -- but it's still much faster than the HDDs. When I write data to the setup, bcachefs and ZFS had very different behaviour. ZFS: - Writes to the HDDs first. - Blocks further writes until the first ones are on disk, unless I use sync=disabled. (Which I do; it's safe for my use-case.) - Is in any case limited by how much can be buffered in RAM. - Doesn't write to the SSD (which is configured as cache) unless it first reads data into RAM and then that data gets dropped from the in-core ARC. BcacheFS: - Writes to the SSD first. - Writes to the HDDs at its leisure. - Can be told the NVMe is more reliable than the HDDs. - Can be configured to leave data uncompressed on the SSD (because even LZ4 is a notable drain at 1.6 GB/s), but use zstd for the HDDs (because it's async so why not, and decompressing zstd at 300 MB/s is no issue). - Uses LRU instead of ARC, so if your working set is larger than your cache ZFS will have much better behaviour... once it's warmed up. The bug I hit is apparently fixed now, so I'm likely to switch back to bcachefs more or less ASAP. My working set is not in fact smaller than my SSD... but I'll just get a second SSD, I think.
- koverstreet 3y ago
  I don't think it's fixed yet? biggest bugfix in the pipeline is that bfoster just figured out the 'missing backpointer' bug, but that's just in the btree write buffer, it wouldn't cause data corruption - and the fix for that isn't in yet

bcrl 3y ago

2GB/s is pretty low performance for storage these days. =-)

Filligree 3y ago

Caching, mostly. My desktop has two HDDs and one NVMe. The NVMe isn't nearly that fast -- it's a 970 Evo which tops out at 1.6 GB/s -- but it's still much faster than the HDDs. When I write data to the setup, bcachefs and ZFS had very different behaviour. ZFS: - Writes to the HDDs first. - Blocks further writes until the first ones are on disk, unless I use sync=disabled. (Which I do; it's safe for my use-case.) - Is in any case limited by how much can be buffered in RAM. - Doesn't write to the SSD (which is configured as cache) unless it first reads data into RAM and then that data gets dropped from the in-core ARC. BcacheFS: - Writes to the SSD first. - Writes to the HDDs at its leisure. - Can be told the NVMe is more reliable than the HDDs. - Can be configured to leave data uncompressed on the SSD (because even LZ4 is a notable drain at 1.6 GB/s), but use zstd for the HDDs (because it's async so why not, and decompressing zstd at 300 MB/s is no issue). - Uses LRU instead of ARC, so if your working set is larger than your cache ZFS will have much better behaviour... once it's warmed up. The bug I hit is apparently fixed now, so I'm likely to switch back to bcachefs more or less ASAP. My working set is not in fact smaller than my SSD... but I'll just get a second SSD, I think.

koverstreet 3y ago
I don't think it's fixed yet? biggest bugfix in the pipeline is that bfoster just figured out the 'missing backpointer' bug, but that's just in the btree write buffer, it wouldn't cause data corruption - and the fix for that isn't in yet