3 ms·

Were you able to report this? If you can get me enough information (e.g. via bcachefs dump), I'd like to track this down.

Filligree 3y ago
We talked on IRC. :-) I'm planning to do that, but it'll have to be after my vacation; don't want to be running bcachefs on a remote server I can't fix. For those following along, this sort of response is why I'm optimistic overall. And bcachefs was dramatically higher performance than ZFS, although the lack of an ARC is a particular pain point. === It's going to be an interesting bug, I'm pretty sure. It affected two sets of files: The Steam Linux runtime (Solder), and everything in /nix/store, but absolutely nothing else. Actually, I think it's probably a bug in the fsck routine; everything worked perfectly until I rebooted.
- vladvasiliu 3y ago
  > And bcachefs was dramatically higher performance than ZFS Would you care to go into some specifics? On my middle of the road machines, the limiting factor has always been the drive. Encrypted ZFS has the same transfer speeds as unencrypted ext4, as tested with fio. My best machine is a zen 3 u laptop, so pcie 3 with a samsung 990 nvme drive. It reads and writes at over 2 GB/s.
  - bcrl 3y ago
    2GB/s is pretty low performance for storage these days. =-)
  - Filligree 3y ago
    Caching, mostly. My desktop has two HDDs and one NVMe. The NVMe isn't nearly that fast -- it's a 970 Evo which tops out at 1.6 GB/s -- but it's still much faster than the HDDs. When I write data to the setup, bcachefs and ZFS had very different behaviour. ZFS: - Writes to the HDDs first. - Blocks further writes until the first ones are on disk, unless I use sync=disabled. (Which I do; it's safe for my use-case.) - Is in any case limited by how much can be buffered in RAM. - Doesn't write to the SSD (which is configured as cache) unless it first reads data into RAM and then that data gets dropped from the in-core ARC. BcacheFS: - Writes to the SSD first. - Writes to the HDDs at its leisure. - Can be told the NVMe is more reliable than the HDDs. - Can be configured to leave data uncompressed on the SSD (because even LZ4 is a notable drain at 1.6 GB/s), but use zstd for the HDDs (because it's async so why not, and decompressing zstd at 300 MB/s is no issue). - Uses LRU instead of ARC, so if your working set is larger than your cache ZFS will have much better behaviour... once it's warmed up. The bug I hit is apparently fixed now, so I'm likely to switch back to bcachefs more or less ASAP. My working set is not in fact smaller than my SSD... but I'll just get a second SSD, I think.
    - koverstreet 3y ago
      I don't think it's fixed yet? biggest bugfix in the pipeline is that bfoster just figured out the 'missing backpointer' bug, but that's just in the btree write buffer, it wouldn't cause data corruption - and the fix for that isn't in yet
- koverstreet 3y ago
  I thought we might have :) If it's a bug in fsck that will be fortunate, that'll be easy to track down.

Filligree 3y ago

We talked on IRC. :-) I'm planning to do that, but it'll have to be after my vacation; don't want to be running bcachefs on a remote server I can't fix. For those following along, this sort of response is why I'm optimistic overall. And bcachefs was dramatically higher performance than ZFS, although the lack of an ARC is a particular pain point. === It's going to be an interesting bug, I'm pretty sure. It affected two sets of files: The Steam Linux runtime (Solder), and everything in /nix/store, but absolutely nothing else. Actually, I think it's probably a bug in the fsck routine; everything worked perfectly until I rebooted.

vladvasiliu 3y ago
> And bcachefs was dramatically higher performance than ZFS Would you care to go into some specifics? On my middle of the road machines, the limiting factor has always been the drive. Encrypted ZFS has the same transfer speeds as unencrypted ext4, as tested with fio. My best machine is a zen 3 u laptop, so pcie 3 with a samsung 990 nvme drive. It reads and writes at over 2 GB/s.
- bcrl 3y ago
  2GB/s is pretty low performance for storage these days. =-)
- Filligree 3y ago
  Caching, mostly. My desktop has two HDDs and one NVMe. The NVMe isn't nearly that fast -- it's a 970 Evo which tops out at 1.6 GB/s -- but it's still much faster than the HDDs. When I write data to the setup, bcachefs and ZFS had very different behaviour. ZFS: - Writes to the HDDs first. - Blocks further writes until the first ones are on disk, unless I use sync=disabled. (Which I do; it's safe for my use-case.) - Is in any case limited by how much can be buffered in RAM. - Doesn't write to the SSD (which is configured as cache) unless it first reads data into RAM and then that data gets dropped from the in-core ARC. BcacheFS: - Writes to the SSD first. - Writes to the HDDs at its leisure. - Can be told the NVMe is more reliable than the HDDs. - Can be configured to leave data uncompressed on the SSD (because even LZ4 is a notable drain at 1.6 GB/s), but use zstd for the HDDs (because it's async so why not, and decompressing zstd at 300 MB/s is no issue). - Uses LRU instead of ARC, so if your working set is larger than your cache ZFS will have much better behaviour... once it's warmed up. The bug I hit is apparently fixed now, so I'm likely to switch back to bcachefs more or less ASAP. My working set is not in fact smaller than my SSD... but I'll just get a second SSD, I think.
  - koverstreet 3y ago
    I don't think it's fixed yet? biggest bugfix in the pipeline is that bfoster just figured out the 'missing backpointer' bug, but that's just in the btree write buffer, it wouldn't cause data corruption - and the fix for that isn't in yet
koverstreet 3y ago
I thought we might have :) If it's a bug in fsck that will be fortunate, that'll be easy to track down.