5 ms·

RIP BCacheFS. I was hopeful I could finally have a modern filesystem in Linux mainlined (I don't trust Btrfs anymore), but I guess I'll keep on having to instal

RIP BCacheFS. I was hopeful I could finally have a modern filesystem in Linux mainlined (I don't trust Btrfs anymore), but I guess I'll keep on having to install ZFS for the foreseeable future I guess. As I predicted, out of tree bcachefs is basically dead on arrival - everybody interested is already on ZFS, btrfs is still around only because ZFS can't be mainlined basically

Ygg2 1y ago
Wait. You don't trust Btrfs but you would trust BCacheFS, that's obviously very experimental?
- rurban 1y ago
  Still more stable than btrfs. btrfs is also dead slow
  - Iridiumkoivu 1y ago
    I agree with this sentiment. Btrfs has destroyed itself on my testing/lab machines three times during last two years up to point where recovery wasn’t possible. Metadata corruption being main issue (or that’s how it looks like to me at least). As of now I trust BCacheFS way more. I’ve given it roughly the same time to prove itself as Btrfs too. BCacheFS has issues but so far I’ve managed to resolve them without major data loss. Please note that I currently use ext4 in all ”really important” desktop/laptop installations and OpenZFS in my server. Performance being the main concern for desktop and reliability for server.
    - bcrl 1y ago
      ext4 has its own issues usually more in terms of scalability. XFS is trustworthy without the glass jaws ext4 has all over the place. Ah, the joy of taking 80 seconds to write out an 8MB file to disk after a fresh mount....
- phire 1y ago
  Btrfs claims to be stable. IMO, it's not. It's generally fine if you stay on the happy path. It will work for 99% of people. But if you fall off that happy path, bad things might happen and nobody is surprised. In my personal experience, nobody associated with the project seems to trust a btrfs filesystem that fell off the happy path, and they strongly recommend you delete it and start from scratch. I was horrified to discover that they don't trust fsck to actually fix a btrfs filesystem into a canonical state. BCacheFS had the massive advantage that it knew it was experimental and embraced it. It took measures to keep data integrity despite the chaos, generally seems to be a better design and has a more trustworthy fsck. It's not that I'd trust BCacheFS, it's still not quite there (even ignoring project management issues). But my trust for Btrfs is just so much lower.
  - ahartmetz 1y ago
    btrfs seems to be a wonky, ill-considered design with ten years of hotfixes. bcachefs seems to be a solid design that is (or has been, it's mostly done) regularly improved where trouble was found. Now it's just fixing basically little coding oversights. In two years, I will trust bcachefs to be a much more reliable filesystem than btrfs.
kiney 1y ago
btrfs has many technical advantages over zfs
- debazel 1y ago
  Yes, like destroying itself and losing all data.
  - natebc 1y ago
    ZFS is perfectly capable of this too. source: worked as a support engineer for a block storage company, witnessed hundreds of customers blowing one or both of their feet off with ZFS.
    - zenoprax 1y ago
      To what extent are these customers blaming the hammer for hitting their thumb? (Legitimate question: I manage several PB with ZFS and would like to know where I should be more cautious.)
      - nubinetwork 1y ago
        Pool feature mismatch on send receive, dedup send receive, new features breaking randomly on bleeding edge releases
        TheNewsIsHere 1y ago
        The intent of feature flags in ZFS is to denote changes in on-disk structures. Replication isn’t supported between pools that don’t support the same flags because otherwise ZFS couldn’t read the data from disk properly on the receiving sides. There are workarounds, with their respective caveats and warnings.
      - natebc 1y ago
        A great deal. Which is why my cringe reflex still activates when I read about people running ZFS in places that aren't super tightly configured. ZFS is just such a massively complex piece of software. There were legitimate bugs in ZFS that we hit. Mostly around ZIL/SLOG and L2ARC and the umpteen million knobs that one can tweak.
sureglymop 1y ago
I've never had any issues with either ZFS or Btrfs after 2020. I wonder what you all are doing to have such issues with them.
- jamesnorden 1y ago
  Ah yes, the famous "holding it wrong".
  - happymellon 1y ago
    I've also not had issues with BTRFS. The question was around usage, because without knowing people's usecases and configurations it'll never be usable for you while working fine for others.
    - pizza234 1y ago
      If 1% of the users report a given issue (say, data corruption), the fact that 99% of the users report that they don't experience it, doesn't mean that the issue is not critical.
      - izacus 1y ago
        The fact that you see an issue reported loudly on social media it doesn't mean it's critical or more common than for other FSes. As usual with all these Linux debates, there's a loud group grinding their old hatreds that can be decade old.
      - happymellon 1y ago
        > If 1% of the users report a given issue (say, data corruption If 0.1% of users say it corrupted for them, and then don't provide any further details and no one can replicate their scenario then it does make it hard to resolve it
        koverstreet 1y ago
        the btrfs devs are also famous for being unresponsive to these sorts of issues. there's a feedback effect: if users know that a filesystem takes these kinds of issues seriously and will drop what they're doing and jump on them, a lot of users will very happily spend the time reporting bugs and working with devs to get it resolved. people don't like wasting their time on bug reports that go into the void. they do like contributing their time when they know it's going to get their issue fixed and make things better for everyone. this is why I regularly tell people "FEED ME YOUR BUG REPORTS! I WANT THEM ALL!" it's just what you have to do if you want your code to be truly bulletproof.
StopDisinfo910 1y ago
> btrfs is still around only because ZFS can't be mainlined basically ZFS is extremely annoying with the way it does extend and the fact that you can’t mismatch drive size. It’s not a panacea. There clearly is space for an improved design.
- nubinetwork 1y ago
  Underprivision your disks, then you don't have to worry about those edge cases...
  - StopDisinfo910 1y ago
    If you need to consider how to buy your drives so you can use a filesystem, that’s a flaw of said filesystem not an edge case. It clearly is an acceptable one for a lot of people but it does leave space for alternative designs.
    - estimator7292 1y ago
      This is the way it's always been. RAID can't really handle mismatched drives either and you must consider that when purchasing drives. It's not a flaw, it's a consequence of geometry
      - StopDisinfo910 1y ago
        It’s strange you say that because my btrfs array handles mismatched sizes just fine.
        Elhana 1y ago
        Let's say you can put a hdd and ssd of different sizes in raid1, but doesnt mean you should.
- cyphar 1y ago
  This is being worked on (they call it AnyRaid), the work is being sponsored by HexOS[1]. [1]: https://hexos.com/blog/introducing-zfs-anyraid-sponsored-by-eshtek https://hexos.com/blog/introducing-zfs-anyraid-sponsored-by-...
accelbred 1y ago
I switched to ZFS for a while but had to switch back because of how much was broken. Overlayfs had issues, reflinks didn't work, etc. Linux-specific stuff that just works on kernel filesystems was missing or buggy. I saw later they added support for some of the missing features but they had data corruption issues. Also I doubt it'll ever support fs-verity. I don't plan on giving ZFS or other filesystems not designed for Linux another go.
EspadaV9 1y ago
Might be harder to keep running ZFS on Linux after 6.18 https://www.phoronix.com/news/Linux-6.18-write-cache-pages https://www.phoronix.com/news/Linux-6.18-write-cache-pages
- qalmakka 1y ago
  Killing ZFS on Linux would basically make Linux unsuitable for lots of usecases. What would you use instead? Btrfs, which keeps having stupid data corruption issues? Bcachefs, which is not yet stable and now it's being struck out of the kernel? LVM2+thin provisioning, which will happily eat your data if your data overlap? I hope some industrial players will force the kernel to drop this nonsense. Heck no native filesystem besides btrfs has compression, I'm saving HUNDREDS of GB with zstd compression on my machines with basically zero overhead
koverstreet 1y ago
The community is still growing (developers too!), and people have been jumping in to help out with getting DKMS support into the distros. bcachefs isn't going away. The SuSE guy also reversed himself after I asked; Debian too, so we have time to get the DKMS packages out.

Ygg2 1y ago

Wait. You don't trust Btrfs but you would trust BCacheFS, that's obviously very experimental?

rurban 1y ago
Still more stable than btrfs. btrfs is also dead slow
- Iridiumkoivu 1y ago
  I agree with this sentiment. Btrfs has destroyed itself on my testing/lab machines three times during last two years up to point where recovery wasn’t possible. Metadata corruption being main issue (or that’s how it looks like to me at least). As of now I trust BCacheFS way more. I’ve given it roughly the same time to prove itself as Btrfs too. BCacheFS has issues but so far I’ve managed to resolve them without major data loss. Please note that I currently use ext4 in all ”really important” desktop/laptop installations and OpenZFS in my server. Performance being the main concern for desktop and reliability for server.
  - bcrl 1y ago
    ext4 has its own issues usually more in terms of scalability. XFS is trustworthy without the glass jaws ext4 has all over the place. Ah, the joy of taking 80 seconds to write out an 8MB file to disk after a fresh mount....
phire 1y ago
Btrfs claims to be stable. IMO, it's not. It's generally fine if you stay on the happy path. It will work for 99% of people. But if you fall off that happy path, bad things might happen and nobody is surprised. In my personal experience, nobody associated with the project seems to trust a btrfs filesystem that fell off the happy path, and they strongly recommend you delete it and start from scratch. I was horrified to discover that they don't trust fsck to actually fix a btrfs filesystem into a canonical state. BCacheFS had the massive advantage that it knew it was experimental and embraced it. It took measures to keep data integrity despite the chaos, generally seems to be a better design and has a more trustworthy fsck. It's not that I'd trust BCacheFS, it's still not quite there (even ignoring project management issues). But my trust for Btrfs is just so much lower.
- ahartmetz 1y ago
  btrfs seems to be a wonky, ill-considered design with ten years of hotfixes. bcachefs seems to be a solid design that is (or has been, it's mostly done) regularly improved where trouble was found. Now it's just fixing basically little coding oversights. In two years, I will trust bcachefs to be a much more reliable filesystem than btrfs.

kiney 1y ago

btrfs has many technical advantages over zfs

debazel 1y ago
Yes, like destroying itself and losing all data.
- natebc 1y ago
  ZFS is perfectly capable of this too. source: worked as a support engineer for a block storage company, witnessed hundreds of customers blowing one or both of their feet off with ZFS.
  - zenoprax 1y ago
    To what extent are these customers blaming the hammer for hitting their thumb? (Legitimate question: I manage several PB with ZFS and would like to know where I should be more cautious.)
    - nubinetwork 1y ago
      Pool feature mismatch on send receive, dedup send receive, new features breaking randomly on bleeding edge releases
      - TheNewsIsHere 1y ago
        The intent of feature flags in ZFS is to denote changes in on-disk structures. Replication isn’t supported between pools that don’t support the same flags because otherwise ZFS couldn’t read the data from disk properly on the receiving sides. There are workarounds, with their respective caveats and warnings.
    - natebc 1y ago
      A great deal. Which is why my cringe reflex still activates when I read about people running ZFS in places that aren't super tightly configured. ZFS is just such a massively complex piece of software. There were legitimate bugs in ZFS that we hit. Mostly around ZIL/SLOG and L2ARC and the umpteen million knobs that one can tweak.

sureglymop 1y ago

I've never had any issues with either ZFS or Btrfs after 2020. I wonder what you all are doing to have such issues with them.

jamesnorden 1y ago
Ah yes, the famous "holding it wrong".
- happymellon 1y ago
  I've also not had issues with BTRFS. The question was around usage, because without knowing people's usecases and configurations it'll never be usable for you while working fine for others.
  - pizza234 1y ago
    If 1% of the users report a given issue (say, data corruption), the fact that 99% of the users report that they don't experience it, doesn't mean that the issue is not critical.
    - izacus 1y ago
      The fact that you see an issue reported loudly on social media it doesn't mean it's critical or more common than for other FSes. As usual with all these Linux debates, there's a loud group grinding their old hatreds that can be decade old.
    - happymellon 1y ago
      > If 1% of the users report a given issue (say, data corruption If 0.1% of users say it corrupted for them, and then don't provide any further details and no one can replicate their scenario then it does make it hard to resolve it
      - koverstreet 1y ago
        the btrfs devs are also famous for being unresponsive to these sorts of issues. there's a feedback effect: if users know that a filesystem takes these kinds of issues seriously and will drop what they're doing and jump on them, a lot of users will very happily spend the time reporting bugs and working with devs to get it resolved. people don't like wasting their time on bug reports that go into the void. they do like contributing their time when they know it's going to get their issue fixed and make things better for everyone. this is why I regularly tell people "FEED ME YOUR BUG REPORTS! I WANT THEM ALL!" it's just what you have to do if you want your code to be truly bulletproof.

StopDisinfo910 1y ago

> btrfs is still around only because ZFS can't be mainlined basically ZFS is extremely annoying with the way it does extend and the fact that you can’t mismatch drive size. It’s not a panacea. There clearly is space for an improved design.

nubinetwork 1y ago
Underprivision your disks, then you don't have to worry about those edge cases...
- StopDisinfo910 1y ago
  If you need to consider how to buy your drives so you can use a filesystem, that’s a flaw of said filesystem not an edge case. It clearly is an acceptable one for a lot of people but it does leave space for alternative designs.
  - estimator7292 1y ago
    This is the way it's always been. RAID can't really handle mismatched drives either and you must consider that when purchasing drives. It's not a flaw, it's a consequence of geometry
    - StopDisinfo910 1y ago
      It’s strange you say that because my btrfs array handles mismatched sizes just fine.
      - Elhana 1y ago
        Let's say you can put a hdd and ssd of different sizes in raid1, but doesnt mean you should.
cyphar 1y ago
This is being worked on (they call it AnyRaid), the work is being sponsored by HexOS[1]. [1]: https://hexos.com/blog/introducing-zfs-anyraid-sponsored-by-eshtek https://hexos.com/blog/introducing-zfs-anyraid-sponsored-by-...

accelbred 1y ago

I switched to ZFS for a while but had to switch back because of how much was broken. Overlayfs had issues, reflinks didn't work, etc. Linux-specific stuff that just works on kernel filesystems was missing or buggy. I saw later they added support for some of the missing features but they had data corruption issues. Also I doubt it'll ever support fs-verity. I don't plan on giving ZFS or other filesystems not designed for Linux another go.

EspadaV9 1y ago

Might be harder to keep running ZFS on Linux after 6.18 https://www.phoronix.com/news/Linux-6.18-write-cache-pages https://www.phoronix.com/news/Linux-6.18-write-cache-pages

qalmakka 1y ago
Killing ZFS on Linux would basically make Linux unsuitable for lots of usecases. What would you use instead? Btrfs, which keeps having stupid data corruption issues? Bcachefs, which is not yet stable and now it's being struck out of the kernel? LVM2+thin provisioning, which will happily eat your data if your data overlap? I hope some industrial players will force the kernel to drop this nonsense. Heck no native filesystem besides btrfs has compression, I'm saving HUNDREDS of GB with zstd compression on my machines with basically zero overhead

koverstreet 1y ago

The community is still growing (developers too!), and people have been jumping in to help out with getting DKMS support into the distros. bcachefs isn't going away. The SuSE guy also reversed himself after I asked; Debian too, so we have time to get the DKMS packages out.