7 ms·

The "why not btrfs" line boils down to "it took a long time to be stable". That's a weird argument. Even if it's true, it is now stable, and has been for a lon

The "why not btrfs" line boils down to "it took a long time to be stable". That's a weird argument. Even if it's true, it is now stable, and has been for a long time. btrfs has long been my default, and I'd be wary of switching to something newer just because someone was mad that development took a long time.

vbezhenar 2y ago
Btrfs lost its credibility and many people would never trust it.
- greenavocado 2y ago
  As little as one year ago I experienced damage on a lightly used btrfs root partition on my laptop. Never again. I use ext4 root and ZFS for /home for snapshots and transparent compression now, all on top of LVM
- BSDobelix 2y ago
  So a year ago i tried to repeat my old trick damaging btrfs (as a user NOT root). Fill the volume with dd if=/dev/urandom of=./file bs=2M && sync && rm ./file then reboot the machine and yes it still works, it's not booting anymore, bravo. BTW: Even SLES SuseLinux Enterprise says use XFS for data btrfs just for the OS i wonder why
  - PlutoIsAPlanet 2y ago
    > BTW: Even SLES SuseLinux Enterprise says use XFS for data btrfs just for the OS i wonder why Because XFS is far quicker for server-related software such as databases and virtual machines, which are weak points on btrfs due to its COW model.
    - deleted 2y ago
      [deleted]
    - BSDobelix 2y ago
      Yeah and maybe additionally you want to keep your data and have a stable filesystem for them ;)
    - josephcsible 2y ago
      Doesn't "chattr +C" give you back that performance, while still letting you keep the rest of the benefits of Btrfs?
      - curt15 2y ago
        Nodatacow is an ugly hack because it disables btrfs's core features for the affected data. It also should not be used with raid1.
  - yjftsjthsd-h 2y ago
    > So a year ago i tried to repeat my old trick damaging btrfs (as a user NOT root). Fill the volume with dd if=/dev/urandom of=./file bs=2M && sync && rm ./file then reboot the machine and yes it still works, it's not booting anymore, bravo. Do you know how ZFS handles that?
    - BSDobelix 2y ago
      Without any problems. No other Filesystem i tested has that problem (ext4, XFS, ZFS, NTFS, JFS, Nilfs2)
      - yjftsjthsd-h 2y ago
        Good to hear:) My understanding is that it's easier to break a CoW filesystem like that because if you run out of space you can't even delete things (because that requires writing that change), so I'm not surprised that the rest (the non-CoW filesystems) did fine, but I'm happy to hear that ZFS also handles it.
eptcyka 2y ago
In 2019, btrfs ate all my data after a power cut. Btrfs peeps said it sounded like my SSD was at fault. Well, ZFS is still chugging along on that drive. I am not surprised btrfs took ages to stabilize, and it will take ages again before I rely on it. I’ve had previous btrfs incidents too. I think the argument against btrfs is that it was not good enough when btrfs devs told people to use it in production for ages.
- greenavocado 2y ago
  Btrfs never actually stabilized it's still garbage compared to ZFS
  - leansensei 2y ago
    Care to substantiate that statement? It seems rather arbitrary to just say that it's garbage when it is running and has been running successfully for the vast majority of its users. It also offers two features that ZFS does not: the ability to grow a pool, and offline duplication.
    - swinglock 2y ago
      Does it even have RAID5?
      - ZhongXina 2y ago
        Why should it matter? It's an extremely niche technology that's only interesting to some home users. I see no reasons why other users should care about a RAID level they're not interested in. (I don't use btrfs or any other COW filesystem because of significantly worse performance with some kinds of workloads, but it has nothing to do with maturity of any of them.)
        throw0101c 2y ago
        > Why should it matter? It's an extremely niche technology that's only interesting to some home users. I use RAID-Z2 in lots of places for bulk storages purposes (HPC). There's a reason why Ceph added erasure coding: * https://ceph.io/en/news/blog/2017/new-luminous-erasure-coding-rbd-cephfs/ https://ceph.io/en/news/blog/2017/new-luminous-erasure-codin... * https://docs.ceph.com/en/latest/rados/operations/erasure-code/ https://docs.ceph.com/en/latest/rados/operations/erasure-cod... When you're talking about PB of data, storage efficiencies add up.
bheadmaster 2y ago
Development taking long usually means that the model itself is too complicated to be done right in a reasonable time, which indicates that the "stable" implementation could still be buggy, but only if you stray away from the common path. It's hard to feel comfortable using such a software in a fundamental role such a file system.
- 7e 2y ago
  In this case I think it’s the case that bcachefs has only a very small set of developers working in it.
  - jeltz 2y ago
    But that was not the case for btrfs.
- humanfromearth9 2y ago
  I think it's not quite so simple. The problem of organising storage is at least complex, on a scale of "simple complicated complex chaotic". The inherent complexity might be impossible to reduce to something simple or even just complicated, except _maybe_ with layering (à la LVM2), each layer tackling one issue independently of the others. But then it's probably at the cost of performance and other efficiency. Each layer should work such that it does not interfere too much with the performance of other layers. Not easy. Given the rather cheap price of durable storage these days, I would favour rock solid, high quality code for storing my data, at the expense of some optimisations. Then again, I still like RAID, instantaneous snapshots, COW, encryption, xattr, resizable partitions, CRC... It's it possible to have all this with acceptable performance and simple code bricks combined and layered on top of each other?
- dspillett 2y ago
  > which indicates that the "stable" implementation could still be buggy, but only if you stray away from the common path Or that the complexity is such that if a new bug is found, it may take a long time to be fixed because of the complexity, or it is fixed fast and has unexpected knock-on effects even for circumstances on the common path. Something that takes a long time to be declared stable/reliable because of its complexity, needs to spend a long time after that declaration without significant issues before I'll actually trust it. Things like btrfs definitely live in this category. bcachefs even won't be something I use for important storage until it has been battle-tested a bit more for a bit longer, though at this point it is much more likely to take over from my current simple ext4-on-RAID arrangement (and when/if it does, my backups might stay on ext4-on-RAID even longer).
rubenbe 2y ago
For me the killer feature of btrfs is "RAID 1 with different sized disks". For a small and cheap setup, this is perfect since a broken disk can be replaced with a bigger one and immediately (part of) the extra new disk space can be used. Other filesystems seem to only increase the size once all disks have been replaced with a bigger capacity one (last time I checked this was still the case for ZFS)
- frankjr 2y ago
  Exactly. Provisioning a completely different set of disks when running out of capacity might be fine for a company but not for home office.
- leansensei 2y ago
  That, plus offline deduplication.
- e12e 2y ago
  How does that work? You have two 100gb drives in raid1, 80% full, you replace one with a 200gb disk and write 50gb to the array - how is your 130gb of data protected against either drive failing?
  - rubenbe 2y ago
    It only works with 3+ disks. All data needs to be on two disks. e.g. you have 3 100GB drives, total capacity in raid 1 is 150GB. If you replace a broken one with a 200GB one, the total capacity will be increased to 200GB.
    - assbuttbuttass 2y ago
      My understanding is that RAID1 is just a mirror, and all disks have identical contents. Are you talking about something else?
      - tuetuopay 2y ago
        Traditional RAID1 will mirror whole drives, yes. BTRFS RAID1 will mirror chunks of data (iirc 1GB) on two drives. So you can have e.g. two 1TB drive and a 2TB one just fine.
GrayShade 2y ago
One interesting titbit I've only recently found out is that btrfs can't really serve reads from different drives in RAID1, it picks a drive based on the process id. ZFS does something smarter here, it keeps track of the queue length for each drive in a mirror, and picks the one with the lowest number of pending requests.
_flux 2y ago
My personal grievances with btrfs are multifaceted. - I never agreed with the btrfs default of root raid 1 system not booting up if a device is missing. I think the point of raid1 is to minimize downtime when losing a device and if you lose the other device before returning it to good state, that's 100% on you. - Poor management tools compared to md (though bcachefs might be in the same boat). Some tools are poorly thought, e.g. there is a tool for defragmentation, but it undoes sharing (so snapshots and dedupped files get expanded). - If a drive in raid1 drops but then later comes back, btrfs is still quite happy. - Need of using btrfs balance, and in a certain way as well: https://github.com/kdave/btrfsmaintenance/blob/master/btrfs-balance.sh https://github.com/kdave/btrfsmaintenance/blob/master/btrfs-... . - At least it used to be difficult to recover when your filesystem becomes full. Helps if you have it on LVM volume with extra space. - Snapshotting or having a clone of a btrfs volume is dangerous (due to the uuid-based volume participant scanning) - I believe raid5/6 is still experimental? - I've lost a filesystem to btrfs raid10 (but my backups are good). - I have also rendered my bcachefs in a state where I could no longer write to the filesystem, but I was still able to read it. So I'm inclined to keep using bcachefs for the time being. Overall I just have the impression that btrfs was complicated and ended up in a design dead-end, making improvements from hard to difficult, and I hope that bcachefs has made different base designs, making future improvements easier. Yes, the number of developers for bcachefs is smaller, but frankly as long as it's possible for a project to advance with a single developer, it is going to be the most effective way to go—at the same time I hope this situation improves in the future.
- viraptor 2y ago
  > I never agreed with the btrfs default of root raid 1 system not booting up if a device is missing. Add "degraded" to default mount options. Solved.
  - cherryteastain 2y ago
    Good luck doing that after the disk shuts down okay but never comes back online
    - 2y ago
qalmakka 2y ago
Btrfs is still unacceptably less reliable than ZFS, after _decades_ of development. This is unacceptable, IMHO. I've lost so much data due to btrfs corruption issues that I've (almost) stopped to use it completely nowadays. It's better to fight to keep the damned OpenZFS modules up to date and get an actual _reliable_ system instead of accepting the risk again.
- BSDobelix 2y ago
  >It's better to fight to keep the damned OpenZFS modules up to date and get an actual _reliable_ system Try CachyOS (or at least the ZFS-Kernel) it has excellent ZFS integration.
  - Volundr 2y ago
    This. I may still give up on running ZFS on Linux due to the common (seemingly intentional from the Linux side) breakage, but for my existing systems switching them over to CachyOS repos has been a blessed relief.
    - BSDobelix 2y ago
      Well i use mainly FreeBSD but have used CachyOS for about 3mo to have some systemd refresher :)
  - magicalhippo 2y ago
    Hadn't heard of CachyOS before, looks very nice! Was looking to move to Arch from KDE Neon, but this might be a much better fit.
    - BSDobelix 2y ago
      Well or don't move from arch and just use the cachyos repos: https://wiki.cachyos.org/de/cachyos_repositories/how_to_add_cachyos_repo/ https://wiki.cachyos.org/de/cachyos_repositories/how_to_add_... No reinstall needed ;)
pantalaimon 2y ago
btrfs has still many weird issues. e.g. you can't remove a drive if it has I/O errors, even if the rest of the array has still enough space to accompany the data. You can do a replace, but then you need to buy a new drive.
nialv7 2y ago
btrfs is not stable, at least not for me. it lost my data only a couple months ago. no power cut, no disk failure, data just gone.
linsomniac 2y ago
It's not simply that it took a long time to become stable; it's that during this time where it was unstable a lot of people got exposed to btrfs by having it lose data. Personally, I was one of those people. Very excited about the prospects of btrfs, switched several machines over to it to test, ended up with filesystem corruption and had to revert to ext. Now, whenever I peek at btrfs, I never see anything that's compelling over running ZFS, which I've run for close to 15+ years, and run hard, and have never had data loss. Even in the early days with zfs+fuse, where I could regularly crash the zfs fuse; the zfs+fuse developers quickly addressed every crash I ran into, once I put together a stress test.
112233 2y ago
> it is now stable, and has been for a long time. Is it really? I must have missed the news. Back when it was released completely raw as a default for many distros, there were fundamental design level issues (e.g. "unbound internal fragmentation" reported by Shishkin). Plus all the reports and personal experiences of getting and trying to recover exotically shaped bricks when volume fills to 100% (which could happen at any time with btrfs). Is it all good now? Where can I read about btrfs behaving robustly when no free space is left?

vbezhenar 2y ago

Btrfs lost its credibility and many people would never trust it.

greenavocado 2y ago
As little as one year ago I experienced damage on a lightly used btrfs root partition on my laptop. Never again. I use ext4 root and ZFS for /home for snapshots and transparent compression now, all on top of LVM
BSDobelix 2y ago
So a year ago i tried to repeat my old trick damaging btrfs (as a user NOT root). Fill the volume with dd if=/dev/urandom of=./file bs=2M && sync && rm ./file then reboot the machine and yes it still works, it's not booting anymore, bravo. BTW: Even SLES SuseLinux Enterprise says use XFS for data btrfs just for the OS i wonder why
- PlutoIsAPlanet 2y ago
  > BTW: Even SLES SuseLinux Enterprise says use XFS for data btrfs just for the OS i wonder why Because XFS is far quicker for server-related software such as databases and virtual machines, which are weak points on btrfs due to its COW model.
  - deleted 2y ago
    [deleted]
  - BSDobelix 2y ago
    Yeah and maybe additionally you want to keep your data and have a stable filesystem for them ;)
  - josephcsible 2y ago
    Doesn't "chattr +C" give you back that performance, while still letting you keep the rest of the benefits of Btrfs?
    - curt15 2y ago
      Nodatacow is an ugly hack because it disables btrfs's core features for the affected data. It also should not be used with raid1.
- yjftsjthsd-h 2y ago
  > So a year ago i tried to repeat my old trick damaging btrfs (as a user NOT root). Fill the volume with dd if=/dev/urandom of=./file bs=2M && sync && rm ./file then reboot the machine and yes it still works, it's not booting anymore, bravo. Do you know how ZFS handles that?
  - BSDobelix 2y ago
    Without any problems. No other Filesystem i tested has that problem (ext4, XFS, ZFS, NTFS, JFS, Nilfs2)
    - yjftsjthsd-h 2y ago
      Good to hear:) My understanding is that it's easier to break a CoW filesystem like that because if you run out of space you can't even delete things (because that requires writing that change), so I'm not surprised that the rest (the non-CoW filesystems) did fine, but I'm happy to hear that ZFS also handles it.

eptcyka 2y ago

In 2019, btrfs ate all my data after a power cut. Btrfs peeps said it sounded like my SSD was at fault. Well, ZFS is still chugging along on that drive. I am not surprised btrfs took ages to stabilize, and it will take ages again before I rely on it. I’ve had previous btrfs incidents too. I think the argument against btrfs is that it was not good enough when btrfs devs told people to use it in production for ages.

greenavocado 2y ago
Btrfs never actually stabilized it's still garbage compared to ZFS
- leansensei 2y ago
  Care to substantiate that statement? It seems rather arbitrary to just say that it's garbage when it is running and has been running successfully for the vast majority of its users. It also offers two features that ZFS does not: the ability to grow a pool, and offline duplication.
  - swinglock 2y ago
    Does it even have RAID5?
    - ZhongXina 2y ago
      Why should it matter? It's an extremely niche technology that's only interesting to some home users. I see no reasons why other users should care about a RAID level they're not interested in. (I don't use btrfs or any other COW filesystem because of significantly worse performance with some kinds of workloads, but it has nothing to do with maturity of any of them.)
      - throw0101c 2y ago
        > Why should it matter? It's an extremely niche technology that's only interesting to some home users. I use RAID-Z2 in lots of places for bulk storages purposes (HPC). There's a reason why Ceph added erasure coding: * https://ceph.io/en/news/blog/2017/new-luminous-erasure-coding-rbd-cephfs/ https://ceph.io/en/news/blog/2017/new-luminous-erasure-codin... * https://docs.ceph.com/en/latest/rados/operations/erasure-code/ https://docs.ceph.com/en/latest/rados/operations/erasure-cod... When you're talking about PB of data, storage efficiencies add up.

bheadmaster 2y ago

Development taking long usually means that the model itself is too complicated to be done right in a reasonable time, which indicates that the "stable" implementation could still be buggy, but only if you stray away from the common path. It's hard to feel comfortable using such a software in a fundamental role such a file system.

7e 2y ago
In this case I think it’s the case that bcachefs has only a very small set of developers working in it.
- jeltz 2y ago
  But that was not the case for btrfs.
humanfromearth9 2y ago
I think it's not quite so simple. The problem of organising storage is at least complex, on a scale of "simple complicated complex chaotic". The inherent complexity might be impossible to reduce to something simple or even just complicated, except _maybe_ with layering (à la LVM2), each layer tackling one issue independently of the others. But then it's probably at the cost of performance and other efficiency. Each layer should work such that it does not interfere too much with the performance of other layers. Not easy. Given the rather cheap price of durable storage these days, I would favour rock solid, high quality code for storing my data, at the expense of some optimisations. Then again, I still like RAID, instantaneous snapshots, COW, encryption, xattr, resizable partitions, CRC... It's it possible to have all this with acceptable performance and simple code bricks combined and layered on top of each other?
dspillett 2y ago
> which indicates that the "stable" implementation could still be buggy, but only if you stray away from the common path Or that the complexity is such that if a new bug is found, it may take a long time to be fixed because of the complexity, or it is fixed fast and has unexpected knock-on effects even for circumstances on the common path. Something that takes a long time to be declared stable/reliable because of its complexity, needs to spend a long time after that declaration without significant issues before I'll actually trust it. Things like btrfs definitely live in this category. bcachefs even won't be something I use for important storage until it has been battle-tested a bit more for a bit longer, though at this point it is much more likely to take over from my current simple ext4-on-RAID arrangement (and when/if it does, my backups might stay on ext4-on-RAID even longer).

rubenbe 2y ago

For me the killer feature of btrfs is "RAID 1 with different sized disks". For a small and cheap setup, this is perfect since a broken disk can be replaced with a bigger one and immediately (part of) the extra new disk space can be used. Other filesystems seem to only increase the size once all disks have been replaced with a bigger capacity one (last time I checked this was still the case for ZFS)

frankjr 2y ago
Exactly. Provisioning a completely different set of disks when running out of capacity might be fine for a company but not for home office.
leansensei 2y ago
That, plus offline deduplication.
e12e 2y ago
How does that work? You have two 100gb drives in raid1, 80% full, you replace one with a 200gb disk and write 50gb to the array - how is your 130gb of data protected against either drive failing?
- rubenbe 2y ago
  It only works with 3+ disks. All data needs to be on two disks. e.g. you have 3 100GB drives, total capacity in raid 1 is 150GB. If you replace a broken one with a 200GB one, the total capacity will be increased to 200GB.
  - assbuttbuttass 2y ago
    My understanding is that RAID1 is just a mirror, and all disks have identical contents. Are you talking about something else?
    - tuetuopay 2y ago
      Traditional RAID1 will mirror whole drives, yes. BTRFS RAID1 will mirror chunks of data (iirc 1GB) on two drives. So you can have e.g. two 1TB drive and a 2TB one just fine.

GrayShade 2y ago

One interesting titbit I've only recently found out is that btrfs can't really serve reads from different drives in RAID1, it picks a drive based on the process id. ZFS does something smarter here, it keeps track of the queue length for each drive in a mirror, and picks the one with the lowest number of pending requests.

_flux 2y ago

My personal grievances with btrfs are multifaceted. - I never agreed with the btrfs default of root raid 1 system not booting up if a device is missing. I think the point of raid1 is to minimize downtime when losing a device and if you lose the other device before returning it to good state, that's 100% on you. - Poor management tools compared to md (though bcachefs might be in the same boat). Some tools are poorly thought, e.g. there is a tool for defragmentation, but it undoes sharing (so snapshots and dedupped files get expanded). - If a drive in raid1 drops but then later comes back, btrfs is still quite happy. - Need of using btrfs balance, and in a certain way as well: https://github.com/kdave/btrfsmaintenance/blob/master/btrfs-balance.sh https://github.com/kdave/btrfsmaintenance/blob/master/btrfs-... . - At least it used to be difficult to recover when your filesystem becomes full. Helps if you have it on LVM volume with extra space. - Snapshotting or having a clone of a btrfs volume is dangerous (due to the uuid-based volume participant scanning) - I believe raid5/6 is still experimental? - I've lost a filesystem to btrfs raid10 (but my backups are good). - I have also rendered my bcachefs in a state where I could no longer write to the filesystem, but I was still able to read it. So I'm inclined to keep using bcachefs for the time being. Overall I just have the impression that btrfs was complicated and ended up in a design dead-end, making improvements from hard to difficult, and I hope that bcachefs has made different base designs, making future improvements easier. Yes, the number of developers for bcachefs is smaller, but frankly as long as it's possible for a project to advance with a single developer, it is going to be the most effective way to go—at the same time I hope this situation improves in the future.

viraptor 2y ago
> I never agreed with the btrfs default of root raid 1 system not booting up if a device is missing. Add "degraded" to default mount options. Solved.
- cherryteastain 2y ago
  Good luck doing that after the disk shuts down okay but never comes back online
  - 2y ago

qalmakka 2y ago

Btrfs is still unacceptably less reliable than ZFS, after _decades_ of development. This is unacceptable, IMHO. I've lost so much data due to btrfs corruption issues that I've (almost) stopped to use it completely nowadays. It's better to fight to keep the damned OpenZFS modules up to date and get an actual _reliable_ system instead of accepting the risk again.

BSDobelix 2y ago
>It's better to fight to keep the damned OpenZFS modules up to date and get an actual _reliable_ system Try CachyOS (or at least the ZFS-Kernel) it has excellent ZFS integration.
- Volundr 2y ago
  This. I may still give up on running ZFS on Linux due to the common (seemingly intentional from the Linux side) breakage, but for my existing systems switching them over to CachyOS repos has been a blessed relief.
  - BSDobelix 2y ago
    Well i use mainly FreeBSD but have used CachyOS for about 3mo to have some systemd refresher :)
- magicalhippo 2y ago
  Hadn't heard of CachyOS before, looks very nice! Was looking to move to Arch from KDE Neon, but this might be a much better fit.
  - BSDobelix 2y ago
    Well or don't move from arch and just use the cachyos repos: https://wiki.cachyos.org/de/cachyos_repositories/how_to_add_cachyos_repo/ https://wiki.cachyos.org/de/cachyos_repositories/how_to_add_... No reinstall needed ;)

pantalaimon 2y ago

btrfs has still many weird issues. e.g. you can't remove a drive if it has I/O errors, even if the rest of the array has still enough space to accompany the data. You can do a replace, but then you need to buy a new drive.

nialv7 2y ago

btrfs is not stable, at least not for me. it lost my data only a couple months ago. no power cut, no disk failure, data just gone.

linsomniac 2y ago

It's not simply that it took a long time to become stable; it's that during this time where it was unstable a lot of people got exposed to btrfs by having it lose data. Personally, I was one of those people. Very excited about the prospects of btrfs, switched several machines over to it to test, ended up with filesystem corruption and had to revert to ext. Now, whenever I peek at btrfs, I never see anything that's compelling over running ZFS, which I've run for close to 15+ years, and run hard, and have never had data loss. Even in the early days with zfs+fuse, where I could regularly crash the zfs fuse; the zfs+fuse developers quickly addressed every crash I ran into, once I put together a stress test.

112233 2y ago

> it is now stable, and has been for a long time. Is it really? I must have missed the news. Back when it was released completely raw as a default for many distros, there were fundamental design level issues (e.g. "unbound internal fragmentation" reported by Shishkin). Plus all the reports and personal experiences of getting and trying to recover exotically shaped bricks when volume fills to 100% (which could happen at any time with btrfs). Is it all good now? Where can I read about btrfs behaving robustly when no free space is left?