6 ms·

> A lot of people that got burned with Btrfs How did they get burned? (I'm thinking of switching to Btrfs on my personal laptop (running Arch), mainly for the

> A lot of people that got burned with Btrfs How did they get burned? (I'm thinking of switching to Btrfs on my personal laptop (running Arch), mainly for the snapshots, subvolumes, checksums+deduplication, etc.)

techdragon 4y ago
Since Btrfs was arguably available in the kernel too soon. Some people who should have avoided a new filesystem for lots of reasons ranging from missing features that were in the plan but not implemented yet or in the earliest days not yet stable... many of these early experimenters developed and fostered a vocal belief that Btrfs was "bad" (there's too much to sum up honestly, you could write an entire decent length blog post on the history of Btrfs, its reception and its criticism) and this group has taken a long time to accept that Btrfs may be ok now. Theres still people with the perception that its not as good or ready enough and should not be used.
- blueflow 4y ago
  From the bcachefs homepage: > We prioritize robustness and reliability over features and hype: we make every effort to ensure you won't lose data. This is exactly what btrfs failed at, this is not just an issue with the code itself, but also with the developers. Why should i trust feature-oriented developers with writing my infrastructure code, where reliability and robustness are top priority?
- viraptor 4y ago
  All this, but also two specific cases: There's a list of features and readiness at https://btrfs.wiki.kernel.org/index.php/Status https://btrfs.wiki.kernel.org/index.php/Status - there were people who didn't check it and used unstable features and experienced issues. This is partially on the project for not making it more explicit. The second is that there are people who read that, see the unstable raid5/6, and are vocal about btrfs being overall broken / not ready rather than accepting that feature just not being available yet.
  - zozbot234 4y ago
    A file system that can eat your data is broken. If a feature is not ready for public consumption, it should not be built-in in the first place.
    - viraptor 4y ago
      Then every filesystem is broken by that definition. You're setting an impossible standard given how non-specialist software is developed. Here's a list of failures from zfs, including many kernel panics for example https://github.com/openzfs/zfs/labels/Type%3A%20Defect https://github.com/openzfs/zfs/labels/Type%3A%20Defect - is it broken and shouldn't have been released?
      - Beldin 4y ago
        Fine, let's be more nuanced: no filesystem should lose data if FAT16/FAT32/NTFS 3.1/EXT2/EXT3 wouldn't.
        vlowther 4y ago
        well, FAT16/FAT32/EXT2 are not journalled, so making them lose data in a giant variety of ways is a trivial affair. You might want a higher bar than that.
        Spivak 4y ago
        Yeah, when you get down to it the opinion really boils down to "the file system is broken iff it doesn't lose data for the use-cases I personally use it for."
        Beldin 4y ago
        That's the point: if file systems that basic would NOT lose data, then it is unacceptable if another file system does. Maybe to put it differently: new file systems should offer improvements wrt reliability for stable features of the file systems of the previous generation; regressions are unacceptable.
deleted 4y ago
[deleted]
oxplot 4y ago
> How did they get burned? In the early days after Fedora made it available, I lost my root twice after a kernel panic and later after a hard power reset. EDIT: in comparison, I've been using xfs for almost a decade now with numerous hard power offs and kernel panics. I've only ever lost select files that were open for write at the time of restart, and not the entire filesystem.
- warmwaffles 4y ago
  I've been running in danger mode of RAID6 on a 15 drive NAS here at my home for the better part of 5 years now, and I have had zero issues. The server is also running Arch and I run package updates every two weeks or so. Maybe I am just lucky.
- mekster 4y ago
  I wonder how Fedora started using btrfs as the default for their desktop version when RedHat had deprecated it years ago. Never used btrfs but I see corruption reports in here everytime I read about it, and it's been in development for over a decade and this thing needs to be put to rest before it eats more people's data. Hopefully bcachefs becomes feature parity and stability against zfs to provide a modern filesystem to wider Linux audience at last. Funny to see we're still using ext4 as the default filesystem which is nothing fancy forever.
  - dralley 4y ago
    > I wonder how Fedora started using btrfs as the default for their desktop version when RedHat had deprecated it years ago. Because the Fedora community wanted to, basically. They're not obligated to go along with whatever Red Hat wants to do, although there is naturally some alignment there. Plus having a different default filesystem really isn't that big of a deal. disclosure: I work for Red Hat.
    - mekster 4y ago
      Well, I always hear about data loss incidents whenever btrfs shows up in HN comments, so I thought making it a default in Fedora desktop sounded like a big deal as there must've been a good reason on why RH dropped it.
rodgerd 4y ago
The main thing I found btrfs was good for was testing the restore process of my backups. I was using it in supposedly known-good configurations (RAID 1 or RAID 10) and it consistently destroyed data. The fact that it turned out that the scrub function spent years corrupting data by mistake was only one of the many flaws with it.
EspadaV9 4y ago
I'd recommend listening to some of the 2.5 Admins podcasts. There are a few episodes (earlier on I think) where one of the hosts lists a number of issues with how it is implemented and basically wouldn't recommend it for anything, even in trivial set ups with a single drive there are far better alternative's.
yjftsjthsd-h 4y ago
I lost a root filesystem twice, on OpenSUSE in a one-disk system (i.e. what should have been the most stable well tested configuration), around 2018-19 IIRC. It now appears most likely that the system ran out of free space (suse ships snapper set to snapshot on every package operation, and the partition was smallish), and that apparently triggered corruption that ... at least, I couldn't figure out how to fix. Power loss may have been involved. Details fade from my memory, but the violent aversion to ever trusting it remains.
- chasil 4y ago
  It is actually wise to shrink a btrfs filesystem on its partition by a few megabytes, so some expansion room is available in an emergency. There are very few filesystems that can shrink like this: btrfs filesystem resize -1G /mount If the "btrfs fi df" command shows any of the data/system/metadata elements close to limits, it is time to take action. They can all push you into read-only mode.
- zepearl 4y ago
  I lost a btrfs-raid5-like NAS out of the blue (after 3-4 months of using it, I didn't even write a lot on it): the filesystem was always cleanly unmounted, I didn't do any crazy experiments with it, it had plenty (>40%, hundreds of GiBs) of free disk space; on one day I noticed that accessing some files was slower than usual, the noise of the HDDs had a weird pattern and from there it went downhill (at that time there was no fsck utility so I didn't even know what to do - if I'm not mistaken, BUT I MIGHT, the general opinion was that a fsck-util was not even needed? but again, I might have just dreamed this) and finally stopped working with a huge amount of error messages in the kernel logs => I had to reformat everything (I went back to mdadm-raid5) and restore from the secondary NAS' backup. In my opinion there were 2 main things that contributed to the bad reputation of the FS: 1) In those days there was a looot of hype about btrfs (as well for me it was a dream come true) which is the reason why I decided to use it. As far as I remember I didn't have to enable any special "experimental" feature in the kernel to use it => that gave me the impression that "it works under normal conditions, the DEVs are probably being a bit conservative for corner cases". 2) All functionalities (e.g. raid5 with rebalancing in my case) were all available from day 1 (at least when I started using it) so if something's available I tend to think that it works (I mean, from a user's point of view why make something available that does not work?) - but no, apparently many people had many problems with many different configurations/usage patterns. - As others have written I hope that Bcachefs will primarily focus on reliability and only secondarily on performance & features, but I admit that that's difficult to balance :). And I hope that the fsck-utility will be available from day1 (at least with a reduced functionality). And I hope that "experimental" features will need some kind of very explicit command line option (e.g. "-might_break_everything") to use them through userland utils. Go Kent!!! :) I'm since a while one of your Patreon follower - If you manage to create something usable I'll print the pic shown on LWM and I'll hang it up in my PC-room, maybe I'll even take my twitter account out of cryosleep to tell Musk that he should use Bcachefs as FS for his Teslas :))
pmarreck 4y ago
I lost not one but TWO entire volumes to it in the past few years. Just one day they wouldn't boot. I wasn't using encryption and I wasn't using any RAID feature, just compression. The people who think BTRFS is a viable substitute for ZFS have just gotten lucky with not getting burned. There's also the issue with BTRFS performance degrading hugely with any significant fragmentation, to the point that "autodefrag" has to be a feature, which of course also adds SSD wear... ZFS has never burned me.
Mister_Snuggles 4y ago
I've been burned enough times to avoid it. My Linux laptop, running OpenSuse Tumbleweed, got its root fs into a state where it would hang when trying to mount RW. I was able to boot off of a rescue image, mount it RO, and copy my data to another machine. None of the repair suggestions were able to get the fs into a mountable state, I had to reinstall it and copy my data back. Every VM at work that uses btrfs gets into a state where some maintenance process (rebalancing, I think - I'm not one of the Linux admins) will consume all CPU in the VM and prevent it from doing any work. The only thing that varies is how long it takes - it seems to be related to how much write activity there is on the fs, VMs with heavier write activity get into this state faster. This has led to outages and the Linux vendor was unable to resolve this issue. We've since removed all traces of btrfs from our data center.
- cyphar 4y ago
  If you used Docker on your machine, this was a bug in the remount code (they forgot to re-trigger the qgroup scan if it was interrupted when you did an rw-remount) which I fixed several years ago[1]. I don't know what that will do for your confidence in btrfs, but hopefully it'll give you some closure. [1]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=6c6b5a39c4bf3dbd8cf629c9f5450e983c19dbb9 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
  - Mister_Snuggles 4y ago
    The incident with my Linux laptop was definitely not Docker-related, but the description of the issue sounds like it could still apply. We didn't use Docker at work, at least not on the VMs that I was affected by. This one was a gradual degradation over time - the maintenance process would run fine, but after a while it would start hanging the machine. It really felt like the fs was only good for some amount of writes before the maintenance process got overwhelmed and would hang the VM whenever it ran. I'm happy to see that things are getting better, but the reputational damage will take a long time to repair. All that being said, I still use btrfs on my Linux laptop, but I've also become more conscientious about backing it up.
alexghr 4y ago
I'm going to add my N=1 anecdata if no one minds: I've been running full Btrfs on my workstation since December 2021 with 0 issues (kernel 5.10). It survived multiple hard reboots and power offs, Ryzen 5000 CPUs seems to lose power every now and then on Linux (it's a known issue for years now that no one has figured out yet). Having compression is great. compsize reports I've saved 33GB on my /home subvolume so far (https://github.com/kilobyte/compsize https://github.com/kilobyte/compsize). I don't use snapshots, I just backup the whole thing. Obviously this doesn't mean that in the future Btrfs won't eat my data, but so far it hasn't and I'm happy.
rkeene2 4y ago
BtrFS is the only filesystem I've ever lost data with, and I've used a lot of filesystems over the decades (UFS, VxFS, JFS, XFS, EXT2/3/4, ZFS). I have a comparison of features between BtrFS, ZFS, and EXT4+LVM at [0], and as I note at the top of that post I was looking to move to BtrFS. I did so and lost data. [0] https://rkeene.org/projects/info/wiki/BtrFS https://rkeene.org/projects/info/wiki/BtrFS
chasil 4y ago
Many people used btrfs while it was still experimental, and lost data. Attempting to create a btrfs filesystem returned a prompt with this warning (until 2013): Btrfs is a new filesystem with extents, writable snapshotting, support for multiple devices and many more features. Btrfs is highly experimental, and THE DISK FORMAT IS NOT YET FINALIZED. You should say N here unless you are interested in testing Btrfs with non-critical data. That prompt was removed when things stabilized, and Fedora's use of btrfs as the default root has not been accompanied by complaints of excessive filesystem corruption (although there are still many rough edges and unfinished features). https://arstechnica.com/gadgets/2021/09/examining-btrfs-linuxs-perpetually-half-finished-filesystem/ https://arstechnica.com/gadgets/2021/09/examining-btrfs-linu...

techdragon 4y ago

Since Btrfs was arguably available in the kernel too soon. Some people who should have avoided a new filesystem for lots of reasons ranging from missing features that were in the plan but not implemented yet or in the earliest days not yet stable... many of these early experimenters developed and fostered a vocal belief that Btrfs was "bad" (there's too much to sum up honestly, you could write an entire decent length blog post on the history of Btrfs, its reception and its criticism) and this group has taken a long time to accept that Btrfs may be ok now. Theres still people with the perception that its not as good or ready enough and should not be used.

blueflow 4y ago
From the bcachefs homepage: > We prioritize robustness and reliability over features and hype: we make every effort to ensure you won't lose data. This is exactly what btrfs failed at, this is not just an issue with the code itself, but also with the developers. Why should i trust feature-oriented developers with writing my infrastructure code, where reliability and robustness are top priority?
viraptor 4y ago
All this, but also two specific cases: There's a list of features and readiness at https://btrfs.wiki.kernel.org/index.php/Status https://btrfs.wiki.kernel.org/index.php/Status - there were people who didn't check it and used unstable features and experienced issues. This is partially on the project for not making it more explicit. The second is that there are people who read that, see the unstable raid5/6, and are vocal about btrfs being overall broken / not ready rather than accepting that feature just not being available yet.
- zozbot234 4y ago
  A file system that can eat your data is broken. If a feature is not ready for public consumption, it should not be built-in in the first place.
  - viraptor 4y ago
    Then every filesystem is broken by that definition. You're setting an impossible standard given how non-specialist software is developed. Here's a list of failures from zfs, including many kernel panics for example https://github.com/openzfs/zfs/labels/Type%3A%20Defect https://github.com/openzfs/zfs/labels/Type%3A%20Defect - is it broken and shouldn't have been released?
    - Beldin 4y ago
      Fine, let's be more nuanced: no filesystem should lose data if FAT16/FAT32/NTFS 3.1/EXT2/EXT3 wouldn't.
      - vlowther 4y ago
        well, FAT16/FAT32/EXT2 are not journalled, so making them lose data in a giant variety of ways is a trivial affair. You might want a higher bar than that.
        Spivak 4y ago
        Yeah, when you get down to it the opinion really boils down to "the file system is broken iff it doesn't lose data for the use-cases I personally use it for."
        Beldin 4y ago
        That's the point: if file systems that basic would NOT lose data, then it is unacceptable if another file system does. Maybe to put it differently: new file systems should offer improvements wrt reliability for stable features of the file systems of the previous generation; regressions are unacceptable.

deleted 4y ago

[deleted]

oxplot 4y ago

> How did they get burned? In the early days after Fedora made it available, I lost my root twice after a kernel panic and later after a hard power reset. EDIT: in comparison, I've been using xfs for almost a decade now with numerous hard power offs and kernel panics. I've only ever lost select files that were open for write at the time of restart, and not the entire filesystem.

warmwaffles 4y ago
I've been running in danger mode of RAID6 on a 15 drive NAS here at my home for the better part of 5 years now, and I have had zero issues. The server is also running Arch and I run package updates every two weeks or so. Maybe I am just lucky.
mekster 4y ago
I wonder how Fedora started using btrfs as the default for their desktop version when RedHat had deprecated it years ago. Never used btrfs but I see corruption reports in here everytime I read about it, and it's been in development for over a decade and this thing needs to be put to rest before it eats more people's data. Hopefully bcachefs becomes feature parity and stability against zfs to provide a modern filesystem to wider Linux audience at last. Funny to see we're still using ext4 as the default filesystem which is nothing fancy forever.
- dralley 4y ago
  > I wonder how Fedora started using btrfs as the default for their desktop version when RedHat had deprecated it years ago. Because the Fedora community wanted to, basically. They're not obligated to go along with whatever Red Hat wants to do, although there is naturally some alignment there. Plus having a different default filesystem really isn't that big of a deal. disclosure: I work for Red Hat.
  - mekster 4y ago
    Well, I always hear about data loss incidents whenever btrfs shows up in HN comments, so I thought making it a default in Fedora desktop sounded like a big deal as there must've been a good reason on why RH dropped it.

rodgerd 4y ago

The main thing I found btrfs was good for was testing the restore process of my backups. I was using it in supposedly known-good configurations (RAID 1 or RAID 10) and it consistently destroyed data. The fact that it turned out that the scrub function spent years corrupting data by mistake was only one of the many flaws with it.

EspadaV9 4y ago

I'd recommend listening to some of the 2.5 Admins podcasts. There are a few episodes (earlier on I think) where one of the hosts lists a number of issues with how it is implemented and basically wouldn't recommend it for anything, even in trivial set ups with a single drive there are far better alternative's.

yjftsjthsd-h 4y ago

I lost a root filesystem twice, on OpenSUSE in a one-disk system (i.e. what should have been the most stable well tested configuration), around 2018-19 IIRC. It now appears most likely that the system ran out of free space (suse ships snapper set to snapshot on every package operation, and the partition was smallish), and that apparently triggered corruption that ... at least, I couldn't figure out how to fix. Power loss may have been involved. Details fade from my memory, but the violent aversion to ever trusting it remains.

chasil 4y ago
It is actually wise to shrink a btrfs filesystem on its partition by a few megabytes, so some expansion room is available in an emergency. There are very few filesystems that can shrink like this: btrfs filesystem resize -1G /mount If the "btrfs fi df" command shows any of the data/system/metadata elements close to limits, it is time to take action. They can all push you into read-only mode.
zepearl 4y ago
I lost a btrfs-raid5-like NAS out of the blue (after 3-4 months of using it, I didn't even write a lot on it): the filesystem was always cleanly unmounted, I didn't do any crazy experiments with it, it had plenty (>40%, hundreds of GiBs) of free disk space; on one day I noticed that accessing some files was slower than usual, the noise of the HDDs had a weird pattern and from there it went downhill (at that time there was no fsck utility so I didn't even know what to do - if I'm not mistaken, BUT I MIGHT, the general opinion was that a fsck-util was not even needed? but again, I might have just dreamed this) and finally stopped working with a huge amount of error messages in the kernel logs => I had to reformat everything (I went back to mdadm-raid5) and restore from the secondary NAS' backup. In my opinion there were 2 main things that contributed to the bad reputation of the FS: 1) In those days there was a looot of hype about btrfs (as well for me it was a dream come true) which is the reason why I decided to use it. As far as I remember I didn't have to enable any special "experimental" feature in the kernel to use it => that gave me the impression that "it works under normal conditions, the DEVs are probably being a bit conservative for corner cases". 2) All functionalities (e.g. raid5 with rebalancing in my case) were all available from day 1 (at least when I started using it) so if something's available I tend to think that it works (I mean, from a user's point of view why make something available that does not work?) - but no, apparently many people had many problems with many different configurations/usage patterns. - As others have written I hope that Bcachefs will primarily focus on reliability and only secondarily on performance & features, but I admit that that's difficult to balance :). And I hope that the fsck-utility will be available from day1 (at least with a reduced functionality). And I hope that "experimental" features will need some kind of very explicit command line option (e.g. "-might_break_everything") to use them through userland utils. Go Kent!!! :) I'm since a while one of your Patreon follower - If you manage to create something usable I'll print the pic shown on LWM and I'll hang it up in my PC-room, maybe I'll even take my twitter account out of cryosleep to tell Musk that he should use Bcachefs as FS for his Teslas :))

pmarreck 4y ago

I lost not one but TWO entire volumes to it in the past few years. Just one day they wouldn't boot. I wasn't using encryption and I wasn't using any RAID feature, just compression. The people who think BTRFS is a viable substitute for ZFS have just gotten lucky with not getting burned. There's also the issue with BTRFS performance degrading hugely with any significant fragmentation, to the point that "autodefrag" has to be a feature, which of course also adds SSD wear... ZFS has never burned me.

Mister_Snuggles 4y ago

I've been burned enough times to avoid it. My Linux laptop, running OpenSuse Tumbleweed, got its root fs into a state where it would hang when trying to mount RW. I was able to boot off of a rescue image, mount it RO, and copy my data to another machine. None of the repair suggestions were able to get the fs into a mountable state, I had to reinstall it and copy my data back. Every VM at work that uses btrfs gets into a state where some maintenance process (rebalancing, I think - I'm not one of the Linux admins) will consume all CPU in the VM and prevent it from doing any work. The only thing that varies is how long it takes - it seems to be related to how much write activity there is on the fs, VMs with heavier write activity get into this state faster. This has led to outages and the Linux vendor was unable to resolve this issue. We've since removed all traces of btrfs from our data center.

cyphar 4y ago
If you used Docker on your machine, this was a bug in the remount code (they forgot to re-trigger the qgroup scan if it was interrupted when you did an rw-remount) which I fixed several years ago[1]. I don't know what that will do for your confidence in btrfs, but hopefully it'll give you some closure. [1]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=6c6b5a39c4bf3dbd8cf629c9f5450e983c19dbb9 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
- Mister_Snuggles 4y ago
  The incident with my Linux laptop was definitely not Docker-related, but the description of the issue sounds like it could still apply. We didn't use Docker at work, at least not on the VMs that I was affected by. This one was a gradual degradation over time - the maintenance process would run fine, but after a while it would start hanging the machine. It really felt like the fs was only good for some amount of writes before the maintenance process got overwhelmed and would hang the VM whenever it ran. I'm happy to see that things are getting better, but the reputational damage will take a long time to repair. All that being said, I still use btrfs on my Linux laptop, but I've also become more conscientious about backing it up.

alexghr 4y ago

I'm going to add my N=1 anecdata if no one minds: I've been running full Btrfs on my workstation since December 2021 with 0 issues (kernel 5.10). It survived multiple hard reboots and power offs, Ryzen 5000 CPUs seems to lose power every now and then on Linux (it's a known issue for years now that no one has figured out yet). Having compression is great. compsize reports I've saved 33GB on my /home subvolume so far (https://github.com/kilobyte/compsize https://github.com/kilobyte/compsize). I don't use snapshots, I just backup the whole thing. Obviously this doesn't mean that in the future Btrfs won't eat my data, but so far it hasn't and I'm happy.

rkeene2 4y ago

BtrFS is the only filesystem I've ever lost data with, and I've used a lot of filesystems over the decades (UFS, VxFS, JFS, XFS, EXT2/3/4, ZFS). I have a comparison of features between BtrFS, ZFS, and EXT4+LVM at [0], and as I note at the top of that post I was looking to move to BtrFS. I did so and lost data. [0] https://rkeene.org/projects/info/wiki/BtrFS https://rkeene.org/projects/info/wiki/BtrFS

chasil 4y ago

Many people used btrfs while it was still experimental, and lost data. Attempting to create a btrfs filesystem returned a prompt with this warning (until 2013): Btrfs is a new filesystem with extents, writable snapshotting, support for multiple devices and many more features. Btrfs is highly experimental, and THE DISK FORMAT IS NOT YET FINALIZED. You should say N here unless you are interested in testing Btrfs with non-critical data. That prompt was removed when things stabilized, and Fedora's use of btrfs as the default root has not been accompanied by complaints of excessive filesystem corruption (although there are still many rough edges and unfinished features). https://arstechnica.com/gadgets/2021/09/examining-btrfs-linuxs-perpetually-half-finished-filesystem/ https://arstechnica.com/gadgets/2021/09/examining-btrfs-linu...