5 ms·

I agree fully with the desire for resilience throughout the kernel, especially in today's world. It was very different 20 years ago when Linux was still young.

I agree fully with the desire for resilience throughout the kernel, especially in today's world. It was very different 20 years ago when Linux was still young. Otoh it's one of the things that I think made Linux succeed in the beginning. Everyone could upstream a half arsed driver for something and it would get fixed while people use it and encounter bugs. Now that Linux is used professionally everywhere that just isn't feasible anymore. On another note, I remember that some time ago there was a talk about Linux file system fuzzing given at some conference and ext4 fared the best by far, which is why I'm still using that exclusively, although some of the features of btrfs would come in handy at times.

est31 7y ago
I think the main reason why btrfs has not fully replaced the ext family by now is its bugs. It's understandable that they don't want to add more buggy filesystems.
- pletnes 7y ago
  Personally, I’ve never noticed bugs, but the performance characteristics are truly weird. If you fill up a large partition, expect using a few days of running various incantations to free space, even after removing plenty of files. I’m completely amazed at how fast snapshots can be done, though.
  - kzrdude 7y ago
    It really makes it seem like handling free space was an afterthought in btrfs. Infinite memory and infinite storage are great abstractions, easy to use models and a lot of software runs on those models... but in this case it seems like ENOSPC should have been designed in from the start.
    - sitkack 7y ago
      I have noticed lots of things break when the resource horizon shrinks. Weirdness happens as on crosses into 1/2 (for things that like to double, like amortized data structures) and then again at <5% of free space/cpu/network/etc. Often the best mitigation is to have excess reserve and remove the false blocker (file on disk, optional process eating ram, etc).
  - est31 7y ago
    Yeah I'd classify that bad performance with nearly full drives as a bug. Compare that to one of my ext4 partitions which has been > 90% since years of active use without major issues. Occasionally I get errors that it can't write stuff any more due to capacity, I just delete some unneeded stuff and go on with life.
    - seized 7y ago
      It's not really a bug though taken in context of how the file systems work with copy on write. You need somewhere to write, so if you low free space and have the drive heads seeking...
  - dnr 7y ago
    Just more anecdotes: I accidentally filled up a 2tb btrfs filesystem on ssd with a buggy backup script. I deleted the stuff that shouldn't be there, ran a few recommended incantations, and was back to normal in about ten minutes. It was much less of a big deal than I expected based on what I'd read.
- iikoolpp 7y ago
  After an experience where btrfs decided to break my combined filesystem I vow to never use it again.
  - pas 7y ago
    Could you explain/describe what happened and why and how to prevent it?
    - iikoolpp 7y ago
      Sure, I was trying to remove a disk from my array (1TB, with 50GB used, the other disk was 4TB and had ~600GB used). I tried doing `btrfs remove /dev/disk1 /mnt` and it refused, claiming there was no free space. No amount of arcane commands worked. Eventually I just copied all the files to somewhere else and nuked the filesystem entirely.
      - cmurf 7y ago
        Due to the way btrfs allocates in "collections" of extents called block groups, it's possible all the space was allocated by mostly empty block groups, which could make enospc possible. But that's a rather old set of bugs that I haven't seen in a very long time and predates all the modern space handling code. It must have been pre-4.0 kernels. And I did run into it myself on purpose many times while trying to help improve the behavior. Non-obvious, but very straight forward way around such a wedged in file system, is to add a 3rd device. It could even be a USB stick, back then I was using small 4GiB sticks and it would work. That was enough to allocate a couple metadata only block groups to the stick, to write out the file system changes necessary to back out the second device. And once that completes, a brief filtered balance (e.g. btrfs balance start -dusage=10 is usually sufficien) allows enough free space on the 1st device, to back out the 3rd (the USB stick). The non-obvious thing about any COW file system is that deletion always requires free space. There is no such thing as deleting a file with COW unless the fs can write that deletion change to all the affected trees into free space. Once the entire set of metadata changes is committed, then the data and metadata extents for those deleted files can be freed. Anyway, a lot has changed even in one year in Btrfs, let alone the past five years. It's thousands of line changes per kernel release.
- nicoburns 7y ago
  Yep. I personally won't use btrfs for this reason. The last thing I want to deal with is filesystem corruption. If I needed the advanced features, I'd just use zfs.
  - vetinari 7y ago
    "just use zfs" is it's own can of worms. Maybe Canonical will solve that, we will see how distributing zfs together with kernel will go through.
    - sigstoat 7y ago
      > "just use zfs" is it's own can of worms. it's been fine for some of us for, what, nearly 10 years now?
      - vetinari 7y ago
        If you consider normal to self-compile kernel modules on your production machines (or have compiler there, or use mutable system at all), well, then yes. Otherwise, it is a massive pita. I do have one machine with zfs, and I made the mistake of placing the root into a subvolume. Won't do that again.
        laumars 7y ago
        A few distros include ZoL drivers in their repos. There's also FreeBSD as well as the various platforms derived from the, now defunct, OpenSolaris. I've never had compile ZFS myself (though I have, on occasions, chosen to because I've wanted to try features before they hit the repos).
        nicoburns 7y ago
        I'd rather do that than use a buggy filesystem!
        vetinari 7y ago
- cmurf 7y ago
  Btrfs has had its fair share of bugs, but rarely do they result in the inability to at least mount the file system read-only and get your data out. What has further been exposed by Btrfs are device bugs, in particular firmware. When write ordering doesn't happen the way the file system expects, it's a big problem, and Btrfs does not tolerate them, it should abruptly go read only in order to prevent amplifying file system confusion. This is separate from silent data corruption which Btrfs also detects and complains about, but usually just results in EIO as Btrfs will not propagate data it think is corrupt due to csum mismatches. But even in that case you can get your data out.

est31 7y ago

I think the main reason why btrfs has not fully replaced the ext family by now is its bugs. It's understandable that they don't want to add more buggy filesystems.

pletnes 7y ago
Personally, I’ve never noticed bugs, but the performance characteristics are truly weird. If you fill up a large partition, expect using a few days of running various incantations to free space, even after removing plenty of files. I’m completely amazed at how fast snapshots can be done, though.
- kzrdude 7y ago
  It really makes it seem like handling free space was an afterthought in btrfs. Infinite memory and infinite storage are great abstractions, easy to use models and a lot of software runs on those models... but in this case it seems like ENOSPC should have been designed in from the start.
  - sitkack 7y ago
    I have noticed lots of things break when the resource horizon shrinks. Weirdness happens as on crosses into 1/2 (for things that like to double, like amortized data structures) and then again at <5% of free space/cpu/network/etc. Often the best mitigation is to have excess reserve and remove the false blocker (file on disk, optional process eating ram, etc).
- est31 7y ago
  Yeah I'd classify that bad performance with nearly full drives as a bug. Compare that to one of my ext4 partitions which has been > 90% since years of active use without major issues. Occasionally I get errors that it can't write stuff any more due to capacity, I just delete some unneeded stuff and go on with life.
  - seized 7y ago
    It's not really a bug though taken in context of how the file systems work with copy on write. You need somewhere to write, so if you low free space and have the drive heads seeking...
- dnr 7y ago
  Just more anecdotes: I accidentally filled up a 2tb btrfs filesystem on ssd with a buggy backup script. I deleted the stuff that shouldn't be there, ran a few recommended incantations, and was back to normal in about ten minutes. It was much less of a big deal than I expected based on what I'd read.
iikoolpp 7y ago
After an experience where btrfs decided to break my combined filesystem I vow to never use it again.
- pas 7y ago
  Could you explain/describe what happened and why and how to prevent it?
  - iikoolpp 7y ago
    Sure, I was trying to remove a disk from my array (1TB, with 50GB used, the other disk was 4TB and had ~600GB used). I tried doing `btrfs remove /dev/disk1 /mnt` and it refused, claiming there was no free space. No amount of arcane commands worked. Eventually I just copied all the files to somewhere else and nuked the filesystem entirely.
    - cmurf 7y ago
      Due to the way btrfs allocates in "collections" of extents called block groups, it's possible all the space was allocated by mostly empty block groups, which could make enospc possible. But that's a rather old set of bugs that I haven't seen in a very long time and predates all the modern space handling code. It must have been pre-4.0 kernels. And I did run into it myself on purpose many times while trying to help improve the behavior. Non-obvious, but very straight forward way around such a wedged in file system, is to add a 3rd device. It could even be a USB stick, back then I was using small 4GiB sticks and it would work. That was enough to allocate a couple metadata only block groups to the stick, to write out the file system changes necessary to back out the second device. And once that completes, a brief filtered balance (e.g. btrfs balance start -dusage=10 is usually sufficien) allows enough free space on the 1st device, to back out the 3rd (the USB stick). The non-obvious thing about any COW file system is that deletion always requires free space. There is no such thing as deleting a file with COW unless the fs can write that deletion change to all the affected trees into free space. Once the entire set of metadata changes is committed, then the data and metadata extents for those deleted files can be freed. Anyway, a lot has changed even in one year in Btrfs, let alone the past five years. It's thousands of line changes per kernel release.
nicoburns 7y ago
Yep. I personally won't use btrfs for this reason. The last thing I want to deal with is filesystem corruption. If I needed the advanced features, I'd just use zfs.
- vetinari 7y ago
  "just use zfs" is it's own can of worms. Maybe Canonical will solve that, we will see how distributing zfs together with kernel will go through.
  - sigstoat 7y ago
    > "just use zfs" is it's own can of worms. it's been fine for some of us for, what, nearly 10 years now?
    - vetinari 7y ago
      If you consider normal to self-compile kernel modules on your production machines (or have compiler there, or use mutable system at all), well, then yes. Otherwise, it is a massive pita. I do have one machine with zfs, and I made the mistake of placing the root into a subvolume. Won't do that again.
      - laumars 7y ago
        A few distros include ZoL drivers in their repos. There's also FreeBSD as well as the various platforms derived from the, now defunct, OpenSolaris. I've never had compile ZFS myself (though I have, on occasions, chosen to because I've wanted to try features before they hit the repos).
      - nicoburns 7y ago
        I'd rather do that than use a buggy filesystem!
        vetinari 7y ago
cmurf 7y ago
Btrfs has had its fair share of bugs, but rarely do they result in the inability to at least mount the file system read-only and get your data out. What has further been exposed by Btrfs are device bugs, in particular firmware. When write ordering doesn't happen the way the file system expects, it's a big problem, and Btrfs does not tolerate them, it should abruptly go read only in order to prevent amplifying file system confusion. This is separate from silent data corruption which Btrfs also detects and complains about, but usually just results in EIO as Btrfs will not propagate data it think is corrupt due to csum mismatches. But even in that case you can get your data out.