7 ms·

ZFS will never be integrated into the Linux kernel due to it's licence. btrfs is complicated to use and has many pitfalls that can lead to it eating your data.

ndsipa_pomu 3y ago
> btrfs is complicated to use and has many pitfalls that can lead to it eating your data I use btrfs in preference over ext4 for Linux filesystems and turn on zstd compression for performance and a bit of space saving. It seems simple enough for my use case, though I'm not doing any snapshots etc. What are some of the potential pitfalls?
- Ridj48dhsnsh 3y ago
  Just an anecdote, but when I used gocryptfs on a btrfs partition, I'd always end up with a few corrupted files on power failure. After switching to gocryptfs on ext4, I never have any corruption.
  - redneb 3y ago
    I have used heavily[1] the combination btrfs+gocryptfs for many years and had no problems. The only quirk is that I have to pass the -noprealloc flag to gocryptfs, otherwise the performance is really bad. [1] by heavily, I mean that I use it for my home directory
    - Ridj48dhsnsh 3y ago
      I used it for my home directory too for several years up until a few months ago, but with default settings. The corrupted files were almost always Firefox cookies db and cache files. Maybe there's something specific about how Firefox writes them that makes them prone to corruption.
- pantalaimon 3y ago
  I was very excited about btrfs' advanced features, but that meant that btrfs would bite me multiple times when I expected it to 'just work™': - RAID5/6 are still not stable - it will not mount a RAID in degraded mode automatically, failing the high availability promise that might tempt you towards RAID. - swapfile support exists, but it breaks snapshots (and I don't want to snapshot the swapfile) - Just an Ubuntu/Debian thing, but snapshots are not integrated into the update process unless you install `apt-btrfs-snapshot` (and know that package exists)
  - ndsipa_pomu 3y ago
    > Just an Ubuntu/Debian thing, but snapshots are not integrated into the update process unless you install `apt-btrfs-snapshot` (and know that package exists) Thanks - I did not know about that. I agree with the stance of not mounting a degraded RAID automatically as then the danger is that someone might not notice it and be subjected to total data loss later on. The best option would be to allow over-riding that choice if the RAID is otherwise monitored.
  - webstrand 3y ago
    Can you not just place the swapfile inside of another subvolume? Since subvolumes are not included in snapshots. There's generally a bunch of stuff in /var that you don't want to include in snapshots, too, so it's not like putting the swapfile inside of a subvolume is an exotic task.
    - pantalaimon 3y ago
      I guess so, I just didn't know when I created the swapfile and was surprised later when I tried to create a snapshot.
      - xorcist 3y ago
        Be careful, just like disk images, you probably don't want to place swapfiles on a CoW filesystem.
- throw0101c 3y ago
  > What are some of the potential pitfalls? RAID-5/6: * https://btrfs.readthedocs.io/en/latest/btrfs-man5.html#raid56-status-and-recommended-practices https://btrfs.readthedocs.io/en/latest/btrfs-man5.html#raid5...
  - viraptor 3y ago
    Wouldn't describe that as a pitfall. As far as I can tell, every place in the docs/help which talks about those raid modes tell you they're experimental and shouldn't be used. At that point it's "you've done a stupid thing and discovered the consequences you were told about".
    - throw0101c 3y ago
      It's a feature that other file systems have: why should I choose Btrfs when (e.g.) ZFS can do everything it can and more on top of that? I remember when Btrfs was announced in 2007 as I was already running Solaris 10 with ZFS in production (and ZFS had non-"experimental" RAID-5-like RAID-Z from day one). Here we are 15+ years later and Btrfs still doesn't have it?
      - pantalaimon 3y ago
        The nice thing about btrfs is that you can add/remove drives at will. That allows for easy expansion of an existing pool.
      - viraptor 3y ago
        I'm just saying it's not a pitfall, because it's clearly documented up front and warned about. That's orthogonal to whether you should choose this filesystem and why a feature is implemented or not.
      - chasil 3y ago
        The one thing that ZFS cannot do is defragment a pool. There are abuse patterns that are toxic for ZFS pools (and all other filesystems). Btrfs appears to be able to repair this damage. https://tim.cexx.org/?p=1236 https://tim.cexx.org/?p=1236 https://www.usenix.org/system/files/login/articles/login_summer17_02_conway.pdf https://www.usenix.org/system/files/login/articles/login_sum...
- viraptor 3y ago
  One that seems to catch out many people is that you need to explicitly add the "degraded" option to allow mounting raid volumes which contain a broken drive. This is opposite to almost all other filesystems. I've seen people confused and thinking they lost the whole volume, even though they just need to replace the bad drive and rebuild as usual.
  - ndsipa_pomu 3y ago
    Personally, I agree with that. It's good practise for systems to fail quickly if they find themselves in an uncertain state and with RAID, it's important for the operator to know if they've suddenly lost redundancy so that they can resolve the issue.
    - viraptor 3y ago
      It really depends on what's your situation. At home I probably want the disks to stay idle until replacement. In a bigger production system I want to keep the availability and ping the on-call person to replace the drive in the background.
      - ndsipa_pomu 3y ago
        Exactly - the default should be to get someone to pay attention to it when something breaks, and if you're planning on high availability, then you can choose that, assuming you keep an eye on the health of the RAID.
    - keep_reading 3y ago
      If you can't even boot because of a mystery error how are you supposed to resolve the issue? It should instead be giving the user error messages written to their terminal, in logs, etc instead of breaking the entire system until the user finds the manual
      - wtallis 3y ago
        If your NAS has a hot spare drive installed then it should probably include the degraded mount option by default and automatically add the hot spare drive to the filesystem in the event of a failure. Alternatively, if there's enough free space on the surviving drives to rebalance the array and restore redundancy without replacing the failed drive, that operation could be kicked off automatically. Or the filesystem can be (hopefully temporarily) set to not store new data redundantly, if that is an acceptable risk for the user. But the filesystem cannot know which method the user would prefer; automatically rebuilding the array involves policy decisions that are outside the scope of the filesystem and requires userspace tooling. If the system doesn't have spare capacity ready, the only sane response is to not boot/mount normally. "giving the user error messages written to their terminal, in logs, etc" isn't a real solution for something like a NAS with no terminal connected and nobody looking at the logs as long as they can still establish a SMB connection; it's too likely to be a silent failure in practice. Mounting the filesystem degraded but read-only makes sense if it's necessary to boot the system so that the user (or their pre-configured userspace tooling) can decide how to deal with the problem, but a lot of Linux distros aren't happy with the root filesystem being read-only. In summary: there's no single right answer to the problem of a failed drive, and btrfs defaults to what is the safest behavior based on the information available to the filesystem itself. Userspace tooling with more information can make other, less universal choices. A distro that tries to simply adopt btrfs as a drop-in replacement for ext4 probably doesn't have all the tooling necessary to make good use of the unique features of btrfs.
- curt15 3y ago
  It performs poorly for certain workloads -- notably DBs -- unless you disable copy-on-write, compression, and checksumming. But then why use it over ext4 in the first place?
  - ndsipa_pomu 3y ago
    That's not surprising though, as DBs usually work best when given raw storage - the features of the filesystem are being duplicated by the DB and thus doing almost twice the work. What seems particularly interesting about Bcachefs is how much it seems to be using database concepts to implement a general filesystem. Ultimately, it seems inevitable that filesystems and databases will converge as they're both supposed to manage data.
    - PlutoIsAPlanet 3y ago
      The "fix" to databases and VMs on btrfs by disabling CoW unfortunately disables nearly all the useful features.
- lproven 3y ago
  Aside from the issues described further downthread: • On Btrfs the `df` command lies. You can't get an accurate count of free space. • There is no working `fsck` and the existing repair tools come with dire warnings. Take these very very seriously. I have tested them. They do not work and will destroy data. • The main point of Btrfs is snapshots. [open]SUSE, Spiral Linux, Garuda Linux and siduction all use these heavily for transactional updates. But the snapshot tool cannot test that there's enough free space for the snapshot, because `df` lies. So, it will fill up your disk. Writing to a full Btrfs volume will corrupt it. In my testing it destroyed my root partition roughly once per year. It was the most unstable fs I have tried since the era of ext2 in the mid-1990s. (Yes I am that old.)
  - VancouverMan 3y ago
    I've experienced corruption and data loss with Btrfs each of the times I've tried using it, too, after only about a week of use at most. Thankfully, all of those incidents were with some non-critical, throw-away VMs where the data loss wasn't really an issue. I've also used ext4 under the same circumstances for years, and I can't think of a single time that I've lost data, nor have I experienced corruption that fsck couldn't easily deal with. I, too, would have to go back to the 1990s to think of a filesystem I used that was that unreliable. After what I experienced, I don't trust Btrfs at all, and I have no plans to ever use it again.
    - lproven 3y ago
      I am glad to hear it's not just me! I worked at SUSE for 4Y and used it every day. The company is in deep denial about its problems, or that there are any problems, and when I pointed at ZFS as a more mature tool, this was actually mocked.
  - jhoechtl 3y ago
    When did you run in any of these issues recently? These issues were a fact in the past but have been sorted out. df is not lying because the fs layer reports corrupt data but because of dedup and fs layering.
- chasil 3y ago
  With ZFS, if a redundant member of a pool goes offline for a time but is then returned, only the updated blocks are written to bring it up to date, and this happens automatically. Unfortunately, btrfs is not that smart, and you must trigger a rebalance event to rewrite every block in the filesystem to return to full redundancy. This rebalance behavior is a deal-killer for many uses.

ndsipa_pomu 3y ago

> btrfs is complicated to use and has many pitfalls that can lead to it eating your data I use btrfs in preference over ext4 for Linux filesystems and turn on zstd compression for performance and a bit of space saving. It seems simple enough for my use case, though I'm not doing any snapshots etc. What are some of the potential pitfalls?

Ridj48dhsnsh 3y ago
Just an anecdote, but when I used gocryptfs on a btrfs partition, I'd always end up with a few corrupted files on power failure. After switching to gocryptfs on ext4, I never have any corruption.
- redneb 3y ago
  I have used heavily[1] the combination btrfs+gocryptfs for many years and had no problems. The only quirk is that I have to pass the -noprealloc flag to gocryptfs, otherwise the performance is really bad. [1] by heavily, I mean that I use it for my home directory
  - Ridj48dhsnsh 3y ago
    I used it for my home directory too for several years up until a few months ago, but with default settings. The corrupted files were almost always Firefox cookies db and cache files. Maybe there's something specific about how Firefox writes them that makes them prone to corruption.
pantalaimon 3y ago
I was very excited about btrfs' advanced features, but that meant that btrfs would bite me multiple times when I expected it to 'just work™': - RAID5/6 are still not stable - it will not mount a RAID in degraded mode automatically, failing the high availability promise that might tempt you towards RAID. - swapfile support exists, but it breaks snapshots (and I don't want to snapshot the swapfile) - Just an Ubuntu/Debian thing, but snapshots are not integrated into the update process unless you install `apt-btrfs-snapshot` (and know that package exists)
- ndsipa_pomu 3y ago
  > Just an Ubuntu/Debian thing, but snapshots are not integrated into the update process unless you install `apt-btrfs-snapshot` (and know that package exists) Thanks - I did not know about that. I agree with the stance of not mounting a degraded RAID automatically as then the danger is that someone might not notice it and be subjected to total data loss later on. The best option would be to allow over-riding that choice if the RAID is otherwise monitored.
- webstrand 3y ago
  Can you not just place the swapfile inside of another subvolume? Since subvolumes are not included in snapshots. There's generally a bunch of stuff in /var that you don't want to include in snapshots, too, so it's not like putting the swapfile inside of a subvolume is an exotic task.
  - pantalaimon 3y ago
    I guess so, I just didn't know when I created the swapfile and was surprised later when I tried to create a snapshot.
    - xorcist 3y ago
      Be careful, just like disk images, you probably don't want to place swapfiles on a CoW filesystem.
throw0101c 3y ago
> What are some of the potential pitfalls? RAID-5/6: * https://btrfs.readthedocs.io/en/latest/btrfs-man5.html#raid56-status-and-recommended-practices https://btrfs.readthedocs.io/en/latest/btrfs-man5.html#raid5...
- viraptor 3y ago
  Wouldn't describe that as a pitfall. As far as I can tell, every place in the docs/help which talks about those raid modes tell you they're experimental and shouldn't be used. At that point it's "you've done a stupid thing and discovered the consequences you were told about".
  - throw0101c 3y ago
    It's a feature that other file systems have: why should I choose Btrfs when (e.g.) ZFS can do everything it can and more on top of that? I remember when Btrfs was announced in 2007 as I was already running Solaris 10 with ZFS in production (and ZFS had non-"experimental" RAID-5-like RAID-Z from day one). Here we are 15+ years later and Btrfs still doesn't have it?
    - pantalaimon 3y ago
      The nice thing about btrfs is that you can add/remove drives at will. That allows for easy expansion of an existing pool.
    - viraptor 3y ago
      I'm just saying it's not a pitfall, because it's clearly documented up front and warned about. That's orthogonal to whether you should choose this filesystem and why a feature is implemented or not.
    - chasil 3y ago
      The one thing that ZFS cannot do is defragment a pool. There are abuse patterns that are toxic for ZFS pools (and all other filesystems). Btrfs appears to be able to repair this damage. https://tim.cexx.org/?p=1236 https://tim.cexx.org/?p=1236 https://www.usenix.org/system/files/login/articles/login_summer17_02_conway.pdf https://www.usenix.org/system/files/login/articles/login_sum...
viraptor 3y ago
One that seems to catch out many people is that you need to explicitly add the "degraded" option to allow mounting raid volumes which contain a broken drive. This is opposite to almost all other filesystems. I've seen people confused and thinking they lost the whole volume, even though they just need to replace the bad drive and rebuild as usual.
- ndsipa_pomu 3y ago
  Personally, I agree with that. It's good practise for systems to fail quickly if they find themselves in an uncertain state and with RAID, it's important for the operator to know if they've suddenly lost redundancy so that they can resolve the issue.
  - viraptor 3y ago
    It really depends on what's your situation. At home I probably want the disks to stay idle until replacement. In a bigger production system I want to keep the availability and ping the on-call person to replace the drive in the background.
    - ndsipa_pomu 3y ago
      Exactly - the default should be to get someone to pay attention to it when something breaks, and if you're planning on high availability, then you can choose that, assuming you keep an eye on the health of the RAID.
  - keep_reading 3y ago
    If you can't even boot because of a mystery error how are you supposed to resolve the issue? It should instead be giving the user error messages written to their terminal, in logs, etc instead of breaking the entire system until the user finds the manual
    - wtallis 3y ago
      If your NAS has a hot spare drive installed then it should probably include the degraded mount option by default and automatically add the hot spare drive to the filesystem in the event of a failure. Alternatively, if there's enough free space on the surviving drives to rebalance the array and restore redundancy without replacing the failed drive, that operation could be kicked off automatically. Or the filesystem can be (hopefully temporarily) set to not store new data redundantly, if that is an acceptable risk for the user. But the filesystem cannot know which method the user would prefer; automatically rebuilding the array involves policy decisions that are outside the scope of the filesystem and requires userspace tooling. If the system doesn't have spare capacity ready, the only sane response is to not boot/mount normally. "giving the user error messages written to their terminal, in logs, etc" isn't a real solution for something like a NAS with no terminal connected and nobody looking at the logs as long as they can still establish a SMB connection; it's too likely to be a silent failure in practice. Mounting the filesystem degraded but read-only makes sense if it's necessary to boot the system so that the user (or their pre-configured userspace tooling) can decide how to deal with the problem, but a lot of Linux distros aren't happy with the root filesystem being read-only. In summary: there's no single right answer to the problem of a failed drive, and btrfs defaults to what is the safest behavior based on the information available to the filesystem itself. Userspace tooling with more information can make other, less universal choices. A distro that tries to simply adopt btrfs as a drop-in replacement for ext4 probably doesn't have all the tooling necessary to make good use of the unique features of btrfs.
curt15 3y ago
It performs poorly for certain workloads -- notably DBs -- unless you disable copy-on-write, compression, and checksumming. But then why use it over ext4 in the first place?
- ndsipa_pomu 3y ago
  That's not surprising though, as DBs usually work best when given raw storage - the features of the filesystem are being duplicated by the DB and thus doing almost twice the work. What seems particularly interesting about Bcachefs is how much it seems to be using database concepts to implement a general filesystem. Ultimately, it seems inevitable that filesystems and databases will converge as they're both supposed to manage data.
  - PlutoIsAPlanet 3y ago
    The "fix" to databases and VMs on btrfs by disabling CoW unfortunately disables nearly all the useful features.
lproven 3y ago
Aside from the issues described further downthread: • On Btrfs the `df` command lies. You can't get an accurate count of free space. • There is no working `fsck` and the existing repair tools come with dire warnings. Take these very very seriously. I have tested them. They do not work and will destroy data. • The main point of Btrfs is snapshots. [open]SUSE, Spiral Linux, Garuda Linux and siduction all use these heavily for transactional updates. But the snapshot tool cannot test that there's enough free space for the snapshot, because `df` lies. So, it will fill up your disk. Writing to a full Btrfs volume will corrupt it. In my testing it destroyed my root partition roughly once per year. It was the most unstable fs I have tried since the era of ext2 in the mid-1990s. (Yes I am that old.)
- VancouverMan 3y ago
  I've experienced corruption and data loss with Btrfs each of the times I've tried using it, too, after only about a week of use at most. Thankfully, all of those incidents were with some non-critical, throw-away VMs where the data loss wasn't really an issue. I've also used ext4 under the same circumstances for years, and I can't think of a single time that I've lost data, nor have I experienced corruption that fsck couldn't easily deal with. I, too, would have to go back to the 1990s to think of a filesystem I used that was that unreliable. After what I experienced, I don't trust Btrfs at all, and I have no plans to ever use it again.
  - lproven 3y ago
    I am glad to hear it's not just me! I worked at SUSE for 4Y and used it every day. The company is in deep denial about its problems, or that there are any problems, and when I pointed at ZFS as a more mature tool, this was actually mocked.
- jhoechtl 3y ago
  When did you run in any of these issues recently? These issues were a fact in the past but have been sorted out. df is not lying because the fs layer reports corrupt data but because of dedup and fs layering.
chasil 3y ago
With ZFS, if a redundant member of a pool goes offline for a time but is then returned, only the updated blocks are written to bring it up to date, and this happens automatically. Unfortunately, btrfs is not that smart, and you must trigger a rebalance event to rewrite every block in the filesystem to return to full redundancy. This rebalance behavior is a deal-killer for many uses.