6 ms·

Aside from the issues described further downthread: • On Btrfs the `df` command lies. You can't get an accurate count of free space. • There is no working `fs

Aside from the issues described further downthread: • On Btrfs the `df` command lies. You can't get an accurate count of free space. • There is no working `fsck` and the existing repair tools come with dire warnings. Take these very very seriously. I have tested them. They do not work and will destroy data. • The main point of Btrfs is snapshots. [open]SUSE, Spiral Linux, Garuda Linux and siduction all use these heavily for transactional updates. But the snapshot tool cannot test that there's enough free space for the snapshot, because `df` lies. So, it will fill up your disk. Writing to a full Btrfs volume will corrupt it. In my testing it destroyed my root partition roughly once per year. It was the most unstable fs I have tried since the era of ext2 in the mid-1990s. (Yes I am that old.)

VancouverMan 3y ago
I've experienced corruption and data loss with Btrfs each of the times I've tried using it, too, after only about a week of use at most. Thankfully, all of those incidents were with some non-critical, throw-away VMs where the data loss wasn't really an issue. I've also used ext4 under the same circumstances for years, and I can't think of a single time that I've lost data, nor have I experienced corruption that fsck couldn't easily deal with. I, too, would have to go back to the 1990s to think of a filesystem I used that was that unreliable. After what I experienced, I don't trust Btrfs at all, and I have no plans to ever use it again.
- lproven 3y ago
  I am glad to hear it's not just me! I worked at SUSE for 4Y and used it every day. The company is in deep denial about its problems, or that there are any problems, and when I pointed at ZFS as a more mature tool, this was actually mocked.
jhoechtl 3y ago
When did you run in any of these issues recently? These issues were a fact in the past but have been sorted out. df is not lying because the fs layer reports corrupt data but because of dedup and fs layering.
- lproven 3y ago
  I left SUSE close to the end of 2021, and I had had to reinstall my work laptop twice that year alone. I consider that recent enough to call it current. > df is not lying To me, that reads as "df isn't lying because $EXCUSES." I disagree. I don't care about excuses. I want a 100% accurate accounting of free space at all times via the standard xNix free-disk-space reporting command, and the same from the APIs that command uses so that applications can also get an accurate report of free space. If a filesystem cannot report free space reliably and accurately, then that filesystem is IMHO broken. Excuses do not exonerate the FS, and having other FS-specific commands that can report free space do not exonerate it. The `df` command must work, or the FS is broken. The primary point of Btrfs is that it is the only GPL snapshot-capable FS. The other stuff is gravy: it's a bonus. There are distros that use Btrfs that don't use snapshots, such as Fedora. Some Btrfs advocates use this to claim that the problems are not problematic. If the filesystem is of interest on the basis of feature $FOO, then "product $BAR does not exhibit this problem" is not an endorsement or a refutation if $BAR does not use feature $FOO. Btrfs RAID is broken in important ways, but that is not a deal-breaker because there are other perfectly good ways of obtaining that functionality using other parts of the Linux stack. If no feature or functionality is lost considering the OS and stack as a whole, then that isn't a problem. However, this remains serious and an issue. Additional problems include: • Poor integration into the overall industry-wide OS stack. Examples: - Existing commands do not work or give inconsistent results. - Duplication of functionality (e.g. overlap with `mdraid`) • Poor integration into specific vendors' OS stacks. Examples: - SUSE uses Btrfs heavily. But SUSE's `zypper` package manager is not integrated with its `snapper` tool. Zypper doesn't include snapshot space used by Snapper in its space estimation. Snapper is integrated with Btrfs; licence restrictions notwithstanding, I would be much reassured if Snapper supported other COW filesystems. (This has been attempted but I don't think anything shipped -- https://github.com/openSUSE/snapper/issues/145 https://github.com/openSUSE/snapper/issues/145 . I welcome correction on this!) The transactional features of SUSE's MicroOS family of distros rely heavily on Btrfs. As such, this lack of awareness of snapshot space utilization deeply worries me. I have raised this with SUSE management, but my concerns were dismissed. That worries me. What I want to see, for clarity, is for Zypper to look at what packages will be replaced, then ask the FS how big the consequent snapshot will be, and include that snapshot in its space estimation checks before beginning the operation so that at least the packaging operation can be safely aborted before starting. A better implementation would be to integrate package management with snapshot management so that older snapshots could be automatically pruned to ensure necessary space is made available, while also ensuring that a pre-operation snapshot is retained for rollback. That's harder but would work better. As it is, currently neither is attempted, and Zypper will start actions that result in filling the disk and thus trashing the FS, and there are no working repair tools to recover. - Red Hat removed Btrfs support from RHEL. As a result it has had to bodge transactional package management together by grafting Git-like functionality into OStree, then building two entirely new packaging systems around OStree, one for the OS itself and a different one for GUI-level packages. The latter is Flatpak, of course. This strikes me as prime evidence that: 1. Btrfs isn't ready. 2. Linux needs an in-kernel COW filesystem -- because much of the complexity of OStree, Flatpak, Nix/NixOS, Guix, SUSE's `transactional-update` commands and so on would be rendered unnecessary if it were in there.
creatonez 3y ago
> There is no working `fsck` and the existing repair tools come with dire warnings This stems from a misunderstanding. Fsck and fsck-adjacent tools have three purposes: 1. Replay journal entries in a journalled filesystem, so that the filesystem is repaired to a good state for mounting 2. Scrub through checksums and recover any data/metadata that has a redundant copy 3. In rare cases, a fsck tool encountering invalid data can make guesses as to how the filesystems should be structured -- basically, shot in the dark attempts at recovery. Btrfs does not need #1 because it is not journalled. Assuming write barriers are working, any partially written copy of the filesystem is valid and will simply appear as if the pending writes had been rolled back. This aspect alone greatly diminishes the need for a fsck tool that filesystems like ext4 have. As for #2, Btrfs already has scrub support. No issues there. As for #3, it's questionable whether you should ever rely on such functionality, and fsck tools that do implement such functionality tend to have little maneuverability in the first place.
- lproven 3y ago
  See my reply below. I disagree. You seem to be attempting to justify a profound failing by quibbling about the meaning of words or commands. The real problem is: Btrfs corrupts readily, and it lacks tools to fix the corruption. What the tools are called, what their functional role is theoretically meant to be, and whether this is justified in a tool of a given name is tangential and relatively speaking unimportant.
  - curt15 3y ago
    >The real problem is: Btrfs corrupts readily, and it lacks tools to fix the corruption. Here's a recent example of corruption unearthed by users after Fedora started defaulting to btrfs: https://bugzilla.redhat.com/show_bug.cgi?id=2169947 https://bugzilla.redhat.com/show_bug.cgi?id=2169947.
- hulitu 3y ago
  I thought that the main purpose of fsck was to rebuild the inode table and to put the filesystem in a correct state (all inodes linked in inode table).
ThatPlayer 3y ago
The df issue to me seems like an artifact of new data's RAID level being undecided. Because one of the planned features is per-subvolume raid level. Or at the very least metadata and data can have different RAID levels.
- lproven 3y ago
  Nope. It's still an issue on single volume on a single disk.
  - ThatPlayer 3y ago
    You can use duplicate on a single disk. IIRC metadata defaults to that.

VancouverMan 3y ago

I've experienced corruption and data loss with Btrfs each of the times I've tried using it, too, after only about a week of use at most. Thankfully, all of those incidents were with some non-critical, throw-away VMs where the data loss wasn't really an issue. I've also used ext4 under the same circumstances for years, and I can't think of a single time that I've lost data, nor have I experienced corruption that fsck couldn't easily deal with. I, too, would have to go back to the 1990s to think of a filesystem I used that was that unreliable. After what I experienced, I don't trust Btrfs at all, and I have no plans to ever use it again.

lproven 3y ago
I am glad to hear it's not just me! I worked at SUSE for 4Y and used it every day. The company is in deep denial about its problems, or that there are any problems, and when I pointed at ZFS as a more mature tool, this was actually mocked.

jhoechtl 3y ago

When did you run in any of these issues recently? These issues were a fact in the past but have been sorted out. df is not lying because the fs layer reports corrupt data but because of dedup and fs layering.

lproven 3y ago
I left SUSE close to the end of 2021, and I had had to reinstall my work laptop twice that year alone. I consider that recent enough to call it current. > df is not lying To me, that reads as "df isn't lying because $EXCUSES." I disagree. I don't care about excuses. I want a 100% accurate accounting of free space at all times via the standard xNix free-disk-space reporting command, and the same from the APIs that command uses so that applications can also get an accurate report of free space. If a filesystem cannot report free space reliably and accurately, then that filesystem is IMHO broken. Excuses do not exonerate the FS, and having other FS-specific commands that can report free space do not exonerate it. The `df` command must work, or the FS is broken. The primary point of Btrfs is that it is the only GPL snapshot-capable FS. The other stuff is gravy: it's a bonus. There are distros that use Btrfs that don't use snapshots, such as Fedora. Some Btrfs advocates use this to claim that the problems are not problematic. If the filesystem is of interest on the basis of feature $FOO, then "product $BAR does not exhibit this problem" is not an endorsement or a refutation if $BAR does not use feature $FOO. Btrfs RAID is broken in important ways, but that is not a deal-breaker because there are other perfectly good ways of obtaining that functionality using other parts of the Linux stack. If no feature or functionality is lost considering the OS and stack as a whole, then that isn't a problem. However, this remains serious and an issue. Additional problems include: • Poor integration into the overall industry-wide OS stack. Examples: - Existing commands do not work or give inconsistent results. - Duplication of functionality (e.g. overlap with `mdraid`) • Poor integration into specific vendors' OS stacks. Examples: - SUSE uses Btrfs heavily. But SUSE's `zypper` package manager is not integrated with its `snapper` tool. Zypper doesn't include snapshot space used by Snapper in its space estimation. Snapper is integrated with Btrfs; licence restrictions notwithstanding, I would be much reassured if Snapper supported other COW filesystems. (This has been attempted but I don't think anything shipped -- https://github.com/openSUSE/snapper/issues/145 https://github.com/openSUSE/snapper/issues/145 . I welcome correction on this!) The transactional features of SUSE's MicroOS family of distros rely heavily on Btrfs. As such, this lack of awareness of snapshot space utilization deeply worries me. I have raised this with SUSE management, but my concerns were dismissed. That worries me. What I want to see, for clarity, is for Zypper to look at what packages will be replaced, then ask the FS how big the consequent snapshot will be, and include that snapshot in its space estimation checks before beginning the operation so that at least the packaging operation can be safely aborted before starting. A better implementation would be to integrate package management with snapshot management so that older snapshots could be automatically pruned to ensure necessary space is made available, while also ensuring that a pre-operation snapshot is retained for rollback. That's harder but would work better. As it is, currently neither is attempted, and Zypper will start actions that result in filling the disk and thus trashing the FS, and there are no working repair tools to recover. - Red Hat removed Btrfs support from RHEL. As a result it has had to bodge transactional package management together by grafting Git-like functionality into OStree, then building two entirely new packaging systems around OStree, one for the OS itself and a different one for GUI-level packages. The latter is Flatpak, of course. This strikes me as prime evidence that: 1. Btrfs isn't ready. 2. Linux needs an in-kernel COW filesystem -- because much of the complexity of OStree, Flatpak, Nix/NixOS, Guix, SUSE's `transactional-update` commands and so on would be rendered unnecessary if it were in there.

creatonez 3y ago

> There is no working `fsck` and the existing repair tools come with dire warnings This stems from a misunderstanding. Fsck and fsck-adjacent tools have three purposes: 1. Replay journal entries in a journalled filesystem, so that the filesystem is repaired to a good state for mounting 2. Scrub through checksums and recover any data/metadata that has a redundant copy 3. In rare cases, a fsck tool encountering invalid data can make guesses as to how the filesystems should be structured -- basically, shot in the dark attempts at recovery. Btrfs does not need #1 because it is not journalled. Assuming write barriers are working, any partially written copy of the filesystem is valid and will simply appear as if the pending writes had been rolled back. This aspect alone greatly diminishes the need for a fsck tool that filesystems like ext4 have. As for #2, Btrfs already has scrub support. No issues there. As for #3, it's questionable whether you should ever rely on such functionality, and fsck tools that do implement such functionality tend to have little maneuverability in the first place.

lproven 3y ago
See my reply below. I disagree. You seem to be attempting to justify a profound failing by quibbling about the meaning of words or commands. The real problem is: Btrfs corrupts readily, and it lacks tools to fix the corruption. What the tools are called, what their functional role is theoretically meant to be, and whether this is justified in a tool of a given name is tangential and relatively speaking unimportant.
- curt15 3y ago
  >The real problem is: Btrfs corrupts readily, and it lacks tools to fix the corruption. Here's a recent example of corruption unearthed by users after Fedora started defaulting to btrfs: https://bugzilla.redhat.com/show_bug.cgi?id=2169947 https://bugzilla.redhat.com/show_bug.cgi?id=2169947.
hulitu 3y ago
I thought that the main purpose of fsck was to rebuild the inode table and to put the filesystem in a correct state (all inodes linked in inode table).

ThatPlayer 3y ago

The df issue to me seems like an artifact of new data's RAID level being undecided. Because one of the planned features is per-subvolume raid level. Or at the very least metadata and data can have different RAID levels.

lproven 3y ago
Nope. It's still an issue on single volume on a single disk.
- ThatPlayer 3y ago
  You can use duplicate on a single disk. IIRC metadata defaults to that.