5 ms·

Is anyone here using BTRFS and can comment on its current-day stability? I used to read horror stories about it

by oDot 2y ago

quotemstr 2y ago
I've used it for my personal machines and backups (via btrbk) for years without any issues
einsteinx2 2y ago
I’ve been using it for a few years on my NAS for the all the data drives (with Snapraid for parity and data validation), and as the boot drive on a few SBCs that run various services. Also use it as the boot drive for my Linux desktop PC. So far no problems at all and I make heavy use of snapshots, I have also had various things like power outages that have shut down the various machines multiple times. I’ve never used BTRFS raid so can’t speak to that, but in my personal experience I’ve found BTRFS and the snapshot system to be reliable. Seems like most (all?) stories I hear about corruption and other problems are all from years ago when it was less stable (years before I started using it). Or maybe I just got lucky ¯\_(ツ)_/¯
- AtlasBarfed 2y ago
  Why use BTRfS raid rather than good old MDadm raid?
  - einsteinx2 2y ago
    I don’t use BTRFS raid, I don’t actually use any RAID. I use SnapRAID which is really more of a parity system than real RAID. I have a bunch of data disks that are formatted BTRFS, then 2 parity disks formatted using ext4 since they don’t require any BTRFS features. Then I use snapraid-btrfs which is a wrapper around SnapRAID to automatically generate BTRFS snapshots on the data disks when doing a SnapRAID sync. Since the parity is file based, it’s best to use it with snapshots, so that’s the solution I went with. I’m sure you could also use LVM snapshots with ext4 or ZFS snapshots, but BTRFS with SnapRAID is well supported and I like how BTRFS snapshots/subvolumes works so I went with that. Also BTRFS has some nice features over ext4 like CoW and checksumming. I considered regular RAID but I don’t need the bandwidth increase over single disks and I didn’t ever want the chance of losing a whole RAID pool. With my SnapRAID setup I can lose any 2 drives and not lose any data, and if I lose 3 drives, I only lose the data on any lost data drives, not all the data. Also it’s easy to add a single drive at a time as I need more space. That was my thought process when choosing it anyway and it’s worked for my use case (I don’t need much IOPS or bandwidth, just lots of cheap fairly resilient and easy to expand storage).
  - tremon 2y ago
    BTRFS raid is usage-aware, so a rebuild will not need to do a bit-for-bit copy of the entire disk, but only the parts that are actually in use. Also, because btrfs has data checksumming, it can detect read errors even when the disk reports a successful read (however, it will not verify the checksum during regular operation, only during scrub).
  - gmokki 2y ago
    BTRFS Raid10 can seemlessly cpmbole multiple raw disks without trying to match capacity. Next time I just replace my 4T disk in my 5 disk Raid10 with 20T. Currently I have 4+8+8+16+20 disks. MD raid does not do checksumming. Although I believe XFS is about to add support for it in the future. I have had my BTRFS raid filesystem survive a lot during the past 14 years: - burned power: no loss of data - failed ram that started corrupting memory: after a little hack 1) BTRFS scrub saved most of data even though the situation got so bad kernel would crash in 10 minutes - buggy pcie SATA extension card: I tried to add 6th disk, but noticed after fee million write errors to one disk that it just randomly stopped passing data through: no data corruption, although btrfs write error counters are in 10s of millions now - 4 disk failures: I have only one original disk still running and it is showing a lot of bad sectors 1) one of the corrupted sectors was in the btrfs tree that contains the checksums for rest of the filesystem and both copies were broken. It prevented access to some 200 files. I patched the kernel to log the exact sector in addition to the expected and actual value. Turns our it was just a single bit flip. So I used hex editor to flip it back to correct value and got the files back
  - ThatPlayer 2y ago
    More flexibility in drives. Btrfs's RAID1, isn't actually RAID1 where everything is written to all the drives, but closer to RAID10 it writes all data to copies on 2 drives. So you can have a 1+2+3 TB drive in an array and still get 3TB of usable storage, or even 1+1+1+1+4. And you can add/remove single drives easily. You can also set different RAID levels for metadata versus data, because the raid knows the difference. At some point in the future you might be able to set it per-file too.
jcalvinowens 2y ago
I've used BTRFS exclusively for over a decade now on all my personal laptops, servers, and embedded devices. I've never had a single problem. It's the flagship Linux filesystem: outside of database workloads, I don't understand why anybody uses anything else.
- KennyBlanken 2y ago
  "Flagship"? I don't know a single person who uses it in production systems. It's the only filesystem I've lost data to. Ditto for friends. Please go look up survivor bias. That's what all you btrfs fanboys don't seem to understand. It doesn't matter how well it has worked for 99.9% of you. Filesystems have to be the most reliable component in an operating system. It's a flagship whose fsck requires you to contact developers to seek advice on how to use it because otherwise it might destroy your filesystem. It's a flagship whose userspace tools, fifteen years in, are still seeing major changes. It's a flagship whose design is so poor that fifteen years in the developers are making major changes to its structure and depreciating old features in ways that do not trigger an automatic upgrade or informative error to upgrade, but cause the filesystem to panic with error messages for which there is no documentation and little clue what the problem is. No other filesystem has these issues.
  - jchw 2y ago
    Btrfs is in production all over the damn place, at big corporations and all kinds of different deployments. Synology has their own btrfs setup that they ship to customers with their NAS software for example. I found it incredibly annoying the first time I ran out of disk space on btrfs, but many of these points are hyperbolic and honestly just silly. For example, btrfs doesn't really do offline fsck. fsck.btrfs has a zero percent chance of destroying your volume because it does nothing. As for the user space utilities changing... I'm not sure how that demonstrates the filesystem is not production ready. Personally I usually use either XFS or btrfs as my root filesystem. While I've caught some snags with btrfs, I've never lost any data. I don't actually know anyone who has, I've merely just heard about it. And it's not like other well-regarded filesystems have never ran into data loss situations: even OpenZFS recently (about a year ago) uncovered a data-eating bug that called its reliability into question. I'm sure some people will angrily tell me that actually btrfs is shit and the worst thing to ever be created and honestly whatever. I am not passionate about filesystems. Wake me up when there's a better one and it's mainlined. Maybe it will eventually be bcachefs. (Edit: and just to be clear, I do realize bcachefs is mainline and Kent Overstreet considers it to be stable and safe. However, it's still young and it's upstream future has been called into question. For non-technical reasons, but still; it does make me less confident.)
badsectoracula 2y ago
I've been using it for a few years now on my main PC (has a couple SSDs and a large HDD) and my laptop, it was the default of openSUSE and just used that. Then i realized that snapshots are a feature i didn't knew i wanted :-P. Never had a problem, though it is annoying that whatever BTRFS thinks is free space and what the rest of the OS thinks is free space do not always align. It has rarely been a problem in practice though.
KennyBlanken 2y ago
The documentation describes 'btrfs check' as being dangerous to run without consulting the mailing list first. That sums up btrfs pretty well. Fifteen years in, and the filesystem's design is still so half-baked that their "check" program can't reliably identify problems and fix them correctly. You have to have a developer look at the errors and then tell you wha to do. Fifteen years in. Nobody cares about btrfs anymore because everyone knows someone who has been burned by it. Which is a shame, because it can do both metadata and data rebalancing and defragmentation, as well as do things like spread N copies of data across X drives (though this feature is almost entirely negated by metadata not having this capability. Again, fifteen years in, why is this still a thing?) and one can add/remove drives from a btrfs volume without consequence. But...it's not able to do stuff like have a volume made up of mirrored pairs, RAID5/6 are (still) unstable (fifteen years in, why is this still a thing?) Do yourself a favor and just stick with ext4 for smaller/simple filesystem needs, XFS where you need the best possible speed or for anything big with lots of files (on md if necessary), or OpenZFS. Now that the BSD and Linux folks have combined forces and are developing OpenZFS together it keeps getting better and better; btrfs's advantages over ZFS just aren't worth the headaches. ZFS's major failing is that it offers no way to address inevitable filesystem data and free space fragmentation, and while you can remove devices from a ZFS pool, it incurs a permanent performance penalty, because they work around ZFS's architectural inflexibilities by adding a mapping table so it can find the moved chunks of data. That mapping table never goes away unless you erase and re-create the file. Which I suppose isn't the end of the world; technically, you could have a script that walked the filesystem re-creating files, but that brings its own problems. That the fs can't address this stuff internally is particularly a bummer considering that ZFS is intended to be used in massive (petabyte to exobyte) filesystems where it would be completely impractical to "just" move data to a fresh ZFS filesystem and back again (the main suggestion for fragmentation.) But...btrfs doesn't offer external (and mirrored!) transaction logging devices, SSD cache, or concepts like pairs of mirrored drives being used in stripes or contiguous chunks. If ZFS ever manages to add maintenance to the list of things it excels at, there will be few arguments against it except for situations where its memory use isn't practical.
- oDot 2y ago
  Thank you for the thorough explanation
anonymousiam 2y ago
I tried btrfs for the first time a few weeks ago. I had been looking for mature r/w filesystems that support realtime compression, and btrfs seemed like a good choice. My use case is this: I normally make full disk images of my systems and store them on a (100TB) NAS. As the number of systems grows, the space available for multiple backup generations shrinks. So compression of disk images is good, until you want to recover something from a compressed disk image without doing a full restore. If I put an uncompressed disk image in a compressed (zstd) btrfs filesystem, I can mount volumes and access specific files without waiting days to uncompress. So I gave it a try and did a backup of an 8TB SSD image to a btrfs filesystem, and it consumed less than 4TB, which was great. I was able to mount partitions and access individual files within the compressed image. The next thing I tried, was refreshing the backup of a specific partition within the disk image. That did not go well. Here's what I did to make the initial backup: (This was done on an up-to-date Ubuntu 24.04 desktop.) cd /btrfs.backups truncate -s 7696581394432 8TB-Thinkpad.btrfs mkfs.btrfs -L btrfs.backup 8TB-Thinkpad.btrfs mount -o compress=zstd 8TB-Thinkpad.btrfs /mnt/1 pv < /dev/nvme0n1 > /mnt/1/8TB-Thinkpad.nvme0n1 All good so far. The backup took about three hours, but would probably go twice as fast if I had used my TB4 dock instead of a regular USB-C port for the backup media. Things went bad when I tried to update one of the backed up partitions: kpartx -a /mnt/1/8TB-Thinkpad.nvme0n1 pv < /dev/nvme0n1p5 > /dev/mapper/loop20p5 This sort of thing works just fine on a normal, uncompressed ext4 filesystem. I did not really expect this to work here, and my expectations were met. The result was a bunch of kernel errors, the backup device being remounted r/o, and a corrupt btrfs filesystem with a corrupt backup file. So for this use case, btrfs is a big improvement for reading compressed disk images on the fly, but it is not suitable for re-writing sections of disk images.
gavinsyancey 2y ago
Btrfs has been slowly eating my data; randomly small files or sectors of larger files will be replaced with all nulls.
yarg 2y ago
I had it go boom on Tumbleweed (when the drive filled up) less than a year ago. I tried accessing and fixing the fubar partition from a parallel install, but to no avail.
seanw444 2y ago
Haven't had any issues with it after using it for years on my work and home PCs. I use transparent compression, snapshots, and send/receive, and they all work great. The main complaint was always about parity RAID, which I still wouldn't recommend running from what I've heard. But RAID 1-10 have been stable.

Is anyone here using BTRFS and can comment on its current-day stability? I used to read horror stories about it

by oDot 2y ago

Is anyone here using BTRFS and can comment on its current-day stability? I used to read horror stories about it

quotemstr 2y ago

I've used it for my personal machines and backups (via btrbk) for years without any issues

einsteinx2 2y ago

I’ve been using it for a few years on my NAS for the all the data drives (with Snapraid for parity and data validation), and as the boot drive on a few SBCs that run various services. Also use it as the boot drive for my Linux desktop PC. So far no problems at all and I make heavy use of snapshots, I have also had various things like power outages that have shut down the various machines multiple times. I’ve never used BTRFS raid so can’t speak to that, but in my personal experience I’ve found BTRFS and the snapshot system to be reliable. Seems like most (all?) stories I hear about corruption and other problems are all from years ago when it was less stable (years before I started using it). Or maybe I just got lucky ¯\_(ツ)_/¯

AtlasBarfed 2y ago
Why use BTRfS raid rather than good old MDadm raid?
- einsteinx2 2y ago
  I don’t use BTRFS raid, I don’t actually use any RAID. I use SnapRAID which is really more of a parity system than real RAID. I have a bunch of data disks that are formatted BTRFS, then 2 parity disks formatted using ext4 since they don’t require any BTRFS features. Then I use snapraid-btrfs which is a wrapper around SnapRAID to automatically generate BTRFS snapshots on the data disks when doing a SnapRAID sync. Since the parity is file based, it’s best to use it with snapshots, so that’s the solution I went with. I’m sure you could also use LVM snapshots with ext4 or ZFS snapshots, but BTRFS with SnapRAID is well supported and I like how BTRFS snapshots/subvolumes works so I went with that. Also BTRFS has some nice features over ext4 like CoW and checksumming. I considered regular RAID but I don’t need the bandwidth increase over single disks and I didn’t ever want the chance of losing a whole RAID pool. With my SnapRAID setup I can lose any 2 drives and not lose any data, and if I lose 3 drives, I only lose the data on any lost data drives, not all the data. Also it’s easy to add a single drive at a time as I need more space. That was my thought process when choosing it anyway and it’s worked for my use case (I don’t need much IOPS or bandwidth, just lots of cheap fairly resilient and easy to expand storage).
- tremon 2y ago
  BTRFS raid is usage-aware, so a rebuild will not need to do a bit-for-bit copy of the entire disk, but only the parts that are actually in use. Also, because btrfs has data checksumming, it can detect read errors even when the disk reports a successful read (however, it will not verify the checksum during regular operation, only during scrub).
- gmokki 2y ago
  BTRFS Raid10 can seemlessly cpmbole multiple raw disks without trying to match capacity. Next time I just replace my 4T disk in my 5 disk Raid10 with 20T. Currently I have 4+8+8+16+20 disks. MD raid does not do checksumming. Although I believe XFS is about to add support for it in the future. I have had my BTRFS raid filesystem survive a lot during the past 14 years: - burned power: no loss of data - failed ram that started corrupting memory: after a little hack 1) BTRFS scrub saved most of data even though the situation got so bad kernel would crash in 10 minutes - buggy pcie SATA extension card: I tried to add 6th disk, but noticed after fee million write errors to one disk that it just randomly stopped passing data through: no data corruption, although btrfs write error counters are in 10s of millions now - 4 disk failures: I have only one original disk still running and it is showing a lot of bad sectors 1) one of the corrupted sectors was in the btrfs tree that contains the checksums for rest of the filesystem and both copies were broken. It prevented access to some 200 files. I patched the kernel to log the exact sector in addition to the expected and actual value. Turns our it was just a single bit flip. So I used hex editor to flip it back to correct value and got the files back
- ThatPlayer 2y ago
  More flexibility in drives. Btrfs's RAID1, isn't actually RAID1 where everything is written to all the drives, but closer to RAID10 it writes all data to copies on 2 drives. So you can have a 1+2+3 TB drive in an array and still get 3TB of usable storage, or even 1+1+1+1+4. And you can add/remove single drives easily. You can also set different RAID levels for metadata versus data, because the raid knows the difference. At some point in the future you might be able to set it per-file too.

jcalvinowens 2y ago

I've used BTRFS exclusively for over a decade now on all my personal laptops, servers, and embedded devices. I've never had a single problem. It's the flagship Linux filesystem: outside of database workloads, I don't understand why anybody uses anything else.

KennyBlanken 2y ago
"Flagship"? I don't know a single person who uses it in production systems. It's the only filesystem I've lost data to. Ditto for friends. Please go look up survivor bias. That's what all you btrfs fanboys don't seem to understand. It doesn't matter how well it has worked for 99.9% of you. Filesystems have to be the most reliable component in an operating system. It's a flagship whose fsck requires you to contact developers to seek advice on how to use it because otherwise it might destroy your filesystem. It's a flagship whose userspace tools, fifteen years in, are still seeing major changes. It's a flagship whose design is so poor that fifteen years in the developers are making major changes to its structure and depreciating old features in ways that do not trigger an automatic upgrade or informative error to upgrade, but cause the filesystem to panic with error messages for which there is no documentation and little clue what the problem is. No other filesystem has these issues.
- jchw 2y ago
  Btrfs is in production all over the damn place, at big corporations and all kinds of different deployments. Synology has their own btrfs setup that they ship to customers with their NAS software for example. I found it incredibly annoying the first time I ran out of disk space on btrfs, but many of these points are hyperbolic and honestly just silly. For example, btrfs doesn't really do offline fsck. fsck.btrfs has a zero percent chance of destroying your volume because it does nothing. As for the user space utilities changing... I'm not sure how that demonstrates the filesystem is not production ready. Personally I usually use either XFS or btrfs as my root filesystem. While I've caught some snags with btrfs, I've never lost any data. I don't actually know anyone who has, I've merely just heard about it. And it's not like other well-regarded filesystems have never ran into data loss situations: even OpenZFS recently (about a year ago) uncovered a data-eating bug that called its reliability into question. I'm sure some people will angrily tell me that actually btrfs is shit and the worst thing to ever be created and honestly whatever. I am not passionate about filesystems. Wake me up when there's a better one and it's mainlined. Maybe it will eventually be bcachefs. (Edit: and just to be clear, I do realize bcachefs is mainline and Kent Overstreet considers it to be stable and safe. However, it's still young and it's upstream future has been called into question. For non-technical reasons, but still; it does make me less confident.)

badsectoracula 2y ago

I've been using it for a few years now on my main PC (has a couple SSDs and a large HDD) and my laptop, it was the default of openSUSE and just used that. Then i realized that snapshots are a feature i didn't knew i wanted :-P. Never had a problem, though it is annoying that whatever BTRFS thinks is free space and what the rest of the OS thinks is free space do not always align. It has rarely been a problem in practice though.

KennyBlanken 2y ago

The documentation describes 'btrfs check' as being dangerous to run without consulting the mailing list first. That sums up btrfs pretty well. Fifteen years in, and the filesystem's design is still so half-baked that their "check" program can't reliably identify problems and fix them correctly. You have to have a developer look at the errors and then tell you wha to do. Fifteen years in. Nobody cares about btrfs anymore because everyone knows someone who has been burned by it. Which is a shame, because it can do both metadata and data rebalancing and defragmentation, as well as do things like spread N copies of data across X drives (though this feature is almost entirely negated by metadata not having this capability. Again, fifteen years in, why is this still a thing?) and one can add/remove drives from a btrfs volume without consequence. But...it's not able to do stuff like have a volume made up of mirrored pairs, RAID5/6 are (still) unstable (fifteen years in, why is this still a thing?) Do yourself a favor and just stick with ext4 for smaller/simple filesystem needs, XFS where you need the best possible speed or for anything big with lots of files (on md if necessary), or OpenZFS. Now that the BSD and Linux folks have combined forces and are developing OpenZFS together it keeps getting better and better; btrfs's advantages over ZFS just aren't worth the headaches. ZFS's major failing is that it offers no way to address inevitable filesystem data and free space fragmentation, and while you can remove devices from a ZFS pool, it incurs a permanent performance penalty, because they work around ZFS's architectural inflexibilities by adding a mapping table so it can find the moved chunks of data. That mapping table never goes away unless you erase and re-create the file. Which I suppose isn't the end of the world; technically, you could have a script that walked the filesystem re-creating files, but that brings its own problems. That the fs can't address this stuff internally is particularly a bummer considering that ZFS is intended to be used in massive (petabyte to exobyte) filesystems where it would be completely impractical to "just" move data to a fresh ZFS filesystem and back again (the main suggestion for fragmentation.) But...btrfs doesn't offer external (and mirrored!) transaction logging devices, SSD cache, or concepts like pairs of mirrored drives being used in stripes or contiguous chunks. If ZFS ever manages to add maintenance to the list of things it excels at, there will be few arguments against it except for situations where its memory use isn't practical.

oDot 2y ago
Thank you for the thorough explanation

anonymousiam 2y ago

I tried btrfs for the first time a few weeks ago. I had been looking for mature r/w filesystems that support realtime compression, and btrfs seemed like a good choice. My use case is this: I normally make full disk images of my systems and store them on a (100TB) NAS. As the number of systems grows, the space available for multiple backup generations shrinks. So compression of disk images is good, until you want to recover something from a compressed disk image without doing a full restore. If I put an uncompressed disk image in a compressed (zstd) btrfs filesystem, I can mount volumes and access specific files without waiting days to uncompress. So I gave it a try and did a backup of an 8TB SSD image to a btrfs filesystem, and it consumed less than 4TB, which was great. I was able to mount partitions and access individual files within the compressed image. The next thing I tried, was refreshing the backup of a specific partition within the disk image. That did not go well. Here's what I did to make the initial backup: (This was done on an up-to-date Ubuntu 24.04 desktop.) cd /btrfs.backups truncate -s 7696581394432 8TB-Thinkpad.btrfs mkfs.btrfs -L btrfs.backup 8TB-Thinkpad.btrfs mount -o compress=zstd 8TB-Thinkpad.btrfs /mnt/1 pv < /dev/nvme0n1 > /mnt/1/8TB-Thinkpad.nvme0n1 All good so far. The backup took about three hours, but would probably go twice as fast if I had used my TB4 dock instead of a regular USB-C port for the backup media. Things went bad when I tried to update one of the backed up partitions: kpartx -a /mnt/1/8TB-Thinkpad.nvme0n1 pv < /dev/nvme0n1p5 > /dev/mapper/loop20p5 This sort of thing works just fine on a normal, uncompressed ext4 filesystem. I did not really expect this to work here, and my expectations were met. The result was a bunch of kernel errors, the backup device being remounted r/o, and a corrupt btrfs filesystem with a corrupt backup file. So for this use case, btrfs is a big improvement for reading compressed disk images on the fly, but it is not suitable for re-writing sections of disk images.

gavinsyancey 2y ago

Btrfs has been slowly eating my data; randomly small files or sectors of larger files will be replaced with all nulls.

yarg 2y ago

I had it go boom on Tumbleweed (when the drive filled up) less than a year ago. I tried accessing and fixing the fubar partition from a parallel install, but to no avail.

seanw444 2y ago

Haven't had any issues with it after using it for years on my work and home PCs. I use transparent compression, snapshots, and send/receive, and they all work great. The main complaint was always about parity RAID, which I still wouldn't recommend running from what I've heard. But RAID 1-10 have been stable.