13 ms·

Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

251 points by pantalaimon 4y ago

alschwalm 4y ago
I'm curious how the benchmarks are for Btrfs on 6.1, given the improvements that (I think) landed in it: https://www.phoronix.com/news/Linux-6.1-Btrfs https://www.phoronix.com/news/Linux-6.1-Btrfs
- DominoTree 4y ago
  This is exactly the sort of thing that Phoronix (will probably) benchmark
nix23 4y ago
Oh yeah, i like reliable filesystems like xfs or zfs ;)
- formerly_proven 4y ago
  extraneous files seroed Though I've used XFS a lot over the years. Mostly because the Debian installer gave 12yo me the choice between ext2, ext3 and xfs, so XFS it was because it sounded cooler.
  - nix23 4y ago
    >so XFS it was because it sounded cooler. That's what redhat though ;)
  - doubled112 4y ago
    I've had pacman and ext4 zero a whole bunch of system files on a forced shutdown during updates. Reminded me of XFS back when.
    - ilyt 4y ago
      Come to think of it vast majority of fs voes was ext3/4 family, I've seen zeroed file, I've also seen file replaced by content of another file on unclean shutdowns
- kcb 4y ago
  Btrfs let me down one time and for a file system one time is too many.
  - warmwaffles 4y ago
    I've been let down by ZFS once before when I wanted to add more drives to an existing pool.
    - sliken 4y ago
      Did you lose data?
      - warmwaffles 4y ago
        No, and I haven't lost data with BTRFS in RAID6 either.
        nix23 4y ago
        >I haven't lost data User error...both with btrfs and zfs ;)
mberning 4y ago
Wonder if this will make it into Synology DSM 7.2. Seems unlikely based on the timing.
- imhoguy 4y ago
  Unlikely, I am on DSM 7.1 and it is 4.4.180+, although I know heavily patched. I have read DSM 7.2 will land 5.10.
  - ksec 4y ago
    I don't think Synology ever update their kernel with DSM. So the kernel you get when you buy it will be the one that last for its life time. But I do hope there will be model, especially entry model that comes with 5.10. Synology's Linux timeline is simply too slow.
    - vetinari 4y ago
      They might update their kernel, just not the kernel version. They seem to be using SoCs in their devices and then respective BSPs by the SoC vendor, which are notorious for not being updated for a new kernel. I have one of the infamous Atom C2538-based Synology there, and it is running kernel 3.10.108, built in october 2022. The device is 2019 model (the SoC was introduced in 2013).
- fetzu 4y ago
  Aren’t Kernel versions tied to device model for Synology? My DS918+ returns “4.4.180+” as its kernel version. That’s pretty.. old? Do/can they downstream some of the changes without changing the Kernel version?
walrus01 4y ago
I wonder how this compares to just using mdadm block device level raid5 or raid6. And then a normal filesystem on top.
- warmwaffles 4y ago
  The flexibility of software RAID is nice that you can mix and match hard drive manufacturers and generally have zero issues. For hardware RAID, I've always been told to stick to one drive family from one manufacturer and not to mix and match.
  - loeg 4y ago
    mdadm is software raid; it's just at the block device layer, rather than part of the filesystem.
    - warmwaffles 4y ago
      Oh that's interesting. I've never used mdadm before. At least not knowingly.
  - reisse 4y ago
    The biggest problem for hardware RAID is controller compatibility. If the controller dies, chances are the whole array is dead, if you couldn't find the exactly same model.
  - sliken 4y ago
    Generally every recommendation I've heard is don't use hardware RAID. Some of the problems: * RAID cards usually have under powered CPUs, and can easily be a bottleneck. Often the best performance with a RAID card is with the RAID disabled. * Even battery backed RAM is often pretty slow and on the wrong end of a high latency connection between CPU/RAM and disks. Generally investments in ram or intentlog/writelog/slog is a better investment. * Metadata is often undocumented, often needs backed up via obscure device dependent methods, and is required for a RAID adapter failure * The firmware is often buggy, especially in the handling of the numerous failure modes. * Recovery often requires the same card with the same firmware and a copy of the backed up metadata. Not all cards can recover all metadata from just the drives. * Often are "too" smart, and won't export raw drives for use with more advanced filesystems like btrfs or ZFS. Some require ugly work arounds like exporting each disks as a single disk RAID0. * Often RAID cards hide the SMART info from the OS, often crippling the ability to predict drive failures. Or hides the functionality behind a weird software stack that assumes a SMTP server and integrated poorly into whatever monitoring/alerting system you use for the operating system. * Some RAID cards require a network connection and run a buggy and insecure out of data web stack on some undocumented and rarely (if ever) patched CPU that was obsolete the day it shipped.
candiddevmike 4y ago
Maybe I'm just cynical but I think the ship has sailed for BTRFS RAID 5/6. It's now part of the global mindshare that BTRFS RAID 5/6 == data loss, no one wants to be the guinea pig that proves it works. Better to direct resources towards bcachefs or ZFS IMO.
- viraptor 4y ago
  Ideally we'd be working based on actual information rather than global mindshare. There will be guinea pigs to test it. There are already quite a few people running it despite the warnings. As much as I want to see a wide use of bcachefs, it's still years away. As someone who actually wants to store data - why would you direct resources to bcachefs which is known experimental, rather than btrfs which plainly documented raid5 as not ready and now may decide to change it to ready... if it is? (I'm donating to bcachefs patreon, but commenting from the PoV of someone choosing a solution today)
  - gjs278 4y ago
  - doublepg23 4y ago
    Is bcachefs years away? My understanding is the critical chain of code is pretty well tested - bcache.
    - viraptor 4y ago
      From widespread use, yes. Bcache was already stable at least 7 years ago. Bcachefs it's not even included in upstream kernel yet, much less any distro as an official option. There's only some plan to start submitting it soon™ and it will take months to complete.
  - ilyt 4y ago
    Right but "actual information" here is "developers over and over fail to nail what ZFS and MDRAID did well decades ago". There is no reason to trust their next try will be "the one"
seanw444 4y ago
Glad to see Btrfs getting continual updates. It's my favorite filesystem for my personal machines (work and home PCs). The feature set is just awesome. I just hope it doesn't get completely abandoned as its development seems to have slowed significantly. The only thing it's missing before I consider it full-featured is stable RAID 5/6. But it looks like that hasn't been forgotten.
- warmwaffles 4y ago
  > The only thing it's missing before I consider it full-featured is stable RAID 5/6. But it looks like that hasn't been forgotten. I've been running BTRFS RAID6 since 2016 and have only had one issue (arch kernel needed to be rolled back) and never suffered anything catastrophic. It's perfectly happy humming along with a 15x8TB raid array.
  - Tuna-Fish 4y ago
    Are you using Raid6 for both data and metadata? If yes, you could just as well have been running raid 0 and having more space and performance out of the same disks. Because of well documented issues, metadata on raid6 is not yet safe in power/hw failure situations. And if you have no safety in those situations, what is raid even for? (It's perfectly safe to run metadata on raid1 and data on raid5/6.)
    - ludocode 4y ago
      With data on raid6 you should be running metadata on raid1c3 or raid1c4, not raid1 nor raid5/6.
      - nwallin 4y ago
        Wait, what's wrong with metadata on raid1?
        wtallis 4y ago
        Metadata on raid1 doesn't guarantee the resilience to two-drive failures that raid6 is used for, but metadata on raid1c3 does give you that degree of resilience (with more space overhead, but that's seldom a problem for the metadata).
herpderperator 4y ago
I've been managing a raid6 ext4 array with mdadm for 10 years. Started with 4 x 4TB disks and kept adding, up to 11 disks now. It works reliably and as designed. Had a few disk failures and replaced them without issues. That's one of the nice things about mdadm vs ZFS: you can add and remove disks from the array as you see fit, rather than being forced to upgrade all disks if you want to increase the size of your array.
- oarsinsync 4y ago
  I did that for a long time. I reached 15 disks. Then something went wrong, and I lost everything. I now run multiple 6 disk raidz2s, in part because I wanted to remove the option that led me down the bad path. I hope your journey ends better than mine did.
  - imiric 4y ago
    Reading these horror stories just makes me not want to rely on smart filesystems at all. If a bug can take out my whole array, or if the system is as inflexible and magical as ZFS, I'd rather not rely on it at all. I've been managing a home NAS for years with SnapRAID and plain old ext4. Sure, it doesn't have the bells and whistles of something like ZFS, but it's simple to use and understand, scales with anything you throw at it, and there's no way to lose the whole array. I'm currently transitioning to a multi-node setup, and will probably move to Ceph, otherwise I would stick with SnapRAID. Every so often I look into btrfs/ZFS/mdadm, but keep reaching the same conclusion that it's just not worth the risk.
    - michaelmrose 4y ago
      Cars require regular maintenance and have known failure modes if you don't bother to do it its neither magical nor dangerous. Snapraid + ext4 doesn't particularly simpler than zfs just different. The fundamental tradeoff of space vs reliability seems to be mathematically tied. It's impossible to do better in one dimension without sacrificing the other. Making it possible to lose SOME of your data in fact seems to be the worst choice. ZFS makes it trivial to lose nothing. Use enough hardware and do cheap replication regularly and scrub periodically. If anything the fact that snapraid by not live replicating every change to every disk in the array makes it impossible to achieve the reliability offered by ZFS. It's forever the inferior cousin and in fact more complicated for layering a dissimilar technology on top of the other.
dale_glass 4y ago
But is the RAID handling remotely sane yet? See https://arstechnica.com/gadgets/2021/09/examining-btrfs-linuxs-perpetually-half-finished-filesystem/ https://arstechnica.com/gadgets/2021/09/examining-btrfs-linu... It has gems such as: * It won't boot on a degraded array by default, requiring manual action to mount it * It won't complain if one of the disks is stale * It won't resilver automatically if a disk is re-added to the array I think the first is the killer. RAID is a High Availability measure. Your system is not Available if it fails to boot.
- klysm 4y ago
  > * It won't boot on a degraded array by default, requiring manual action to mount it That by itself is a complete deal breaker.
  - chomp 4y ago
    Why? Are you in a position where you can’t modify grub or fstab?
    - dale_glass 4y ago
      Because RAID is a High Availability technology. It's not a backup. It doesn't protect against mistakes, bugs, or the server failing in some other way. What it's good for is for ensuring that work keeps happening if a disk fails. And disk failures happen to be fairly frequent, since spinning rust is a rather delicate technology. If somebody has to connect to the machine and fiddle with it by hand it means that the system has been down, possibly for hours, when that was the exact thing you were trying to prevent by setting up RAID on it.
      - viraptor 4y ago
        You don't have to modify it after a failure. You can set it right now to explicitly say "when I end up in a degraded state, I want to boot anyway". > What it's good for is for ensuring that work keeps happening if a disk fails. Different services have different priorities. I want my work servers to keep running, but my home server under the desk to stop until I replace the failed drive. Btrfs gives you a choice.
DominoTree 4y ago
I've never lost data with btrfs, but I have purposefully avoided anything other than basic disk configurations. I use it on my workstations because it's the default in Fedora these days and haven't had any complaints. On the flip-side, I've done some of the most horrible things possible and screwed up my 30-disk ZFS array a number of times and I've never lost data. I doubt btrfs could recover from anything I've done to break my ZFS pool. Overall I think it's a huge shame that it was determined that the CDDL was incompatible with the GPL, because it wasn't intentional, and we would be in a completely different spot for storage otherwise.
- fguerraz 4y ago
  Never lost any data either in 10y of use as main driver. I've ditched RAID 5 mostly because performance sucked so much, but I have never lost any data despite going through many changes of hard drives, changes of RAID mode, rebalancing, etc. The closest I've come to losing data is probably rotten HDD sectors, but it's always been caught and remapped by scrubs. Yes, there is the infamous write hole, but that's not a BTRFS only problem.
- Gigachad 4y ago
  I feel like the data loss risk doesn't really make sense to be worried about. It doesn't matter what FS or storage system you use, you _need_ a backup if you actually care about the data. And if you have backups, you are fine. As a random anecdote I've been using BTRFS for everything for over 7 years now and had no issues. Think the only FS I've ever had problems with was ExFAT.
  - pavon 4y ago
    I think it is important for two reasons. First, the primary purpose of RAID is to improve availability of your data, such that you can continue operating in spite of a hardware failure. If the RAID feature of btrfs has serious data loss problems, then it is taking a feature that is supposed to increase availability, and instead decreasing it. It is like an UPS that is more likely to cause a power outage than to protect you from one. The other reason is that the atomic snapshot feature, and ability to easily transfer diffs of snapshots makes a wonderful foundation for incremental backups. But if I can't trust the filesystem to avoid corrupting my data, how can I trust it to avoid corrupting my snapshots? Hence I can't trust my backups. So I need to go back to a more independent backup processes like rsync. With those two features gone, my whole motivation for using btrfs over simpler file systems like ext4 is gone, so why bother.
helf 4y ago
I’ll trust btrfs in another 2 decades maybe. hugs zfs
- mekster 4y ago
  It'll be irrelevant by then.
Pr0ject217 4y ago
I haven't had any issues with BTRFS yet (2yrs, arch). I've had to rollback to a previous snapshot once (a self-inflicted mistake during an update).
helf 4y ago
I ran into a major problem with performance on btrfs when used on an extremely high write volume that filled up. Between fragmentation and full disk I couldn’t even /copy/ data off the drive at a reasonable rate. It was a perfect storm of me being an idiot and heavy use. But I dunno if they have remotely resolved the issues
- Teknoman117 4y ago
  I actually hit this for the first time in all my years of using btrfs a few weeks ago. Every month or so I spin up external drives to write out monthly backups of my NAS and personal computers. Forgot I'd taken on the home video collection and overfilled the array. Deleting some old subvolumes and having the space reclaimed took hours (on a 99.9% full 5 TB RAID-1 volume). Part of that is that I use compress=zstd:10 on the external drives, but I wasn't expecting it. Writes speeds tanked to tens of KB/s. Since these are those Seagate drives from Costco, probably also SMR drives...
  - helf 4y ago
    Yes! I actually gave up on the one drive. It was a torrent disk (like I said worst case scenario for a lot of things) and btrfs ate itself lol.
    - Teknoman117 4y ago
      To be fair, it didn't break or lose data. Just got really, really slow until I let it finish reclaiming space.
      - helf 4y ago
        Yeah. On my end it just slowed to a completely unusable level. So while it technically didn’t lose data it effectively did since I was looking at weeks to replicate any data off it. I let it run for a week and it was still crawling when I gave up (4TB drive)
        4y ago
freeqaz 4y ago
We run BTRFS on our work laptops and have been for a few years now. Keep in mind that we're only 4 devs, so our sample size is small, but we haven't had any issues with it so far and it's been very pleasant to use so far! We published our internal doc for how we install our Arch setup with fully encrypted BTRFS if anybody is curious. Happy to answer any questions too! https://www.lunasec.io/docs/blog/arch-linux-installation-guide/ https://www.lunasec.io/docs/blog/arch-linux-installation-gui... (We run Thinkpad X1 Extremes (+ the business P1 equivalents) as our dev boxes.)
- encryptluks2 4y ago
  That is a pretty awesome article. I recently reinstalled Arch on a new PC and used archinstall to setup encryption, but they had a few bad defaults that I had to later correct... like using a sha512 hash algo, which slowed down opening the drives significantly, and somehow adding 2 different types of compression to the fstab. Once that was fixed, it has been working great. You have a typo in your article BTW: > aes-exts-plain64
- em-bee 4y ago
  i am running fedora on encrypted btrfs, on top of a mirror raid, and i have a lot of problem with the performance of the disk. I/O frequently hangs the GUI for a few seconds. throughput is fine once it gets going, but random access to many files appears to be an issue. searches pointing to btrfs on encryption being slow, but i think the raid is also a factor.
  - M911T 4y ago
    Check both of these links, I added the configurations to my laptop with BTRFS + NVME SSD + encryption and it's snappy. https://wiki.archlinux.org/title/Dm-crypt/Specialties#Discard/TRIM_support_for_solid_state_drives_(SSD) https://wiki.archlinux.org/title/Dm-crypt/Specialties#Discar... https://wiki.archlinux.org/title/Dm-crypt/Specialties#Disable_workqueue_for_increased_solid_state_drive_(SSD)_performance https://wiki.archlinux.org/title/Dm-crypt/Specialties#Disabl...
Krisjohn 4y ago
We never had a BTRFS event on OS drives that didn't result in total loss of the OS. We have finally removed it from the last server it was put on.
i13e 4y ago
I use BTRFS on my laptop and in case anyone is interested in trying it, please be sure you know how to chroot into the system in case it won't start. Arch Linux had a problematic GRUB package pushed to stable a few months ago and it bricked people's computers by booting directly to the BIOS. Whatever you do, make sure to take notes on the configuration (if it is DIY) because there are many difference between BTRFS and EXT4, so you can't use a normal chroot guide to fix your system.
stormking 4y ago
If you value your data, you use ZFS.
shmerl 4y ago
I'm still waiting for bcachefs upstreaming.
- ahartmetz 4y ago
  Yeah, looks like that one is actually designed to do what it does, not just "somehow implemented". The BTRFS bugs fixed in every kernel release for how many years now? are horrific and show that BTRFS is crawling with corner cases.
  - leetnewb 4y ago
    Are you saying that complex software getting bug fixes is an indication of something? I don't see how it's a bad thing that btrfs has been squashing bugs and corner cases.
unixhero 4y ago
I have used btrfs for a low memory footprint raid 0 solution to span drives. Mind you for expendable datasets. It has worked really well, never had failures due to btrfs shall we say -rot. Filesystem is light on requirements and works really well. Compression works great too. The only thing I haven't played around with are reflinks. For critical datasets I apply 3-2-1 backup strategy and use ZFS.
boardwaalk 4y ago
As someone who is in the middle of pulling apart a BTRFS volume by hand (read: writing code to interpret the data structures) to try and recover it, I think being burnt enough is once. No indication of any hardware issue: No recent power loss (& it's on a UPS), no SMART issues, no memory test positives. But the block tree (at least, WIP) is f*cked across all the disks (looks like two competing writers went at it) and none of the available tools can deal with it. It wasn't a super exotic setup either: RAID10 with 4 disks (2 stripes), fairly full and regular snapshots/cleanup, but that's it. I already converted my root to ext4 because paranoia and I'm probably going to move bulk data (what can be recovered) to ZFS.
- stingraycharles 4y ago
  For what it's worth, about a year ago I had data being corrupted with ZFS (something about NVMe + raidz + dedup if I recall correctly) -- this was fortunately while I was testing deployments, and quickly confirmed as a bug (and fixed) by the ZFS team. But it left me equally burned. Using xfs|ext4 + mdraid is just a lot simpler and much faster, and it addresses most of my use cases.
- AshamedCaptain 4y ago
  This just can't happen with btrfs since the _old_ block tree should be still in the disk somewhere. i.e. to corrupt the image so that btrfs shits itself is easy to do, but the _previous_ version of the data is still there by construction and if you are already manipulating the data structures it should be easy enough to just point the sb to it. The worst issue I've ever had with btrfs simply required zeroing the journal (by hand -- both the kernel and *fsck tools would crash when reading it). The first time I reported the issue to the mailing list and the underlying issue was promptly fixed. However, it still happened a second time, and I didn't bother reporting it. As many people say on this thread, once is too many when it comes to filesystems.
- AdamJacobMuller 4y ago
  I've been there, not with BTRFS but with reiserfs many many years ago. I don't envy you.
aborsy 4y ago
For important data, we use ZFS in our shop.
- timbit42 4y ago
  For important data, I use backups.
whatever1 4y ago
I read all these horror stories here for RAID config and data availability/integrity and I wonder how do the Cloud providers resolve all these issues at their scale? Why bare-metal admins have to spend countless man-hours on troubleshooting these?
- BatteryMountain 4y ago
  BSD + ZFS
- CoolCold 4y ago
  You can start with backblaze articles on their data storage tier, will give some hints Also note, comments on this thread are heavily biased on DYI things and not covering enterprise storage areas, which for instance can come as Blackbox with exported LUNs
vermaden 4y ago
Just 'better' (less fuckups) or it works 100% and its production ready? :) ... just use FreeBSD with OpenZFS for RAIDZ/RAIDZ2/RAIDZ3 pools instead. ... or even OpenZFS on Linux.
crest 4y ago
"This should ..." doesn't belong in a fucking file system! Take a deep breath and try again. Maybe limit the scope to something within your understanding this time.
mattpallissard 4y ago
I, as many, have a BTRFS raid 5/6 story. I once worked at a place that had double digit TB databases. A few of the servers that predated me were set up with BTRFS raid 10 as their backup strategy. Basically stop the server, snapshot the volume, backup from the snapshot. It worked well enough. There came a point where we were going to have to move to a larger server as there were no more slots for additional disks. New hardware procured, provisioned, prepped, ready to roll with the exception of an actual cut over. However, c-suite got involved as they tend to do from time to time, and kept kicking the can down the road. Eventually disk space was a critical issue and a junior sysadmin decided it was a good idea to tell the COO that btrfs can be converted to raid 5 on the fly. I over the phone told them that this would guarantee data loss and refused. They literally came to my office to force my hand. Some people just have to learn the hard way. That story aside, I'm a big fan of BTRFS and have been using in various configurations on my desktops for years. I'm glad to see this progressing.

251 points by pantalaimon 4y ago

I'm curious how the benchmarks are for Btrfs on 6.1, given the improvements that (I think) landed in it: https://www.phoronix.com/news/Linux-6.1-Btrfs https://www.phoronix.com/news/Linux-6.1-Btrfs

DominoTree 4y ago
This is exactly the sort of thing that Phoronix (will probably) benchmark

nix23 4y ago

Oh yeah, i like reliable filesystems like xfs or zfs ;)

formerly_proven 4y ago
extraneous files seroed Though I've used XFS a lot over the years. Mostly because the Debian installer gave 12yo me the choice between ext2, ext3 and xfs, so XFS it was because it sounded cooler.
- nix23 4y ago
  >so XFS it was because it sounded cooler. That's what redhat though ;)
- doubled112 4y ago
  I've had pacman and ext4 zero a whole bunch of system files on a forced shutdown during updates. Reminded me of XFS back when.
  - ilyt 4y ago
    Come to think of it vast majority of fs voes was ext3/4 family, I've seen zeroed file, I've also seen file replaced by content of another file on unclean shutdowns
kcb 4y ago
Btrfs let me down one time and for a file system one time is too many.
- warmwaffles 4y ago
  I've been let down by ZFS once before when I wanted to add more drives to an existing pool.
  - sliken 4y ago
    Did you lose data?
    - warmwaffles 4y ago
      No, and I haven't lost data with BTRFS in RAID6 either.
      - nix23 4y ago
        >I haven't lost data User error...both with btrfs and zfs ;)

mberning 4y ago

Wonder if this will make it into Synology DSM 7.2. Seems unlikely based on the timing.

imhoguy 4y ago
Unlikely, I am on DSM 7.1 and it is 4.4.180+, although I know heavily patched. I have read DSM 7.2 will land 5.10.
- ksec 4y ago
  I don't think Synology ever update their kernel with DSM. So the kernel you get when you buy it will be the one that last for its life time. But I do hope there will be model, especially entry model that comes with 5.10. Synology's Linux timeline is simply too slow.
  - vetinari 4y ago
    They might update their kernel, just not the kernel version. They seem to be using SoCs in their devices and then respective BSPs by the SoC vendor, which are notorious for not being updated for a new kernel. I have one of the infamous Atom C2538-based Synology there, and it is running kernel 3.10.108, built in october 2022. The device is 2019 model (the SoC was introduced in 2013).
fetzu 4y ago
Aren’t Kernel versions tied to device model for Synology? My DS918+ returns “4.4.180+” as its kernel version. That’s pretty.. old? Do/can they downstream some of the changes without changing the Kernel version?

walrus01 4y ago

I wonder how this compares to just using mdadm block device level raid5 or raid6. And then a normal filesystem on top.

warmwaffles 4y ago
The flexibility of software RAID is nice that you can mix and match hard drive manufacturers and generally have zero issues. For hardware RAID, I've always been told to stick to one drive family from one manufacturer and not to mix and match.
- loeg 4y ago
  mdadm is software raid; it's just at the block device layer, rather than part of the filesystem.
  - warmwaffles 4y ago
    Oh that's interesting. I've never used mdadm before. At least not knowingly.
- reisse 4y ago
  The biggest problem for hardware RAID is controller compatibility. If the controller dies, chances are the whole array is dead, if you couldn't find the exactly same model.
- sliken 4y ago
  Generally every recommendation I've heard is don't use hardware RAID. Some of the problems: * RAID cards usually have under powered CPUs, and can easily be a bottleneck. Often the best performance with a RAID card is with the RAID disabled. * Even battery backed RAM is often pretty slow and on the wrong end of a high latency connection between CPU/RAM and disks. Generally investments in ram or intentlog/writelog/slog is a better investment. * Metadata is often undocumented, often needs backed up via obscure device dependent methods, and is required for a RAID adapter failure * The firmware is often buggy, especially in the handling of the numerous failure modes. * Recovery often requires the same card with the same firmware and a copy of the backed up metadata. Not all cards can recover all metadata from just the drives. * Often are "too" smart, and won't export raw drives for use with more advanced filesystems like btrfs or ZFS. Some require ugly work arounds like exporting each disks as a single disk RAID0. * Often RAID cards hide the SMART info from the OS, often crippling the ability to predict drive failures. Or hides the functionality behind a weird software stack that assumes a SMTP server and integrated poorly into whatever monitoring/alerting system you use for the operating system. * Some RAID cards require a network connection and run a buggy and insecure out of data web stack on some undocumented and rarely (if ever) patched CPU that was obsolete the day it shipped.

candiddevmike 4y ago

Maybe I'm just cynical but I think the ship has sailed for BTRFS RAID 5/6. It's now part of the global mindshare that BTRFS RAID 5/6 == data loss, no one wants to be the guinea pig that proves it works. Better to direct resources towards bcachefs or ZFS IMO.

viraptor 4y ago
Ideally we'd be working based on actual information rather than global mindshare. There will be guinea pigs to test it. There are already quite a few people running it despite the warnings. As much as I want to see a wide use of bcachefs, it's still years away. As someone who actually wants to store data - why would you direct resources to bcachefs which is known experimental, rather than btrfs which plainly documented raid5 as not ready and now may decide to change it to ready... if it is? (I'm donating to bcachefs patreon, but commenting from the PoV of someone choosing a solution today)
- gjs278 4y ago
- doublepg23 4y ago
  Is bcachefs years away? My understanding is the critical chain of code is pretty well tested - bcache.
  - viraptor 4y ago
    From widespread use, yes. Bcache was already stable at least 7 years ago. Bcachefs it's not even included in upstream kernel yet, much less any distro as an official option. There's only some plan to start submitting it soon™ and it will take months to complete.
- ilyt 4y ago
  Right but "actual information" here is "developers over and over fail to nail what ZFS and MDRAID did well decades ago". There is no reason to trust their next try will be "the one"

seanw444 4y ago

Glad to see Btrfs getting continual updates. It's my favorite filesystem for my personal machines (work and home PCs). The feature set is just awesome. I just hope it doesn't get completely abandoned as its development seems to have slowed significantly. The only thing it's missing before I consider it full-featured is stable RAID 5/6. But it looks like that hasn't been forgotten.

warmwaffles 4y ago
> The only thing it's missing before I consider it full-featured is stable RAID 5/6. But it looks like that hasn't been forgotten. I've been running BTRFS RAID6 since 2016 and have only had one issue (arch kernel needed to be rolled back) and never suffered anything catastrophic. It's perfectly happy humming along with a 15x8TB raid array.
- Tuna-Fish 4y ago
  Are you using Raid6 for both data and metadata? If yes, you could just as well have been running raid 0 and having more space and performance out of the same disks. Because of well documented issues, metadata on raid6 is not yet safe in power/hw failure situations. And if you have no safety in those situations, what is raid even for? (It's perfectly safe to run metadata on raid1 and data on raid5/6.)
  - ludocode 4y ago
    With data on raid6 you should be running metadata on raid1c3 or raid1c4, not raid1 nor raid5/6.
    - nwallin 4y ago
      Wait, what's wrong with metadata on raid1?
      - wtallis 4y ago
        Metadata on raid1 doesn't guarantee the resilience to two-drive failures that raid6 is used for, but metadata on raid1c3 does give you that degree of resilience (with more space overhead, but that's seldom a problem for the metadata).

herpderperator 4y ago

I've been managing a raid6 ext4 array with mdadm for 10 years. Started with 4 x 4TB disks and kept adding, up to 11 disks now. It works reliably and as designed. Had a few disk failures and replaced them without issues. That's one of the nice things about mdadm vs ZFS: you can add and remove disks from the array as you see fit, rather than being forced to upgrade all disks if you want to increase the size of your array.

oarsinsync 4y ago
I did that for a long time. I reached 15 disks. Then something went wrong, and I lost everything. I now run multiple 6 disk raidz2s, in part because I wanted to remove the option that led me down the bad path. I hope your journey ends better than mine did.
- imiric 4y ago
  Reading these horror stories just makes me not want to rely on smart filesystems at all. If a bug can take out my whole array, or if the system is as inflexible and magical as ZFS, I'd rather not rely on it at all. I've been managing a home NAS for years with SnapRAID and plain old ext4. Sure, it doesn't have the bells and whistles of something like ZFS, but it's simple to use and understand, scales with anything you throw at it, and there's no way to lose the whole array. I'm currently transitioning to a multi-node setup, and will probably move to Ceph, otherwise I would stick with SnapRAID. Every so often I look into btrfs/ZFS/mdadm, but keep reaching the same conclusion that it's just not worth the risk.
  - michaelmrose 4y ago
    Cars require regular maintenance and have known failure modes if you don't bother to do it its neither magical nor dangerous. Snapraid + ext4 doesn't particularly simpler than zfs just different. The fundamental tradeoff of space vs reliability seems to be mathematically tied. It's impossible to do better in one dimension without sacrificing the other. Making it possible to lose SOME of your data in fact seems to be the worst choice. ZFS makes it trivial to lose nothing. Use enough hardware and do cheap replication regularly and scrub periodically. If anything the fact that snapraid by not live replicating every change to every disk in the array makes it impossible to achieve the reliability offered by ZFS. It's forever the inferior cousin and in fact more complicated for layering a dissimilar technology on top of the other.

dale_glass 4y ago

But is the RAID handling remotely sane yet? See https://arstechnica.com/gadgets/2021/09/examining-btrfs-linuxs-perpetually-half-finished-filesystem/ https://arstechnica.com/gadgets/2021/09/examining-btrfs-linu... It has gems such as: * It won't boot on a degraded array by default, requiring manual action to mount it * It won't complain if one of the disks is stale * It won't resilver automatically if a disk is re-added to the array I think the first is the killer. RAID is a High Availability measure. Your system is not Available if it fails to boot.

klysm 4y ago
> * It won't boot on a degraded array by default, requiring manual action to mount it That by itself is a complete deal breaker.
- chomp 4y ago
  Why? Are you in a position where you can’t modify grub or fstab?
  - dale_glass 4y ago
    Because RAID is a High Availability technology. It's not a backup. It doesn't protect against mistakes, bugs, or the server failing in some other way. What it's good for is for ensuring that work keeps happening if a disk fails. And disk failures happen to be fairly frequent, since spinning rust is a rather delicate technology. If somebody has to connect to the machine and fiddle with it by hand it means that the system has been down, possibly for hours, when that was the exact thing you were trying to prevent by setting up RAID on it.
    - viraptor 4y ago
      You don't have to modify it after a failure. You can set it right now to explicitly say "when I end up in a degraded state, I want to boot anyway". > What it's good for is for ensuring that work keeps happening if a disk fails. Different services have different priorities. I want my work servers to keep running, but my home server under the desk to stop until I replace the failed drive. Btrfs gives you a choice.

DominoTree 4y ago

I've never lost data with btrfs, but I have purposefully avoided anything other than basic disk configurations. I use it on my workstations because it's the default in Fedora these days and haven't had any complaints. On the flip-side, I've done some of the most horrible things possible and screwed up my 30-disk ZFS array a number of times and I've never lost data. I doubt btrfs could recover from anything I've done to break my ZFS pool. Overall I think it's a huge shame that it was determined that the CDDL was incompatible with the GPL, because it wasn't intentional, and we would be in a completely different spot for storage otherwise.

fguerraz 4y ago
Never lost any data either in 10y of use as main driver. I've ditched RAID 5 mostly because performance sucked so much, but I have never lost any data despite going through many changes of hard drives, changes of RAID mode, rebalancing, etc. The closest I've come to losing data is probably rotten HDD sectors, but it's always been caught and remapped by scrubs. Yes, there is the infamous write hole, but that's not a BTRFS only problem.
Gigachad 4y ago
I feel like the data loss risk doesn't really make sense to be worried about. It doesn't matter what FS or storage system you use, you _need_ a backup if you actually care about the data. And if you have backups, you are fine. As a random anecdote I've been using BTRFS for everything for over 7 years now and had no issues. Think the only FS I've ever had problems with was ExFAT.
- pavon 4y ago
  I think it is important for two reasons. First, the primary purpose of RAID is to improve availability of your data, such that you can continue operating in spite of a hardware failure. If the RAID feature of btrfs has serious data loss problems, then it is taking a feature that is supposed to increase availability, and instead decreasing it. It is like an UPS that is more likely to cause a power outage than to protect you from one. The other reason is that the atomic snapshot feature, and ability to easily transfer diffs of snapshots makes a wonderful foundation for incremental backups. But if I can't trust the filesystem to avoid corrupting my data, how can I trust it to avoid corrupting my snapshots? Hence I can't trust my backups. So I need to go back to a more independent backup processes like rsync. With those two features gone, my whole motivation for using btrfs over simpler file systems like ext4 is gone, so why bother.

helf 4y ago

I’ll trust btrfs in another 2 decades maybe. hugs zfs

mekster 4y ago
It'll be irrelevant by then.

Pr0ject217 4y ago

I haven't had any issues with BTRFS yet (2yrs, arch). I've had to rollback to a previous snapshot once (a self-inflicted mistake during an update).

helf 4y ago

I ran into a major problem with performance on btrfs when used on an extremely high write volume that filled up. Between fragmentation and full disk I couldn’t even /copy/ data off the drive at a reasonable rate. It was a perfect storm of me being an idiot and heavy use. But I dunno if they have remotely resolved the issues

Teknoman117 4y ago
I actually hit this for the first time in all my years of using btrfs a few weeks ago. Every month or so I spin up external drives to write out monthly backups of my NAS and personal computers. Forgot I'd taken on the home video collection and overfilled the array. Deleting some old subvolumes and having the space reclaimed took hours (on a 99.9% full 5 TB RAID-1 volume). Part of that is that I use compress=zstd:10 on the external drives, but I wasn't expecting it. Writes speeds tanked to tens of KB/s. Since these are those Seagate drives from Costco, probably also SMR drives...
- helf 4y ago
  Yes! I actually gave up on the one drive. It was a torrent disk (like I said worst case scenario for a lot of things) and btrfs ate itself lol.
  - Teknoman117 4y ago
    To be fair, it didn't break or lose data. Just got really, really slow until I let it finish reclaiming space.
    - helf 4y ago
      Yeah. On my end it just slowed to a completely unusable level. So while it technically didn’t lose data it effectively did since I was looking at weeks to replicate any data off it. I let it run for a week and it was still crawling when I gave up (4TB drive)
      - 4y ago

freeqaz 4y ago

We run BTRFS on our work laptops and have been for a few years now. Keep in mind that we're only 4 devs, so our sample size is small, but we haven't had any issues with it so far and it's been very pleasant to use so far! We published our internal doc for how we install our Arch setup with fully encrypted BTRFS if anybody is curious. Happy to answer any questions too! https://www.lunasec.io/docs/blog/arch-linux-installation-guide/ https://www.lunasec.io/docs/blog/arch-linux-installation-gui... (We run Thinkpad X1 Extremes (+ the business P1 equivalents) as our dev boxes.)

encryptluks2 4y ago
That is a pretty awesome article. I recently reinstalled Arch on a new PC and used archinstall to setup encryption, but they had a few bad defaults that I had to later correct... like using a sha512 hash algo, which slowed down opening the drives significantly, and somehow adding 2 different types of compression to the fstab. Once that was fixed, it has been working great. You have a typo in your article BTW: > aes-exts-plain64
em-bee 4y ago
i am running fedora on encrypted btrfs, on top of a mirror raid, and i have a lot of problem with the performance of the disk. I/O frequently hangs the GUI for a few seconds. throughput is fine once it gets going, but random access to many files appears to be an issue. searches pointing to btrfs on encryption being slow, but i think the raid is also a factor.
- M911T 4y ago
  Check both of these links, I added the configurations to my laptop with BTRFS + NVME SSD + encryption and it's snappy. https://wiki.archlinux.org/title/Dm-crypt/Specialties#Discard/TRIM_support_for_solid_state_drives_(SSD) https://wiki.archlinux.org/title/Dm-crypt/Specialties#Discar... https://wiki.archlinux.org/title/Dm-crypt/Specialties#Disable_workqueue_for_increased_solid_state_drive_(SSD)_performance https://wiki.archlinux.org/title/Dm-crypt/Specialties#Disabl...

Krisjohn 4y ago

We never had a BTRFS event on OS drives that didn't result in total loss of the OS. We have finally removed it from the last server it was put on.

i13e 4y ago

I use BTRFS on my laptop and in case anyone is interested in trying it, please be sure you know how to chroot into the system in case it won't start. Arch Linux had a problematic GRUB package pushed to stable a few months ago and it bricked people's computers by booting directly to the BIOS. Whatever you do, make sure to take notes on the configuration (if it is DIY) because there are many difference between BTRFS and EXT4, so you can't use a normal chroot guide to fix your system.

stormking 4y ago

If you value your data, you use ZFS.

shmerl 4y ago

I'm still waiting for bcachefs upstreaming.

ahartmetz 4y ago
Yeah, looks like that one is actually designed to do what it does, not just "somehow implemented". The BTRFS bugs fixed in every kernel release for how many years now? are horrific and show that BTRFS is crawling with corner cases.
- leetnewb 4y ago
  Are you saying that complex software getting bug fixes is an indication of something? I don't see how it's a bad thing that btrfs has been squashing bugs and corner cases.

unixhero 4y ago

I have used btrfs for a low memory footprint raid 0 solution to span drives. Mind you for expendable datasets. It has worked really well, never had failures due to btrfs shall we say -rot. Filesystem is light on requirements and works really well. Compression works great too. The only thing I haven't played around with are reflinks. For critical datasets I apply 3-2-1 backup strategy and use ZFS.

boardwaalk 4y ago

As someone who is in the middle of pulling apart a BTRFS volume by hand (read: writing code to interpret the data structures) to try and recover it, I think being burnt enough is once. No indication of any hardware issue: No recent power loss (& it's on a UPS), no SMART issues, no memory test positives. But the block tree (at least, WIP) is f*cked across all the disks (looks like two competing writers went at it) and none of the available tools can deal with it. It wasn't a super exotic setup either: RAID10 with 4 disks (2 stripes), fairly full and regular snapshots/cleanup, but that's it. I already converted my root to ext4 because paranoia and I'm probably going to move bulk data (what can be recovered) to ZFS.

stingraycharles 4y ago
For what it's worth, about a year ago I had data being corrupted with ZFS (something about NVMe + raidz + dedup if I recall correctly) -- this was fortunately while I was testing deployments, and quickly confirmed as a bug (and fixed) by the ZFS team. But it left me equally burned. Using xfs|ext4 + mdraid is just a lot simpler and much faster, and it addresses most of my use cases.
AshamedCaptain 4y ago
This just can't happen with btrfs since the _old_ block tree should be still in the disk somewhere. i.e. to corrupt the image so that btrfs shits itself is easy to do, but the _previous_ version of the data is still there by construction and if you are already manipulating the data structures it should be easy enough to just point the sb to it. The worst issue I've ever had with btrfs simply required zeroing the journal (by hand -- both the kernel and *fsck tools would crash when reading it). The first time I reported the issue to the mailing list and the underlying issue was promptly fixed. However, it still happened a second time, and I didn't bother reporting it. As many people say on this thread, once is too many when it comes to filesystems.
AdamJacobMuller 4y ago
I've been there, not with BTRFS but with reiserfs many many years ago. I don't envy you.

aborsy 4y ago

For important data, we use ZFS in our shop.

timbit42 4y ago
For important data, I use backups.

whatever1 4y ago

I read all these horror stories here for RAID config and data availability/integrity and I wonder how do the Cloud providers resolve all these issues at their scale? Why bare-metal admins have to spend countless man-hours on troubleshooting these?

BatteryMountain 4y ago
BSD + ZFS
CoolCold 4y ago
You can start with backblaze articles on their data storage tier, will give some hints Also note, comments on this thread are heavily biased on DYI things and not covering enterprise storage areas, which for instance can come as Blackbox with exported LUNs

vermaden 4y ago

Just 'better' (less fuckups) or it works 100% and its production ready? :) ... just use FreeBSD with OpenZFS for RAIDZ/RAIDZ2/RAIDZ3 pools instead. ... or even OpenZFS on Linux.

crest 4y ago

"This should ..." doesn't belong in a fucking file system! Take a deep breath and try again. Maybe limit the scope to something within your understanding this time.

mattpallissard 4y ago

I, as many, have a BTRFS raid 5/6 story. I once worked at a place that had double digit TB databases. A few of the servers that predated me were set up with BTRFS raid 10 as their backup strategy. Basically stop the server, snapshot the volume, backup from the snapshot. It worked well enough. There came a point where we were going to have to move to a larger server as there were no more slots for additional disks. New hardware procured, provisioned, prepped, ready to roll with the exception of an actual cut over. However, c-suite got involved as they tend to do from time to time, and kept kicking the can down the road. Eventually disk space was a critical issue and a junior sysadmin decided it was a good idea to tell the COO that btrfs can be converted to raid 5 on the fly. I over the phone told them that this would guarantee data loss and refused. They literally came to my office to force my hand. Some people just have to learn the hard way. That story aside, I'm a big fan of BTRFS and have been using in various configurations on my desktops for years. I'm glad to see this progressing.