16 ms·

Linux 5.10 BTRFS performance regression

141 points by shrikant 6y ago

kzrdude 6y ago
Assuming it can be fixed in a 5.10.x patch release, it's a rather minor story - a bug that is quickly found after release and fixed, hopefully. It will be investigated: https://marc.info/?l=linux-btrfs&m=160869337604422&w=2 https://marc.info/?l=linux-btrfs&m=160869337604422&w=2
- turminal 6y ago
  The problem with btrfs is that it has quirks like that (and worse, much worse) all the time.
  - throwii 6y ago
    Why does btrfs have those issues compared to other filesystems? I'd love to use btrfs too. Note that I deeply respect people who can write such complicated code, which I couldn't. Would Rust solve the non-speed issues? Rust-in-kernel discussion from August: https://lwn.net/Articles/829858/ https://lwn.net/Articles/829858/
    - panpanna 6y ago
      Please understand that the rust memory and thread security mainly applies to "normal" applications. In kernel, you can run a privileged cache or mmu instruction or a write to some magical memory position and all the sudden the "normal" rules don't apply anymore. (But I think there are other parts of rust that are nice to have in kernel or any complex software).
      - throwii 6y ago
        I thought the Rust compiler solves issues that you wouldn't immediately see with pure C, which is why I had the idea. I didn't know this requires certain features which are not available inside the kernel. I only knew all existing interfaces may be unsafe because they are in C though. Rust does not seem as useful then. Thank you for your input.
        simcop2387 6y ago
        It's not so much that rust the language requires them as much as it is that other non-rust parts can quite easily stomp all over the guarantees of rust without there ever being a way of knowing it happened. So rust alone won't solve many problems, but it would let you say "this code can't do these things itself", which is still a useful distinction. It also doesn't allow you to deal with misbehaving hardware that changes memory underneath you in ways it said won't happen. Hardware sucks.
        throwii 6y ago
        Do other parts stomp often? :) But true that can happen. Especially on non-ECC systems. I didn't think about the hardware issues, hmm. I can't see how to do that, when the compiler guarantees get invalidated by hardware. Checks are also needed like in C? (assuming there are checks which do not get compiled out..)
        simcop2387 6y ago
        When the hardware can't make the guarantees, then software really can't do anything about it. There's really not any checks you can do, but modern hardware is getting the capabilities to try to prevent those kinds of issues with the IOMMU units, but operating system support is still hit or miss for most hardware and it won't prevent everything (just devices stomping on each-other with DMA). That's basically how the thunderbolt attacks have worked and the solutions to them.
    - turminal 6y ago
      I sincerely hope you are joking but I realize this mindset is quite common these days, so let me reiterate: Rust does not magically solve problems for you. Btrfs has a lot of issues and some of them may well be of the not possible in Rust sort, but I'm quite sure most of them are not and there is nothing Rust can fix about them. Rewriting a 13 years old and very complex thing in another language is a massive effort and a big opportunity for introducing some more, possibly worse bugs along the way. EDIT: also see panpanna's comment
      - ho_schi 6y ago
        It is like a repeating story, every time someone sells the magically solution to all problems and apply a name on it people blindly believe it. This are tools. A screwdriver is not the right tool, when you need a power drill. And vice versa. Yes, sometimes you can use both and stick with your accustomed tool. Anyway. C and C++ and the tool chain are constantly improving like others.The moern memory sanitizers in GCC and LLVM are awesome.
      - throwii 6y ago
        Any rewrite must be carefully evaluated, true. And it may not be advisable. I'm trying to understand the issues. Are they not solvable because the kernel does not give enough guarantees as it gives to userspace, because the c-interfaces of the kernel have to be wrapped in unsafe or because of other reasons (architecture, data model, kernel constraints, ...)?
        ldrndll 6y ago
        I think the issue is that you haven’t understood the issue, and just proposed using Rust. Don’t get me wrong, I’m a huge fan of Rust, but seeing people blindly suggest it as a panacea gets frustrating for people. The reality is that when people talk about Rust being fact, it’s because a decently optimised Rust program might be comparable to an equivalent C program. C is generally not the problem when it comes to performance issues in the kernel, which means rewriting that thing in Rust wouldn’t magically make it faster. The issue will probably be some sort of pathological case in an algorithm being used, or perhaps from a poorly chosen algorithm. The point being, it’s not clear yet, and to solve that requires understanding the problem, not effecting a needless rewrite in a new language.
    - thomaszander 6y ago
      > Why does btrfs have those issues compared to other filesystems? As someone that has built infrastructure on BtrFS for years, the scary stories are mostly just hot air and the stability of other filesystems is really not significantly better. Bugs like this happen, this is why Linus releases many release-candiates every kernel, this one got through as 10 was a rather massive kernel and there were several regressions. Including one that caused a new release just hours after the supposedly final one. Distros wait a bit longer before shipping an updated kernel and none of these hit actual users. As far as I know there is no reason to abstain from using Btrfs. When Fedora talked about not using it, they had as reason that they had no in-house expertise.
      - throwii 6y ago
        Thank you for posting your experience!
      - circularfoyers 6y ago
        Does Fedora now have in-house expertise now that they've decided to make Btrfs default?
        curt15 6y ago
        Some Facebook btrfs devs agreed to help out. FB runs Fedora/CentOS + btrfs root on most of its systems.
      - yjftsjthsd-h 6y ago
        > As someone that has built infrastructure on BtrFS for years, the scary stories are mostly just hot air and the stability of other filesystems is really not significantly better. I've lost 2 root filesystems to btrfs, on a laptop with only 1 drive (read: not even using RAID). Have you considered that you're just lucky?
      - Volundr 6y ago
        In my experience, the stories are real. Our entire company was offline for a day when our central storage server quit accepting writes despite having over 50% free space. That's when I learned the hard way about the data/metadata split (something I was aware of but wasn't exactly top of mind) and BTRFS balance. You can certainly say it was my fault for not reading ALL the documentation before using BTRFS, and I'd find it hard to disagree, but any other filesystem wouldn't have had this problem. I can't speak to if there are other foot-guns waiting around or how common problems like this are because we migrated back to FreeBSD and ZFS shortly after that experience. I do know they have since updated BTRFS to make that scenario less likely (but still not impossible).
    - jeromenerf 6y ago
      > Why does btrfs have those issues compared to other filesystems? Mostly because it has lots of features and as a consequence, is pretty large and complex. Closer to ZFS than ext2. Btrfs suffers from a initial bad rep, which is difficult to overcome.
      - nicoburns 6y ago
        ZFS itself seems very stable and not to suffer from these issues though.
        Quekid5 6y ago
        ZFS is also extremely well designed at a system level... which is not the impression I get from BtrFS. (Disclaimer: I have not bothered looking at BtrFS for years because ZFS has handled everything I've thrown at it very admirably. Including complicated setups with RAID-Z, etc.) Granted, there are some limitations to the design, but it doesn't affect my use cases, so whatever...
        jpeloquin 6y ago
        Anecdotally, I've encountered issues with both ZFS and BTRFS at about the same rate. A public example of an apparent ZFS performance issue is https://github.com/openzfs/zfs/issues/9375 https://github.com/openzfs/zfs/issues/9375 Both are much more quirky than simpler filesystems like ext4. Data integrity verification from checksumming makes it worth it though. The ZFS vs. BTRFS choice, I think, depends more on whether you need specific features like offline deduplication or L2ARC / SLOG cache devices. And which one you're more familiar with (can troubleshoot better).
      - throwii 6y ago
        Good theory, thank you.
    - tobias3 6y ago
      My guess is this (compared to ZFS): With a CoW-file system like btrfs you have to problem that you need new file system space to delete something. This is problematic if the file system is full and you want to be able to write to it again by deleting something. ZFS solved this by just saying one can only fill a file system to 90% usage. At some point they even decreased this (during upgrade) and I had the issue that I couldn't write to the ZFS file system because this was lowered. Btrfs tries to fully use the space and gets all the associated complexities. Additionally, because data/metadata ratio is not fixed one can get into situations where the file system is full and there is no more metadata space. For every action it needs to carefully check if there is enough space to actually perform the action even if the file system is nearly full. Improvements in this area caused this regression. And no Rust wouldn't help. How often do you get a kernel Oops, dead lock or memory leak? Rust would help with those.
      - throwii 6y ago
        Interesting decision they made, I wonder whether they would decide differently now after seeing all the complexity. So Rust does not decrease the complexity, but only removes certain kinds of errors which the compiler can detect. Neither logic errors nor speed regressions. Thank you for your input.
        yjftsjthsd-h 6y ago
        > I wonder whether they would decide differently now after seeing all the complexity. Decide what? Cow is a fundamental part of how btrfs works, and Rust didn't exist for the majority of Linux's life. (Although if you're into that, look at Redox)
        throwii 6y ago
        Sorry I was unclear :) Maybe they would limit storage to 90% now, seeing that it increases complexity. Maybe I misunderstood the prior point though. Thanks fr mentioning Redox!
    - rleigh 6y ago
      > Why does btrfs have those issues compared to other filesystems? Why? There are several reasons, but if you go right back to the beginning, there's a single reason which caused all the other problems: they started coding before they had finished the design. All of the other problems are fallout from that. Changing the design and the implementation to fix bugs after the initial implementation was done. Introducing more bugs in the process. And leaving unresolved design flaws after freezing the on-disc format. When you look at ZFS as a comparison, the design was done and validated before they started implementing it. Not unsurprisingly, it worked as designed once the implementation was done. Up-front design work is necessary for engineering complex systems, it really goes without saying. This isn't even unique to Btrfs, but filesystems are one thing you can't hack around with without coming to grief; you have to get it right first time when their sole purpose is to store and retrieve data reliably. Many open source projects are ridden with problems because their developers were more interested in bashing out code than stopping and thinking beforehand. Same with a lot of closed source projects as well for that matter. In the case of Btrfs, which was aiming from the start to be a "better ZFS", they didn't even take the time to fully understand some of the design choices and compromises made in ZFS, because they ended up making choices which had terrible implications. Examples: using B-trees rather than Merkle hashes; this is at the root of many of its performance problems. Not having immutable snapshots; again has performance implications as well as safety implications, and is rooted in not having pool transaction numbers and deadlists. Not separating datasets/subvols from the directory hierarchy; presents logistical and administration challenges, while ZFS datasets can freely inherit metadata from parents and the mount locations are a separate property. ZFS isn't perfect of course, there are improvements and new features that could be made, but what is there is well designed, well thought out, and is a joy to work with.
      - throwii 6y ago
        Can you tell how such evaluation on a design is done? Is some kind of formal verification, analysis or rather experimentation to figure out its properties normal? Thank you for your input!
        rleigh 6y ago
  - Aardwolf 6y ago
    I had been using btrfs for 5+ years without any issues. Then a minor issue did happen, some files got missing after a system crash. That's not a problem in itself, that happens. But then I discovered that the tooling of btrfs is awful to use. Tools to repair, restore, check, scrub, ... I think I got it to finally print some filenames of missing files after a huge amount of hoops. Just see how complex btrfs restore is: https://btrfs.wiki.kernel.org/index.php/Restore https://btrfs.wiki.kernel.org/index.php/Restore. Ideally those tools would know how btrfs works internally including all indirections and be able to translate that into recovering files to the user, not have, for example, a user manually puzzle together objectid numbers from a huge list. But anyway, that made me simply format everything back to ext4 and go with that again for now. So, I was a happy btrfs user until I discovered how bad its tooling is. No issues with bugs or anything though. I really want btrfs to become better given that it's present in the kernel and has checksums, cow, .... I hope it'll keep improving!
    - solarkraft 6y ago
      I've also been using btrfs for 5+ years. When the USB stick I keep my home server's file system on didn't boot after a power outage I stuck it into my PC, performed the simplest repair on it (don't remember which) and everything was great again. The most annoying issue I've run into (and I really hate it) was being low on space and it being really hard to free some since it's very intransparent what's actually using it - but with btdu I mostly got that under control as well. That said when repairing the boot drive I did experience some confusion due to there being multiple different ways - the documentation could be better on that front. For simple things at least (subvolume management, snapshot creation, checking free space) I've been quite happy with the btrfs command line tool.
  - bitL 6y ago
    I'd prefer performance regressions any day compared to losing my data because BTRFS had a bad day like was common in the past. I also no longer use BTRFS, once was enough. If a performance regression leads to ZFS-like stability, bring it on!
  - baggy_trough 6y ago
    In 10 years of running a production service, the only two long duration outages were due to btrfs quotas (unwanted in the first place) slowing things to a crawl.
  - shmerl 6y ago
    I hope bcachefs will catch up and will become a better alternative.
Daho0n 6y ago
A link to a few Reddit comments? This is just like how 9gag link to Reddit that link to Facebook that link to ........ that link to reddit and so on. Now HN really is at Reddit level. Now we just need a link to HN in the comments.
m1gu3l 6y ago
The ratio of bad to good posts about BTRFS always made me wonder why anyone would roll with it. Don’t put your fs in crazy.
- thomond 6y ago
  Didn't Fedora transition to BTRFS as default in 33? I'm eager to see reports from that.
  - enz 6y ago
    It did, but as far as I know it's effective on fresh installs only. It would be too risky to migrate automatically to Btrfs from ext4 during the 32->33 upgrade. I suspect a lot of users upgraded to 33 from 32 without seeing any problem with sticking with ext4 (including me). As a result, a lot of users are still using ext4.
- encom 6y ago
  Nobody posts about things that are working fine.
  - curt15 6y ago
    Does OpenZFS have a similar ratio of bad to good posts?
    - washadjeffmad 6y ago
      ZFS's killer feature is religious inspiration. Write holes during resilvers are healthy and natural, but the same in a software RAID 5/6 configuration on BTRFS is unacceptably risky. Brought to you by the same people who don't think SMR disks are used with ZFS in enterprise.
      - dmm 6y ago
        Forgive me but I'm completely missing what you're saying here. Are you saying that ZFS has a write hole when reslivering?
panpanna 6y ago
With brtfs having all kind of crazy bugs and zfs never being finished, I guess it's back to ext4...
- flemhans 6y ago
  Is ZFS not ready for use? I was going to use it in production next year.
  - washadjeffmad 6y ago
    ZoL != OpenZFS != ZoF. A lot has changed recently, so you'll have to clarify the platform and version. ZFS on FUSE should be deprecated by now, and ZoL I believe merged with OpenZFS for the 2.0 release. FreeBSD ZFS is fine, and just check your versions if using it on Linux. Ie- don't use anything pre-2.0.
  - NickNameNick 6y ago
    My home server is hardly production, but it has been running a ZFS raidz+1 for more than 5 years, and survived a disk failure. (Nb, avoid SMR hard drives - the rebuild took more than a week!)
    - StavrosK 6y ago
      Mine is RAIDZ too, and has become unusably slow since a few years back. Now I need to find a way to recreate the pool, but it's too much data to store on another disk... Not in love with ZFS so far.
      - magicalhippo 6y ago
        ZFS is great for storing data. It's very unlikely it will lose your data when using ZFS. It does however require a bit of care to maintain performance, and you need to know your expected workload going in. Otherwise you can find yourself in a situation with a very poorly performing pool where the only realistic route to recovery is a send/receive to a fresh pool and back. If you do not have a lot of sync workload (VMs, DBs) and don't have super-high performance needs, say a home NAS, then mainly you just need to think about not filling up the pool too much. A nice way to do this is to create a root dataset where you set a quota to say 75% of capacity, and then create all other datasets below this one. You should not go above 85% space usage, as ZFS switches allocation strategy then to one which can significantly increase fragmentation. If you do have a lot of sync writes, a SLOG device is basically mandatory. The SLOG device does not have to be large, it only stores about 5-10 seconds worth of writes, so 10-20GB can be plenty. I've partitioned up my SSDs and created a mirror out of two small partitions, using the remaining SSD space for other things. One thing to keep in mind is that while an L2ARC device sounds like a great thing, depending on your configuration you can actually slow things down with one. A bunch of disks has a lot more bandwidth than a single SATA SSD. An L2ARC device also requires some memory overhead, so reduces your primary ARC. Again depending on load this can be detrimental. And finally, don't ever think about using deduplication, unless you've read about the consequences, measured the performance benefits and ensured the memory overhead is acceptable. It sounds great on paper but has a lot of associated downsides that can ruin pool performance, and disabling it does not make it go away. At least that's what I've picked up so far.
esgwpl 6y ago
This version has got to be the worst kernel released in a while in terms of regression, from AMDGPU null pointer dereference crash[0] to f2fs data corruption bug[1] and now this. Fixes for these are on their way as far as I can tell but since the stable team are probably on Christmas vacation it might take a while. [0] https://bbs.archlinux.org/viewtopic.php?pid=1943906#p1943906 https://bbs.archlinux.org/viewtopic.php?pid=1943906#p1943906 [1] https://bugzilla.kernel.org/show_bug.cgi?id=210765 https://bugzilla.kernel.org/show_bug.cgi?id=210765
- gregkh 6y ago
  I get a vacation? Hah!
  - nasir_hm 6y ago
    Thank You very much for the hard work you do Greg, You're awesome :D
  - mr_sturd 6y ago
    Merry Christmas, Greg!
  - muxator 6y ago
    Thanks for your work, Greg!
  - tuldia 6y ago
    Every software has bugs, and is easier to criticize than to help. Don't focus on the negativity of HN and keep it up! Thanks for your hard work, Greg!
    - gregkh 6y ago
      If people don't report bugs, we don't know they are there as it "works for me!". This isn't "negativity", this is people not understanding how the process works :) And you're welcome!
redis_mlc 6y ago
I wish to god people would get a clue with BTRFS: - don't use it because you think the name is "cool". Every time I go to a linux meetup, some idiot starts talking about the name. - it takes 5-10 years for a filesystem or database to mature after it's released. Otherwise you will lose data and cry. - the recent BTRFS performance improvements are against itself, not other filesystems. EXT4 is an excellent fs, as good as XFS performance-wise overall. Source: DBA and storage engineer.
- dmm 6y ago
  > - it takes 5-10 years for a filesystem or database to mature after it's released. Great news! BTRFS was introduced in the mainline kernel in March 2009! It's been ten years.
  - tremon 6y ago
    And it's been the default filesystem for openSUSE (system partitions only) since December 2015 [0], so there's been plenty of real-world usage too. [0] https://en.opensuse.org/Portal:13.2 https://en.opensuse.org/Portal:13.2
    - therealrootuser 6y ago
      Yeah, I remember that. I never do a separate home partition though, so I ended up with the entire disk in Btrfs. I wasn't paying as much attention as I should have when setting up partitions during installation, and didn't totally understand the implications of using Btrfs back then. Eventually though, it filled up my entire drive with snapshots, causing me much confusion and many disk full errors until I could figure out what had happened. That experience turned me off from openSUSE for several years, until I finally started using it (with a more conservative choice of Ext4) again for a few machines a couple years ago.
      - viraptor 6y ago
        The snapshots in this case come from suse, not btrfs itself. It's their package tooling creating them https://doc.opensuse.org/documentation/leap/reference/html/book.opensuse.reference/cha-snapper.html https://doc.opensuse.org/documentation/leap/reference/html/b... If you're on lvm, you're still getting snapshots now, even with ext4.
snvzz 6y ago
HAMMER2 cannot be ready soon enough.
- yjftsjthsd-h 6y ago
  Is there any talk of porting to Linux?
  - snvzz 6y ago
    No idea. I have only heard there is interest from openbsd. At the time, Matt told them off because it's wasn't "ready for porting yet" and wanted to save them useless work.
solarkraft 6y ago
Watching René talk about this stuff on his Youtube channel is delightful. I highly recommend it.
- Bayart 6y ago
  I'm assuming you mean René Rebe [1] ;) [1] https://www.youtube.com/user/renerebe https://www.youtube.com/user/renerebe
champtar 6y ago
I had multiple corruption on a "server" at my parents, checked the disk and reinstalled from scratch 2 times, then I ran memtest86+ and found out one of the memory stick was faulty. Btrfs allowed me to detect that and be confident that all data recovered from the disks were ok because of the data checksumming. With ext4 I might have never found out.
- knorker 6y ago
  I wouldn't count on that though. I'd say you were lucky. If the corruption happens before it gets sent to the disk, then you have checksummed and valid but broken data.
  - diegocg 6y ago
    Btrfs has added some checks to prevent these issues. It can't catch everything, but some people have apparently hit them so it certainly helps https://btrfs.wiki.kernel.org/index.php/Tree-checker https://btrfs.wiki.kernel.org/index.php/Tree-checker
    - tremon 6y ago
      The parent's point was that if the user data (not the tree metadata) was corrupted in-memory before writing to disk, no amount of checksumming will save you because the checksum was calculated over corrupted data.
xaduha 6y ago
You can get 'surplus' HP P410 from Ali or Ebay, they are cheap and work just fine. Always was told that you can't beat hardware RAID and now that they are available people don't seem to be interested. Just looking at it I can say that apart from CPU/GPU chips themselves it's the best piece of electronics in my PC.
- speakeron 6y ago
  The problem with hardware RAID is that if the hardware fails, you need to replace that specific card with the same type to recover the data. If they're cheap enough and you want to go down that route, then buy two so you've got a backup. On the server side, the tendency is now towards vertical striping (directly addressing disks on multiple servers through the network with software).
- buran77 6y ago
  > Always was told that you can't beat hardware RAID Recently with super powerful multi-core CPUs being the norm especially in places where RAID is a consideration, hardware RAID makes less and less sense. You just create additional complications like when moving your RAID array between 2 otherwise identical cards but with different firmwares fails. So you end up always buying spares and testing the migration before putting any data, then you freeze the config in place. On the other side you may be stuck forever with a buggy firmware.
- BuildTheRobots 6y ago
  Where as I was always told to avoid hardware raid if possible, and that's been a mantra for 15+ years. The reason I was given is twofold. 1) If your RAID card dies it's nice to be able to plug drives into any controller and be able to access the data. 2) "look at how often MDADM fix bugs. Now look at how often your RAID card firmware gets fixed... use MDADM". Modern filesystems like btrfs & zfs both prefer to deal with the RAID aspects themselves as it gives them far more control. Even if you're a hardware raid afficianado, I wouldn't recommend formatting (eg) ZFS on top of a hardware raid pool.
  - xaduha 6y ago
teekert 6y ago
I work with TBs of genomics data and I am at this point pretty convinced that some of that data changes on-disk. Every now and then. It's very weird. Recently I heard an interview (on one of the Jupiter broadcasting shows) about how btrfs is exposing errors of underlying hardware using their checksumming features, I had a bit of an Aha! moment. The files in question (BAM files) have a sort of checksumming built in (called cigar strings) and I am getting more and more convinced that the discrepancies I run into are actually bit flips on the hardware. Anybody else have any experience with this? BTRFS would be a good tool to prevent such bitrot, imo...
- sethhochberg 6y ago
  Depending on how you store this data (the number of physical disks involved) you may also want to explore ZFS. ZFS and BTRFS have pretty comparable features, but given how battle-hardened ZFS is, and its ever-increasing integration into the Linux ecosystem, I'm seeing less and less reason to prefer BTRFS. Both do checksumming and regular scrubbing to help detect bitrot, but both need sufficient redundancy configured to actually be able to repair the corrupted data when it is detected.
  - ZFSKing 6y ago
    That, this, is the most diplomatical way of saying 'just use ZFS' we've seen in a decayed. Mod parent up. BTRFS will never be a suck-zess. Use ext4 or ZFS for life.
  - hda111 6y ago
    There is exactly one distro that comes with ZFS, so it's a bit exaggeration to say it's well integrated with Linux, considering how much effort is applied by kernel developers to make it even harder to use ZFS. And this one distro that includes ZFS is just one lawsuit away from removing it again.
    - jacob019 6y ago
      It's easy to use zfs on Debian and derivatives apt install dkms spl-dkms apt install zfs-dkms zfsutils-linux
renewiltord 6y ago
Not sure why anyone would use btrfs over zfs. And that's coming from someone who used to use old XFS in an unreliable power environment.
- cesarb 6y ago
  A good reason for using btrfs over zfs is that btrfs is available by default on the Linux kernel, while zfs is only available as a third-party out-of-tree module, with all the usual problems out-of-tree modules have.
- pkulak 6y ago
  Because it's in the kernel.
shmerl 6y ago
Still waiting to switch to bcachefs.
- viraptor 6y ago
  You can help it arrive faster by supporting the development: https://www.patreon.com/bcachefs https://www.patreon.com/bcachefs

141 points by shrikant 6y ago

Assuming it can be fixed in a 5.10.x patch release, it's a rather minor story - a bug that is quickly found after release and fixed, hopefully. It will be investigated: https://marc.info/?l=linux-btrfs&m=160869337604422&w=2 https://marc.info/?l=linux-btrfs&m=160869337604422&w=2

turminal 6y ago
The problem with btrfs is that it has quirks like that (and worse, much worse) all the time.
- throwii 6y ago
  Why does btrfs have those issues compared to other filesystems? I'd love to use btrfs too. Note that I deeply respect people who can write such complicated code, which I couldn't. Would Rust solve the non-speed issues? Rust-in-kernel discussion from August: https://lwn.net/Articles/829858/ https://lwn.net/Articles/829858/
  - panpanna 6y ago
    Please understand that the rust memory and thread security mainly applies to "normal" applications. In kernel, you can run a privileged cache or mmu instruction or a write to some magical memory position and all the sudden the "normal" rules don't apply anymore. (But I think there are other parts of rust that are nice to have in kernel or any complex software).
    - throwii 6y ago
      I thought the Rust compiler solves issues that you wouldn't immediately see with pure C, which is why I had the idea. I didn't know this requires certain features which are not available inside the kernel. I only knew all existing interfaces may be unsafe because they are in C though. Rust does not seem as useful then. Thank you for your input.
      - simcop2387 6y ago
        It's not so much that rust the language requires them as much as it is that other non-rust parts can quite easily stomp all over the guarantees of rust without there ever being a way of knowing it happened. So rust alone won't solve many problems, but it would let you say "this code can't do these things itself", which is still a useful distinction. It also doesn't allow you to deal with misbehaving hardware that changes memory underneath you in ways it said won't happen. Hardware sucks.
        throwii 6y ago
        Do other parts stomp often? :) But true that can happen. Especially on non-ECC systems. I didn't think about the hardware issues, hmm. I can't see how to do that, when the compiler guarantees get invalidated by hardware. Checks are also needed like in C? (assuming there are checks which do not get compiled out..)
        simcop2387 6y ago
        When the hardware can't make the guarantees, then software really can't do anything about it. There's really not any checks you can do, but modern hardware is getting the capabilities to try to prevent those kinds of issues with the IOMMU units, but operating system support is still hit or miss for most hardware and it won't prevent everything (just devices stomping on each-other with DMA). That's basically how the thunderbolt attacks have worked and the solutions to them.
  - turminal 6y ago
    I sincerely hope you are joking but I realize this mindset is quite common these days, so let me reiterate: Rust does not magically solve problems for you. Btrfs has a lot of issues and some of them may well be of the not possible in Rust sort, but I'm quite sure most of them are not and there is nothing Rust can fix about them. Rewriting a 13 years old and very complex thing in another language is a massive effort and a big opportunity for introducing some more, possibly worse bugs along the way. EDIT: also see panpanna's comment
    - ho_schi 6y ago
      It is like a repeating story, every time someone sells the magically solution to all problems and apply a name on it people blindly believe it. This are tools. A screwdriver is not the right tool, when you need a power drill. And vice versa. Yes, sometimes you can use both and stick with your accustomed tool. Anyway. C and C++ and the tool chain are constantly improving like others.The moern memory sanitizers in GCC and LLVM are awesome.
    - throwii 6y ago
      Any rewrite must be carefully evaluated, true. And it may not be advisable. I'm trying to understand the issues. Are they not solvable because the kernel does not give enough guarantees as it gives to userspace, because the c-interfaces of the kernel have to be wrapped in unsafe or because of other reasons (architecture, data model, kernel constraints, ...)?
      - ldrndll 6y ago
        I think the issue is that you haven’t understood the issue, and just proposed using Rust. Don’t get me wrong, I’m a huge fan of Rust, but seeing people blindly suggest it as a panacea gets frustrating for people. The reality is that when people talk about Rust being fact, it’s because a decently optimised Rust program might be comparable to an equivalent C program. C is generally not the problem when it comes to performance issues in the kernel, which means rewriting that thing in Rust wouldn’t magically make it faster. The issue will probably be some sort of pathological case in an algorithm being used, or perhaps from a poorly chosen algorithm. The point being, it’s not clear yet, and to solve that requires understanding the problem, not effecting a needless rewrite in a new language.
  - thomaszander 6y ago
    > Why does btrfs have those issues compared to other filesystems? As someone that has built infrastructure on BtrFS for years, the scary stories are mostly just hot air and the stability of other filesystems is really not significantly better. Bugs like this happen, this is why Linus releases many release-candiates every kernel, this one got through as 10 was a rather massive kernel and there were several regressions. Including one that caused a new release just hours after the supposedly final one. Distros wait a bit longer before shipping an updated kernel and none of these hit actual users. As far as I know there is no reason to abstain from using Btrfs. When Fedora talked about not using it, they had as reason that they had no in-house expertise.
    - throwii 6y ago
      Thank you for posting your experience!
    - circularfoyers 6y ago
      Does Fedora now have in-house expertise now that they've decided to make Btrfs default?
      - curt15 6y ago
        Some Facebook btrfs devs agreed to help out. FB runs Fedora/CentOS + btrfs root on most of its systems.
    - yjftsjthsd-h 6y ago
      > As someone that has built infrastructure on BtrFS for years, the scary stories are mostly just hot air and the stability of other filesystems is really not significantly better. I've lost 2 root filesystems to btrfs, on a laptop with only 1 drive (read: not even using RAID). Have you considered that you're just lucky?
    - Volundr 6y ago
      In my experience, the stories are real. Our entire company was offline for a day when our central storage server quit accepting writes despite having over 50% free space. That's when I learned the hard way about the data/metadata split (something I was aware of but wasn't exactly top of mind) and BTRFS balance. You can certainly say it was my fault for not reading ALL the documentation before using BTRFS, and I'd find it hard to disagree, but any other filesystem wouldn't have had this problem. I can't speak to if there are other foot-guns waiting around or how common problems like this are because we migrated back to FreeBSD and ZFS shortly after that experience. I do know they have since updated BTRFS to make that scenario less likely (but still not impossible).
  - jeromenerf 6y ago
    > Why does btrfs have those issues compared to other filesystems? Mostly because it has lots of features and as a consequence, is pretty large and complex. Closer to ZFS than ext2. Btrfs suffers from a initial bad rep, which is difficult to overcome.
    - nicoburns 6y ago
      ZFS itself seems very stable and not to suffer from these issues though.
      - Quekid5 6y ago
        ZFS is also extremely well designed at a system level... which is not the impression I get from BtrFS. (Disclaimer: I have not bothered looking at BtrFS for years because ZFS has handled everything I've thrown at it very admirably. Including complicated setups with RAID-Z, etc.) Granted, there are some limitations to the design, but it doesn't affect my use cases, so whatever...
      - jpeloquin 6y ago
        Anecdotally, I've encountered issues with both ZFS and BTRFS at about the same rate. A public example of an apparent ZFS performance issue is https://github.com/openzfs/zfs/issues/9375 https://github.com/openzfs/zfs/issues/9375 Both are much more quirky than simpler filesystems like ext4. Data integrity verification from checksumming makes it worth it though. The ZFS vs. BTRFS choice, I think, depends more on whether you need specific features like offline deduplication or L2ARC / SLOG cache devices. And which one you're more familiar with (can troubleshoot better).
    - throwii 6y ago
      Good theory, thank you.
  - tobias3 6y ago
    My guess is this (compared to ZFS): With a CoW-file system like btrfs you have to problem that you need new file system space to delete something. This is problematic if the file system is full and you want to be able to write to it again by deleting something. ZFS solved this by just saying one can only fill a file system to 90% usage. At some point they even decreased this (during upgrade) and I had the issue that I couldn't write to the ZFS file system because this was lowered. Btrfs tries to fully use the space and gets all the associated complexities. Additionally, because data/metadata ratio is not fixed one can get into situations where the file system is full and there is no more metadata space. For every action it needs to carefully check if there is enough space to actually perform the action even if the file system is nearly full. Improvements in this area caused this regression. And no Rust wouldn't help. How often do you get a kernel Oops, dead lock or memory leak? Rust would help with those.
    - throwii 6y ago
      Interesting decision they made, I wonder whether they would decide differently now after seeing all the complexity. So Rust does not decrease the complexity, but only removes certain kinds of errors which the compiler can detect. Neither logic errors nor speed regressions. Thank you for your input.
      - yjftsjthsd-h 6y ago
        > I wonder whether they would decide differently now after seeing all the complexity. Decide what? Cow is a fundamental part of how btrfs works, and Rust didn't exist for the majority of Linux's life. (Although if you're into that, look at Redox)
        throwii 6y ago
        Sorry I was unclear :) Maybe they would limit storage to 90% now, seeing that it increases complexity. Maybe I misunderstood the prior point though. Thanks fr mentioning Redox!
  - rleigh 6y ago
    > Why does btrfs have those issues compared to other filesystems? Why? There are several reasons, but if you go right back to the beginning, there's a single reason which caused all the other problems: they started coding before they had finished the design. All of the other problems are fallout from that. Changing the design and the implementation to fix bugs after the initial implementation was done. Introducing more bugs in the process. And leaving unresolved design flaws after freezing the on-disc format. When you look at ZFS as a comparison, the design was done and validated before they started implementing it. Not unsurprisingly, it worked as designed once the implementation was done. Up-front design work is necessary for engineering complex systems, it really goes without saying. This isn't even unique to Btrfs, but filesystems are one thing you can't hack around with without coming to grief; you have to get it right first time when their sole purpose is to store and retrieve data reliably. Many open source projects are ridden with problems because their developers were more interested in bashing out code than stopping and thinking beforehand. Same with a lot of closed source projects as well for that matter. In the case of Btrfs, which was aiming from the start to be a "better ZFS", they didn't even take the time to fully understand some of the design choices and compromises made in ZFS, because they ended up making choices which had terrible implications. Examples: using B-trees rather than Merkle hashes; this is at the root of many of its performance problems. Not having immutable snapshots; again has performance implications as well as safety implications, and is rooted in not having pool transaction numbers and deadlists. Not separating datasets/subvols from the directory hierarchy; presents logistical and administration challenges, while ZFS datasets can freely inherit metadata from parents and the mount locations are a separate property. ZFS isn't perfect of course, there are improvements and new features that could be made, but what is there is well designed, well thought out, and is a joy to work with.
    - throwii 6y ago
      Can you tell how such evaluation on a design is done? Is some kind of formal verification, analysis or rather experimentation to figure out its properties normal? Thank you for your input!
      - rleigh 6y ago
- Aardwolf 6y ago
  I had been using btrfs for 5+ years without any issues. Then a minor issue did happen, some files got missing after a system crash. That's not a problem in itself, that happens. But then I discovered that the tooling of btrfs is awful to use. Tools to repair, restore, check, scrub, ... I think I got it to finally print some filenames of missing files after a huge amount of hoops. Just see how complex btrfs restore is: https://btrfs.wiki.kernel.org/index.php/Restore https://btrfs.wiki.kernel.org/index.php/Restore. Ideally those tools would know how btrfs works internally including all indirections and be able to translate that into recovering files to the user, not have, for example, a user manually puzzle together objectid numbers from a huge list. But anyway, that made me simply format everything back to ext4 and go with that again for now. So, I was a happy btrfs user until I discovered how bad its tooling is. No issues with bugs or anything though. I really want btrfs to become better given that it's present in the kernel and has checksums, cow, .... I hope it'll keep improving!
  - solarkraft 6y ago
    I've also been using btrfs for 5+ years. When the USB stick I keep my home server's file system on didn't boot after a power outage I stuck it into my PC, performed the simplest repair on it (don't remember which) and everything was great again. The most annoying issue I've run into (and I really hate it) was being low on space and it being really hard to free some since it's very intransparent what's actually using it - but with btdu I mostly got that under control as well. That said when repairing the boot drive I did experience some confusion due to there being multiple different ways - the documentation could be better on that front. For simple things at least (subvolume management, snapshot creation, checking free space) I've been quite happy with the btrfs command line tool.
- bitL 6y ago
  I'd prefer performance regressions any day compared to losing my data because BTRFS had a bad day like was common in the past. I also no longer use BTRFS, once was enough. If a performance regression leads to ZFS-like stability, bring it on!
- baggy_trough 6y ago
  In 10 years of running a production service, the only two long duration outages were due to btrfs quotas (unwanted in the first place) slowing things to a crawl.
- shmerl 6y ago
  I hope bcachefs will catch up and will become a better alternative.

Daho0n 6y ago

A link to a few Reddit comments? This is just like how 9gag link to Reddit that link to Facebook that link to ........ that link to reddit and so on. Now HN really is at Reddit level. Now we just need a link to HN in the comments.

m1gu3l 6y ago

The ratio of bad to good posts about BTRFS always made me wonder why anyone would roll with it. Don’t put your fs in crazy.

thomond 6y ago
Didn't Fedora transition to BTRFS as default in 33? I'm eager to see reports from that.
- enz 6y ago
  It did, but as far as I know it's effective on fresh installs only. It would be too risky to migrate automatically to Btrfs from ext4 during the 32->33 upgrade. I suspect a lot of users upgraded to 33 from 32 without seeing any problem with sticking with ext4 (including me). As a result, a lot of users are still using ext4.
encom 6y ago
Nobody posts about things that are working fine.
- curt15 6y ago
  Does OpenZFS have a similar ratio of bad to good posts?
  - washadjeffmad 6y ago
    ZFS's killer feature is religious inspiration. Write holes during resilvers are healthy and natural, but the same in a software RAID 5/6 configuration on BTRFS is unacceptably risky. Brought to you by the same people who don't think SMR disks are used with ZFS in enterprise.
    - dmm 6y ago
      Forgive me but I'm completely missing what you're saying here. Are you saying that ZFS has a write hole when reslivering?

panpanna 6y ago

With brtfs having all kind of crazy bugs and zfs never being finished, I guess it's back to ext4...

flemhans 6y ago
Is ZFS not ready for use? I was going to use it in production next year.
- washadjeffmad 6y ago
  ZoL != OpenZFS != ZoF. A lot has changed recently, so you'll have to clarify the platform and version. ZFS on FUSE should be deprecated by now, and ZoL I believe merged with OpenZFS for the 2.0 release. FreeBSD ZFS is fine, and just check your versions if using it on Linux. Ie- don't use anything pre-2.0.
- NickNameNick 6y ago
  My home server is hardly production, but it has been running a ZFS raidz+1 for more than 5 years, and survived a disk failure. (Nb, avoid SMR hard drives - the rebuild took more than a week!)
  - StavrosK 6y ago
    Mine is RAIDZ too, and has become unusably slow since a few years back. Now I need to find a way to recreate the pool, but it's too much data to store on another disk... Not in love with ZFS so far.
    - magicalhippo 6y ago
      ZFS is great for storing data. It's very unlikely it will lose your data when using ZFS. It does however require a bit of care to maintain performance, and you need to know your expected workload going in. Otherwise you can find yourself in a situation with a very poorly performing pool where the only realistic route to recovery is a send/receive to a fresh pool and back. If you do not have a lot of sync workload (VMs, DBs) and don't have super-high performance needs, say a home NAS, then mainly you just need to think about not filling up the pool too much. A nice way to do this is to create a root dataset where you set a quota to say 75% of capacity, and then create all other datasets below this one. You should not go above 85% space usage, as ZFS switches allocation strategy then to one which can significantly increase fragmentation. If you do have a lot of sync writes, a SLOG device is basically mandatory. The SLOG device does not have to be large, it only stores about 5-10 seconds worth of writes, so 10-20GB can be plenty. I've partitioned up my SSDs and created a mirror out of two small partitions, using the remaining SSD space for other things. One thing to keep in mind is that while an L2ARC device sounds like a great thing, depending on your configuration you can actually slow things down with one. A bunch of disks has a lot more bandwidth than a single SATA SSD. An L2ARC device also requires some memory overhead, so reduces your primary ARC. Again depending on load this can be detrimental. And finally, don't ever think about using deduplication, unless you've read about the consequences, measured the performance benefits and ensured the memory overhead is acceptable. It sounds great on paper but has a lot of associated downsides that can ruin pool performance, and disabling it does not make it go away. At least that's what I've picked up so far.

esgwpl 6y ago

This version has got to be the worst kernel released in a while in terms of regression, from AMDGPU null pointer dereference crash[0] to f2fs data corruption bug[1] and now this. Fixes for these are on their way as far as I can tell but since the stable team are probably on Christmas vacation it might take a while. [0] https://bbs.archlinux.org/viewtopic.php?pid=1943906#p1943906 https://bbs.archlinux.org/viewtopic.php?pid=1943906#p1943906 [1] https://bugzilla.kernel.org/show_bug.cgi?id=210765 https://bugzilla.kernel.org/show_bug.cgi?id=210765

gregkh 6y ago
I get a vacation? Hah!
- nasir_hm 6y ago
  Thank You very much for the hard work you do Greg, You're awesome :D
- mr_sturd 6y ago
  Merry Christmas, Greg!
- muxator 6y ago
  Thanks for your work, Greg!
- tuldia 6y ago
  Every software has bugs, and is easier to criticize than to help. Don't focus on the negativity of HN and keep it up! Thanks for your hard work, Greg!
  - gregkh 6y ago
    If people don't report bugs, we don't know they are there as it "works for me!". This isn't "negativity", this is people not understanding how the process works :) And you're welcome!

redis_mlc 6y ago

I wish to god people would get a clue with BTRFS: - don't use it because you think the name is "cool". Every time I go to a linux meetup, some idiot starts talking about the name. - it takes 5-10 years for a filesystem or database to mature after it's released. Otherwise you will lose data and cry. - the recent BTRFS performance improvements are against itself, not other filesystems. EXT4 is an excellent fs, as good as XFS performance-wise overall. Source: DBA and storage engineer.

dmm 6y ago
> - it takes 5-10 years for a filesystem or database to mature after it's released. Great news! BTRFS was introduced in the mainline kernel in March 2009! It's been ten years.
- tremon 6y ago
  And it's been the default filesystem for openSUSE (system partitions only) since December 2015 [0], so there's been plenty of real-world usage too. [0] https://en.opensuse.org/Portal:13.2 https://en.opensuse.org/Portal:13.2
  - therealrootuser 6y ago
    Yeah, I remember that. I never do a separate home partition though, so I ended up with the entire disk in Btrfs. I wasn't paying as much attention as I should have when setting up partitions during installation, and didn't totally understand the implications of using Btrfs back then. Eventually though, it filled up my entire drive with snapshots, causing me much confusion and many disk full errors until I could figure out what had happened. That experience turned me off from openSUSE for several years, until I finally started using it (with a more conservative choice of Ext4) again for a few machines a couple years ago.
    - viraptor 6y ago
      The snapshots in this case come from suse, not btrfs itself. It's their package tooling creating them https://doc.opensuse.org/documentation/leap/reference/html/book.opensuse.reference/cha-snapper.html https://doc.opensuse.org/documentation/leap/reference/html/b... If you're on lvm, you're still getting snapshots now, even with ext4.

snvzz 6y ago

HAMMER2 cannot be ready soon enough.

yjftsjthsd-h 6y ago
Is there any talk of porting to Linux?
- snvzz 6y ago
  No idea. I have only heard there is interest from openbsd. At the time, Matt told them off because it's wasn't "ready for porting yet" and wanted to save them useless work.

solarkraft 6y ago

Watching René talk about this stuff on his Youtube channel is delightful. I highly recommend it.

Bayart 6y ago
I'm assuming you mean René Rebe [1] ;) [1] https://www.youtube.com/user/renerebe https://www.youtube.com/user/renerebe

champtar 6y ago

I had multiple corruption on a "server" at my parents, checked the disk and reinstalled from scratch 2 times, then I ran memtest86+ and found out one of the memory stick was faulty. Btrfs allowed me to detect that and be confident that all data recovered from the disks were ok because of the data checksumming. With ext4 I might have never found out.

knorker 6y ago
I wouldn't count on that though. I'd say you were lucky. If the corruption happens before it gets sent to the disk, then you have checksummed and valid but broken data.
- diegocg 6y ago
  Btrfs has added some checks to prevent these issues. It can't catch everything, but some people have apparently hit them so it certainly helps https://btrfs.wiki.kernel.org/index.php/Tree-checker https://btrfs.wiki.kernel.org/index.php/Tree-checker
  - tremon 6y ago
    The parent's point was that if the user data (not the tree metadata) was corrupted in-memory before writing to disk, no amount of checksumming will save you because the checksum was calculated over corrupted data.

xaduha 6y ago

You can get 'surplus' HP P410 from Ali or Ebay, they are cheap and work just fine. Always was told that you can't beat hardware RAID and now that they are available people don't seem to be interested. Just looking at it I can say that apart from CPU/GPU chips themselves it's the best piece of electronics in my PC.

speakeron 6y ago
The problem with hardware RAID is that if the hardware fails, you need to replace that specific card with the same type to recover the data. If they're cheap enough and you want to go down that route, then buy two so you've got a backup. On the server side, the tendency is now towards vertical striping (directly addressing disks on multiple servers through the network with software).
buran77 6y ago
> Always was told that you can't beat hardware RAID Recently with super powerful multi-core CPUs being the norm especially in places where RAID is a consideration, hardware RAID makes less and less sense. You just create additional complications like when moving your RAID array between 2 otherwise identical cards but with different firmwares fails. So you end up always buying spares and testing the migration before putting any data, then you freeze the config in place. On the other side you may be stuck forever with a buggy firmware.
BuildTheRobots 6y ago
Where as I was always told to avoid hardware raid if possible, and that's been a mantra for 15+ years. The reason I was given is twofold. 1) If your RAID card dies it's nice to be able to plug drives into any controller and be able to access the data. 2) "look at how often MDADM fix bugs. Now look at how often your RAID card firmware gets fixed... use MDADM". Modern filesystems like btrfs & zfs both prefer to deal with the RAID aspects themselves as it gives them far more control. Even if you're a hardware raid afficianado, I wouldn't recommend formatting (eg) ZFS on top of a hardware raid pool.
- xaduha 6y ago

teekert 6y ago

I work with TBs of genomics data and I am at this point pretty convinced that some of that data changes on-disk. Every now and then. It's very weird. Recently I heard an interview (on one of the Jupiter broadcasting shows) about how btrfs is exposing errors of underlying hardware using their checksumming features, I had a bit of an Aha! moment. The files in question (BAM files) have a sort of checksumming built in (called cigar strings) and I am getting more and more convinced that the discrepancies I run into are actually bit flips on the hardware. Anybody else have any experience with this? BTRFS would be a good tool to prevent such bitrot, imo...

sethhochberg 6y ago
Depending on how you store this data (the number of physical disks involved) you may also want to explore ZFS. ZFS and BTRFS have pretty comparable features, but given how battle-hardened ZFS is, and its ever-increasing integration into the Linux ecosystem, I'm seeing less and less reason to prefer BTRFS. Both do checksumming and regular scrubbing to help detect bitrot, but both need sufficient redundancy configured to actually be able to repair the corrupted data when it is detected.
- ZFSKing 6y ago
  That, this, is the most diplomatical way of saying 'just use ZFS' we've seen in a decayed. Mod parent up. BTRFS will never be a suck-zess. Use ext4 or ZFS for life.
- hda111 6y ago
  There is exactly one distro that comes with ZFS, so it's a bit exaggeration to say it's well integrated with Linux, considering how much effort is applied by kernel developers to make it even harder to use ZFS. And this one distro that includes ZFS is just one lawsuit away from removing it again.
  - jacob019 6y ago
    It's easy to use zfs on Debian and derivatives apt install dkms spl-dkms apt install zfs-dkms zfsutils-linux

renewiltord 6y ago

Not sure why anyone would use btrfs over zfs. And that's coming from someone who used to use old XFS in an unreliable power environment.

cesarb 6y ago
A good reason for using btrfs over zfs is that btrfs is available by default on the Linux kernel, while zfs is only available as a third-party out-of-tree module, with all the usual problems out-of-tree modules have.
pkulak 6y ago
Because it's in the kernel.

shmerl 6y ago

Still waiting to switch to bcachefs.

viraptor 6y ago
You can help it arrive faster by supporting the development: https://www.patreon.com/bcachefs https://www.patreon.com/bcachefs