9 ms·
Bug hunting in Btrfs
- throw0101d 3y agoGiven that (Open)ZFS[1] is quite mature, and Bcachefs[2] seems to be gaining popularity, how much of a future does Btrfs have? [1] https://en.wikipedia.org/wiki/ZFS https://en.wikipedia.org/wiki/ZFS [2] https://en.wikipedia.org/wiki/Bcachefs https://en.wikipedia.org/wiki/Bcachefs
- opengears 3y agoI big future (unless ZFS licensing incompatibility is solved)
- maxloh 3y agoMaybe replace Oracle-owned code with a clean room implementation?
- DaSHacka 3y agoWould you not basically be starting over at that point, though?
- arp242 3y agoThey did that. It's called "btrfs". A stable clean-room ZFS with on-disk compatibility would be a huge task. How long did stable NTFS write capability take? And NTFS is a much simpler filesystem. It would also be a huge waste of time given that btrfs and bcachefs exist, and that ZFS is fine to use license-wise – it's just distribution that's a tad awkward (but not overly so).
- rascul 3y agoInteresting to note here that btrfs came from Oracle.
- tadfisher 3y ago"Chris Mason is the founding developer of btrfs, which he began working on in 2007 while working at Oracle. This leads many people to believe that btrfs is an Oracle project—it is not. The project belonged to Mason, not to his employer, and it remains a community project unencumbered by corporate ownership to this day." https://arstechnica.com/gadgets/2021/09/examining-btrfs-linuxs-perpetually-half-finished-filesystem/ https://arstechnica.com/gadgets/2021/09/examining-btrfs-linu...
- chungy 3y agoThat's countered by btrfs's source code always beginning with "Copyright (C) 2007 Oracle." Maybe it was Mason's pet project within the company, but there is no ambiguity that Oracle owns it. It is an Oracle project.
- arp242 3y agoA copyright line doesn't make it an "Oracle project". That implies a high level of control/involvement in the project.
- chungy 3y agoIt shows belonging, at the very least. The quote from Jim Salter was "The project belonged to Mason, not to his employer" (emphasis added). The copyright line demonstrably and incontestably refutes this claim. btrfs belongs to Oracle.
- yjftsjthsd-h 3y agoI don't think that would work; all of the changes since the fork are also CDDL and they aren't owned by any one entity/person. (IANAL)
- maxloh 3y agoGPL only forbids ZFS to be distributed alongside Linux, it doesn't prevent users from installing it manually. (IANAL)
- deleted 3y ago[deleted]
- mustache_kimono 3y ago> GPL only forbids ZFS to be distributed alongside Linux But does it even do that? You might be surprised when/if you read a little more widely. Position of OpenZFS project[0] is which I find persuasive (my emphasis added): In the case of the Linux Kernel, this prevents us from distributing OpenZFS as part of the Linux Kernel binary. *However, there is nothing in either license that prevents distributing it in the form of a binary module* or in the form of source code. [0]: https://openzfs.github.io/openzfs-docs/License.html https://openzfs.github.io/openzfs-docs/License.html You might see also: [1]: https://www.networkworld.com/article/836039/smb-encouraging-closed-source-modules-part-1-copyright-and-software.html https://www.networkworld.com/article/836039/smb-encouraging-... [2]: https://law.resource.org/pub/us/case/reporter/F2/977/977.F2d.1510.92-15655.html https://law.resource.org/pub/us/case/reporter/F2/977/977.F2d...
- lmz 3y agoIt doesn't matter what the project thinks as long as there is code owned by Oracle in the FS.
- mustache_kimono 3y ago> It doesn't matter what the project thinks as long as there is code owned by Oracle in the FS. Might we agree then that the only thing that really matters is the law? And we should ignore other opinions coughlike from the FSF/the SFCcough which don't make reference to the law? Or which ignore long held copyright law principles, like fair use? Please take a look at the case law. The so far theoretical claim of OpenZFS/Linux incompatibility is especially weak re: a binary kernel module.
- throw0101d 3y agoWhy would Btrfs have a big(ger) future than Bcachefs when the latter seems to have the same functionality and less 'historical baggage'†? I remember when Btrfs was initially released (I was doing Solaris sysadmining, and it was billed as "ZFS for Linux"), and yet here we are all these years later and it still seems to be 'meh'. † E.g., RAID5+ that's less likely to eat data: * https://btrfs.readthedocs.io/en/latest/btrfs-man5.html#raid56-status-and-recommended-practices https://btrfs.readthedocs.io/en/latest/btrfs-man5.html#raid5... * https://bcachefs.org/ErasureCoding/ https://bcachefs.org/ErasureCoding/ / https://github.com/koverstreet/bcachefs/issues/657 https://github.com/koverstreet/bcachefs/issues/657
- bhaney 3y agoBy the time future-bcachefs has feature parity with present-btrfs, who knows what more will be in future-btrfs? For your specific example, bcachefs's erasure coding is very experimental and currently pretty much unusable, while btrfs is actively working towards fixing the raid56 write hole with the recent addition of the raid-stripe-tree. By the time bcachefs has a trustworthy parity profile, btrfs's may be just as good.
- curt15 3y agoBcachefs has not advertised erasure coding as production ready only to renege on that claim later. So nobody has been unwittingly burned yet.
- throw0101d 3y ago> For your specific example My specific example says that the bcachefs are "actively working towards fixing the raid56 write hole" as well—or rather, their way of doing things doesn't have one in the first place.
- bhaney 3y ago> bcachefs are "actively working towards fixing the raid56 write hole" as well Yep, that's my point. Neither btrfs nor bcachefs have a write-hole-less parity raid profile implementation yet, and both are working towards one. We don't know if one will be finished and battle tested significantly before the other, or if one will prove to be more performant or reliable. Just have to wait and see.
- e145bc455f1 3y agoWhen can we expect debian to ship Bcachefs?
- throw0101d 3y agoBcachefs is in the Linux 6.7 kernel, and that is available in Debian unstable and experimental: * https://packages.debian.org/search?keywords=linux-image-6.7 https://packages.debian.org/search?keywords=linux-image-6.7 * Search "6.7.1-1~exp1": https://metadata.ftp-master.debian.org/changelogs//main/l/linux/linux_6.7.4-1~exp1_changelog https://metadata.ftp-master.debian.org/changelogs//main/l/li...
- candiddevmike 3y agoBcachefs is not a drop-in replacement for btrfs yet. It's still missing critical things like scrub: https://bcachefs.org/Roadmap/ https://bcachefs.org/Roadmap/
- bhaney 3y agoAs a heavy btrfs user, I do expect bcachefs to fully replace it eventually. But that's still many years off.
- nolist_policy 3y agoI fully expect bcachefs will initially hit similar issues like btrfs. Until now (bcachefs has been merged) it has only been used by people running custom kernels. As more people try it, it will hit more and more drives with buggy firmware and whatnot.
- viraptor 3y agoThey're in a slightly different position though. Bcache itself has existed for many years and has been used in production. Bcachefs changes it quite a bit, but I wouldn't expect any basic issues we've seen elsewhere.
- e145bc455f1 3y agoJust last week my btrfs filesystem got irrecoverably corrupted. This is like the fourth time it has happened to me in the last 10 years. Do not use it in consumer grade hardware. Compared to this, ext4 is rock solid. It was even able to survive me accidentally passing the currently running host's hard disk to a VM guest, which booted from it.
- londons_explore 3y ago> last week my btrfs filesystem got irrecoverably corrupted. This is 2 bugs really. 1, the file system got corrupted. 2, tooling didn't exist to automatically scan through the disk data structures and recover as much of your drive as possible from whatever fragments of metadata and data were left. For 2, it should happen by default. Most users don't want a 'disk is corrupt, refusing to mount' error. Most users want any errors to auto-correct if possible and get on with their day. Keep a recovery logfile with all the info needed to reverse any repairs for that small percentage of users who want to use a hex editor to dive into data corruption by hand.
- mrob 3y agoWhere is that log file supposed to be stored? It can't be on the same filesystem it was created for or it negates the purpose of its creation.
- londons_explore 3y agoIf I were designing it, the recovery process would: * scan through the whole disk and, for every sector, decide if it is "definitely free space (part of the free space table, not referenced by any metadata)", "definitely metadata/file data", "unknown/unsure (ie. perhaps referenced by some dangling metadata/an old version of some tree nodes)". * I would then make a new file containing a complete image of the whole filesystem pre-repair, but leaving out the 'definitely free space' parts. * such a file takes nearly zero space, considering btrfs's copy-on-write and sparse-file abilities. * I would then repair the filesystem to make everything consistent. The pre-repair file would still be available for any tooling wanting to see what the filesystem looked like before it was repaired. You could even loopmount it or try other repair options on it. * I would probably encourage distros to auto-delete this recovery file if disk space is low/after some time, since otherwise the recovery image will end up pinning user data to using up disk space for years and users will be unhappy. The above fails in only one case: Free space on the drive is very low. In that case, I would probably just do the repairs in-RAM and mount the filesystem readonly, and have a link to a wiki page on possible manual repair routes.
- Daunk 3y agoI recently tried (for the first time) Btrfs on my low-end laptop (no snapshots), and I was surprised to see that the laptop ran even worse than it usually does! Turns out there was something like a "btrfs-cleaner" (or similar) running in the background, eating up almost all the CPU at all time. After about 2 days I jumped over to ext4 and everything ran just fine.
- eru 3y agoInteresting that the 'cleaner' doesn't run as nice?
- rcthompson 3y agoI'm pretty sure it's a kernel thread, not a process, since it's part of the filesystem. So it can't be renice'd.
- eru 3y agoOh, that's annoying. I wonder if there's a way with kernel configs etc to only run that thread when there's nothing better to do? Sounds like something someone would have thought of in general for background house keeping?
- mustache_kimono 3y ago> I recently tried Btrfs on my low-end laptop (no snapshots) Do snapshots degrade the performance of btrfs?
- j16sdiz 3y agoNo noticeably. you see the difference when enabling CoW though
- nolist_policy 3y agoWhat was your workload? Do you have quotas enabled? Compression? Are you running OpenSuse by any chance?
- londons_explore 3y agoOne btrfs bug which is 100% reproducible: * Start with an ext3 filesystem 70% full. * Convert to btrfs using btrfs-convert. * Delete the ext3_saved snapshot of the original filesystem as recommended by the convert utility. * Enable compression (-o compress) and defrag the filesystem as recommended by the man page for how to compress all existing data. It fails with out of disk space, leaving a filesystem which isn't repairable - deleting files will not free any space. The fact such a bug seems to have existed for years, with such a basic following of the man pages for a common use case (migration to btrfs to make use of its compression abilities to get more free space), tells me that it isn't yet ready for primetime.
- cesarb 3y ago> deleting files will not free any space. Does a rebalance fix it? I have once (and only once, back when it was new) hit a "out of disk space" situation with btrfs, and IIRC rebalancing was enough to fix it. > for a common use case It might have been a common use case back when btrfs was new (though I doubt it, most users of btrfs probably created the filesystem from scratch even back then), but I doubt it's a common use case nowadays.
- eru 3y ago> It might have been a common use case back when btrfs was new (though I doubt it, most users of btrfs probably created the filesystem from scratch even back then), but I doubt it's a common use case nowadays. It's perhaps not as common as it once was, but you'd expect it to be common enough to work, and not some obscure corner case.
- bayindirh 3y agoFrom my perspective, A filesystem is a critical infrastructure in an OS, and failing here and there and not fixing these bugs because they're not common is not acceptable. Same for the RAID5/6 bugs in BTRFS. What's their solution? A simple warning in the docs: > RAID5/6 has known problems and should not be used in production. [0] Also the CLI discourages you from creating these things. Brilliant. This is why I don't use BTRFS anywhere. An FS shall be bulletproof. Errors must only cause from hardware problems. Not random bugs in a filesystem. [0]: https://btrfs.readthedocs.io/en/latest/mkfs.btrfs.html#multiple-devices https://btrfs.readthedocs.io/en/latest/mkfs.btrfs.html#multi...
- zaggynl 3y agoFor what it's worth, I'm happy with using Btrfs on OpenSUSE Tumbleweed and the provided tooling like snapper and restoring Btrfs snapshots from grub, saved me a few times. SSD used: Samsung SSD 970 PRO 1T, same installation since 2020-05-02.
- xyzzy_plugh 3y agoI'm so sad. I was in the btrfs corner for over a decade, and it saddens me to say that ZFS has won. But it has. And ZFS is actually good. I'm happy with it. I don't think about it. I've moved on. Sorry, btrfs, but I don't think it's ever going to work out between us. Maybe in a different life.
- doublepg23 3y agoI’m personally cheering for Bcachefs now.
- matheusmoreira 3y agoWhat saddens me is the fact they still haven't managed to put ZFS into the Linux kernel tree because of licensing nonsense.
- different_base 3y agoIt's not Linux's fault for not merging ZFS. Blame Oracle.
- matheusmoreira 3y agoI know. I was blaming Oracle, and Sun before them. Why can't they just relicense the entire thing as GPL or some other license that's actually open source and GPL-compatible such as MIT? They own the copyright, don't they? Surely they have the power to do that. ZFS is too important to not be in Linux. Yet it's 2024 and it's still not in there. Because of licensing copyright nonsense. The blame falls solely on the copyright owners.
- mardifoufs 3y ago> I'm so sad. I was in the btrfs corner for over a decade, and it saddens me to say that ZFS has won. But it has. > And ZFS is actually good. I'm happy with it. I don't think about it. I've moved on. > Sorry, btrfs, but I don't think it's ever going to work out between us. Maybe in a different life. ZFS has had a worse bug not even 4 months ago. They are both very good filesystems, bugs happen
- west0n 3y agoThe vast majority of databases currently recommend using XFS.
- west0n 3y agoI'm very curious if there are any databases running in production environments on Btrfs or ZFS.
- yjftsjthsd-h 3y agoIt wasn't a company you've heard of, but I can absolutely tell you that there are:) We really benefited from compressing our postgres databases; not only did it save space, but in some cases it actually improved throughput because data could be read and uncompressed faster than the disks were physically capable of. Edit: Also there was a time when it was normal for databases to be running on Solaris, in which case ZFS is expected; my recollection is that Sun even marketed how great of a combination that was.
- eru 3y agoInteresting. The compression options that postgres offers itself were not enough?
- yjftsjthsd-h 3y agoIt went the other way; we were using ZFS to protect against data errors, then found that it did compression as well. But looking now, the only native postgres options I see are TOAST, which seems to only work for certain data types in the database, and WAL compression (that's only existed since pg 15), so unless I've missed something I would tend to say yes it's far superior to the native options.
- eru 3y agoI mostly only looked into the options offered by MariaDB a while ago, and they seemed quite neat. I had just assumed that postgres was at least on par. Thanks for reporting!
- applied_heat 3y agoBtrfs seems to work perfectly well in synology NAS. It must be some other combination of options or functions in use from what is available in synology that garners the bad reputation
- gh02t 3y agoSynology uses BTRFS weirdly, on top of dmraid, which negates the most well-known BTRFS bugs in their RAID5/6 implementation*. Best I can tell they also have some custom modifications in their implementation as well, though it's hard to find much info on it. * FWIW, I used native BTRFS RAID5 for years and never had an issue but that's just anecdata
- deleted 3y ago[deleted]
- jcalvinowens 3y agoThis bug is very very rare in practice: all my dev and testing machines run btrfs, and I haven't hit it once in 100+ machine-hours of running on 6.8-rc. The actual patch is buried at the end the article: https://lore.kernel.org/linux-btrfs/1ca6e688950ee82b1526bb3098852af99b75e6ba.1710551459.git.tavianator@tavianator.com/ https://lore.kernel.org/linux-btrfs/1ca6e688950ee82b1526bb30...
- tavianator 3y agoDo they run btrfs on top of dm-crypt? I suspect it's impossible to reproduce on a regular block device.
- jcalvinowens 3y agoYes, most of them do, precisely because I'm trying to catch more bugs :)
- jcalvinowens 3y agoDid this ever trip any of the debugging ASSERT stuff for you? I'm really curious if some more generic debugging instrument might be able to flag this failure mode more explicitly, it's far from the first ABA problem with ->bflags. Also, if you don't mind sharing your kconfig that would be interesting to see too.
- tavianator 3y agoNo, but I may have been missing some debugging CONFIGs. I was just using the stock Arch kconfig. I did submit a patch to add a WARN_ON() for this case: https://lore.kernel.org/linux-btrfs/d4a055317bdb8ecbd7e6d9bdb5ebb074fa93f7f8.1710769876.git.tavianator@tavianator.com/ https://lore.kernel.org/linux-btrfs/d4a055317bdb8ecbd7e6d9bd... But to catch the general class of bugs you'd need something like KCSAN. I did try that but the kernel is not KCSAN-clean so it was hard to know if any reports were relevant.
- jcalvinowens 3y ago
- disdba 3y ago[dead]
- streb-lo 3y agoAs someone who uses a rolling release, I use btrfs because I don't want to deal with keeping ZFS up to date. It's been really good for me. And btrbk is the best backup solution I've had on Linux, btrfs send/receive is a lot faster than rsync even when sending non-incremental snapshots.
- kccqzy 3y agoSame here: I use a rolling release and btrfs. Personally I really enjoy btrfs's snapshot feature. Most of the time when I need backups it's not because of a hardware failure but because of a fat finger mistake where I rm'ed a file I need. Periodic snapshots completely solved that problem for me. (Of course, backing up to another disk is absolutely still needed, but you probably need it less than you think.)
- matja 3y ago> I don't want to deal with keeping ZFS up to date That's what DKMS is for, which most distros use for ZFS. Install and forget.
- streb-lo 3y agoI think the larger issue is that openzfs doesn't move in sync with the kernel so you have to check https://raw.githubusercontent.com/openzfs/zfs/master/META https://raw.githubusercontent.com/openzfs/zfs/master/META to make sure you can actually upgrade each time. On a rolling distro this is a pretty common breakage AFAIK. It's not the end of the world, but it is annoying.
- yjftsjthsd-h 3y agoDepends on the rolling release; some distros specifically provide a kernel package that is still rolling, but is also always the latest version compatible with ZFS.
- deleted 3y ago[deleted]
- TZubiri 3y agoreminder that btrfs stands for butter filesystem and not better filesystem
- cies 3y agoI had my /home on a subvolume (great as the sub and super share the same space). When I wanted to reinstall I naively thought I could format the root super volume and keep /home subvolume -- but this was impossible: I had to format /home as well according to the OpenSUSE Tumbleweed installer. Major problem for me. I now have separate root (btrfs) and home (ext4) partitions.
- stryan 3y agoYou can do it but it's not a very happy path. Easiest way is probably to map the old home subvolume into a different path and either re-label the subvolumes once you're installed or just copy everything over. Separate BTRFS root and ext4 home partitions is either the default filesystem layout now if you're not doing FDE or the second recommended one.
- cies 3y agoI do disk encryption of root and home. Not of boot and swap (the attack needed to steal my data in that scenario is too involved: according to my assessment my data is not that valuable)
- vetinari 3y agoThat's how subvolumes work; not just on btrfs, but on zfs as well. The non-footgun way is to have several subvolumes (@root, @boot, @home for btrfs, rpool/ROOT/system_instance and rpool/USERDATA for zfs), and nothing in else in the volume itself. Then, you wipe the system subvolume and create a new one, with new system. Or just keep the old system subvolume and create a new one.
- apitman 3y agoI really wish you could use custom filesystems such as btrfs with WSL2. I don't think there's currently any way to do snapshotting, which means you can never be sure a backup taken within WSL is corrupt.
- chasil 3y agoCan this be used? I knew ReactOS would use it natively. https://github.com/maharmstone/btrfs https://github.com/maharmstone/btrfs
- apitman 3y agoMaybe that could be adapted, but I don't think it would solve my problem currently. Basically I want to be able to do btrfs snapshots from within my WSL distros, so that I can run restic or similar on the snapshots.
- aseipp 3y agoHell, even just being able to use XFS would be an improvement, because ext4 has painful degradation scenarios when you hit cases like exhausting the inode count. (Somewhat related, but there has been a WIP 6.1 kernel for WSL2 "in preview" for a while now... I wonder why it hasn't become the default considering both it and 5.12 are LTS... For filesystems like btrfs I often want a newer kernel to pick up every bugfix.)
- cogman10 3y agoYou can, it's just a bit of a pain. https://blog.bryanroessler.com/2020-12-14-btrfs-on-wsl2/ https://blog.bryanroessler.com/2020-12-14-btrfs-on-wsl2/
- apitman 3y agoNice. Unfortunately for my use case I can't use physical storage devices. I need a something similar to a qcow2 that can be created and completely managed by my app.
- mappu 3y ago
- JonChesterfield 3y agoThe visualisation of the data race in this post is superb. Worth reading just for that. Handrolled concurrency using atomic integers in C. Without the proof system or state machine model to support it. Seems likely that's not going to be their only race condition.
- re 3y agoThe animations also stood out to me. I took a look at the source to see if they used a library or were written completely by hand and was surprised to see that they were entirely CSS-based, no JavaScript required (although the "metadata" overwrite animation is glitched if you disable JS, for some reason not immediately apparent to me).
- bhaney 3y ago> for some reason not immediately apparent to me The CSS that sets the background of the code elements (so they properly cover up the ones under them) is attached to the hljs class, which is added to those elements by highlight.js. No JS > no classname added > no background style > jank
- tavianator 3y agoThanks for pointing this out to me! I'll see if I can pre-render the syntax highlighting like I do with KaTeX and stuff. No reason to do that on the client.
- Sakos 3y agoAnybody know how the visualisation was done?
- tavianator 3y agoI did it by hand with CSS: https://github.com/tavianator/tavianator.com/blob/main/src/2024/btrfs_bug.md?plain=1#L248 https://github.com/tavianator/tavianator.com/blob/main/src/2...
- deleted 3y ago[deleted]
- n8henrie 3y agoI've been overall very happy with: - Arch on BTRFS RAID1 root, across 3 dissimilar NVME drives (about 7 years, 3 drive replacements for hardware failure, I don't think ZFS supports this configuration) - numerous low-power systems (like Pi3s) on BTRFS root (also going on 7 years for several of these, lighter on resources than ZFS) - Asahi NixOS on BTRFS root (kernel doesn't support ZFS) My NAS and several larger datasets are on ZFS based on reputation alone, but honestly I've had more data loss scares with ZFS than BTRFS (drives that have disappeared for no reason then reappeared hours later, soft locked and unable to unmount indefinitely, several unfortunate user-error issues with automounting datasets overlaying necessary top-level directories and preventing successful boots), and I find the BTRFS tooling more intuitive. For my hobbyist-level homelab type needs, I would say I'm overall pretty happy with BTRFS. The only issue I've never been able to resolve is regarding lockups when issuing quotas -- another reason I stick to ZFS for my spinning rust storage drives. Oh, and the ZFS ability to mount a zvol as a foreign filesystem (words?) lets me `btrfs send` backups to ZFS, which is nice!
- frankjr 3y ago> about 7 years, 3 drive replacements for hardware failure You've had 3 SSD failures in just 7 years? Any more details about those disks?
- n8henrie 3y agoCheap NVME drives. And to be fair, 2 of 3 were only suspicious for having a hardware problem so I replaced with a larger drive (and was super nice to have such a straightforward was to expand the size as compared to ZFS).