13 ms·
Btrfs RAID 5/6 Code Found to Be Very Unsafe and Will Likely Require a Rewrite
- PaulHoule 10y agoFancy file systems are for people who think their life isn't exciting enough.
- lmm 10y agoDepends how you're defining "fancy". I use ZFS because I don't want any drama, and it has delivered that (Linux md was more complicated to set up, to the extent that I lost data). But it's a mature, solid filesystem in a way that BTRFS isn't yet.
- click170 10y agoDownvoted because I frequently hear this sentiment from both sides of the argument but rarely is anything more than an anecdote offered to support it. Feels more and more like a mud slinging competition instead of an assessment of merit.
- finnh 10y agoYou may wish to read the article we are discussing, then.
- rleigh 10y agoMany people use both filesystems, and you'll see many claims of how both work really well and that the user hasn't had any problems. The problem with these people's experiences is not that they are untrue, but that they are primarily from people who haven't had hardware failure/glitches, and who have never had their systems run the failure-case codepaths. Everything's fine and dandy up until the point you lose all your data. I've run Btrfs on many systems since just after it started to be usable, and written software with Btrfs-specific support which hammers it (and LVM) like nothing else creating and destroying tens of thousands of transient snapshots. I've also now run ZFS on several systems, admittedly over a smaller timeframe (3 years vs 7-8ish). I've had Btrfs totally trash a RAID1 mirror from a transient SATA cable connector glitch. On this test system, I had half the disk using Btrfs, half using mdraid/LVM. The mdraid half recovered and resynced transparently as soon as I reseated the connector; no service interruption or dataloss. Btrfs ceased to function, and on reboot toasted both mirrors resulting in total unrecoverable dataloss and repeated kernel panics. That's been fixed a while, but right here we're seeing the same thing. The failure codepaths, which are of critical importance, are untested and buggy. And even non-failure codepaths are still bad. Take the snapshotting case above, I had to take the system offline and do a full manual rebalance every 18 hours. The time from fresh new filesystem to read-only unbalanced disaster was just 18 hours when thrashed continuously, at most using 10% of the total space. And lastly, the performance of some things such as fsync are truly abysmal, to the extent that we had to use "eatmydata" to completely disable it for apt/dpkg operations! When under heavy parallel workloads, it could take many tens of minutes or hours(!) to complete writes which ext4 would complete in a minute or so. I've yet to experience any problems at all with ZFS. Now that might have been luck on my part, but it might also be down to better design and quality of implementation. It's certainly been battle tested in high end installations. That's not to say that Btrfs doesn't have some neat features; it does a few things ZFS doesn't, like rebalancing data over its devices while ZFS only does that on write. But Btrfs has let me down badly every time I've used it in anger, and those few neat features don't make up for its lack of robustness--the primary purpose of the filesystem is to reliably store data, and it fails at that. I don't like to see "mud slinging", since such fanboyism is unobjective and uninformed. I've reached my opinion based upon several years of practical intensive use of Btrfs for various things, the most demanding of which was repeated whole-archive rebuilds of the whole of Debian when I was maintaining the Debian build tools, and wrote btrfs snapshot support specifically for them, doing over 30 parallel builds on a single system using independent snapshots per build with over 20000 snapshots per run, creating and destroying several per second. The experiment was disastrous, and showed Btrfs to be unsuitable for such intensive workloads. When your filesystem is guaranteed be turned read-only at some unpredictable and unknown point in the future, you can't rely on it. Regular rebalancing mitigates but doesn't solve this, and has a terrible performance impact. Not dataloss per se (unless it makes you lose writes when it turns read-only), but it's a serious design or implementation flaw. I did all this testing and adding of Btrfs support to various tools because I had high hopes for its potential; unfortunately they exposed serious shortcomings, many of which exist to this day. Today I'm using ZFS, not because of any irrational prejudice against Btrfs, but because Btrfs has never managed to deliver a robust and well tested filesystem!
- nisa 10y agoI'm not having nearly your experience but I just want to say I'm agreeing 100% to you conclusions based on my experience. We ran at Uni a Hadoop Cluster that had disks slowly dying (some bad sectors every few days, but otherwise fine) and lacked the money to replace them. We replaced ext4 with ZFS (no raid, just plain zpools with failmode=continue) and ZFS ran mostly fine and scrubs kept the metadata sane. Never had data loss (HDFS has it's own replication and checksumming, we just need sane metadata for Hadoop to run and intermediate MapReduce outputs where send to directories with zfs set copies=2) and we only replaced the botched disks that had longer scrub times or couldn't survive a scrub. I'm still surprised how ZFS managed to pull that off. Only bugs I've found where related to ZoL at this time but could be worked around. btrfs switched to readonly as fast as ext4 (which is probably the correct thing to do) but was useless for this problem. On new hardware with new enterprise disks we choose btrfs and we had a painful tour of crashes, data corruption, metadata corruption (undeletable files), deadlocks until kernel 4.4 where things got a little bit better. Here disks and server where enterprise class and fully working. Just btrfs bugs. Also no RAID. I'm not doing that anymore so I don't know if any new bugs appeared but the whole experience will keep me from ever using btrfs. This was ~2years ago and you could easily find lot's of slides Fujitsu or Suse that btrfs is stable and you can use it (around kernel 3.13-3.16). It's probably fine for your notebook or even your backup HDD but don't think you can stress it without experiencing pain (be it corruption, hangups or dataloss) or just abysmal performance. That beeing said ZFS on Linux is also a far cry from rock solid but I'm optimistic that they flesh out the problems and tackle them in a solid way. As a Linux fanboy for years this gave me some solid appreciation for Solaris engineering.
- lomnakkus 10y agoUnstable file systems are for people who think their life isn't exciting enough. If you've been following along at all in btrfs development, this doesn't really come as a huge surprise.
- pmarreck 10y agoHad nothing but great results from ZFS on a FreeNAS server.
- Philipp__ 10y agoIt is amazing how stubborn Linux people are. They constantly refuse and reject superior engineering that was done outside of their realm. (yeah I was pointing at ZFS, but that would be just one of many)
- lomnakkus 10y agoThe lack of ZFS is due to licensing.
- pantalaimon 10y agoUbuntu has it
- LukeShu 10y agoIn violation of the license; probably.
- cm3 10y agoAFAIK it's DKMS'ed just like Nvidia's kernel driver, so it's a user decision, just like media codec installation. Also, OpenZFS doesn't reuse kernel interfaces it's not allowed to, and that's why they carry their own crypto code now.
- lathiat 10y agoIt's actually not DKMS in 16.04 - crypto problem still applies though.
- vonmoltke 10y agoHow is the license being violated?
- lomnakkus 10y agoNot necessarily. It's a run-time loaded module (as e.g. NVIDIA's graphics driver is) and 'taints' the kernel with CDDL, but since you -- the user -- aren't redistributing anything you are not breaking any copyright laws. EDIT: ... and neither are Ubuntu since what they're doing is "mere aggregation" (see the GPL).
- sp332 10y agoBtrfs people have been telling everyone that RAID5/6 code is incomplete for years now, so hopefully people weren't depending on it for important things. The wiki link is broken, here's the page https://btrfs.wiki.kernel.org/index.php/RAID56 https://btrfs.wiki.kernel.org/index.php/RAID56 and you can see in the history that on the very first version of the page, in the first sentence, "the recovery and rebuild code isn't bulletproof or complete yet."
- pantalaimon 10y agoThis warning has only been added recently https://btrfs.wiki.kernel.org/index.php?title=RAID56&diff=30445&oldid=29769 https://btrfs.wiki.kernel.org/index.php?title=RAID56&diff=30...
- sp332 10y agoOh, so that's ~7 months of people maybe using it for important things. That could be really bad then.
- pantalaimon 10y agoWell that line about it stabilizing over the next couple of kernel releases was added in February 2015, so when I installed Ubuntu 16.04 on my NAS I thought things probably had calmed down sufficiently. As I learned the hard way, I was wrong and that warning is very much justified. https://btrfs.wiki.kernel.org/index.php?title=RAID56&diff=29294&oldid=25681 https://btrfs.wiki.kernel.org/index.php?title=RAID56&diff=29...
- gbrown_ 10y agoBryan Cantrill expresses this[1] in a manner I certainly agree with. Software that is designed/ intended to be reliable should not go through large periods of instability only to be written off as "prepubescence". [1]http://www.youtube.com/watch?v=79fvDDPaIoY&t=18m28s http://www.youtube.com/watch?v=79fvDDPaIoY&t=18m28s
- cpeterso 10y agoBryan Cantrill is such a great storyteller. "My therapist says I need to talk about the firmware revision numbers, just to let it out so it doesn't corrode me." He then shares a story about a firmware bug where the polarity would be reversed of the disk read/write head. :)
- rdtsc 10y agoCompletely agree. I often just search for videos by him and just listen to him present a topic or talk about anything. I learned about things normally I wouldn't go and read about otherwise -- history of Sun, ZFS, dtrace, containerization (Joyent, zones legacy, KVM, comparison with Docker). In that respect he is like David Beazley (of Python fame). I could just listen to David talk all day and not get tired of it.
- rdtsc 10y agoOoh, like that analogy. Pure gold: (Paraphrasing): --- ZFS never went through a period where people lost data using it. ... amazing how people are willing to write that off like 'Oh well, it kind of lost my data but Omg! A million eyeballs'. This is like a 12 year old who is stealing cars -- this is a serious issue. Not like "Well...he's only 12" it is more like "He stole a car at gunpoint, having just robbed a liquor store". Can't say let's just wait till he's 18. Would you want to know what he's like when he's 18? ---
- makomk 10y agoTechnically, ZFS never went through a period where people lost data using it. It just kernel paniced when they tried to access their data and besides it was probably their fault for using hardware that was slightly flaky. (Early ZFS adopters had a really interesting time of it, judging from the complaints people had back then.)
- teraflop 10y agoI've never really understood the rationale behind integrating RAID into the filesystem. It seems like a giant layering violation, and unnecessary given that mdadm already exists and has many years of testing behind it. The btrfs FAQ [1] says that "unlike MD-RAID, btrfs knows what blocks are actually used by data/metadata, and can use that information in a rebuild/recovery situation", but is that really a good enough reason to reimplement the entire RAID subsystem from scratch? And couldn't TRIM/discard provide the same benefits? For me, the clincher is that mdadm has a very stable, well-defined on-disk format, which is a huge bonus if you hit a bug or make a mistake. I once almost lost a personal RAID10 array by recklessly trying to add an extra drive with no backup, and without fully understanding what I was doing. I was able to recover all of my data by hacking together a Python script to reassemble all of the blocks in the correct order. I can't imagine how much effort that would have taken if I had to build something that understood the full details of btrfs's data structures. [1]: https://btrfs.wiki.kernel.org/index.php/FAQ#Case_study:_btrfs-raid_5.2F6_versus_MD-RAID_5.2F6 https://btrfs.wiki.kernel.org/index.php/FAQ#Case_study:_btrf...
- erkkie 10y agoSometimes collapsing arbitrary layers gives you more context to work on. Judging from ZFS history, this is the correct path. (and if you've ever had to rebuild a mostly free, large, raid5-6 array, you will appreciate filesystem aware rebuilds)
- snuxoll 10y agoIntegrating RAID into the filesystem solves a lot of problems, one big one being data integrity. When you have a dumb RAID layer unaware of the data on disk it will just replicate or shard it between disks, when you request that data back and you have replicas it has no way to verify that data is correct, if one replica is bad but the other is good your filesystem has no way to correct for it since it just sees one big block device. Moving storage pooling functionality to the filesystem is the right call for anything with server workloads, ZFS's self-healing functionality wouldn't work without it (and this is precisely why I use FreeNAS instead Unraid or other alternatives, I care about the long-term integrity of my data, especially my VMWare images and photo collection).
- LeoPanthera 10y agoIs it possible to install Linux on HAMMER?
- reitanqild 10y agoI'm interested to know as well. I read up on it yesterday but couldn't find anything but a student project that only provided read access.
- X86BSD 10y agoYou mean is it possible to port and run hammer on Linux? Anything is possible but the work required would be a big challenge.
- deleted 10y ago[deleted]
- moogly 10y agoDoes this impact Synology's Hybrid RAID running on Btrfs? Btrfs is nowadays the default fs on Synology boxes.
- Calms 10y agoNo, there is no impact to Synology's Hybrid RAID running on btrfs. Synology uses mdadm/lvm for RAID.
- koverstreet 10y agoIf anyone wants to help out with bcachefs, Reed-Solomon might be a fun place to jump in. And bcachefs so far has a better track record with people's data than btrfs :)