10 ms·

Examining Btrfs

102 points by mrlonglong 5y ago

jqpabc123 5y ago
Do one thing and do it well. Do too much and you may end up like Btrfs.
- nikbackm 5y ago
  What about ZFS?
  - rwaksmunski 5y ago
    What about it? It does big storage really, really well.
  - jqpabc123 5y ago
    Suspect as well --- for the same reason. System level data handling functionality is critical and needs to be rock solid. Complexity makes this more difficult to achieve. Not to say it's impossible, just increasingly more difficult.
    - db48x 5y ago
      I disagree. I think that the traditional split between block devices and file systems is a premature optimization that ZFS fixes. And in practice ZFS is extraordinarily solid.
      - krageon 5y ago
        > And in practice ZFS is extraordinarily solid. As long as you never ever run into an edge case it's great, but then again the same goes for btrfs :)
        db48x 5y ago
        Did you have anything specific in mind? Because otherwise that statement could apply to anything.
  - uniqueuid 5y ago
    I have to second this - the rampant layering violation that is ZFS has in my experience been extremely robust. Maybe it's simply doing the "one coherent tool to manage large chunks of data and disks" thing well.
    - creshal 5y ago
      "Do only one thing and do it well" is highly dependent on how you define "thing", yes. Is awk bad because you can't compose the internals of its built-in functions? Are vim and emacs bad because they have multiple modes of operation? Is bash violating the unix spirit by doing both the job of an interactive shell and a C API extensible scripting engine? I don't think so. ZFS, BTRFS, and advanced ext4 features like encryption are more arguments that layered block devices may have been a poor choice of abstraction.
  - ajross 5y ago
    ZFS is a more mature and (maybe?) more performant solution to... the same wrong problem. The modern world doesn't want this stuff from its filesystems. It just doesn't. Modern flash devices are extremely reliably and outrageously fast, and they pervasively sit on high bandwidth internet connections which provide reliable backup for virtually every application imaginable. RAID and checksumming[1] just isn't used at the level of small/personal devices anywhere anymore, and really it never will be. And as you scale higher, single filesystems just don't cut it anyway. Cloud storage paradigms work on much larger bits, and worry about things like multi-site failover and atomicity. ZFS/btrfs has nothing to offer someone working on an S3-like service, or a globalized database, either. I mean, it's fine. Use it if you like it and you have an esoteric application that can leverage it (lots of hackers' personal networks tend to have toys like that). For that matter btrfs is fine too; my main data server at home is scripted around btrfs snapshots because they're easy. But really... fancy filesystems are just software in need of a home. Run ext4 or whatever and don't sweat it. [1] Non-encryption checksumming anyway. dm-verity doesn't work at the filesystem layer, nor should it.
    - uniqueuid 5y ago
      Although I completely agree with your observations - sitting in a weird spot between very large and very small data, there are use cases where ZFS is extremely useful. I'm regularly working with datasets in the single to two digit TB range. They need to sit on disk because the university doesn't have 10GE. If I ran a regular raid and would encounter a disk or data error, I would have to re-create the data from tape archive or worse, lose it; and getting a replacement disk would surely take weeks. So instead, I've been running raidz pools for ~10 years and have never skipped a beat. I agree that this is not very typical, but it suits my needs perfectly.
    - curt15 5y ago
      Isn't Lustre designed specifically for ZFS as its backing filesystem?
      - ajross 5y ago
        No, Lustre shipped on ext4 for years and years. Not an expert, but IIRC there aren't any core/major/whatever features in ZFS being leveraged in an irreplaceable way. Again, ZFS works fine, it's there (so is btrfs). Use it if it does something you like. It's just not really "worth" the level of complexity involved, and the window has long since closed for fancy filesystems to change any major paradigms of computation. This is dinosaur technology, basically.
gootler 5y ago
Butters fs!
uniqueuid 5y ago
One thing I always found curious is that Synology supports Btrfs. Granted, they are not exactly what I'd call a company with a rock-solid track record, but they are still a rather large premium manufacturer in the NAS space. It seems they have either managed to contain Btrfs' complexity and prevent users from doing the dangerous things, or their own hybrid raid was even worse, or they are just very risk friendly.
- vanburen 5y ago
  I think Synology runs BTRFS on top of LVM, so that may mitigate some of the issues.
  - cvubrugier 5y ago
    Synology SHR is btrfs (or ext4) on top of LVM and MD. MD is used for redundancy. LVM is used to aggregate multiple MD arrays into a volume group and to allow creating one or move volumes from that volume group.
- zionic 5y ago
  I’ve been incredibly happy with my Synology products FWIW. BTRFS/SHR have been rock solid.
  - uniqueuid 5y ago
    The hardware is very nice indeed, and the software is smooth as well. What has irritated me in the past was that they sometimes took very long to fix vulnerabilities [1]. [1] https://www.cvedetails.com/vendor/11138/Synology.html https://www.cvedetails.com/vendor/11138/Synology.html
    - sumtechguy 5y ago
      What is getting me is the EOL bits they have done with some of their HW. It is a NAS. I can see EOL on parts of the stuff they add. But the fileshare bits? That is really the only reason I bought the thing. Does anyone know any alt distros to replace theirs?
gabella 5y ago
nice site http://essaypapers.reviews/ http://essaypapers.reviews/
terrywang 5y ago
Btrfs is flexible and works well for self-hosted home storage use cases, especially running cheap hardware & HDDs with mixed RPM + capacity specs (many consider it a major advantage over OpenZFS on Linux which inherits ZFS pros and cons). As far as data corruption is concerned, avoid using raid{5,6} profile (that governs how a chunk is replicated within or across a member device) for data and you should be fine. Running OpenZFS on Linux also requires dealing with DKM(ES)S, Ubuntu being an exception (hmm)... NOTE: Self-hosted podcast show episode 25 contains excellent coverage (ZFS vs. Btrfs) by Jonathan Panozzo from Unraid.
- uniqueuid 5y ago
  Yes, canonical was brave enough to assume they don't need DKMS for zfs.ko ... [1] From my experience, Ubuntu's ZFS support is pretty solid, even for root disks. There were some limitations in the past, however - for example, you might not want to use ZFS on a ssd root because of its write amplification. By the way, there's a handy PPA with newer versions [2] - but this does use DKMS. [1] https://ubuntu.com/blog/zfs-licensing-and-linux https://ubuntu.com/blog/zfs-licensing-and-linux [2] https://launchpad.net/~jonathonf/+archive/ubuntu/zfs https://launchpad.net/~jonathonf/+archive/ubuntu/zfs
- mnw21cam 5y ago
  I use ZFS on my main computer, but it sends its backups to a tiny computer with two laptop USB drives plugged in, running Btrfs RAID-0. (The second drive was added when the first one filled up.) I did it that way because it's the second copy of the data, and I'll only need to call on it if my main computer dies. I have a sentry program running on the tiny computer that regularly checks that all the data is present, correct, and readable. The likelihood that it breaks at the same time as my main computer is slim. I wouldn't trust Btrfs for much more than that. I specified ZFS for some of work's very large servers (>100 drives). They wanted to do hardware RAID and XFS instead, and I can just image the nightmares that would have involved.
  - naranha 5y ago
    What is kind of neat for your case maybe is that you can run btrfs scrub even on raid0 or a single drive. The scrub command reads the whole filesystem and verifies checksums, so you can be sure that the data is readable and correct. Disclaimer: I only tried it on a single disk.
uniqueuid 5y ago
TL;DR: Btrfs is usable in mirror mode. Raid is still unusable (as in: you will lose data) Usage and tooling are unintuitive and dangerous. I have to say, it doesn't sound so bad when you know what you're doing. The example from the article - the fact that you need to re-silver and re-balance a mirror after a failure - yes that is dangerous. But then again, when you're starting with zero ZFS knowledge, you also need to learn the particular way it's designed. And let's remember that some mdadm raids don't auto-scrub periodically. So the essence is: Storage is still hard, it's still necessary to think before you manage disks.
- creshal 5y ago
  > But then again, when you're starting with zero ZFS knowledge, you also need to learn the particular way it's designed. Except… there's nothing to really learn. All these weird edge cases that can kill your BTRFS simply don't exist in ZFS. The tooling is also much more polished and in the few cases that something does require user attention, it'll tell you in easily understandable terms, so as a new user the only command you really need to learn is `zpool status`, which you now did.
  - WastingMyTime89 5y ago
    > Except… there's nothing to really learn. Would you mind explaining to us how to add a disk to an array with ZFS? Then same question but with a disk which is not the same size that the rest of the array.
    - uniqueuid 5y ago
      You're teasing, of course, but it shows the real problems. To take you up on that with a dangerous example that reinforces the point: To add a disk to an array whose used space is less than the capacity of a single disk, you can (DON'T DO THIS, IT'S DANGEROUS) (1) remove a disk from the array (it's degraded now) (2) reformat the single disk, put all data on there (3) create a sparse file with the size of one disk (4) create a new array with N disks, in place of the temporary disk use the sparse file as a device (5) remove the sparse file (the new array is now degraded) (6) bring the new array up, copy all data from temp disk on it (7) replace the sparse file with the temp disk (8) copy files around to make them re-balance Congratulations, you've now grown your array, and put your data seriously at risk in the process! :)
jacknews 5y ago
The ONLY time I've actually lost data, was with BTRFS, and only using it as a basic single-volume filesystem, no RAID, snapshots, or anything complex. BTRFS does not seem to recover well from media errors. I've had plenty of those, and always been able to recover at least partial data (minus the bad media) with other FS's.
- codys 5y ago
  I've also recently had bad experiences where hard-power-offs of a fleet of systems with a btrfs filesystem repeatedly cause corruption (resulting in the btrfs filesystem no longer being mountable) Some testing seems to indicate even ext4 is more reliable in the face of hard power cycles.
- labawi 5y ago
  I've had funny experiences with disappearing files on ext4 - reading a file while there are transient SATA errors (on a flaky SATA link) would cause a the file to be removed. To be fair, I think I've lost more data on btrfs.
- ok123456 5y ago
  And the "rescue" commands tell you not to use them unless you've been blessed by someone on a mailing list, and that they can actually cause MORE damage. I was using BTRFS on my workstation and all of a sudden some garbage got written to the extent-tree, and it was toast. Never using BTRFS again. I had backups and was able to recover 99.99%, but the recuse tools were garbage. I finally killed it after it had spent a month, from July 4 until August 4, doing apparently nothing. And this was this year. This was a normal desktop configuration, single drive. I wasn't using any of the real features of the FS other than COW.
- jbotz 5y ago
  How long ago was this? My experience about a year ago was the opposite, btrfs held up really well on a failing drive and I managed to copy the filesystem with 'btrfs send' without suffering any data loss or corruption as far as I've been able to tell. I also had a btrfs filesystem on an external drive with a lose USB plug that caused it to disconnect once in a while, and didn't have any corruptions or loss on that one either.
DiabloD3 5y ago
Brtfs only exists as Oracle's "replacement" for ZFS, to hedge their bets in case they couldn't buy Sun. Oracle bought sun in 2010, that has been 12 years. They merged the Btrfs and ZFS teams, they then worked on ZFS entirely with no Btrfs work, and then basically reassigned, fired, or otherwise dissolved the ZFS team when it became clear that OpenZFS (Sun's chosen steward for ZFS) could not be swayed. Btrfs has had no real work done on it since then, and anyone that is actually serious about data storage has moved to ZFS, or has left POSIX filesystems entirely.
- EastToWest 5y ago
  > Btrfs has had no real work done on it since then, and anyone that is actually serious about data storage has moved to ZFS, or has left POSIX filesystems entirely. I'm not very familiar with this space, but if this were true why would Fedora use Btrfs by default? Also on Fedora wiki [0] -- Btrfs is a mature, well-understood, and battle-tested file system, used on both desktop/container and server/cloud use-cases -- which seems to contradict what you said. [0] https://fedoraproject.org/wiki/Changes/BtrfsByDefault https://fedoraproject.org/wiki/Changes/BtrfsByDefault
  - naranha 5y ago
    Apparently it used at facebook a lot too. https://lwn.net/Articles/824855/ https://lwn.net/Articles/824855/
- jcalvinowens 5y ago
  This is one of the most laughably ignorant posts I've ever seen on the internet. > Btrfs has had no real work done on it since then, and anyone that is actually serious about data storage has moved to ZFS, or has left POSIX filesystems entirely Btrfs is the most actively developed filesystem in the kernel, both by LOC changed and the number of contributors. Facebook publicly stated that they run BTRFS on every webserver years ago. ZFS is an antiquated joke. It is wed to the memory allocator in Solaris, and requires that you permanently reserve 1GB of RAM per 1TB of disk space, which is an impractical constraint in the modern world. ZFS is also full of cute little wasteful "innovations" like 128-bit file size support. I don't know what planet you're from, but it's not mine :)
mhd 5y ago
Well, at least there's a chance. Now, ReiserFS 4...
- pdmccormick 5y ago
  His sentence will be served as of sometime in 2023.
- kakd 5y ago
  The development might resume in 2023
jl6 5y ago
Happy user of btrfs-raid1 for more than ten years here, for what it’s worth.
- plainnoodles 5y ago
  Seconded, though I think my array is perhaps only... 8ish years old?
COGlory 5y ago
>Moving beyond the question of individual disk reliability, btrfs-raid1 can only tolerate a single disk failure, no matter how large the total array is. The remaining copies of the blocks that were on a lost disk are distributed throughout the entire array—so losing any second disk loses you the array along with it. (This is in contrast to RAID10 arrays, which can survive any number of disk failures as long as no two are from the same mirror pair.) This feels a little disingenuous to me. This is a distinctly RAID-1 problem. This doesn't feel like a BTRFS problem. Everyone knows RAID-1 = 1 drive loss. >I don't have anything specific to say about btrfs-raid0, other than the fact that it's raid zero. Any failure of any disk loses all data on the array. This is not a storage system, it's a virtual woodchipper. Avoid. If the previous paragraph felt disingenuous, this one feels downright malicious. Not only does BTRFS RAID-0 have no differences from normal RAID-0, but the blanket recommendation to avoid it makes no sense. There are many use cases for RAID-0, but the author is acting like this is some kind of problem with BTRFS.
- tw04 5y ago
  >This feels a little disingenuous to me. This is a distinctly RAID-1 problem. This doesn't feel like a BTRFS problem. Everyone knows RAID-1 = 1 drive loss. That's simply not true. I can do a RAID-1 (mirror) with 3 drives on ZFS or mdadm and lose 2 drives. I can do it with 10 drives and lose 9. https://www.thegeekdiary.com/how-to-add-a-3rd-disk-to-create-a-3-way-mirror-raid1-md-device-centos-rhel-7/ https://www.thegeekdiary.com/how-to-add-a-3rd-disk-to-create... >There are many use cases for RAID-0, but the author is acting like this is some kind of problem with BTRFS. Sure, but for a novice the default advice needs to be: don't do that. And he's writing for folks with all sorts of levels of technical knowledge. Someone who knows nothing about RAID reading the article should be scared away unless they have a very, very good reason for doing RAID-0.
  - nybble41 5y ago
    > I can do a RAID-1 (mirror) with 3 drives on ZFS or mdadm and lose 2 drives. You can do this with BTRFS also; it's called RAID1C3 (for three copies). RAID1C4 is also an option with larger arrays. The issue is that if you have, say, four drives and standard RAID-1 (two copies) it doesn't divide the drives into mirrored pairs where you could lose one drive from each pair without any ill effect (like RAID-10), but rather just ensures that each piece of data is stored on any two different drives. With 4-drive RAID-10 if you lose two drives you have a 50% chance of losing the whole array, since both failing drives could be from the same pair. With BTRFS RAID1 in a four-drive array you have a high probability of losing both copies of some of the data no matter which two drives fail. RAID-6 is strictly better than either, though, since you can lose any two drives and still access all your data.
derbOac 5y ago
Can someone recommend a good overview of modern filesystems that covers performance and licensing issues? I liked this article but was left with the question "so what else do you use?" It's been awhile (too long really) since I really tried to read up on this area. Last time I did I concluded that it was sort of a mess but lots of things were still in progress.
- bluGill 5y ago
  Use ZFS if you can. However for legal reasons it is hard to use on linux, which pushes you to FreeBSD (I'm not sure about the others). FreeBSD can replace linux, but you will find a number of places where the software you use assumes linux and so things don't work right. If you are doing NAS all on your own, then don't hesitate to run to FreeBSD. However if you want to a not do it yourself NAS it is a little harder. The most popular/polished commercial NAS is synology based on BTRFS. I personally use TrueNAS which is a great file server, but synology has some nice integration to photo and media applications and so I'm thinking about switching.
  - bluGill 5y ago
    Too late to edit, but I just discovered the synology isn't use BRTF for redundancy, they are somehow using linux device manager in a way that I don't understand. Since I don't understand what they are doing I don't know how it compares to ZFS.
deleted 5y ago
[deleted]
mrlonglong 5y ago
I would definitely look again at btrfs once it gains raid5/6 capability. Currently happily using ZFS. The only issue is not being able to use latest kernel releases as the ZFS folks usually don't release updates for a while, but overall it doesn't really bother me much nowadays.
dosman33 5y ago
Yet another knock-off of Lustre. Lustre is not perfect, and I'm not suggesting it has a place in the mainline kernel, but it sounds like neither does BTRFS. When you get to large scale (multi-petabyte) filesystems there are not many options, it's GPFS, Lustre, or roll your own. Lustre development has been indirectly sponsored by the DoE for a long time now. And if Lustre didn't exist IBM would charge even more for GPFS. But at least Lustre is free. But it's not a walk in the park to use.
- guenthert 5y ago
  Uh, BTRFS and Lustre can't easily be compared. The former is exclusively a single-host fs, the latter is a cluster-only fs. The former focuses (more or less successfully) on data resilience and availability, the latter on performance. There are quite a few large scale distributed file systems. XtreemFS, Hadoop and Ceph come to mind. There are others, e.g. OrangeFS (successor of PVFS), PanFS, etc. .
anuraaga 5y ago
How to kill your array - reboot your computer at a bad time. Who'd have thought raid1 could be so unreliable. https://btrfs.wiki.kernel.org/index.php/Gotchas#raid1_volumes_only_mountable_once_RW_if_degraded https://btrfs.wiki.kernel.org/index.php/Gotchas#raid1_volume... I did end up having to copy the array out because of this. Copied it into a zfs array and am glad to be out of the land of gotchas.
flas9sd 5y ago
I understand the audience of the ars article is people running larger setups with redundant disk arrays, managing lots of data reliably. But when it comes to what you use in a laptop or desktop, with maybe one more slot for a ssd/nvme or an external disk, btrfs snapshots with incremental send+receive to the second disk or over the network is a good argument in favor of giving it a try.
kzrdude 5y ago
I guess it's unfortunate for btrfs that it has so much "raid" functionality. Otherwise it could be known for its useful other functionality - snapshots, subvolumes, checksums, compression!

Examining Btrfs

102 points by mrlonglong 5y ago

jqpabc123 5y ago

Do one thing and do it well. Do too much and you may end up like Btrfs.

nikbackm 5y ago
What about ZFS?
- rwaksmunski 5y ago
  What about it? It does big storage really, really well.
- jqpabc123 5y ago
  Suspect as well --- for the same reason. System level data handling functionality is critical and needs to be rock solid. Complexity makes this more difficult to achieve. Not to say it's impossible, just increasingly more difficult.
  - db48x 5y ago
    I disagree. I think that the traditional split between block devices and file systems is a premature optimization that ZFS fixes. And in practice ZFS is extraordinarily solid.
    - krageon 5y ago
      > And in practice ZFS is extraordinarily solid. As long as you never ever run into an edge case it's great, but then again the same goes for btrfs :)
      - db48x 5y ago
        Did you have anything specific in mind? Because otherwise that statement could apply to anything.
- uniqueuid 5y ago
  I have to second this - the rampant layering violation that is ZFS has in my experience been extremely robust. Maybe it's simply doing the "one coherent tool to manage large chunks of data and disks" thing well.
  - creshal 5y ago
    "Do only one thing and do it well" is highly dependent on how you define "thing", yes. Is awk bad because you can't compose the internals of its built-in functions? Are vim and emacs bad because they have multiple modes of operation? Is bash violating the unix spirit by doing both the job of an interactive shell and a C API extensible scripting engine? I don't think so. ZFS, BTRFS, and advanced ext4 features like encryption are more arguments that layered block devices may have been a poor choice of abstraction.
- ajross 5y ago
  ZFS is a more mature and (maybe?) more performant solution to... the same wrong problem. The modern world doesn't want this stuff from its filesystems. It just doesn't. Modern flash devices are extremely reliably and outrageously fast, and they pervasively sit on high bandwidth internet connections which provide reliable backup for virtually every application imaginable. RAID and checksumming[1] just isn't used at the level of small/personal devices anywhere anymore, and really it never will be. And as you scale higher, single filesystems just don't cut it anyway. Cloud storage paradigms work on much larger bits, and worry about things like multi-site failover and atomicity. ZFS/btrfs has nothing to offer someone working on an S3-like service, or a globalized database, either. I mean, it's fine. Use it if you like it and you have an esoteric application that can leverage it (lots of hackers' personal networks tend to have toys like that). For that matter btrfs is fine too; my main data server at home is scripted around btrfs snapshots because they're easy. But really... fancy filesystems are just software in need of a home. Run ext4 or whatever and don't sweat it. [1] Non-encryption checksumming anyway. dm-verity doesn't work at the filesystem layer, nor should it.
  - uniqueuid 5y ago
    Although I completely agree with your observations - sitting in a weird spot between very large and very small data, there are use cases where ZFS is extremely useful. I'm regularly working with datasets in the single to two digit TB range. They need to sit on disk because the university doesn't have 10GE. If I ran a regular raid and would encounter a disk or data error, I would have to re-create the data from tape archive or worse, lose it; and getting a replacement disk would surely take weeks. So instead, I've been running raidz pools for ~10 years and have never skipped a beat. I agree that this is not very typical, but it suits my needs perfectly.
  - curt15 5y ago
    Isn't Lustre designed specifically for ZFS as its backing filesystem?
    - ajross 5y ago
      No, Lustre shipped on ext4 for years and years. Not an expert, but IIRC there aren't any core/major/whatever features in ZFS being leveraged in an irreplaceable way. Again, ZFS works fine, it's there (so is btrfs). Use it if it does something you like. It's just not really "worth" the level of complexity involved, and the window has long since closed for fancy filesystems to change any major paradigms of computation. This is dinosaur technology, basically.

gootler 5y ago

Butters fs!

uniqueuid 5y ago

One thing I always found curious is that Synology supports Btrfs. Granted, they are not exactly what I'd call a company with a rock-solid track record, but they are still a rather large premium manufacturer in the NAS space. It seems they have either managed to contain Btrfs' complexity and prevent users from doing the dangerous things, or their own hybrid raid was even worse, or they are just very risk friendly.

vanburen 5y ago
I think Synology runs BTRFS on top of LVM, so that may mitigate some of the issues.
- cvubrugier 5y ago
  Synology SHR is btrfs (or ext4) on top of LVM and MD. MD is used for redundancy. LVM is used to aggregate multiple MD arrays into a volume group and to allow creating one or move volumes from that volume group.
zionic 5y ago
I’ve been incredibly happy with my Synology products FWIW. BTRFS/SHR have been rock solid.
- uniqueuid 5y ago
  The hardware is very nice indeed, and the software is smooth as well. What has irritated me in the past was that they sometimes took very long to fix vulnerabilities [1]. [1] https://www.cvedetails.com/vendor/11138/Synology.html https://www.cvedetails.com/vendor/11138/Synology.html
  - sumtechguy 5y ago
    What is getting me is the EOL bits they have done with some of their HW. It is a NAS. I can see EOL on parts of the stuff they add. But the fileshare bits? That is really the only reason I bought the thing. Does anyone know any alt distros to replace theirs?

gabella 5y ago

nice site http://essaypapers.reviews/ http://essaypapers.reviews/

terrywang 5y ago

Btrfs is flexible and works well for self-hosted home storage use cases, especially running cheap hardware & HDDs with mixed RPM + capacity specs (many consider it a major advantage over OpenZFS on Linux which inherits ZFS pros and cons). As far as data corruption is concerned, avoid using raid{5,6} profile (that governs how a chunk is replicated within or across a member device) for data and you should be fine. Running OpenZFS on Linux also requires dealing with DKM(ES)S, Ubuntu being an exception (hmm)... NOTE: Self-hosted podcast show episode 25 contains excellent coverage (ZFS vs. Btrfs) by Jonathan Panozzo from Unraid.

uniqueuid 5y ago
Yes, canonical was brave enough to assume they don't need DKMS for zfs.ko ... [1] From my experience, Ubuntu's ZFS support is pretty solid, even for root disks. There were some limitations in the past, however - for example, you might not want to use ZFS on a ssd root because of its write amplification. By the way, there's a handy PPA with newer versions [2] - but this does use DKMS. [1] https://ubuntu.com/blog/zfs-licensing-and-linux https://ubuntu.com/blog/zfs-licensing-and-linux [2] https://launchpad.net/~jonathonf/+archive/ubuntu/zfs https://launchpad.net/~jonathonf/+archive/ubuntu/zfs
mnw21cam 5y ago
I use ZFS on my main computer, but it sends its backups to a tiny computer with two laptop USB drives plugged in, running Btrfs RAID-0. (The second drive was added when the first one filled up.) I did it that way because it's the second copy of the data, and I'll only need to call on it if my main computer dies. I have a sentry program running on the tiny computer that regularly checks that all the data is present, correct, and readable. The likelihood that it breaks at the same time as my main computer is slim. I wouldn't trust Btrfs for much more than that. I specified ZFS for some of work's very large servers (>100 drives). They wanted to do hardware RAID and XFS instead, and I can just image the nightmares that would have involved.
- naranha 5y ago
  What is kind of neat for your case maybe is that you can run btrfs scrub even on raid0 or a single drive. The scrub command reads the whole filesystem and verifies checksums, so you can be sure that the data is readable and correct. Disclaimer: I only tried it on a single disk.

uniqueuid 5y ago

TL;DR: Btrfs is usable in mirror mode. Raid is still unusable (as in: you will lose data) Usage and tooling are unintuitive and dangerous. I have to say, it doesn't sound so bad when you know what you're doing. The example from the article - the fact that you need to re-silver and re-balance a mirror after a failure - yes that is dangerous. But then again, when you're starting with zero ZFS knowledge, you also need to learn the particular way it's designed. And let's remember that some mdadm raids don't auto-scrub periodically. So the essence is: Storage is still hard, it's still necessary to think before you manage disks.

creshal 5y ago
> But then again, when you're starting with zero ZFS knowledge, you also need to learn the particular way it's designed. Except… there's nothing to really learn. All these weird edge cases that can kill your BTRFS simply don't exist in ZFS. The tooling is also much more polished and in the few cases that something does require user attention, it'll tell you in easily understandable terms, so as a new user the only command you really need to learn is `zpool status`, which you now did.
- WastingMyTime89 5y ago
  > Except… there's nothing to really learn. Would you mind explaining to us how to add a disk to an array with ZFS? Then same question but with a disk which is not the same size that the rest of the array.
  - uniqueuid 5y ago
    You're teasing, of course, but it shows the real problems. To take you up on that with a dangerous example that reinforces the point: To add a disk to an array whose used space is less than the capacity of a single disk, you can (DON'T DO THIS, IT'S DANGEROUS) (1) remove a disk from the array (it's degraded now) (2) reformat the single disk, put all data on there (3) create a sparse file with the size of one disk (4) create a new array with N disks, in place of the temporary disk use the sparse file as a device (5) remove the sparse file (the new array is now degraded) (6) bring the new array up, copy all data from temp disk on it (7) replace the sparse file with the temp disk (8) copy files around to make them re-balance Congratulations, you've now grown your array, and put your data seriously at risk in the process! :)

jacknews 5y ago

The ONLY time I've actually lost data, was with BTRFS, and only using it as a basic single-volume filesystem, no RAID, snapshots, or anything complex. BTRFS does not seem to recover well from media errors. I've had plenty of those, and always been able to recover at least partial data (minus the bad media) with other FS's.

codys 5y ago
I've also recently had bad experiences where hard-power-offs of a fleet of systems with a btrfs filesystem repeatedly cause corruption (resulting in the btrfs filesystem no longer being mountable) Some testing seems to indicate even ext4 is more reliable in the face of hard power cycles.
labawi 5y ago
I've had funny experiences with disappearing files on ext4 - reading a file while there are transient SATA errors (on a flaky SATA link) would cause a the file to be removed. To be fair, I think I've lost more data on btrfs.
ok123456 5y ago
And the "rescue" commands tell you not to use them unless you've been blessed by someone on a mailing list, and that they can actually cause MORE damage. I was using BTRFS on my workstation and all of a sudden some garbage got written to the extent-tree, and it was toast. Never using BTRFS again. I had backups and was able to recover 99.99%, but the recuse tools were garbage. I finally killed it after it had spent a month, from July 4 until August 4, doing apparently nothing. And this was this year. This was a normal desktop configuration, single drive. I wasn't using any of the real features of the FS other than COW.
jbotz 5y ago
How long ago was this? My experience about a year ago was the opposite, btrfs held up really well on a failing drive and I managed to copy the filesystem with 'btrfs send' without suffering any data loss or corruption as far as I've been able to tell. I also had a btrfs filesystem on an external drive with a lose USB plug that caused it to disconnect once in a while, and didn't have any corruptions or loss on that one either.

DiabloD3 5y ago

Brtfs only exists as Oracle's "replacement" for ZFS, to hedge their bets in case they couldn't buy Sun. Oracle bought sun in 2010, that has been 12 years. They merged the Btrfs and ZFS teams, they then worked on ZFS entirely with no Btrfs work, and then basically reassigned, fired, or otherwise dissolved the ZFS team when it became clear that OpenZFS (Sun's chosen steward for ZFS) could not be swayed. Btrfs has had no real work done on it since then, and anyone that is actually serious about data storage has moved to ZFS, or has left POSIX filesystems entirely.

EastToWest 5y ago
> Btrfs has had no real work done on it since then, and anyone that is actually serious about data storage has moved to ZFS, or has left POSIX filesystems entirely. I'm not very familiar with this space, but if this were true why would Fedora use Btrfs by default? Also on Fedora wiki [0] -- Btrfs is a mature, well-understood, and battle-tested file system, used on both desktop/container and server/cloud use-cases -- which seems to contradict what you said. [0] https://fedoraproject.org/wiki/Changes/BtrfsByDefault https://fedoraproject.org/wiki/Changes/BtrfsByDefault
- naranha 5y ago
  Apparently it used at facebook a lot too. https://lwn.net/Articles/824855/ https://lwn.net/Articles/824855/
jcalvinowens 5y ago
This is one of the most laughably ignorant posts I've ever seen on the internet. > Btrfs has had no real work done on it since then, and anyone that is actually serious about data storage has moved to ZFS, or has left POSIX filesystems entirely Btrfs is the most actively developed filesystem in the kernel, both by LOC changed and the number of contributors. Facebook publicly stated that they run BTRFS on every webserver years ago. ZFS is an antiquated joke. It is wed to the memory allocator in Solaris, and requires that you permanently reserve 1GB of RAM per 1TB of disk space, which is an impractical constraint in the modern world. ZFS is also full of cute little wasteful "innovations" like 128-bit file size support. I don't know what planet you're from, but it's not mine :)

mhd 5y ago

Well, at least there's a chance. Now, ReiserFS 4...

pdmccormick 5y ago
His sentence will be served as of sometime in 2023.
kakd 5y ago
The development might resume in 2023

jl6 5y ago

Happy user of btrfs-raid1 for more than ten years here, for what it’s worth.

plainnoodles 5y ago
Seconded, though I think my array is perhaps only... 8ish years old?

COGlory 5y ago

>Moving beyond the question of individual disk reliability, btrfs-raid1 can only tolerate a single disk failure, no matter how large the total array is. The remaining copies of the blocks that were on a lost disk are distributed throughout the entire array—so losing any second disk loses you the array along with it. (This is in contrast to RAID10 arrays, which can survive any number of disk failures as long as no two are from the same mirror pair.) This feels a little disingenuous to me. This is a distinctly RAID-1 problem. This doesn't feel like a BTRFS problem. Everyone knows RAID-1 = 1 drive loss. >I don't have anything specific to say about btrfs-raid0, other than the fact that it's raid zero. Any failure of any disk loses all data on the array. This is not a storage system, it's a virtual woodchipper. Avoid. If the previous paragraph felt disingenuous, this one feels downright malicious. Not only does BTRFS RAID-0 have no differences from normal RAID-0, but the blanket recommendation to avoid it makes no sense. There are many use cases for RAID-0, but the author is acting like this is some kind of problem with BTRFS.

tw04 5y ago
>This feels a little disingenuous to me. This is a distinctly RAID-1 problem. This doesn't feel like a BTRFS problem. Everyone knows RAID-1 = 1 drive loss. That's simply not true. I can do a RAID-1 (mirror) with 3 drives on ZFS or mdadm and lose 2 drives. I can do it with 10 drives and lose 9. https://www.thegeekdiary.com/how-to-add-a-3rd-disk-to-create-a-3-way-mirror-raid1-md-device-centos-rhel-7/ https://www.thegeekdiary.com/how-to-add-a-3rd-disk-to-create... >There are many use cases for RAID-0, but the author is acting like this is some kind of problem with BTRFS. Sure, but for a novice the default advice needs to be: don't do that. And he's writing for folks with all sorts of levels of technical knowledge. Someone who knows nothing about RAID reading the article should be scared away unless they have a very, very good reason for doing RAID-0.
- nybble41 5y ago
  > I can do a RAID-1 (mirror) with 3 drives on ZFS or mdadm and lose 2 drives. You can do this with BTRFS also; it's called RAID1C3 (for three copies). RAID1C4 is also an option with larger arrays. The issue is that if you have, say, four drives and standard RAID-1 (two copies) it doesn't divide the drives into mirrored pairs where you could lose one drive from each pair without any ill effect (like RAID-10), but rather just ensures that each piece of data is stored on any two different drives. With 4-drive RAID-10 if you lose two drives you have a 50% chance of losing the whole array, since both failing drives could be from the same pair. With BTRFS RAID1 in a four-drive array you have a high probability of losing both copies of some of the data no matter which two drives fail. RAID-6 is strictly better than either, though, since you can lose any two drives and still access all your data.

derbOac 5y ago

Can someone recommend a good overview of modern filesystems that covers performance and licensing issues? I liked this article but was left with the question "so what else do you use?" It's been awhile (too long really) since I really tried to read up on this area. Last time I did I concluded that it was sort of a mess but lots of things were still in progress.

bluGill 5y ago
Use ZFS if you can. However for legal reasons it is hard to use on linux, which pushes you to FreeBSD (I'm not sure about the others). FreeBSD can replace linux, but you will find a number of places where the software you use assumes linux and so things don't work right. If you are doing NAS all on your own, then don't hesitate to run to FreeBSD. However if you want to a not do it yourself NAS it is a little harder. The most popular/polished commercial NAS is synology based on BTRFS. I personally use TrueNAS which is a great file server, but synology has some nice integration to photo and media applications and so I'm thinking about switching.
- bluGill 5y ago
  Too late to edit, but I just discovered the synology isn't use BRTF for redundancy, they are somehow using linux device manager in a way that I don't understand. Since I don't understand what they are doing I don't know how it compares to ZFS.

deleted 5y ago

[deleted]

mrlonglong 5y ago

I would definitely look again at btrfs once it gains raid5/6 capability. Currently happily using ZFS. The only issue is not being able to use latest kernel releases as the ZFS folks usually don't release updates for a while, but overall it doesn't really bother me much nowadays.

dosman33 5y ago

Yet another knock-off of Lustre. Lustre is not perfect, and I'm not suggesting it has a place in the mainline kernel, but it sounds like neither does BTRFS. When you get to large scale (multi-petabyte) filesystems there are not many options, it's GPFS, Lustre, or roll your own. Lustre development has been indirectly sponsored by the DoE for a long time now. And if Lustre didn't exist IBM would charge even more for GPFS. But at least Lustre is free. But it's not a walk in the park to use.

guenthert 5y ago
Uh, BTRFS and Lustre can't easily be compared. The former is exclusively a single-host fs, the latter is a cluster-only fs. The former focuses (more or less successfully) on data resilience and availability, the latter on performance. There are quite a few large scale distributed file systems. XtreemFS, Hadoop and Ceph come to mind. There are others, e.g. OrangeFS (successor of PVFS), PanFS, etc. .

anuraaga 5y ago

How to kill your array - reboot your computer at a bad time. Who'd have thought raid1 could be so unreliable. https://btrfs.wiki.kernel.org/index.php/Gotchas#raid1_volumes_only_mountable_once_RW_if_degraded https://btrfs.wiki.kernel.org/index.php/Gotchas#raid1_volume... I did end up having to copy the array out because of this. Copied it into a zfs array and am glad to be out of the land of gotchas.

flas9sd 5y ago

I understand the audience of the ars article is people running larger setups with redundant disk arrays, managing lots of data reliably. But when it comes to what you use in a laptop or desktop, with maybe one more slot for a ssd/nvme or an external disk, btrfs snapshots with incremental send+receive to the second disk or over the network is a good argument in favor of giving it a try.

kzrdude 5y ago

I guess it's unfortunate for btrfs that it has so much "raid" functionality. Otherwise it could be known for its useful other functionality - snapshots, subvolumes, checksums, compression!