16 ms·

So is btrfs dead?

by grigio 10y ago

So is btrfs dead?

eatonphil 10y ago
btrfs is immature.
- krondor 10y ago
  btrfs is stable as of kernel 3.14 or so. The features in btrfs that are unstable at this point are raid56 (write hole on power failure risk), autodefrag (and mainly just with high transaction files like vm images and databases), and ext34/in place filesystem conversion. We've been running btrfs in production since Ubuntu 14.04 with excellent results. The feature set vastly outweighs the few risks that remain.
  - DominoTree 10y ago
    "stable" != "mature"
    - krondor 10y ago
      That's a fair point. I guess we can debate what 'mature' means. In development since 2007, stable since 2014. ZFS, in development since 2001 (correction I erroneously listed 2005 earlier), stable since 2006 (or at least Solaris included since then). Do you consider the Solaris years or just the Linux years and then do you consider the Linux Debian/Ubuntu sanctioned years or the ZoL years? I'm fine with mature since included in the default installer on Ubuntu/Redhat/Oracle/SUSE/etc... for my definition of btrfs maturity.
      - eatonphil 10y ago
        I think you've confused your dates. ZFS has been in development since 2001. It was first introduced in 2005.
        krondor 10y ago
        Yes, thanks for catching that, I've edited the parent. As you can tell I'm not that strong on ZFS other than what I've learned in some small tinkering and discussion with others.
  - deleted 10y ago
    [deleted]
  - XorNot 10y ago
    The problem is "in production" you can do a lot of things which ironically don't work for smaller users, because "in production" you specifically optimize and assume hardware and applications will crash in catastrophic ways pretty much all the time and you code quite differently (i.e. it is totally reasonable to ask applications to be distributed and deal with it).
    - krondor 10y ago
      Agreed, my workload is probably not representative of a small organization or user. I absolutely leverage resiliency in software, distributed computing, and mitigate single points of failure, etc... This is a good point. Now, I do think these methods are increasingly approachable for all users, however. A lot of that is actually enabled by the feature sets of ZFS and btrfs. The default Ubuntu installer, for example, will create snapshots during OS upgrade for rollback if the upgrade fails. ZFS and btrfs send/receive feature allows for efficient DR clones (not to mention seed images and snapshots). LXD leverages ZFS if you choose to for rapid containerization and snapshots. These intrinsic abilities of these two filesystems allow for smaller users and organizations to improve those workflows to be more risk averse in general (even while assuming some risk in a newer FS). Just, if possible, be up to date on the known issues and run more recent kernels and userland utilities. https://btrfs.wiki.kernel.org/index.php/Gotchas https://btrfs.wiki.kernel.org/index.php/Gotchas
    - deleted 10y ago
      [deleted]
  - bogomipz 10y ago
    It seems odd that after all this time that btrfs wouldn't be able to handle database workloads. If it can't handle hight I/O it doens't have much use except for a bootdisk. Do you have any reading you could suggest on this issue?
    - zanny 10y ago
      btrfs is only particularly slow on database files if they are not marked for inplace writing because the filesystem is default copy on write, which is horribly slow for huge constantly changing files (they end up highly fragmented across data blocks if you don't turn off cow).
    - krondor 10y ago
      As zanny said, cow presents an issue for transactional workloads (databases, vm images, etc...) as each write fragments the file, by nature of the process of cow. btrfs can handle database workloads but you have to disable cow for them (which you can do at the file, directory, or subvolume level in btrfs). You would specify the nodatacow mount option, or chattr +C (file/directory). The btrfs autodefrag is still rather new, and needs some work. I expect that could be the long term fix (manual defrag is fine now, but you wouldn't want to call it frequently on a db file), I'm not sure how ZFS handles the fragmentation, but I do know in the past ZFS observed similar issues (seems to have been mostly resolved). I should also point out that disabling cow doesn't really fully disable it, snapshots can still function, etc... however, I'm sure that once you start to use the other cow functions you might observe slipping performance due to fragmentation of these types of files. https://blog.pgaddict.com/posts/friends-dont-let-friends-use-btrfs-for-oltp https://blog.pgaddict.com/posts/friends-dont-let-friends-use... https://bartsjerps.wordpress.com/2013/02/26/zfs-ora-database-fragmentation/ https://bartsjerps.wordpress.com/2013/02/26/zfs-ora-database...
  - jamesblonde 10y ago
    You forgot about how sensitive it is to low disk space. Don't go above 80% in production is what I've been told - that's a lot of wasted disk space.
    - krondor 10y ago
      You are correct that btrfs and ZFS are sensitive to low disk space. This is also related to how cow filesystems function, they need that free space to commit writes and for snapshots because they, by nature, don't overwrite blocks. See this for the ZFS example; http://serverfault.com/a/556892/79238 http://serverfault.com/a/556892/79238 In both cases the exact amount of free space you desire is a mix of workload, fragmentation, and snapshots. However, I disagree they waste a lot of space. I think both ZFS and btrfs more than make up for the overhead of free space commits through their space saving features. cp --reflink, block suballocation, compresssion, and efficient snapshots outweight the overhead, in my case. Your mileage may vary.
      - cm3 10y ago
        So, isn't there a way to set aside a buffer space so that you don't run into ENOSPACE problems? Maybe like ext's 5% reserve.
        krondor 10y ago
        I believe you're misunderstanding the problems that occur on a cow filesystem. In fact btrfs already has an overcommit disk buffer, and is already doing many ENOSPACE handling tricks. Have a look; https://btrfs.wiki.kernel.org/index.php/ENOSPC https://btrfs.wiki.kernel.org/index.php/ENOSPC Reading more for my own interest it seams ZFS uses ZIL to help convert random writes to sequential, which helps with fragmentation under low space. I am curious if bcache, can operate similarly. In addition, I should also point out that this is less of an issue on SSD, due to the nature of how random reads/writes work there anyway (btrfs does a good job of being SSD aware).
        Elhana 10y ago
        As I understand it in btrfs case there is two problems: 1) Metadata in btrfs can use lots of space, especially when you convert from ext4. It might happen that you have gigabytes of free space reserved for metadata, so it can not be used anymore. This can be solved with rebalancing, but that can take ages, which is actually one of the reasons zfs doesn't have bpr rewrite feature. 2) btrfs can have mixed raid levels and in that scenario calculating free space is tricky, but people still rely on common tools, that simply give some estimates in that case. Change the way it estimates free space and you'll have less clueless people complaining about fs running out of it, but more will say btrfs shows too little.
  - EyesBlue1988 10y ago
    btrfs is not stable!, xfs is stable
- zanny 10y ago
  ext4 = started in 2006, stable in 2008, universal on all distros by 2010 btrfs = started in 2007, stable in 2014, mature never? There is no magical point in time where a file system is suddenly "mature". If anything I'd argue once OpenSUSE started defaulting to it one should consider it stable. I just wonder why Ubuntu / Fedora are not prioritizing making it default since snapshotted upgrades are such a huge usability gain and are trivially easy with btrfs.
  - _asummers 10y ago
    Why the 7 year gap for stabilization? Especially after ext4 only took 2. Jusding from the timelines, btrfs is much more complicated and I'd guess more Linux users exposed more corner cases, but a 3.5x lengthening is substantial.
    - lallysingh 10y ago
      AFAICT, btrfs is a while new system, while ext4 is a set of changes to an already tested coffee base.
      - lallysingh 10y ago
        *code base. Friggin' autocomplete.
  - eatonphil 10y ago
    Meanwhile, just for reference, ZFS started in 2001. Introduced in Solaris in 2005 and FreeBSD in 2007. I'm not necessarily trying to make a point here, just adding some related facts.
  - cm3 10y ago
    ext4 isn't a rewrite and magnitudes smaller in code size and complexity, so the comparison isn't really fair.
  - rcthompson 10y ago
    I don't know if it's fair to say that ext4 "started" in 2006, given that it was an evolution of ext3, which was an evolution of ext2 (which I assume was an evolution of ext?), to the point that you can still mount an ext2 filesystem as a very feature-limited ext4 filesystem.
ThatPlayer 10y ago
Btrfs is a different use-case. ZFS doesn't support different sized hard drives. I could also be wrong, but ZFS doesn't expand as easily, and you can't remove drives from it. Synology NAS just added support for Btrfs in their software
- stewartbutler 10y ago
  ZFS does support different sized drives, and you can add disks to a pool easily. It does not support different sized drives in a raidz (equivalent to an mdadm array), as far as I know, but you can mix raidz and singular drives in a pool, though that is a bit silly. It also does not support adding disks to a raidz, which is equivalent to growing a raid5 or raid6 array, which is admittedly a pain in the ass.
  - mbreese 10y ago
    You can also upgrade an entire raidz vdev if you replace each drive in the vdev with a larger one (one at a time) and enable "autoexpand" on the zpool. It's a manual and time consuming process, waiting for each drive to resilver. Still an pain, but at least possible.
  - Annatar 10y ago
    > It does not support different sized drives in a raidz % mkfile -v 64m /var/tmp/d0 /var/tmp/d0 67108864 bytes % mkfile -v 96m /var/tmp/d1 /var/tmp/d1 100663296 bytes % mkfile -v 128m /var/tmp/d2 /var/tmp/d2 134217728 bytes % sudo zpool create -f pool0 raidz /var/tmp/d0 /var/tmp/d1 /var/tmp/d2 Password: % zpool status pool: pool0 state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM pool0 ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 /var/tmp/d0 ONLINE 0 0 0 /var/tmp/d1 ONLINE 0 0 0 /var/tmp/d2 ONLINE 0 0 0 errors: No known data errors
    - stewartbutler 10y ago
      You can build it, but it is limited to the smallest device in the vdev, so the array will appear to only have three devices of 64m available. Though you can replace all smaller disks and grow it.
the8472 10y ago
btrfs supports a bunch of features that ZoL does not which are useful especially on smaller systems. among others: the ability to resize/change raid layout in place, reflink copies, on-demand deduplication.
- Dylan16807 10y ago
  And it does the reflink copies and on-demand deduplication with minimal overhead. ZFS handles snapshots fine but its deduplication can cripple speeds, even when reading. Maybe some day it'll get block pointer rewrite, but for now it can be a big problem, depending on the type of data you have.
- jamesblonde 10y ago
  But btrfs RAID-5 still suffers from the 'write hole' problem where data can be permanently lost. We looked at RAID-5 performance for both ZFS and btrfs, and btrfs is about twice the throughput. One of the reasons for the performance difference is the following: when ZFS writes a stripe, it waits for all blocks and the parity block to confirm they have been written before atomically updating the metadata for that stripe; when btrfs writes a stripe, it just updates the metadata and all blocks in parallel. Hence if btrfs crashes, metadata may say a stripe is written but not all blocks are written. It will be interesting to see the performance hit on btrfs when they finally fix the write hole problem. My prediction is that it will perform similarly to ZFS for RAID5. Currently, for writes, we can get about 400MB/s for btrfs RAID5, but only 220MB/s for ZFS with five 5400 rpm 2.5" disks.
  - poizan42 10y ago
    That should be exposed as an option so the user can choose whether they need performance or reliability.
  - krondor 10y ago
    Interesting findings. I'm curious also to see how fixed width stripes change the performance profile in btrfs (presently not supported). To be clear though, the raid5/6 write hole only applies to power failures with data in flight. It is still a concern, of course, but I think depending on your environment is acceptable to some (redundant PSUs, well engineered PDUs and UPS systems). Personally, I'm of increasing opinion that parity rebuilds aren't worth it anyway. I'd rather raid10, raid1, or raid0 depending on use. If I have to take a system out of production during parity rebuild (because IO activity is too intense for performant use), might as well not parity rebuild and simply reload the system on failure and rely on other cluster nodes.
krondor 10y ago
btrfs is most certainly not dead. I actually think it's really gaining traction presently. It only became stable near Ubuntu 14.04 or so, and it takes people awhile (understandably) to warm up to a new filesystem. It's great to see ZFS on Linux get a more stable footing. It's an excellent filesystem. As others have said I think the use case differs slightly from btrfs (though they are very similar in capabilities). ZFS, to my eyes, seems more resilient. It has more levels of data checksums, The RAIDz model allows for more redundancy, and it just feels like a stronger enterprise offering (meaning stable and built for large systems and disk quantities). btrfs brings many of the ZFS features to Linux in a GPL wrapping. What it lacks in resiliency, it makes up for with flexibility. Raid in btrfs, for instance, occurs within data chunks across disks, not at the disk level, meaning mixed disk capacities, and on the fly raid changes. I also appreciate the way it divides namespace across subvolumes while maintaining block awareness within the pool (cp --reflink across subvolumes, snapshots across subvolumes). It also doesn't have the ram requirements of ZFS (which aren't much of a data center concern, but are definitely a client level concern for workstations). Either way it's a win, both great filesystems for Linux. With bcache supporting btrfs properly now, I personally don't have much of a reason for ZFS now. Two years ago I would have jumped easily to it. Your workloads and needs may differ, it's great to have choices!
ansible 10y ago
No, I don't think so. I'm a small-time admin, and I think that btrfs is working pretty good these days. We have recently switched all our servers to LXC containers so that we can take full advantage of btrfs features. I doubt I need to explain the advantages of containers to anyone here... but in short we've broken out all the network services (file service, LDAP, DNS, etc.) to separate containers. Each container is in a separate btrfs subvolume. This allows us to take snapshots of the running systems every 10 minutes, and using btrfs send/receive, cheaply back up those snapshots to alternate container hosts. The send/receive stuff works better with the btrfs v4.4 tools that ship with Ubuntu 16.04. Since the network interfaces for all the containers are bridged with the container host, we can configure each container with its own static IP address. So if a container host fails, those containers can be booted up on the alternate host, and keep their IP and MAC addresses. So that's convenient, and causes minimal disruption. The main improvement I'd like to see with btrfs is a configurable RAID redundancy level. Currently, RAID-1 means that there are two copies of each piece of data / metadata. So in a 3-drive RAID-1 system that gives you extra capacity, but two drives failing at the same time will cause data loss.
- cmurf 10y ago
  Being lazy, I'm going to ask if you know of some LXC + Btrfs pros in contrast to Docker + Btrfs? Right now a gotcha with Docker + Btrfs is that SELinux contexts for each container can be different, but the context= mount option currently is once per superblock (thus per fs volume, rather than per fs tree or subvolume). So Docker's work around in 1.10.x is they do a snapshot and then relabel it with the new selinux context then start the container. For my containers (very basic) this adds an almost imperceptible one time delay for that container. I doubt it's even 1 second, which in container start times might seem massive to some.
  - ansible 10y ago
    If you have already adapted to Docker, then I'm sure you don't want to use LXC. Containers are long-lived and mutable. I treat them like old school servers, just not tied to physical hardware.
- deleted 10y ago
zxcvcxz 10y ago
It's the default file system in OpenSUSE.
cmurf 10y ago
Not even close. Each merge window sees many dozens of bug fixes and enhancements, thousands of line additions and deletions. Development is very active. There are even a couple developers who are getting ants in the pants about focusing more on stabilization. The drawback of stabilization is that it increases the burden of adding new features, because any new features could reduce stability. So it's a balancing act.
leonroy 10y ago
BTRFS has much lower memory requirements than ZFS which needs at least 16GB RAM for anything performant. Also for NAS boxes expecting the user to compile the ZFS libs from source wouldn't be optimal.

So is btrfs dead?

by grigio 10y ago

So is btrfs dead?

eatonphil 10y ago

btrfs is immature.

krondor 10y ago
btrfs is stable as of kernel 3.14 or so. The features in btrfs that are unstable at this point are raid56 (write hole on power failure risk), autodefrag (and mainly just with high transaction files like vm images and databases), and ext34/in place filesystem conversion. We've been running btrfs in production since Ubuntu 14.04 with excellent results. The feature set vastly outweighs the few risks that remain.
- DominoTree 10y ago
  "stable" != "mature"
  - krondor 10y ago
    That's a fair point. I guess we can debate what 'mature' means. In development since 2007, stable since 2014. ZFS, in development since 2001 (correction I erroneously listed 2005 earlier), stable since 2006 (or at least Solaris included since then). Do you consider the Solaris years or just the Linux years and then do you consider the Linux Debian/Ubuntu sanctioned years or the ZoL years? I'm fine with mature since included in the default installer on Ubuntu/Redhat/Oracle/SUSE/etc... for my definition of btrfs maturity.
    - eatonphil 10y ago
      I think you've confused your dates. ZFS has been in development since 2001. It was first introduced in 2005.
      - krondor 10y ago
        Yes, thanks for catching that, I've edited the parent. As you can tell I'm not that strong on ZFS other than what I've learned in some small tinkering and discussion with others.
- deleted 10y ago
  [deleted]
- XorNot 10y ago
  The problem is "in production" you can do a lot of things which ironically don't work for smaller users, because "in production" you specifically optimize and assume hardware and applications will crash in catastrophic ways pretty much all the time and you code quite differently (i.e. it is totally reasonable to ask applications to be distributed and deal with it).
  - krondor 10y ago
    Agreed, my workload is probably not representative of a small organization or user. I absolutely leverage resiliency in software, distributed computing, and mitigate single points of failure, etc... This is a good point. Now, I do think these methods are increasingly approachable for all users, however. A lot of that is actually enabled by the feature sets of ZFS and btrfs. The default Ubuntu installer, for example, will create snapshots during OS upgrade for rollback if the upgrade fails. ZFS and btrfs send/receive feature allows for efficient DR clones (not to mention seed images and snapshots). LXD leverages ZFS if you choose to for rapid containerization and snapshots. These intrinsic abilities of these two filesystems allow for smaller users and organizations to improve those workflows to be more risk averse in general (even while assuming some risk in a newer FS). Just, if possible, be up to date on the known issues and run more recent kernels and userland utilities. https://btrfs.wiki.kernel.org/index.php/Gotchas https://btrfs.wiki.kernel.org/index.php/Gotchas
  - deleted 10y ago
    [deleted]
- bogomipz 10y ago
  It seems odd that after all this time that btrfs wouldn't be able to handle database workloads. If it can't handle hight I/O it doens't have much use except for a bootdisk. Do you have any reading you could suggest on this issue?
  - zanny 10y ago
    btrfs is only particularly slow on database files if they are not marked for inplace writing because the filesystem is default copy on write, which is horribly slow for huge constantly changing files (they end up highly fragmented across data blocks if you don't turn off cow).
  - krondor 10y ago
    As zanny said, cow presents an issue for transactional workloads (databases, vm images, etc...) as each write fragments the file, by nature of the process of cow. btrfs can handle database workloads but you have to disable cow for them (which you can do at the file, directory, or subvolume level in btrfs). You would specify the nodatacow mount option, or chattr +C (file/directory). The btrfs autodefrag is still rather new, and needs some work. I expect that could be the long term fix (manual defrag is fine now, but you wouldn't want to call it frequently on a db file), I'm not sure how ZFS handles the fragmentation, but I do know in the past ZFS observed similar issues (seems to have been mostly resolved). I should also point out that disabling cow doesn't really fully disable it, snapshots can still function, etc... however, I'm sure that once you start to use the other cow functions you might observe slipping performance due to fragmentation of these types of files. https://blog.pgaddict.com/posts/friends-dont-let-friends-use-btrfs-for-oltp https://blog.pgaddict.com/posts/friends-dont-let-friends-use... https://bartsjerps.wordpress.com/2013/02/26/zfs-ora-database-fragmentation/ https://bartsjerps.wordpress.com/2013/02/26/zfs-ora-database...
- jamesblonde 10y ago
  You forgot about how sensitive it is to low disk space. Don't go above 80% in production is what I've been told - that's a lot of wasted disk space.
  - krondor 10y ago
    You are correct that btrfs and ZFS are sensitive to low disk space. This is also related to how cow filesystems function, they need that free space to commit writes and for snapshots because they, by nature, don't overwrite blocks. See this for the ZFS example; http://serverfault.com/a/556892/79238 http://serverfault.com/a/556892/79238 In both cases the exact amount of free space you desire is a mix of workload, fragmentation, and snapshots. However, I disagree they waste a lot of space. I think both ZFS and btrfs more than make up for the overhead of free space commits through their space saving features. cp --reflink, block suballocation, compresssion, and efficient snapshots outweight the overhead, in my case. Your mileage may vary.
    - cm3 10y ago
      So, isn't there a way to set aside a buffer space so that you don't run into ENOSPACE problems? Maybe like ext's 5% reserve.
      - krondor 10y ago
        I believe you're misunderstanding the problems that occur on a cow filesystem. In fact btrfs already has an overcommit disk buffer, and is already doing many ENOSPACE handling tricks. Have a look; https://btrfs.wiki.kernel.org/index.php/ENOSPC https://btrfs.wiki.kernel.org/index.php/ENOSPC Reading more for my own interest it seams ZFS uses ZIL to help convert random writes to sequential, which helps with fragmentation under low space. I am curious if bcache, can operate similarly. In addition, I should also point out that this is less of an issue on SSD, due to the nature of how random reads/writes work there anyway (btrfs does a good job of being SSD aware).
      - Elhana 10y ago
        As I understand it in btrfs case there is two problems: 1) Metadata in btrfs can use lots of space, especially when you convert from ext4. It might happen that you have gigabytes of free space reserved for metadata, so it can not be used anymore. This can be solved with rebalancing, but that can take ages, which is actually one of the reasons zfs doesn't have bpr rewrite feature. 2) btrfs can have mixed raid levels and in that scenario calculating free space is tricky, but people still rely on common tools, that simply give some estimates in that case. Change the way it estimates free space and you'll have less clueless people complaining about fs running out of it, but more will say btrfs shows too little.
- EyesBlue1988 10y ago
  btrfs is not stable!, xfs is stable
zanny 10y ago
ext4 = started in 2006, stable in 2008, universal on all distros by 2010 btrfs = started in 2007, stable in 2014, mature never? There is no magical point in time where a file system is suddenly "mature". If anything I'd argue once OpenSUSE started defaulting to it one should consider it stable. I just wonder why Ubuntu / Fedora are not prioritizing making it default since snapshotted upgrades are such a huge usability gain and are trivially easy with btrfs.
- _asummers 10y ago
  Why the 7 year gap for stabilization? Especially after ext4 only took 2. Jusding from the timelines, btrfs is much more complicated and I'd guess more Linux users exposed more corner cases, but a 3.5x lengthening is substantial.
  - lallysingh 10y ago
    AFAICT, btrfs is a while new system, while ext4 is a set of changes to an already tested coffee base.
    - lallysingh 10y ago
      *code base. Friggin' autocomplete.
- eatonphil 10y ago
  Meanwhile, just for reference, ZFS started in 2001. Introduced in Solaris in 2005 and FreeBSD in 2007. I'm not necessarily trying to make a point here, just adding some related facts.
- cm3 10y ago
  ext4 isn't a rewrite and magnitudes smaller in code size and complexity, so the comparison isn't really fair.
- rcthompson 10y ago
  I don't know if it's fair to say that ext4 "started" in 2006, given that it was an evolution of ext3, which was an evolution of ext2 (which I assume was an evolution of ext?), to the point that you can still mount an ext2 filesystem as a very feature-limited ext4 filesystem.

ThatPlayer 10y ago

Btrfs is a different use-case. ZFS doesn't support different sized hard drives. I could also be wrong, but ZFS doesn't expand as easily, and you can't remove drives from it. Synology NAS just added support for Btrfs in their software

stewartbutler 10y ago
ZFS does support different sized drives, and you can add disks to a pool easily. It does not support different sized drives in a raidz (equivalent to an mdadm array), as far as I know, but you can mix raidz and singular drives in a pool, though that is a bit silly. It also does not support adding disks to a raidz, which is equivalent to growing a raid5 or raid6 array, which is admittedly a pain in the ass.
- mbreese 10y ago
  You can also upgrade an entire raidz vdev if you replace each drive in the vdev with a larger one (one at a time) and enable "autoexpand" on the zpool. It's a manual and time consuming process, waiting for each drive to resilver. Still an pain, but at least possible.
- Annatar 10y ago
  > It does not support different sized drives in a raidz % mkfile -v 64m /var/tmp/d0 /var/tmp/d0 67108864 bytes % mkfile -v 96m /var/tmp/d1 /var/tmp/d1 100663296 bytes % mkfile -v 128m /var/tmp/d2 /var/tmp/d2 134217728 bytes % sudo zpool create -f pool0 raidz /var/tmp/d0 /var/tmp/d1 /var/tmp/d2 Password: % zpool status pool: pool0 state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM pool0 ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 /var/tmp/d0 ONLINE 0 0 0 /var/tmp/d1 ONLINE 0 0 0 /var/tmp/d2 ONLINE 0 0 0 errors: No known data errors
  - stewartbutler 10y ago
    You can build it, but it is limited to the smallest device in the vdev, so the array will appear to only have three devices of 64m available. Though you can replace all smaller disks and grow it.

the8472 10y ago

btrfs supports a bunch of features that ZoL does not which are useful especially on smaller systems. among others: the ability to resize/change raid layout in place, reflink copies, on-demand deduplication.

Dylan16807 10y ago
And it does the reflink copies and on-demand deduplication with minimal overhead. ZFS handles snapshots fine but its deduplication can cripple speeds, even when reading. Maybe some day it'll get block pointer rewrite, but for now it can be a big problem, depending on the type of data you have.
jamesblonde 10y ago
But btrfs RAID-5 still suffers from the 'write hole' problem where data can be permanently lost. We looked at RAID-5 performance for both ZFS and btrfs, and btrfs is about twice the throughput. One of the reasons for the performance difference is the following: when ZFS writes a stripe, it waits for all blocks and the parity block to confirm they have been written before atomically updating the metadata for that stripe; when btrfs writes a stripe, it just updates the metadata and all blocks in parallel. Hence if btrfs crashes, metadata may say a stripe is written but not all blocks are written. It will be interesting to see the performance hit on btrfs when they finally fix the write hole problem. My prediction is that it will perform similarly to ZFS for RAID5. Currently, for writes, we can get about 400MB/s for btrfs RAID5, but only 220MB/s for ZFS with five 5400 rpm 2.5" disks.
- poizan42 10y ago
  That should be exposed as an option so the user can choose whether they need performance or reliability.
- krondor 10y ago
  Interesting findings. I'm curious also to see how fixed width stripes change the performance profile in btrfs (presently not supported). To be clear though, the raid5/6 write hole only applies to power failures with data in flight. It is still a concern, of course, but I think depending on your environment is acceptable to some (redundant PSUs, well engineered PDUs and UPS systems). Personally, I'm of increasing opinion that parity rebuilds aren't worth it anyway. I'd rather raid10, raid1, or raid0 depending on use. If I have to take a system out of production during parity rebuild (because IO activity is too intense for performant use), might as well not parity rebuild and simply reload the system on failure and rely on other cluster nodes.

krondor 10y ago

btrfs is most certainly not dead. I actually think it's really gaining traction presently. It only became stable near Ubuntu 14.04 or so, and it takes people awhile (understandably) to warm up to a new filesystem. It's great to see ZFS on Linux get a more stable footing. It's an excellent filesystem. As others have said I think the use case differs slightly from btrfs (though they are very similar in capabilities). ZFS, to my eyes, seems more resilient. It has more levels of data checksums, The RAIDz model allows for more redundancy, and it just feels like a stronger enterprise offering (meaning stable and built for large systems and disk quantities). btrfs brings many of the ZFS features to Linux in a GPL wrapping. What it lacks in resiliency, it makes up for with flexibility. Raid in btrfs, for instance, occurs within data chunks across disks, not at the disk level, meaning mixed disk capacities, and on the fly raid changes. I also appreciate the way it divides namespace across subvolumes while maintaining block awareness within the pool (cp --reflink across subvolumes, snapshots across subvolumes). It also doesn't have the ram requirements of ZFS (which aren't much of a data center concern, but are definitely a client level concern for workstations). Either way it's a win, both great filesystems for Linux. With bcache supporting btrfs properly now, I personally don't have much of a reason for ZFS now. Two years ago I would have jumped easily to it. Your workloads and needs may differ, it's great to have choices!

ansible 10y ago

No, I don't think so. I'm a small-time admin, and I think that btrfs is working pretty good these days. We have recently switched all our servers to LXC containers so that we can take full advantage of btrfs features. I doubt I need to explain the advantages of containers to anyone here... but in short we've broken out all the network services (file service, LDAP, DNS, etc.) to separate containers. Each container is in a separate btrfs subvolume. This allows us to take snapshots of the running systems every 10 minutes, and using btrfs send/receive, cheaply back up those snapshots to alternate container hosts. The send/receive stuff works better with the btrfs v4.4 tools that ship with Ubuntu 16.04. Since the network interfaces for all the containers are bridged with the container host, we can configure each container with its own static IP address. So if a container host fails, those containers can be booted up on the alternate host, and keep their IP and MAC addresses. So that's convenient, and causes minimal disruption. The main improvement I'd like to see with btrfs is a configurable RAID redundancy level. Currently, RAID-1 means that there are two copies of each piece of data / metadata. So in a 3-drive RAID-1 system that gives you extra capacity, but two drives failing at the same time will cause data loss.

cmurf 10y ago
Being lazy, I'm going to ask if you know of some LXC + Btrfs pros in contrast to Docker + Btrfs? Right now a gotcha with Docker + Btrfs is that SELinux contexts for each container can be different, but the context= mount option currently is once per superblock (thus per fs volume, rather than per fs tree or subvolume). So Docker's work around in 1.10.x is they do a snapshot and then relabel it with the new selinux context then start the container. For my containers (very basic) this adds an almost imperceptible one time delay for that container. I doubt it's even 1 second, which in container start times might seem massive to some.
- ansible 10y ago
  If you have already adapted to Docker, then I'm sure you don't want to use LXC. Containers are long-lived and mutable. I treat them like old school servers, just not tied to physical hardware.
deleted 10y ago

zxcvcxz 10y ago

It's the default file system in OpenSUSE.

cmurf 10y ago

Not even close. Each merge window sees many dozens of bug fixes and enhancements, thousands of line additions and deletions. Development is very active. There are even a couple developers who are getting ants in the pants about focusing more on stabilization. The drawback of stabilization is that it increases the burden of adding new features, because any new features could reduce stability. So it's a balancing act.

leonroy 10y ago

BTRFS has much lower memory requirements than ZFS which needs at least 16GB RAM for anything performant. Also for NAS boxes expecting the user to compile the ZFS libs from source wouldn't be optimal.