9 ms·

btrfs is stable as of kernel 3.14 or so. The features in btrfs that are unstable at this point are raid56 (write hole on power failure risk), autodefrag (and m

by krondor 10y ago

btrfs is stable as of kernel 3.14 or so. The features in btrfs that are unstable at this point are raid56 (write hole on power failure risk), autodefrag (and mainly just with high transaction files like vm images and databases), and ext34/in place filesystem conversion. We've been running btrfs in production since Ubuntu 14.04 with excellent results. The feature set vastly outweighs the few risks that remain.

DominoTree 10y ago
"stable" != "mature"
- krondor 10y ago
  That's a fair point. I guess we can debate what 'mature' means. In development since 2007, stable since 2014. ZFS, in development since 2001 (correction I erroneously listed 2005 earlier), stable since 2006 (or at least Solaris included since then). Do you consider the Solaris years or just the Linux years and then do you consider the Linux Debian/Ubuntu sanctioned years or the ZoL years? I'm fine with mature since included in the default installer on Ubuntu/Redhat/Oracle/SUSE/etc... for my definition of btrfs maturity.
  - eatonphil 10y ago
    I think you've confused your dates. ZFS has been in development since 2001. It was first introduced in 2005.
    - krondor 10y ago
      Yes, thanks for catching that, I've edited the parent. As you can tell I'm not that strong on ZFS other than what I've learned in some small tinkering and discussion with others.
deleted 10y ago
[deleted]
XorNot 10y ago
The problem is "in production" you can do a lot of things which ironically don't work for smaller users, because "in production" you specifically optimize and assume hardware and applications will crash in catastrophic ways pretty much all the time and you code quite differently (i.e. it is totally reasonable to ask applications to be distributed and deal with it).
- krondor 10y ago
  Agreed, my workload is probably not representative of a small organization or user. I absolutely leverage resiliency in software, distributed computing, and mitigate single points of failure, etc... This is a good point. Now, I do think these methods are increasingly approachable for all users, however. A lot of that is actually enabled by the feature sets of ZFS and btrfs. The default Ubuntu installer, for example, will create snapshots during OS upgrade for rollback if the upgrade fails. ZFS and btrfs send/receive feature allows for efficient DR clones (not to mention seed images and snapshots). LXD leverages ZFS if you choose to for rapid containerization and snapshots. These intrinsic abilities of these two filesystems allow for smaller users and organizations to improve those workflows to be more risk averse in general (even while assuming some risk in a newer FS). Just, if possible, be up to date on the known issues and run more recent kernels and userland utilities. https://btrfs.wiki.kernel.org/index.php/Gotchas https://btrfs.wiki.kernel.org/index.php/Gotchas
- deleted 10y ago
  [deleted]
bogomipz 10y ago
It seems odd that after all this time that btrfs wouldn't be able to handle database workloads. If it can't handle hight I/O it doens't have much use except for a bootdisk. Do you have any reading you could suggest on this issue?
- zanny 10y ago
  btrfs is only particularly slow on database files if they are not marked for inplace writing because the filesystem is default copy on write, which is horribly slow for huge constantly changing files (they end up highly fragmented across data blocks if you don't turn off cow).
- krondor 10y ago
  As zanny said, cow presents an issue for transactional workloads (databases, vm images, etc...) as each write fragments the file, by nature of the process of cow. btrfs can handle database workloads but you have to disable cow for them (which you can do at the file, directory, or subvolume level in btrfs). You would specify the nodatacow mount option, or chattr +C (file/directory). The btrfs autodefrag is still rather new, and needs some work. I expect that could be the long term fix (manual defrag is fine now, but you wouldn't want to call it frequently on a db file), I'm not sure how ZFS handles the fragmentation, but I do know in the past ZFS observed similar issues (seems to have been mostly resolved). I should also point out that disabling cow doesn't really fully disable it, snapshots can still function, etc... however, I'm sure that once you start to use the other cow functions you might observe slipping performance due to fragmentation of these types of files. https://blog.pgaddict.com/posts/friends-dont-let-friends-use-btrfs-for-oltp https://blog.pgaddict.com/posts/friends-dont-let-friends-use... https://bartsjerps.wordpress.com/2013/02/26/zfs-ora-database-fragmentation/ https://bartsjerps.wordpress.com/2013/02/26/zfs-ora-database...
jamesblonde 10y ago
You forgot about how sensitive it is to low disk space. Don't go above 80% in production is what I've been told - that's a lot of wasted disk space.
- krondor 10y ago
  You are correct that btrfs and ZFS are sensitive to low disk space. This is also related to how cow filesystems function, they need that free space to commit writes and for snapshots because they, by nature, don't overwrite blocks. See this for the ZFS example; http://serverfault.com/a/556892/79238 http://serverfault.com/a/556892/79238 In both cases the exact amount of free space you desire is a mix of workload, fragmentation, and snapshots. However, I disagree they waste a lot of space. I think both ZFS and btrfs more than make up for the overhead of free space commits through their space saving features. cp --reflink, block suballocation, compresssion, and efficient snapshots outweight the overhead, in my case. Your mileage may vary.
  - cm3 10y ago
    So, isn't there a way to set aside a buffer space so that you don't run into ENOSPACE problems? Maybe like ext's 5% reserve.
    - krondor 10y ago
      I believe you're misunderstanding the problems that occur on a cow filesystem. In fact btrfs already has an overcommit disk buffer, and is already doing many ENOSPACE handling tricks. Have a look; https://btrfs.wiki.kernel.org/index.php/ENOSPC https://btrfs.wiki.kernel.org/index.php/ENOSPC Reading more for my own interest it seams ZFS uses ZIL to help convert random writes to sequential, which helps with fragmentation under low space. I am curious if bcache, can operate similarly. In addition, I should also point out that this is less of an issue on SSD, due to the nature of how random reads/writes work there anyway (btrfs does a good job of being SSD aware).
    - Elhana 10y ago
      As I understand it in btrfs case there is two problems: 1) Metadata in btrfs can use lots of space, especially when you convert from ext4. It might happen that you have gigabytes of free space reserved for metadata, so it can not be used anymore. This can be solved with rebalancing, but that can take ages, which is actually one of the reasons zfs doesn't have bpr rewrite feature. 2) btrfs can have mixed raid levels and in that scenario calculating free space is tricky, but people still rely on common tools, that simply give some estimates in that case. Change the way it estimates free space and you'll have less clueless people complaining about fs running out of it, but more will say btrfs shows too little.
EyesBlue1988 10y ago
btrfs is not stable!, xfs is stable

by krondor 10y ago

DominoTree 10y ago

"stable" != "mature"

krondor 10y ago
That's a fair point. I guess we can debate what 'mature' means. In development since 2007, stable since 2014. ZFS, in development since 2001 (correction I erroneously listed 2005 earlier), stable since 2006 (or at least Solaris included since then). Do you consider the Solaris years or just the Linux years and then do you consider the Linux Debian/Ubuntu sanctioned years or the ZoL years? I'm fine with mature since included in the default installer on Ubuntu/Redhat/Oracle/SUSE/etc... for my definition of btrfs maturity.
- eatonphil 10y ago
  I think you've confused your dates. ZFS has been in development since 2001. It was first introduced in 2005.
  - krondor 10y ago
    Yes, thanks for catching that, I've edited the parent. As you can tell I'm not that strong on ZFS other than what I've learned in some small tinkering and discussion with others.

deleted 10y ago

[deleted]

XorNot 10y ago

The problem is "in production" you can do a lot of things which ironically don't work for smaller users, because "in production" you specifically optimize and assume hardware and applications will crash in catastrophic ways pretty much all the time and you code quite differently (i.e. it is totally reasonable to ask applications to be distributed and deal with it).

krondor 10y ago
Agreed, my workload is probably not representative of a small organization or user. I absolutely leverage resiliency in software, distributed computing, and mitigate single points of failure, etc... This is a good point. Now, I do think these methods are increasingly approachable for all users, however. A lot of that is actually enabled by the feature sets of ZFS and btrfs. The default Ubuntu installer, for example, will create snapshots during OS upgrade for rollback if the upgrade fails. ZFS and btrfs send/receive feature allows for efficient DR clones (not to mention seed images and snapshots). LXD leverages ZFS if you choose to for rapid containerization and snapshots. These intrinsic abilities of these two filesystems allow for smaller users and organizations to improve those workflows to be more risk averse in general (even while assuming some risk in a newer FS). Just, if possible, be up to date on the known issues and run more recent kernels and userland utilities. https://btrfs.wiki.kernel.org/index.php/Gotchas https://btrfs.wiki.kernel.org/index.php/Gotchas
deleted 10y ago
[deleted]

bogomipz 10y ago

It seems odd that after all this time that btrfs wouldn't be able to handle database workloads. If it can't handle hight I/O it doens't have much use except for a bootdisk. Do you have any reading you could suggest on this issue?

zanny 10y ago
btrfs is only particularly slow on database files if they are not marked for inplace writing because the filesystem is default copy on write, which is horribly slow for huge constantly changing files (they end up highly fragmented across data blocks if you don't turn off cow).
krondor 10y ago
As zanny said, cow presents an issue for transactional workloads (databases, vm images, etc...) as each write fragments the file, by nature of the process of cow. btrfs can handle database workloads but you have to disable cow for them (which you can do at the file, directory, or subvolume level in btrfs). You would specify the nodatacow mount option, or chattr +C (file/directory). The btrfs autodefrag is still rather new, and needs some work. I expect that could be the long term fix (manual defrag is fine now, but you wouldn't want to call it frequently on a db file), I'm not sure how ZFS handles the fragmentation, but I do know in the past ZFS observed similar issues (seems to have been mostly resolved). I should also point out that disabling cow doesn't really fully disable it, snapshots can still function, etc... however, I'm sure that once you start to use the other cow functions you might observe slipping performance due to fragmentation of these types of files. https://blog.pgaddict.com/posts/friends-dont-let-friends-use-btrfs-for-oltp https://blog.pgaddict.com/posts/friends-dont-let-friends-use... https://bartsjerps.wordpress.com/2013/02/26/zfs-ora-database-fragmentation/ https://bartsjerps.wordpress.com/2013/02/26/zfs-ora-database...

jamesblonde 10y ago

You forgot about how sensitive it is to low disk space. Don't go above 80% in production is what I've been told - that's a lot of wasted disk space.

krondor 10y ago
You are correct that btrfs and ZFS are sensitive to low disk space. This is also related to how cow filesystems function, they need that free space to commit writes and for snapshots because they, by nature, don't overwrite blocks. See this for the ZFS example; http://serverfault.com/a/556892/79238 http://serverfault.com/a/556892/79238 In both cases the exact amount of free space you desire is a mix of workload, fragmentation, and snapshots. However, I disagree they waste a lot of space. I think both ZFS and btrfs more than make up for the overhead of free space commits through their space saving features. cp --reflink, block suballocation, compresssion, and efficient snapshots outweight the overhead, in my case. Your mileage may vary.
- cm3 10y ago
  So, isn't there a way to set aside a buffer space so that you don't run into ENOSPACE problems? Maybe like ext's 5% reserve.
  - krondor 10y ago
    I believe you're misunderstanding the problems that occur on a cow filesystem. In fact btrfs already has an overcommit disk buffer, and is already doing many ENOSPACE handling tricks. Have a look; https://btrfs.wiki.kernel.org/index.php/ENOSPC https://btrfs.wiki.kernel.org/index.php/ENOSPC Reading more for my own interest it seams ZFS uses ZIL to help convert random writes to sequential, which helps with fragmentation under low space. I am curious if bcache, can operate similarly. In addition, I should also point out that this is less of an issue on SSD, due to the nature of how random reads/writes work there anyway (btrfs does a good job of being SSD aware).
  - Elhana 10y ago
    As I understand it in btrfs case there is two problems: 1) Metadata in btrfs can use lots of space, especially when you convert from ext4. It might happen that you have gigabytes of free space reserved for metadata, so it can not be used anymore. This can be solved with rebalancing, but that can take ages, which is actually one of the reasons zfs doesn't have bpr rewrite feature. 2) btrfs can have mixed raid levels and in that scenario calculating free space is tricky, but people still rely on common tools, that simply give some estimates in that case. Change the way it estimates free space and you'll have less clueless people complaining about fs running out of it, but more will say btrfs shows too little.

EyesBlue1988 10y ago

btrfs is not stable!, xfs is stable