9 ms·

ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Performance Comparison (2015)

131 points by SXX 10y ago

LeoPanthera 10y ago
*on Linux. ZFS on Linux in particular is not representative of ZFS on BSD or Solaris.
- georgyo 10y ago
  Not sure what you mean here. They showed that ZFS performed well on linux. Are you saying that ZFS performs poorly on BSD and Solaris? Love me some ZFS and it works well pretty much everywhere. OpenZFS shows good promise of keeping (bringing) the BSD, Illumos and Linux versions in line with each other.
  - protomyth 10y ago
    > Are you saying that ZFS performs poorly on BSD and Solaris? I think LeoPanthera is saying it performs better on BSD (well FreeBSD) and Solaris (well Illumos-based in particular).
    - ryao 10y ago
      ClusterHQ's tests were unable to detect a difference between the platforms, with the exception of SELinux where performance was degraded unless xattr=sa was set. The code bases are almost identical, so performance differences tend to be caused by external factors and/or compatibility shims. There are also cases where one platform has a performance improvement before the others because the releases and porting are not done in lock step.
    - tete 10y ago
      Another reason is that ZFS is tightly integrated with other things (for example jails and zones) on those systems, having switches related to other parts. I don't think that it is quiet there yet on Linux, even though I am sure that will come.
beefhash 10y ago
Why this article had to be segmented into 10 pages is beyond me, however.
- koolba 10y ago
  > Why this article had to be segmented into 10 pages is beyond me, however. "I've figured out a way to increase our pageviews by 10x!" For "major" sites there's usually a printable view that consolidates things into one page. I don't see one on this page though.
- _Codemonkeyism 10y ago
  The reason I didn't read the article.
  - frou_dh 10y ago
    Very interesting - thanks for announcing your non-reading!
    - _Codemonkeyism 10y ago
      My pleasure!
  - babuskov 10y ago
    Ditto. When I reach the end of page 1 on such website, I just close the tab and move on to something else.
- SXX 10y ago
  I'm not author, but since I know some Joomla magic this might help: http://www.ilsistemista.net/index.php/virtualization/47-zfs-btrfs-xfs-ext4-and-lvm-with-kvm-a-storage-performance-comparison.html?print=true http://www.ilsistemista.net/index.php/virtualization/47-zfs-...
- ovidiup 10y ago
  To make matters worse, their web site has run out of MySQL database connections, and currently the site is unavailable.
ars 10y ago
This is more than a year old. I've been told that btrfs has improved greatly in that time.
- SXX 10y ago
  I found it just recently when upgraded my VM images storage and BTRFS was still significally slower on HDDs using Ubuntu 16.04. Though I only compared it to EXT4 on LVM with QCOW2 + backing files. Any idea if there was newer performance comparison for VM storage?
- ryao 10y ago
  ZFS has not stood still during that time either.
  - tux1968 10y ago
    But it had a lot less low-hanging-fruit to pick during that time.
    - ryao 10y ago
      I could argue otherwise, but that is beside the point on enterprise distributions where these tests were done. The latest ZFS code is always avaliable on those while the btrfs code is stagnant there. The btrfs developers delegate responsibility of backporting their changes to distribution developers, who never do any backports. Consequently, the newer btrfs code is not very relevant as far as benchmarks on RHEL and others are concerned.
- MertsA 10y ago
  This is more than a year old and the hardware this was tested on wasn't even that exceptional when it was released 7 years ago. I'm kind of assuming that the tester did something strange for BTRFS in particular because his results disagree with every other benchmark out there. Using BTRFS on top of software RAID 10 is also inappropriate as this should be done by creating a filesystem containing all four devices. That being said I'd love to see more benchmarks of BTRFS compared to other filesystems and on hardware that isn't so archaic. I think it's safe to say that this article is not representative of reality as Phoronix has tons of benchmarks that all don't have nearly this big of difference between BTRFS and Ext4. Here's an article that's even more outdated than the parent and it still shows BTRFS performing acceptably across the board. https://www.phoronix.com/scan.php?page=article&item=linux_raid_fs4 https://www.phoronix.com/scan.php?page=article&item=linux_ra...
matt-attack 10y ago
What is with BTRFS? It stands out in every single test (and not in a good way).
- josefbacik 10y ago
  Kvm (or any overwrite workload for that matter) is the worst possible workload for btrfs because of COW. We have ideas to address this but honestly it's not high on the list.
  - loeg 10y ago
    Shouldn't ZFS have similar problems with overwrite workloads?
    - josefbacik 10y ago
      ZFS has a 7 year head start on BTRFS and has traditionally had about 5 times as many core contributors at any given time so I imagine they've solved this in some novel way by now.
      - deleted 10y ago
        [deleted]
      - tolle 10y ago
        Wasn't ZFS even "production ready" before BTRFS development even began? I don't remember the state of it when it was first introduced into Solaris, maybe someones memory is better then mine. Was ZFS better of in 06-07 Then BTRFS is now?
        ryao 10y ago
        btrfs development started in 2007 while ZFS development started in 2001. If you want to do a point in time comparison, look at how ZFS was in 2010. You can get the last copy of OpenSolaris for the comparison. It would need to be done on physical hardware due to poor support for virtualization back then. By the way, ZFS was deemed production ready after 4 years of development.
yashau 10y ago
Mirror?
- gervase 10y ago
  https://archive.is/ZHjTH https://archive.is/ZHjTH Unfortunately, thanks to the way they've partitioned the article, this mirror is not particularly useful.
- ryao 10y ago
  https://web.archive.org/web/20160403001544/http://www.ilsistemista.net/index.php/virtualization/47-zfs-btrfs-xfs-ext4-and-lvm-with-kvm-a-storage-performance-comparison.html https://web.archive.org/web/20160403001544/http://www.ilsist...
- deleted 10y ago
  [deleted]
tete 10y ago
Here a few things to note: BTRFS has had Copy on Write disabled, while ZFS (not possible, cause the whole idea of the FS is intended to be copy on write), which actually makes BTRFS look even worse compared to ZFS, cause BTRFS writes once instead of twice and has most of its features not work, while still performance pretty bad. But then it's also younger. Both ZFS and BTRFS (only know specifics of ZFS) can be configured to be better for DB workloads. On ZFS, I know a couple of people using it cause it has nice properties. Anyway. You might not want to use ZFS or BTRFS for a (pure) database system when performance is the important thing (compared to data security). What I find kind of missing is UFS because of the BSD world (it's kind of what Ext4 is in the sense of "your general purpose FS with good performance for DBs"), but okay. And yeah, that all may sound a bit biased towards ZFS, but so far my experience with ZFS has been rather pleasent compared to BTRFS, but again. You might wanna use UFS or Ext4. It's also kind of "wrong" to compare those file systems. It's like while you could use Redis and PostgreSQL for the same things it's probably not what you want to for one reason or the other. But then of course it's good, cause you might really have a thing where you want to have a comparison of those two things to take the right decision for your specific application. Like those cases where you say "It's slow, but it makes a lot of thing easier" or "It doesn't guarantee that, but when I take care of it in the design I will only need only a fraction of the resources. And the other downsides don' t annoy me thaaat much". Because the question came up. COW means Copy on Write and it really means what it says. You copy the data (so have find and to write blocks, duplicating your data on write) which in case of a full blown database which does that again (WAL, Autovacuum, keeping lots of metadata, etc.) you really shouldn't be surprised that it's a lot slower on write heavy sytems. It is more than expected. On the other hand because the FS does a lot of things in a similar way to a database, other than snapshots and replication and all that cool stuff ZFS does things related to metadata extremely quickly and that's why some CDNs that have many small files use it. Besides just being an amazing thing for managing your data in general. Also if you wanna learn more a about ZFS and have a good understanding of how you can run databases and other things on ZFS and really utilize it (not just gaining more performance) then I highly recommend the book FreeBSD Mastery: Advanced ZFS.
- specialist 10y ago
  Is there something like Aphyr's Jespen for file systems? https://aphyr.com/tags/Jepsen https://aphyr.com/tags/Jepsen I've been curious about ZFS, btrfs, etc. But as a layperson, I don't have the technical chops, gumption, wherewithal to figure what's what. Reading posts (comments) about the edge cases where they fail (data, performance, missing features) leaves me more baffled.
Zardoz84 10y ago
Interesting. But in our case, we are running Hyper-V VMs (son sadly NTFS on host), were the VMs uses EXT4, and a few uses BTRFS. The only problem that we had, was with a uncontrolled poweroff a year ago. Looks that BRTFS had some trouble recovering from it, but we manage to restore all data from the partitions. However, the flexibility that offer BTRFS (transparent compression, increasing hard disk space on demand, etc) is really nice. I hope that it improve more, so we can use it without any issues.
- tremon 10y ago
  increasing hard disk space on demand Note that ext4 on LVM can do this as well, you just need to plan for it in advance. All of my systems are configured as fs-on-lvm, even the single-disk ones. I've found it's just less hassle: I can do all storage migration or storage expansion without ever taking the machine offline.
  - newman314 10y ago
    This. I just realized one of my Ubuntu VMs was built without LVM and only found out because I was going to expand storage on it. Luckily, it's just a toy VM on the home server so no biggie. But yes, having fs-on-lvm should just be the default now since it makes expansion so much easier.
mrmondo 10y ago
- Very old kernel used (3.10!), makes me wonder how old the packages like btrfs-progs are as well. - BTRFS not mounted with compression (compress=lzo) - Don't use QCOW2, just don't, it's slow and you're just adding extra layers where you don't need to. It would be interesting to see you re-run these tests using a modern kernel, say at least 4.4 and either raw block devices or logical volumes along with mounting BTRFS properly with the compress=lzo option
- ryao 10y ago
  The benchmark configuration appears to be designed to evaluate the use of storage technologies for a KVM host. Consequently, saying to use raw block devices when giving tips for improving btrfs performance is contradictory. Also, there are a large number of people that will not run a newer kernel for several years because they are on RHEL6 or RHEL7, so while newer kernels are interesting, we should not discount the results on the basis that the kernel is old. The latest ZFSOnLinux code is able to run on those kernels, so while btrfs remains stagnant there, ZFS will continue to improve. As for rerunning the tests, using recordsize=4K and compression=lz4 on ZFS should improve its performance here too. Putting the VM images on zvols (where it would be volblocksize=4K) rather than qcow2 also would help. In ZoL, zvols are block devices.
  - mrmondo 10y ago
    I disagree (politely) with your assumption about QCOW2 and BTRFS performance, if I get some time this week I'll do some objective benchmarking (I design and build Linux based storage systems) and see hey I might be proven wrong but regardless I can let you know the results if you're interested? People not willing to update their kernel in production environments is a social problem, not a technical problem. The kernel is one of the most reliable, well tested and reviewed software projects in the world. When you upgrade your kernel 99.5% of the time you get new features, performance and bug fixes without any negotiate impact. There are of course rare corner cases especially if running propriety hardware when they are generally slower to release updates that show the benefits of a modern kernel. The problem there is the culture and traditional slow moving operational engineered haven't all embraced the well proven fact that regular, small charges are safer and have various added benefits. There is also a serious language barrier between many engineers and management / project managers who clearly would not be likely to understand the benefit of upgrading to a kernel version that say properly supported the new SCSI blk_mq backend for storage, so there either needs to be degree of trust and respect to (proven) engineers(ing) and teams or they need to be clearly taught the value add of a fast release cycle and practising quick (hopefully automated) patching. That's where I think books like Gene Kim's - The Phoenix Project and his soon to be released book - The DevOps Handbook which may in fact be more useful to people performing PM/PO tasks.
Already__Taken 10y ago
I'd like to see NTFS somewhere in these comparisons just to shake some of that "grass is greener" thinking. The amount I've read on the various file systems I feel like there's a lot on the table there to get out of our hardware by just using a better system. But I've not seen a whole lot about ntfs, because it's all we've got in windows land I suppose.
yAnonymous 10y ago
My ad blocker doesn't care that you separate the article into 11 pages. It only puts more load on your server.
newman314 10y ago
Got a Synology NAS showing up today. Based on the results here and the fuzzing link posted several days ago (and elsewhere ITT), it looks like I'll be going with ext4 for now. Amazingly, I've had a ReadyNAS device last me over 10 years and here's to hoping this one lasts a similar period of time.

131 points by SXX 10y ago

LeoPanthera 10y ago

*on Linux. ZFS on Linux in particular is not representative of ZFS on BSD or Solaris.

georgyo 10y ago
Not sure what you mean here. They showed that ZFS performed well on linux. Are you saying that ZFS performs poorly on BSD and Solaris? Love me some ZFS and it works well pretty much everywhere. OpenZFS shows good promise of keeping (bringing) the BSD, Illumos and Linux versions in line with each other.
- protomyth 10y ago
  > Are you saying that ZFS performs poorly on BSD and Solaris? I think LeoPanthera is saying it performs better on BSD (well FreeBSD) and Solaris (well Illumos-based in particular).
  - ryao 10y ago
    ClusterHQ's tests were unable to detect a difference between the platforms, with the exception of SELinux where performance was degraded unless xattr=sa was set. The code bases are almost identical, so performance differences tend to be caused by external factors and/or compatibility shims. There are also cases where one platform has a performance improvement before the others because the releases and porting are not done in lock step.
  - tete 10y ago
    Another reason is that ZFS is tightly integrated with other things (for example jails and zones) on those systems, having switches related to other parts. I don't think that it is quiet there yet on Linux, even though I am sure that will come.

beefhash 10y ago

Why this article had to be segmented into 10 pages is beyond me, however.

koolba 10y ago
> Why this article had to be segmented into 10 pages is beyond me, however. "I've figured out a way to increase our pageviews by 10x!" For "major" sites there's usually a printable view that consolidates things into one page. I don't see one on this page though.
_Codemonkeyism 10y ago
The reason I didn't read the article.
- frou_dh 10y ago
  Very interesting - thanks for announcing your non-reading!
  - _Codemonkeyism 10y ago
    My pleasure!
- babuskov 10y ago
  Ditto. When I reach the end of page 1 on such website, I just close the tab and move on to something else.
SXX 10y ago
I'm not author, but since I know some Joomla magic this might help: http://www.ilsistemista.net/index.php/virtualization/47-zfs-btrfs-xfs-ext4-and-lvm-with-kvm-a-storage-performance-comparison.html?print=true http://www.ilsistemista.net/index.php/virtualization/47-zfs-...
ovidiup 10y ago
To make matters worse, their web site has run out of MySQL database connections, and currently the site is unavailable.

ars 10y ago

This is more than a year old. I've been told that btrfs has improved greatly in that time.

SXX 10y ago
I found it just recently when upgraded my VM images storage and BTRFS was still significally slower on HDDs using Ubuntu 16.04. Though I only compared it to EXT4 on LVM with QCOW2 + backing files. Any idea if there was newer performance comparison for VM storage?
ryao 10y ago
ZFS has not stood still during that time either.
- tux1968 10y ago
  But it had a lot less low-hanging-fruit to pick during that time.
  - ryao 10y ago
    I could argue otherwise, but that is beside the point on enterprise distributions where these tests were done. The latest ZFS code is always avaliable on those while the btrfs code is stagnant there. The btrfs developers delegate responsibility of backporting their changes to distribution developers, who never do any backports. Consequently, the newer btrfs code is not very relevant as far as benchmarks on RHEL and others are concerned.
MertsA 10y ago
This is more than a year old and the hardware this was tested on wasn't even that exceptional when it was released 7 years ago. I'm kind of assuming that the tester did something strange for BTRFS in particular because his results disagree with every other benchmark out there. Using BTRFS on top of software RAID 10 is also inappropriate as this should be done by creating a filesystem containing all four devices. That being said I'd love to see more benchmarks of BTRFS compared to other filesystems and on hardware that isn't so archaic. I think it's safe to say that this article is not representative of reality as Phoronix has tons of benchmarks that all don't have nearly this big of difference between BTRFS and Ext4. Here's an article that's even more outdated than the parent and it still shows BTRFS performing acceptably across the board. https://www.phoronix.com/scan.php?page=article&item=linux_raid_fs4 https://www.phoronix.com/scan.php?page=article&item=linux_ra...

matt-attack 10y ago

What is with BTRFS? It stands out in every single test (and not in a good way).

josefbacik 10y ago
Kvm (or any overwrite workload for that matter) is the worst possible workload for btrfs because of COW. We have ideas to address this but honestly it's not high on the list.
- loeg 10y ago
  Shouldn't ZFS have similar problems with overwrite workloads?
  - josefbacik 10y ago
    ZFS has a 7 year head start on BTRFS and has traditionally had about 5 times as many core contributors at any given time so I imagine they've solved this in some novel way by now.
    - deleted 10y ago
      [deleted]
    - tolle 10y ago
      Wasn't ZFS even "production ready" before BTRFS development even began? I don't remember the state of it when it was first introduced into Solaris, maybe someones memory is better then mine. Was ZFS better of in 06-07 Then BTRFS is now?
      - ryao 10y ago
        btrfs development started in 2007 while ZFS development started in 2001. If you want to do a point in time comparison, look at how ZFS was in 2010. You can get the last copy of OpenSolaris for the comparison. It would need to be done on physical hardware due to poor support for virtualization back then. By the way, ZFS was deemed production ready after 4 years of development.

yashau 10y ago

Mirror?

gervase 10y ago
https://archive.is/ZHjTH https://archive.is/ZHjTH Unfortunately, thanks to the way they've partitioned the article, this mirror is not particularly useful.
ryao 10y ago
https://web.archive.org/web/20160403001544/http://www.ilsistemista.net/index.php/virtualization/47-zfs-btrfs-xfs-ext4-and-lvm-with-kvm-a-storage-performance-comparison.html https://web.archive.org/web/20160403001544/http://www.ilsist...
deleted 10y ago
[deleted]

tete 10y ago

Here a few things to note: BTRFS has had Copy on Write disabled, while ZFS (not possible, cause the whole idea of the FS is intended to be copy on write), which actually makes BTRFS look even worse compared to ZFS, cause BTRFS writes once instead of twice and has most of its features not work, while still performance pretty bad. But then it's also younger. Both ZFS and BTRFS (only know specifics of ZFS) can be configured to be better for DB workloads. On ZFS, I know a couple of people using it cause it has nice properties. Anyway. You might not want to use ZFS or BTRFS for a (pure) database system when performance is the important thing (compared to data security). What I find kind of missing is UFS because of the BSD world (it's kind of what Ext4 is in the sense of "your general purpose FS with good performance for DBs"), but okay. And yeah, that all may sound a bit biased towards ZFS, but so far my experience with ZFS has been rather pleasent compared to BTRFS, but again. You might wanna use UFS or Ext4. It's also kind of "wrong" to compare those file systems. It's like while you could use Redis and PostgreSQL for the same things it's probably not what you want to for one reason or the other. But then of course it's good, cause you might really have a thing where you want to have a comparison of those two things to take the right decision for your specific application. Like those cases where you say "It's slow, but it makes a lot of thing easier" or "It doesn't guarantee that, but when I take care of it in the design I will only need only a fraction of the resources. And the other downsides don' t annoy me thaaat much". Because the question came up. COW means Copy on Write and it really means what it says. You copy the data (so have find and to write blocks, duplicating your data on write) which in case of a full blown database which does that again (WAL, Autovacuum, keeping lots of metadata, etc.) you really shouldn't be surprised that it's a lot slower on write heavy sytems. It is more than expected. On the other hand because the FS does a lot of things in a similar way to a database, other than snapshots and replication and all that cool stuff ZFS does things related to metadata extremely quickly and that's why some CDNs that have many small files use it. Besides just being an amazing thing for managing your data in general. Also if you wanna learn more a about ZFS and have a good understanding of how you can run databases and other things on ZFS and really utilize it (not just gaining more performance) then I highly recommend the book FreeBSD Mastery: Advanced ZFS.

specialist 10y ago
Is there something like Aphyr's Jespen for file systems? https://aphyr.com/tags/Jepsen https://aphyr.com/tags/Jepsen I've been curious about ZFS, btrfs, etc. But as a layperson, I don't have the technical chops, gumption, wherewithal to figure what's what. Reading posts (comments) about the edge cases where they fail (data, performance, missing features) leaves me more baffled.

Zardoz84 10y ago

Interesting. But in our case, we are running Hyper-V VMs (son sadly NTFS on host), were the VMs uses EXT4, and a few uses BTRFS. The only problem that we had, was with a uncontrolled poweroff a year ago. Looks that BRTFS had some trouble recovering from it, but we manage to restore all data from the partitions. However, the flexibility that offer BTRFS (transparent compression, increasing hard disk space on demand, etc) is really nice. I hope that it improve more, so we can use it without any issues.

tremon 10y ago
increasing hard disk space on demand Note that ext4 on LVM can do this as well, you just need to plan for it in advance. All of my systems are configured as fs-on-lvm, even the single-disk ones. I've found it's just less hassle: I can do all storage migration or storage expansion without ever taking the machine offline.
- newman314 10y ago
  This. I just realized one of my Ubuntu VMs was built without LVM and only found out because I was going to expand storage on it. Luckily, it's just a toy VM on the home server so no biggie. But yes, having fs-on-lvm should just be the default now since it makes expansion so much easier.

mrmondo 10y ago

- Very old kernel used (3.10!), makes me wonder how old the packages like btrfs-progs are as well. - BTRFS not mounted with compression (compress=lzo) - Don't use QCOW2, just don't, it's slow and you're just adding extra layers where you don't need to. It would be interesting to see you re-run these tests using a modern kernel, say at least 4.4 and either raw block devices or logical volumes along with mounting BTRFS properly with the compress=lzo option

ryao 10y ago
The benchmark configuration appears to be designed to evaluate the use of storage technologies for a KVM host. Consequently, saying to use raw block devices when giving tips for improving btrfs performance is contradictory. Also, there are a large number of people that will not run a newer kernel for several years because they are on RHEL6 or RHEL7, so while newer kernels are interesting, we should not discount the results on the basis that the kernel is old. The latest ZFSOnLinux code is able to run on those kernels, so while btrfs remains stagnant there, ZFS will continue to improve. As for rerunning the tests, using recordsize=4K and compression=lz4 on ZFS should improve its performance here too. Putting the VM images on zvols (where it would be volblocksize=4K) rather than qcow2 also would help. In ZoL, zvols are block devices.
- mrmondo 10y ago
  I disagree (politely) with your assumption about QCOW2 and BTRFS performance, if I get some time this week I'll do some objective benchmarking (I design and build Linux based storage systems) and see hey I might be proven wrong but regardless I can let you know the results if you're interested? People not willing to update their kernel in production environments is a social problem, not a technical problem. The kernel is one of the most reliable, well tested and reviewed software projects in the world. When you upgrade your kernel 99.5% of the time you get new features, performance and bug fixes without any negotiate impact. There are of course rare corner cases especially if running propriety hardware when they are generally slower to release updates that show the benefits of a modern kernel. The problem there is the culture and traditional slow moving operational engineered haven't all embraced the well proven fact that regular, small charges are safer and have various added benefits. There is also a serious language barrier between many engineers and management / project managers who clearly would not be likely to understand the benefit of upgrading to a kernel version that say properly supported the new SCSI blk_mq backend for storage, so there either needs to be degree of trust and respect to (proven) engineers(ing) and teams or they need to be clearly taught the value add of a fast release cycle and practising quick (hopefully automated) patching. That's where I think books like Gene Kim's - The Phoenix Project and his soon to be released book - The DevOps Handbook which may in fact be more useful to people performing PM/PO tasks.

Already__Taken 10y ago

I'd like to see NTFS somewhere in these comparisons just to shake some of that "grass is greener" thinking. The amount I've read on the various file systems I feel like there's a lot on the table there to get out of our hardware by just using a better system. But I've not seen a whole lot about ntfs, because it's all we've got in windows land I suppose.

yAnonymous 10y ago

My ad blocker doesn't care that you separate the article into 11 pages. It only puts more load on your server.

newman314 10y ago

Got a Synology NAS showing up today. Based on the results here and the fuzzing link posted several days ago (and elsewhere ITT), it looks like I'll be going with ext4 for now. Amazingly, I've had a ReadyNAS device last me over 10 years and here's to hoping this one lasts a similar period of time.