6 ms·

But is it, performance-wise? How will ZFS compare to btrfs?

pizza234 12y ago
At this point in time, they still can't be compared, because btrfs is still not [considered] production-ready, and subject to significant changes (also performance-related). This is especially important because on use cases where performance differences are significant, that is, not on general desktop usage, the maturity of the FS is funamental, and btrfs is discouraged right now.
- derefr 12y ago
  There are many situations where performance is important, and durability is... not. For example, ephemeral CoreOS cloud instances running Docker containers: they use lots of copy-on-write layers, but they don't actually need to persist any state across reboots (the layers may as well be stored in volatile memory.) Btrfs is perfectly "production-ready" for this particular use-case, so a [current] performance comparison would be pretty useful.
  - ryao 12y ago
    I am working on ZFS support for CoreOS. A snapshot of my WIP proof of concept was posted by my employer yesterday: https://github.com/ClusterHQ/flocker/blob/zfs-on-coreos-tutorial-667/docs/experimental/zfs-on-coreos.rst https://github.com/ClusterHQ/flocker/blob/zfs-on-coreos-tuto... CoreOS uses btrfs as its rootfs. I imagine that the CoreOS developers managed to avoid issues like ENOSPC by virtue of not writing to their rootfs very much. I did not have that luxury since I compiled a Gentoo GNU userland on top of it during the course of development. I encountered numerous ENOSPC errors on btrfs when developing the ZFS port to CoreOS and even hit ENOSPC errors when trying to correct the btrfs ENOSPC errors with `btrfs balance /`. I would not consider btrfs ready for production use, but your mileage will vary.
    - simoncion 12y ago
      When were you hitting these ENOSPC errors, and what kernel were you using when you hit them? $ df -h . Filesystem Size Used Avail Use% Mounted on /dev/dm-1 42G 41G 280M 100% /home $ btrfs fi df . Data, single: total=40.48GiB, used=40.21GiB System, single: total=4.00MiB, used=12.00KiB Metadata, single: total=1.01GiB, used=649.18MiB $ uname -r 3.15.2-hardened As you can see, I've been working with a pretty much full btrfs volume. It used to be TERRIBLE to deal with btrfs in such a situation, but I haven't had an ENOSPC issue in ages.
      - ryao 12y ago
        The Gentoo Prefix boootstrap on a developer image: https://www.gentoo.org/proj/en/gentoo-alt/prefix/bootstrap.xml https://www.gentoo.org/proj/en/gentoo-alt/prefix/bootstrap.x... CoreOS uses Linux 3.15.y.
aidenn0 12y ago
btrfs is almost painfully slow at times, so hopefully ZFS is a lot better.
- mprovost 12y ago
  Even on Solaris, ZFS doesn't have a great reputation for performance.
  - ryao 12y ago
    Can you substantiate that? ZFS on Solaris produced excellent numbers back in 2008: https://blogs.oracle.com/brendan/entry/a_quarter_million_nfs_iops https://blogs.oracle.com/brendan/entry/a_quarter_million_nfs... More recent derivatives of OpenSolaris were reported to do 1.6 million IOPS last year: http://www.high-availability.com/high-availability-zfs-partner-nexenta-achieves-1-6-million-iops/ http://www.high-availability.com/high-availability-zfs-partn... Your mileage will vary, but ZFS has always had strong performance.
    - mprovost 12y ago
      One of those is an 8 year old benchmark, the other used a 9.6TB flash array so hardly a realistic storage platform. I'm not saying that ZFS performance is awful, just that other features like data protection come first. If you're hoping for some salvation from btrfs you might be waiting a while. Of course they're both developed by the same company now! Here's a benchmark from 2013 with most results showing that it has worse performance on Linux than XFS and EXT4: http://www.phoronix.com/scan.php?page=article&item=zfs_linux_062&num=1 http://www.phoronix.com/scan.php?page=article&item=zfs_linux...
      - ryao 12y ago
        Oracle has nothing to do with the Open ZFS community these days. As for the benchmarks you cite, there are several key problems: 1. You say that they show ZFS having worse performance than XFS and ext4, but XFS is not in those benchmarks and the FIO tester shows ZFS as outperforming its competition by a significant margin. 2. They use a single disk. No server does this and while desktops and laptops do this, it is not clear how the benchmarks are of any relevance there. Additionally, LZ4 compression is not in use, when practically everyone deploying ZFSOnLinux would configure it to use LZ4. 3. ext4 manages to perform better than the theoretical limit of the SATA II interface and there is no discussion as to why. 4. ZFSOnLinux 0.6.2 is an old release. The most recent 0.6.3 release includes a new IO elevator and other improvements that enable ZFSOnLinux 0.6.3 to outperform its precedessor by a significant margin in many workloads. Would you post something constructive that you actually did yourself? I am beginning to think that you have never even used ZFS.
kbenson 12y ago
You may be interested in Phoronix[1] linux review site. When I used to follow it through RSS more diligently, I routinely saw filesystem benchmark shootouts such as this[2]. I'm not sure if there's a more recent one than that, but there are more recent benchmarks of Btrfs and ZFS against prior versions of themselves so you can see improvements (or regressions). 1: http://www.phoronix.com http://www.phoronix.com 2: http://www.phoronix.com/scan.php?page=article&item=linux_313fs_8way&num=1 http://www.phoronix.com/scan.php?page=article&item=linux_313...
- ucsdrake 12y ago
  I would be really really careful with Phoronix benchmarking. If I recall correctly, I've seen them 'benchmark' and compare a FreeBSD system with a recent (at the time) Linux release. Issue was, they weren't even running the tests on the same hardware. <sarcasm> Really?! you saw performance differences between and i5 and i7?! Each running a different OS?! Get outa here!! </sarcasm>
  - kbenson 12y ago
    I'd be interested in seeing that. It's been a while since I really followed the site, but I seem to recall he was at least forward about what he was testing and why in the text around the benchmarks, I will admit to him seeming to be overly eager to get articles out, sometimes of ambiguous usefulness, probably due to his advertising model (at least at the time, when I was there earlier it seemed far less riddled with advertising). Edit: Nevermind about the advertising, it's still intrusive, I'm just running adblock now.
    - ryao 12y ago
      Phoronix does try to be honest about its methodology, but the person running benchmarks gives no thought to what his results mean, if anything. It shows in the distinct lack of documentation on why a given benchmark matters. As for Phoronix's ZFS benchmarks, the test hardware used drives that misreported themselves as having 512-byte sectors, which handicapped ZFS performance. Phoronix rejected all suggestions that it correct for this as end-users had been doing. Phoronix refused to meet half way by posting two results (one with proper configuration and one without), and also refused the suggestion that it to mention the existence of that problem in its test hardware. I eventually wrote code to identify drives known to misreport their sector sizes so that ZFS will automatically use the correct settings on them. That lead to the Phoronix August 2013 benchmarks showing a remarkable improvement in ZFS performance in FIO. It was so great that it sparked a discussion among the btrfs developers: http://comments.gmane.org/gmane.comp.file-systems.btrfs/27546 http://comments.gmane.org/gmane.comp.file-systems.btrfs/2754... Later that month, I publicly criticized Phoronix for posting misleading benchmarks: http://phoronix.com/forums/showthread.php?83731-ZFSOnLinux-0-6-2-released http://phoronix.com/forums/showthread.php?83731-ZFSOnLinux-0... Phoronix has not posted ZFS benchmarks since that time.
orkoden 12y ago
ZFS supports better compression algorithms. Those can even speed up slow disks.
ryao 12y ago
I would expect ZFS to outperform btrfs because of ARC and ZIL. ARC allows ZFS to increase the cache hit rate. ZIL permits ZFS to update its on-disk trees asynchronously. I am not aware of any analog to this in btrfs. Desktop users typically report that their systems become more responsive after switching to ZFS. The only case where I suspect btrfs might significantly outperform ZFS involves getdents() when the caches is cold. This is because ZoL does not presently do readahead on directory lookups. This has a noticeable impact on the performance of `ls` on the first access of a directory on mechanical storage, but it is not a problem for subsequent reads or on solid state storage. Prefetch support in getdents() will likely be implemented in either the next release or the one that follows.
deleted 12y ago
[deleted]
javanix 12y ago
Anecdotally I see better performance using ZoL than I did when I was running the same zpool and hardware on FreeBSD.
- ryao 12y ago
  This would suggest that the block device layer under ZFS and/or the VFS above it have room for improvement on FreeBSD.

At this point in time, they still can't be compared, because btrfs is still not [considered] production-ready, and subject to significant changes (also performance-related). This is especially important because on use cases where performance differences are significant, that is, not on general desktop usage, the maturity of the FS is funamental, and btrfs is discouraged right now.

derefr 12y ago
There are many situations where performance is important, and durability is... not. For example, ephemeral CoreOS cloud instances running Docker containers: they use lots of copy-on-write layers, but they don't actually need to persist any state across reboots (the layers may as well be stored in volatile memory.) Btrfs is perfectly "production-ready" for this particular use-case, so a [current] performance comparison would be pretty useful.
- ryao 12y ago
  I am working on ZFS support for CoreOS. A snapshot of my WIP proof of concept was posted by my employer yesterday: https://github.com/ClusterHQ/flocker/blob/zfs-on-coreos-tutorial-667/docs/experimental/zfs-on-coreos.rst https://github.com/ClusterHQ/flocker/blob/zfs-on-coreos-tuto... CoreOS uses btrfs as its rootfs. I imagine that the CoreOS developers managed to avoid issues like ENOSPC by virtue of not writing to their rootfs very much. I did not have that luxury since I compiled a Gentoo GNU userland on top of it during the course of development. I encountered numerous ENOSPC errors on btrfs when developing the ZFS port to CoreOS and even hit ENOSPC errors when trying to correct the btrfs ENOSPC errors with `btrfs balance /`. I would not consider btrfs ready for production use, but your mileage will vary.
  - simoncion 12y ago
    When were you hitting these ENOSPC errors, and what kernel were you using when you hit them? $ df -h . Filesystem Size Used Avail Use% Mounted on /dev/dm-1 42G 41G 280M 100% /home $ btrfs fi df . Data, single: total=40.48GiB, used=40.21GiB System, single: total=4.00MiB, used=12.00KiB Metadata, single: total=1.01GiB, used=649.18MiB $ uname -r 3.15.2-hardened As you can see, I've been working with a pretty much full btrfs volume. It used to be TERRIBLE to deal with btrfs in such a situation, but I haven't had an ENOSPC issue in ages.
    - ryao 12y ago
      The Gentoo Prefix boootstrap on a developer image: https://www.gentoo.org/proj/en/gentoo-alt/prefix/bootstrap.xml https://www.gentoo.org/proj/en/gentoo-alt/prefix/bootstrap.x... CoreOS uses Linux 3.15.y.

aidenn0 12y ago

btrfs is almost painfully slow at times, so hopefully ZFS is a lot better.

mprovost 12y ago
Even on Solaris, ZFS doesn't have a great reputation for performance.
- ryao 12y ago
  Can you substantiate that? ZFS on Solaris produced excellent numbers back in 2008: https://blogs.oracle.com/brendan/entry/a_quarter_million_nfs_iops https://blogs.oracle.com/brendan/entry/a_quarter_million_nfs... More recent derivatives of OpenSolaris were reported to do 1.6 million IOPS last year: http://www.high-availability.com/high-availability-zfs-partner-nexenta-achieves-1-6-million-iops/ http://www.high-availability.com/high-availability-zfs-partn... Your mileage will vary, but ZFS has always had strong performance.
  - mprovost 12y ago
    One of those is an 8 year old benchmark, the other used a 9.6TB flash array so hardly a realistic storage platform. I'm not saying that ZFS performance is awful, just that other features like data protection come first. If you're hoping for some salvation from btrfs you might be waiting a while. Of course they're both developed by the same company now! Here's a benchmark from 2013 with most results showing that it has worse performance on Linux than XFS and EXT4: http://www.phoronix.com/scan.php?page=article&item=zfs_linux_062&num=1 http://www.phoronix.com/scan.php?page=article&item=zfs_linux...
    - ryao 12y ago
      Oracle has nothing to do with the Open ZFS community these days. As for the benchmarks you cite, there are several key problems: 1. You say that they show ZFS having worse performance than XFS and ext4, but XFS is not in those benchmarks and the FIO tester shows ZFS as outperforming its competition by a significant margin. 2. They use a single disk. No server does this and while desktops and laptops do this, it is not clear how the benchmarks are of any relevance there. Additionally, LZ4 compression is not in use, when practically everyone deploying ZFSOnLinux would configure it to use LZ4. 3. ext4 manages to perform better than the theoretical limit of the SATA II interface and there is no discussion as to why. 4. ZFSOnLinux 0.6.2 is an old release. The most recent 0.6.3 release includes a new IO elevator and other improvements that enable ZFSOnLinux 0.6.3 to outperform its precedessor by a significant margin in many workloads. Would you post something constructive that you actually did yourself? I am beginning to think that you have never even used ZFS.

kbenson 12y ago

You may be interested in Phoronix[1] linux review site. When I used to follow it through RSS more diligently, I routinely saw filesystem benchmark shootouts such as this[2]. I'm not sure if there's a more recent one than that, but there are more recent benchmarks of Btrfs and ZFS against prior versions of themselves so you can see improvements (or regressions). 1: http://www.phoronix.com http://www.phoronix.com 2: http://www.phoronix.com/scan.php?page=article&item=linux_313fs_8way&num=1 http://www.phoronix.com/scan.php?page=article&item=linux_313...

ucsdrake 12y ago
I would be really really careful with Phoronix benchmarking. If I recall correctly, I've seen them 'benchmark' and compare a FreeBSD system with a recent (at the time) Linux release. Issue was, they weren't even running the tests on the same hardware. <sarcasm> Really?! you saw performance differences between and i5 and i7?! Each running a different OS?! Get outa here!! </sarcasm>
- kbenson 12y ago
  I'd be interested in seeing that. It's been a while since I really followed the site, but I seem to recall he was at least forward about what he was testing and why in the text around the benchmarks, I will admit to him seeming to be overly eager to get articles out, sometimes of ambiguous usefulness, probably due to his advertising model (at least at the time, when I was there earlier it seemed far less riddled with advertising). Edit: Nevermind about the advertising, it's still intrusive, I'm just running adblock now.
  - ryao 12y ago
    Phoronix does try to be honest about its methodology, but the person running benchmarks gives no thought to what his results mean, if anything. It shows in the distinct lack of documentation on why a given benchmark matters. As for Phoronix's ZFS benchmarks, the test hardware used drives that misreported themselves as having 512-byte sectors, which handicapped ZFS performance. Phoronix rejected all suggestions that it correct for this as end-users had been doing. Phoronix refused to meet half way by posting two results (one with proper configuration and one without), and also refused the suggestion that it to mention the existence of that problem in its test hardware. I eventually wrote code to identify drives known to misreport their sector sizes so that ZFS will automatically use the correct settings on them. That lead to the Phoronix August 2013 benchmarks showing a remarkable improvement in ZFS performance in FIO. It was so great that it sparked a discussion among the btrfs developers: http://comments.gmane.org/gmane.comp.file-systems.btrfs/27546 http://comments.gmane.org/gmane.comp.file-systems.btrfs/2754... Later that month, I publicly criticized Phoronix for posting misleading benchmarks: http://phoronix.com/forums/showthread.php?83731-ZFSOnLinux-0-6-2-released http://phoronix.com/forums/showthread.php?83731-ZFSOnLinux-0... Phoronix has not posted ZFS benchmarks since that time.

orkoden 12y ago

ZFS supports better compression algorithms. Those can even speed up slow disks.

ryao 12y ago

I would expect ZFS to outperform btrfs because of ARC and ZIL. ARC allows ZFS to increase the cache hit rate. ZIL permits ZFS to update its on-disk trees asynchronously. I am not aware of any analog to this in btrfs. Desktop users typically report that their systems become more responsive after switching to ZFS. The only case where I suspect btrfs might significantly outperform ZFS involves getdents() when the caches is cold. This is because ZoL does not presently do readahead on directory lookups. This has a noticeable impact on the performance of `ls` on the first access of a directory on mechanical storage, but it is not a problem for subsequent reads or on solid state storage. Prefetch support in getdents() will likely be implemented in either the next release or the one that follows.

deleted 12y ago

[deleted]

javanix 12y ago

Anecdotally I see better performance using ZoL than I did when I was running the same zpool and hardware on FreeBSD.

ryao 12y ago
This would suggest that the block device layer under ZFS and/or the VFS above it have room for improvement on FreeBSD.