3 ms·

Even on Solaris, ZFS doesn't have a great reputation for performance.

ryao 12y ago
Can you substantiate that? ZFS on Solaris produced excellent numbers back in 2008: https://blogs.oracle.com/brendan/entry/a_quarter_million_nfs_iops https://blogs.oracle.com/brendan/entry/a_quarter_million_nfs... More recent derivatives of OpenSolaris were reported to do 1.6 million IOPS last year: http://www.high-availability.com/high-availability-zfs-partner-nexenta-achieves-1-6-million-iops/ http://www.high-availability.com/high-availability-zfs-partn... Your mileage will vary, but ZFS has always had strong performance.
- mprovost 12y ago
  One of those is an 8 year old benchmark, the other used a 9.6TB flash array so hardly a realistic storage platform. I'm not saying that ZFS performance is awful, just that other features like data protection come first. If you're hoping for some salvation from btrfs you might be waiting a while. Of course they're both developed by the same company now! Here's a benchmark from 2013 with most results showing that it has worse performance on Linux than XFS and EXT4: http://www.phoronix.com/scan.php?page=article&item=zfs_linux_062&num=1 http://www.phoronix.com/scan.php?page=article&item=zfs_linux...
  - ryao 12y ago
    Oracle has nothing to do with the Open ZFS community these days. As for the benchmarks you cite, there are several key problems: 1. You say that they show ZFS having worse performance than XFS and ext4, but XFS is not in those benchmarks and the FIO tester shows ZFS as outperforming its competition by a significant margin. 2. They use a single disk. No server does this and while desktops and laptops do this, it is not clear how the benchmarks are of any relevance there. Additionally, LZ4 compression is not in use, when practically everyone deploying ZFSOnLinux would configure it to use LZ4. 3. ext4 manages to perform better than the theoretical limit of the SATA II interface and there is no discussion as to why. 4. ZFSOnLinux 0.6.2 is an old release. The most recent 0.6.3 release includes a new IO elevator and other improvements that enable ZFSOnLinux 0.6.3 to outperform its precedessor by a significant margin in many workloads. Would you post something constructive that you actually did yourself? I am beginning to think that you have never even used ZFS.
deelowe 12y ago
Compared to what exactly? It's consistently competitive against other similar FSes (XFS, JFS, etc..) When using caching, it's supposedly the best for database storage. And, feature wise, it beats everything else out there. As with most things, this isn't a binary decision, but typically ZFS is a good (if not the best) solution for most storage arrays.
- mprovost 12y ago
  Most people I know that are looking for performance on Linux stick with XFS, but then again people that are serious about Linux aren't running ZFS anyway. And of course there aren't any benchmarks for XFS vs ZFS on Solaris. Where I see consistent problems with ZFS performance is with metadata operations. Sure it has great features for managing disks and protecting data, but those are the primary feature set and performance has always taken a back seat.
  - ryao 12y ago
    Would you elaborate on what metadata operations are slow? If you had problems, I would like to know so that I could look into fixing them. Otherwise, I do not see anything actionable here. Your criticism so far contains zero verifiable claims. As for XFS, it is a single block device filesystem that relies on external shims to scale to multiple disks. ZFS can outscale it when various shims are put into place to allow multiple disks to be used. One user on freenode had difficulty getting good sequential performance from XFS + LVM + MD RAID 5 on 4 disks. He reported that he could not get better than 44MB/sec writes while ZFS managed 210MB/sec. I had a similar problem in 2011 with ext4 + LVM + MD RAID 6 on 6 disks. In that case, I could only manage 20MB/sec. It is why I am a contributor to the ZFSOnLinux project today. To make these anecdotes constructive, it would be nice if we had documentation on how to configure XFS + LVM + MD RAID 5/6 in a way that sequential performance does not suffer. In my case, my performance issue involved KVM/Xen guests. I never confirmed whether that user ran his tests on bare metal, but I suspect that he did. XFS' inability to scale past a single block device is not its only issue. Until a disk format change occurs that will add checksums, it has none to protect itself against corruption. When it gains them, it will not be able to do anything about that corruption when it detects it (aside from keeping the kernel from panicing) and it does nothing to protect your data. That said, ZFS has always focused on obtaining strong performance. This is why it has innovations like ZIL, ARC and L2ARC. The only instances where ZFS purposefully sacrifices performance is when getting a few extra percentage points means jeopardizing data integrity. There would have been no sense in developing ZFS as a replacement for existing filesystems if it did not keep data safe.

ryao 12y ago

Can you substantiate that? ZFS on Solaris produced excellent numbers back in 2008: https://blogs.oracle.com/brendan/entry/a_quarter_million_nfs_iops https://blogs.oracle.com/brendan/entry/a_quarter_million_nfs... More recent derivatives of OpenSolaris were reported to do 1.6 million IOPS last year: http://www.high-availability.com/high-availability-zfs-partner-nexenta-achieves-1-6-million-iops/ http://www.high-availability.com/high-availability-zfs-partn... Your mileage will vary, but ZFS has always had strong performance.

mprovost 12y ago
One of those is an 8 year old benchmark, the other used a 9.6TB flash array so hardly a realistic storage platform. I'm not saying that ZFS performance is awful, just that other features like data protection come first. If you're hoping for some salvation from btrfs you might be waiting a while. Of course they're both developed by the same company now! Here's a benchmark from 2013 with most results showing that it has worse performance on Linux than XFS and EXT4: http://www.phoronix.com/scan.php?page=article&item=zfs_linux_062&num=1 http://www.phoronix.com/scan.php?page=article&item=zfs_linux...
- ryao 12y ago
  Oracle has nothing to do with the Open ZFS community these days. As for the benchmarks you cite, there are several key problems: 1. You say that they show ZFS having worse performance than XFS and ext4, but XFS is not in those benchmarks and the FIO tester shows ZFS as outperforming its competition by a significant margin. 2. They use a single disk. No server does this and while desktops and laptops do this, it is not clear how the benchmarks are of any relevance there. Additionally, LZ4 compression is not in use, when practically everyone deploying ZFSOnLinux would configure it to use LZ4. 3. ext4 manages to perform better than the theoretical limit of the SATA II interface and there is no discussion as to why. 4. ZFSOnLinux 0.6.2 is an old release. The most recent 0.6.3 release includes a new IO elevator and other improvements that enable ZFSOnLinux 0.6.3 to outperform its precedessor by a significant margin in many workloads. Would you post something constructive that you actually did yourself? I am beginning to think that you have never even used ZFS.

deelowe 12y ago

Compared to what exactly? It's consistently competitive against other similar FSes (XFS, JFS, etc..) When using caching, it's supposedly the best for database storage. And, feature wise, it beats everything else out there. As with most things, this isn't a binary decision, but typically ZFS is a good (if not the best) solution for most storage arrays.

mprovost 12y ago
Most people I know that are looking for performance on Linux stick with XFS, but then again people that are serious about Linux aren't running ZFS anyway. And of course there aren't any benchmarks for XFS vs ZFS on Solaris. Where I see consistent problems with ZFS performance is with metadata operations. Sure it has great features for managing disks and protecting data, but those are the primary feature set and performance has always taken a back seat.
- ryao 12y ago
  Would you elaborate on what metadata operations are slow? If you had problems, I would like to know so that I could look into fixing them. Otherwise, I do not see anything actionable here. Your criticism so far contains zero verifiable claims. As for XFS, it is a single block device filesystem that relies on external shims to scale to multiple disks. ZFS can outscale it when various shims are put into place to allow multiple disks to be used. One user on freenode had difficulty getting good sequential performance from XFS + LVM + MD RAID 5 on 4 disks. He reported that he could not get better than 44MB/sec writes while ZFS managed 210MB/sec. I had a similar problem in 2011 with ext4 + LVM + MD RAID 6 on 6 disks. In that case, I could only manage 20MB/sec. It is why I am a contributor to the ZFSOnLinux project today. To make these anecdotes constructive, it would be nice if we had documentation on how to configure XFS + LVM + MD RAID 5/6 in a way that sequential performance does not suffer. In my case, my performance issue involved KVM/Xen guests. I never confirmed whether that user ran his tests on bare metal, but I suspect that he did. XFS' inability to scale past a single block device is not its only issue. Until a disk format change occurs that will add checksums, it has none to protect itself against corruption. When it gains them, it will not be able to do anything about that corruption when it detects it (aside from keeping the kernel from panicing) and it does nothing to protect your data. That said, ZFS has always focused on obtaining strong performance. This is why it has innovations like ZIL, ARC and L2ARC. The only instances where ZFS purposefully sacrifices performance is when getting a few extra percentage points means jeopardizing data integrity. There would have been no sense in developing ZFS as a replacement for existing filesystems if it did not keep data safe.