5 ms·

I wrote the above article. I will try to answer any questions that I see here.

by ryao 12y ago

notacoward 12y ago
Nice article. I think you understate the importance of O_DIRECT, since it's the only way many kinds of software can work around the utter lameness of Linux local filesystems' durability semantics, but other than that I think it's extremely informative and useful.
- otterley 12y ago
  O_DIRECT isn't about durability -- a broken device that falsely reports that data was committed will still be broken when O_DIRECT is used. O_DIRECT is about bypassing the kernel's buffer cache (and, presumably, zfs's ARC) so that the software can implement its own buffering.
  - notacoward 12y ago
    Bypassing the kernel's page cache (not buffer cache any more BTW) is not a complete durability story, but it's an essential part of one. Ditto for bypassing filesystems' own not-always-appropriate queuing/syncing behavior. There are other steps that are also necessary to make sure data really made it to disk, and the failure to expose the block device layer's flush behavior other than through all those other layers is part of the lameness I mentioned. A real I/O system would provide more flexible support for ordering, flushing, completion notification, etc. instead of just forcing everything into a model designed for the workloads and machine characteristics of twenty years ago.
- jfindley 12y ago
  As another commenter noted it's not a complete durability solution. However it is useful in a number of places - and not just to do with durability. AIO is also very extensively used, and bmap is pretty important. It's really important when writing comparison pages like this to not play down the weaknesses of the thing you're trying to sell - it comes across badly. O_DIRECT, AIO and to a lesser extent bmap are feature that are required of a general purpose linux filesystem. I'm sure you'll have them all working in ZoL soon enough, but please don't try to pretend they aren't important. I'm also not sure why you have NFS in there - network filesystems are not local filesystems, and aren't generally going to have identical feature-sets. It feels a bit like you just stuck NFS in there so that there'd at least be something with more missing features. The same could kind of be said of JFS - I haven't heard of anyone using JFS on linux in a serious way in a very long time.
  - ryao 12y ago
    AIO is in HEAD for the next release. I implemented it myself. As for bmap, I can say unequivocally that bmap will never be supported in ZFS. bmap is a nonstandard extension that is incompatible with filesystems that span multiple disks, incompatible with network filesystems that have no local disks and incompatible when doing writes to filesystems that support checksums. ZFS is disqualified from supporting bmap by virtue of #1 and #3. The same is true for others such as btrfs and NFS. As for O_DIRECT, there are no clearly defined semantics for O_DIRECT other than how it is implemented in XFS. My understanding of O_DIRECT is that the I/O is intended to go straight to/from the underlying disk. A true implementation of those semantics cannot be implemented in filesystem that spans multiple disks, CoW or checksums. A lesser form could probably be implemented where we try to make I/Os using O_DIRECT operate as if they were run with primarycache=metadata when primarycache=data and secondarycache=metadata when secondarycache=data. A third option would be to just ignore the O_DIRECT flag. A fourth would be to let users choose between the second and third. We might support that in the future, but I am inclined to think it will do more harm than good. As for why NFS is in the VFS API support chart, NFS is least common denominator for VFS API support that is widely deployed. Nearly all software that works on NFS can be known to work on other filesystems. Those using the chart for its intended purpose can quickly deduce whether software will likely run on ZFSOnLinux when they know that it already works on NFS. The sole exceptions are software that uses O_DIRECT or NFSv4 ACLs. The former can typically be toggled on/off in software that supports it while the latter can be dismissed with the knowledge that it runs on NFSv3 (or any other Linux filesystem at the present time).
justincormack 12y ago
I can't read the tables on my phone as can't shrink them down (Firefox Android)... So as you don't seem to spell it out is it the case that SElinux should be fine on ZFS but PAX might not be?
- ryao 12y ago
  They both work, with two caveats: 1. ZoL is currently incompatible with kernels built with either PAX_USERCOPY_SLABS or PAX_KERNEXEC_PLUGIN_METHOD_OR. The next release should fix PAX_USERCOPY_SLABS and might fix PAX_KERNEXEC_PLUGIN_METHOD_OR. The former should be fixed by a kmem-rework that the project plans to merge before the next release: https://github.com/zfsonlinux/spl/pull/369 https://github.com/zfsonlinux/spl/pull/369 https://github.com/zfsonlinux/zfs/pull/2411 https://github.com/zfsonlinux/zfs/pull/2411 The fix requires both pull requests, but I have yet to refresh the second pull request with the fix. I plan to refresh it before I leave for LinuxCon Europe 2014. The latter might be fixed by changes to drop support for pre-2.6.32 kernels that the project lead would prefer to merge for the next release: https://github.com/zfsonlinux/spl/pull/395 https://github.com/zfsonlinux/spl/pull/395 2. There is a bug specific to ZoL where unlinking a file with a directory extended attribute (Solaris-style) will trigger a full traversal of all objects (i.e. inodes) within a dataset. It manifests itself as the zfs_iput_taskq kernle thread consuming excessive CPU time. It is fixed for the next release: https://github.com/zfsonlinux/zfs/commit/0d5c500d6cf4dd9e133cbffd82440c95d89478a4 https://github.com/zfsonlinux/zfs/commit/0d5c500d6cf4dd9e133...
purplerails 12y ago
According to this article, ZFS is sub-optimal for databases due to fragmentation of the db files. Could you please comment? Thx. http://bartsjerps.wordpress.com/2013/02/26/zfs-ora-database-fragmentation/ http://bartsjerps.wordpress.com/2013/02/26/zfs-ora-database-...
- nabla9 12y ago
  Oracle advice on how to configure ZFS and Oracle Database http://www.oracle.com/technetwork/server-storage/solaris/config-solaris-zfs-wp-167894.pdf http://www.oracle.com/technetwork/server-storage/solaris/con... Some picks: Free space is important 10-20%. >In an environment with a high rate of data update (or churn), it is advisable to maintain a certain amount of free space. ZFS is a copy-on-write file system and relocates all writes to free disk space. Keeping a certain amount of free blocks allows ZFS to easily find space in big chunks, allows ZFS to aggregate writes and reduce the write IOPS demand on hard disks. Streaming aggregated writes can be 50-100 times less demanding on hard drives than doing small random writes. Therefore, the incentive for keeping free space is high, especially in a high churn environment, such as an active OLTP database. >The number one rule for setting up an Oracle database on ZFS is to set ZFS recordsize equal to the database block size for the file systems that contain the Oracle data files. recordsize = db_block_size
jwildeboer 12y ago
Who offers production Level support? Certification? No one AFAICS. I wonder why you didn't mention that.
- ryao 12y ago
  It was not the purpose of the blog post. Anyway, you mentioned a couple of different things here. First, there are a few companies that offer production level support. SoftNAS is one. For full disclosure, my 2014 tax return will include consulting fees earned from fixing bugs for SoftNAS. It is in use at various large companies, including Netflix and Boeing: http://www.softnas.com/ http://www.softnas.com/ Another is PogoStor, which advertises that the hardware that they sell has been certified by Nexenta for use with ZFS: http://www.pogostorage.com/products/nexenta/overview/whycertified.php http://www.pogostorage.com/products/nexenta/overview/whycert... They also have their own Linux distribution based on ZoL, for which I assume that they offer commercial support: http://www.pogostorage.com/products/pogostor/pogostor-overview.php http://www.pogostorage.com/products/pogostor/pogostor-overvi... As for certification, it is not clear if you mean hardware certification or system administrator certification. I am not aware of any filesystems that have system administrator certifications. As for hardware certifications, ZFS is designed for use with commodity hardware. It probably does not make sense to try to certify it for the full range of hardware it supports (although a subset is certified by Nexenta). Specific hardware guidelines are available on the official Open ZFS wiki: http://open-zfs.org/wiki/Hardware http://open-zfs.org/wiki/Hardware
  - stock_toaster 12y ago
    ixsystems also provides support for ZFS on FreeNAS, which is based on FreeBSD.

I wrote the above article. I will try to answer any questions that I see here.

by ryao 12y ago

I wrote the above article. I will try to answer any questions that I see here.

Nice article. I think you understate the importance of O_DIRECT, since it's the only way many kinds of software can work around the utter lameness of Linux local filesystems' durability semantics, but other than that I think it's extremely informative and useful.

otterley 12y ago
O_DIRECT isn't about durability -- a broken device that falsely reports that data was committed will still be broken when O_DIRECT is used. O_DIRECT is about bypassing the kernel's buffer cache (and, presumably, zfs's ARC) so that the software can implement its own buffering.
- notacoward 12y ago
  Bypassing the kernel's page cache (not buffer cache any more BTW) is not a complete durability story, but it's an essential part of one. Ditto for bypassing filesystems' own not-always-appropriate queuing/syncing behavior. There are other steps that are also necessary to make sure data really made it to disk, and the failure to expose the block device layer's flush behavior other than through all those other layers is part of the lameness I mentioned. A real I/O system would provide more flexible support for ordering, flushing, completion notification, etc. instead of just forcing everything into a model designed for the workloads and machine characteristics of twenty years ago.
jfindley 12y ago
As another commenter noted it's not a complete durability solution. However it is useful in a number of places - and not just to do with durability. AIO is also very extensively used, and bmap is pretty important. It's really important when writing comparison pages like this to not play down the weaknesses of the thing you're trying to sell - it comes across badly. O_DIRECT, AIO and to a lesser extent bmap are feature that are required of a general purpose linux filesystem. I'm sure you'll have them all working in ZoL soon enough, but please don't try to pretend they aren't important. I'm also not sure why you have NFS in there - network filesystems are not local filesystems, and aren't generally going to have identical feature-sets. It feels a bit like you just stuck NFS in there so that there'd at least be something with more missing features. The same could kind of be said of JFS - I haven't heard of anyone using JFS on linux in a serious way in a very long time.
- ryao 12y ago
  AIO is in HEAD for the next release. I implemented it myself. As for bmap, I can say unequivocally that bmap will never be supported in ZFS. bmap is a nonstandard extension that is incompatible with filesystems that span multiple disks, incompatible with network filesystems that have no local disks and incompatible when doing writes to filesystems that support checksums. ZFS is disqualified from supporting bmap by virtue of #1 and #3. The same is true for others such as btrfs and NFS. As for O_DIRECT, there are no clearly defined semantics for O_DIRECT other than how it is implemented in XFS. My understanding of O_DIRECT is that the I/O is intended to go straight to/from the underlying disk. A true implementation of those semantics cannot be implemented in filesystem that spans multiple disks, CoW or checksums. A lesser form could probably be implemented where we try to make I/Os using O_DIRECT operate as if they were run with primarycache=metadata when primarycache=data and secondarycache=metadata when secondarycache=data. A third option would be to just ignore the O_DIRECT flag. A fourth would be to let users choose between the second and third. We might support that in the future, but I am inclined to think it will do more harm than good. As for why NFS is in the VFS API support chart, NFS is least common denominator for VFS API support that is widely deployed. Nearly all software that works on NFS can be known to work on other filesystems. Those using the chart for its intended purpose can quickly deduce whether software will likely run on ZFSOnLinux when they know that it already works on NFS. The sole exceptions are software that uses O_DIRECT or NFSv4 ACLs. The former can typically be toggled on/off in software that supports it while the latter can be dismissed with the knowledge that it runs on NFSv3 (or any other Linux filesystem at the present time).

justincormack 12y ago

I can't read the tables on my phone as can't shrink them down (Firefox Android)... So as you don't seem to spell it out is it the case that SElinux should be fine on ZFS but PAX might not be?

ryao 12y ago
They both work, with two caveats: 1. ZoL is currently incompatible with kernels built with either PAX_USERCOPY_SLABS or PAX_KERNEXEC_PLUGIN_METHOD_OR. The next release should fix PAX_USERCOPY_SLABS and might fix PAX_KERNEXEC_PLUGIN_METHOD_OR. The former should be fixed by a kmem-rework that the project plans to merge before the next release: https://github.com/zfsonlinux/spl/pull/369 https://github.com/zfsonlinux/spl/pull/369 https://github.com/zfsonlinux/zfs/pull/2411 https://github.com/zfsonlinux/zfs/pull/2411 The fix requires both pull requests, but I have yet to refresh the second pull request with the fix. I plan to refresh it before I leave for LinuxCon Europe 2014. The latter might be fixed by changes to drop support for pre-2.6.32 kernels that the project lead would prefer to merge for the next release: https://github.com/zfsonlinux/spl/pull/395 https://github.com/zfsonlinux/spl/pull/395 2. There is a bug specific to ZoL where unlinking a file with a directory extended attribute (Solaris-style) will trigger a full traversal of all objects (i.e. inodes) within a dataset. It manifests itself as the zfs_iput_taskq kernle thread consuming excessive CPU time. It is fixed for the next release: https://github.com/zfsonlinux/zfs/commit/0d5c500d6cf4dd9e133cbffd82440c95d89478a4 https://github.com/zfsonlinux/zfs/commit/0d5c500d6cf4dd9e133...

purplerails 12y ago

According to this article, ZFS is sub-optimal for databases due to fragmentation of the db files. Could you please comment? Thx. http://bartsjerps.wordpress.com/2013/02/26/zfs-ora-database-fragmentation/ http://bartsjerps.wordpress.com/2013/02/26/zfs-ora-database-...

nabla9 12y ago
Oracle advice on how to configure ZFS and Oracle Database http://www.oracle.com/technetwork/server-storage/solaris/config-solaris-zfs-wp-167894.pdf http://www.oracle.com/technetwork/server-storage/solaris/con... Some picks: Free space is important 10-20%. >In an environment with a high rate of data update (or churn), it is advisable to maintain a certain amount of free space. ZFS is a copy-on-write file system and relocates all writes to free disk space. Keeping a certain amount of free blocks allows ZFS to easily find space in big chunks, allows ZFS to aggregate writes and reduce the write IOPS demand on hard disks. Streaming aggregated writes can be 50-100 times less demanding on hard drives than doing small random writes. Therefore, the incentive for keeping free space is high, especially in a high churn environment, such as an active OLTP database. >The number one rule for setting up an Oracle database on ZFS is to set ZFS recordsize equal to the database block size for the file systems that contain the Oracle data files. recordsize = db_block_size

jwildeboer 12y ago

Who offers production Level support? Certification? No one AFAICS. I wonder why you didn't mention that.

ryao 12y ago
It was not the purpose of the blog post. Anyway, you mentioned a couple of different things here. First, there are a few companies that offer production level support. SoftNAS is one. For full disclosure, my 2014 tax return will include consulting fees earned from fixing bugs for SoftNAS. It is in use at various large companies, including Netflix and Boeing: http://www.softnas.com/ http://www.softnas.com/ Another is PogoStor, which advertises that the hardware that they sell has been certified by Nexenta for use with ZFS: http://www.pogostorage.com/products/nexenta/overview/whycertified.php http://www.pogostorage.com/products/nexenta/overview/whycert... They also have their own Linux distribution based on ZoL, for which I assume that they offer commercial support: http://www.pogostorage.com/products/pogostor/pogostor-overview.php http://www.pogostorage.com/products/pogostor/pogostor-overvi... As for certification, it is not clear if you mean hardware certification or system administrator certification. I am not aware of any filesystems that have system administrator certifications. As for hardware certifications, ZFS is designed for use with commodity hardware. It probably does not make sense to try to certify it for the full range of hardware it supports (although a subset is certified by Nexenta). Specific hardware guidelines are available on the official Open ZFS wiki: http://open-zfs.org/wiki/Hardware http://open-zfs.org/wiki/Hardware
- stock_toaster 12y ago
  ixsystems also provides support for ZFS on FreeNAS, which is based on FreeBSD.