4 ms·
Nice article. I think you understate the importance of O_DIRECT, since it's the only way many kinds of software can work around the utter lameness of Linux loc
by notacoward 12y ago
Nice article. I think you understate the importance of O_DIRECT, since it's the only way many kinds of software can work around the utter lameness of Linux local filesystems' durability semantics, but other than that I think it's extremely informative and useful.
- otterley 12y agoO_DIRECT isn't about durability -- a broken device that falsely reports that data was committed will still be broken when O_DIRECT is used. O_DIRECT is about bypassing the kernel's buffer cache (and, presumably, zfs's ARC) so that the software can implement its own buffering.
- notacoward 12y agoBypassing the kernel's page cache (not buffer cache any more BTW) is not a complete durability story, but it's an essential part of one. Ditto for bypassing filesystems' own not-always-appropriate queuing/syncing behavior. There are other steps that are also necessary to make sure data really made it to disk, and the failure to expose the block device layer's flush behavior other than through all those other layers is part of the lameness I mentioned. A real I/O system would provide more flexible support for ordering, flushing, completion notification, etc. instead of just forcing everything into a model designed for the workloads and machine characteristics of twenty years ago.
- jfindley 12y agoAs another commenter noted it's not a complete durability solution. However it is useful in a number of places - and not just to do with durability. AIO is also very extensively used, and bmap is pretty important. It's really important when writing comparison pages like this to not play down the weaknesses of the thing you're trying to sell - it comes across badly. O_DIRECT, AIO and to a lesser extent bmap are feature that are required of a general purpose linux filesystem. I'm sure you'll have them all working in ZoL soon enough, but please don't try to pretend they aren't important. I'm also not sure why you have NFS in there - network filesystems are not local filesystems, and aren't generally going to have identical feature-sets. It feels a bit like you just stuck NFS in there so that there'd at least be something with more missing features. The same could kind of be said of JFS - I haven't heard of anyone using JFS on linux in a serious way in a very long time.
- ryao 12y agoAIO is in HEAD for the next release. I implemented it myself. As for bmap, I can say unequivocally that bmap will never be supported in ZFS. bmap is a nonstandard extension that is incompatible with filesystems that span multiple disks, incompatible with network filesystems that have no local disks and incompatible when doing writes to filesystems that support checksums. ZFS is disqualified from supporting bmap by virtue of #1 and #3. The same is true for others such as btrfs and NFS. As for O_DIRECT, there are no clearly defined semantics for O_DIRECT other than how it is implemented in XFS. My understanding of O_DIRECT is that the I/O is intended to go straight to/from the underlying disk. A true implementation of those semantics cannot be implemented in filesystem that spans multiple disks, CoW or checksums. A lesser form could probably be implemented where we try to make I/Os using O_DIRECT operate as if they were run with primarycache=metadata when primarycache=data and secondarycache=metadata when secondarycache=data. A third option would be to just ignore the O_DIRECT flag. A fourth would be to let users choose between the second and third. We might support that in the future, but I am inclined to think it will do more harm than good. As for why NFS is in the VFS API support chart, NFS is least common denominator for VFS API support that is widely deployed. Nearly all software that works on NFS can be known to work on other filesystems. Those using the chart for its intended purpose can quickly deduce whether software will likely run on ZFSOnLinux when they know that it already works on NFS. The sole exceptions are software that uses O_DIRECT or NFSv4 ACLs. The former can typically be toggled on/off in software that supports it while the latter can be dismissed with the knowledge that it runs on NFSv3 (or any other Linux filesystem at the present time).