9 ms·

Is ZFS a suitable replacement for other Linux filesystems?

70 points by lewq 12y ago

ryao 12y ago
I wrote the above article. I will try to answer any questions that I see here.
- notacoward 12y ago
  Nice article. I think you understate the importance of O_DIRECT, since it's the only way many kinds of software can work around the utter lameness of Linux local filesystems' durability semantics, but other than that I think it's extremely informative and useful.
  - otterley 12y ago
    O_DIRECT isn't about durability -- a broken device that falsely reports that data was committed will still be broken when O_DIRECT is used. O_DIRECT is about bypassing the kernel's buffer cache (and, presumably, zfs's ARC) so that the software can implement its own buffering.
    - notacoward 12y ago
      Bypassing the kernel's page cache (not buffer cache any more BTW) is not a complete durability story, but it's an essential part of one. Ditto for bypassing filesystems' own not-always-appropriate queuing/syncing behavior. There are other steps that are also necessary to make sure data really made it to disk, and the failure to expose the block device layer's flush behavior other than through all those other layers is part of the lameness I mentioned. A real I/O system would provide more flexible support for ordering, flushing, completion notification, etc. instead of just forcing everything into a model designed for the workloads and machine characteristics of twenty years ago.
  - jfindley 12y ago
    As another commenter noted it's not a complete durability solution. However it is useful in a number of places - and not just to do with durability. AIO is also very extensively used, and bmap is pretty important. It's really important when writing comparison pages like this to not play down the weaknesses of the thing you're trying to sell - it comes across badly. O_DIRECT, AIO and to a lesser extent bmap are feature that are required of a general purpose linux filesystem. I'm sure you'll have them all working in ZoL soon enough, but please don't try to pretend they aren't important. I'm also not sure why you have NFS in there - network filesystems are not local filesystems, and aren't generally going to have identical feature-sets. It feels a bit like you just stuck NFS in there so that there'd at least be something with more missing features. The same could kind of be said of JFS - I haven't heard of anyone using JFS on linux in a serious way in a very long time.
    - ryao 12y ago
      AIO is in HEAD for the next release. I implemented it myself. As for bmap, I can say unequivocally that bmap will never be supported in ZFS. bmap is a nonstandard extension that is incompatible with filesystems that span multiple disks, incompatible with network filesystems that have no local disks and incompatible when doing writes to filesystems that support checksums. ZFS is disqualified from supporting bmap by virtue of #1 and #3. The same is true for others such as btrfs and NFS. As for O_DIRECT, there are no clearly defined semantics for O_DIRECT other than how it is implemented in XFS. My understanding of O_DIRECT is that the I/O is intended to go straight to/from the underlying disk. A true implementation of those semantics cannot be implemented in filesystem that spans multiple disks, CoW or checksums. A lesser form could probably be implemented where we try to make I/Os using O_DIRECT operate as if they were run with primarycache=metadata when primarycache=data and secondarycache=metadata when secondarycache=data. A third option would be to just ignore the O_DIRECT flag. A fourth would be to let users choose between the second and third. We might support that in the future, but I am inclined to think it will do more harm than good. As for why NFS is in the VFS API support chart, NFS is least common denominator for VFS API support that is widely deployed. Nearly all software that works on NFS can be known to work on other filesystems. Those using the chart for its intended purpose can quickly deduce whether software will likely run on ZFSOnLinux when they know that it already works on NFS. The sole exceptions are software that uses O_DIRECT or NFSv4 ACLs. The former can typically be toggled on/off in software that supports it while the latter can be dismissed with the knowledge that it runs on NFSv3 (or any other Linux filesystem at the present time).
- justincormack 12y ago
  I can't read the tables on my phone as can't shrink them down (Firefox Android)... So as you don't seem to spell it out is it the case that SElinux should be fine on ZFS but PAX might not be?
  - ryao 12y ago
    They both work, with two caveats: 1. ZoL is currently incompatible with kernels built with either PAX_USERCOPY_SLABS or PAX_KERNEXEC_PLUGIN_METHOD_OR. The next release should fix PAX_USERCOPY_SLABS and might fix PAX_KERNEXEC_PLUGIN_METHOD_OR. The former should be fixed by a kmem-rework that the project plans to merge before the next release: https://github.com/zfsonlinux/spl/pull/369 https://github.com/zfsonlinux/spl/pull/369 https://github.com/zfsonlinux/zfs/pull/2411 https://github.com/zfsonlinux/zfs/pull/2411 The fix requires both pull requests, but I have yet to refresh the second pull request with the fix. I plan to refresh it before I leave for LinuxCon Europe 2014. The latter might be fixed by changes to drop support for pre-2.6.32 kernels that the project lead would prefer to merge for the next release: https://github.com/zfsonlinux/spl/pull/395 https://github.com/zfsonlinux/spl/pull/395 2. There is a bug specific to ZoL where unlinking a file with a directory extended attribute (Solaris-style) will trigger a full traversal of all objects (i.e. inodes) within a dataset. It manifests itself as the zfs_iput_taskq kernle thread consuming excessive CPU time. It is fixed for the next release: https://github.com/zfsonlinux/zfs/commit/0d5c500d6cf4dd9e133cbffd82440c95d89478a4 https://github.com/zfsonlinux/zfs/commit/0d5c500d6cf4dd9e133...
- purplerails 12y ago
  According to this article, ZFS is sub-optimal for databases due to fragmentation of the db files. Could you please comment? Thx. http://bartsjerps.wordpress.com/2013/02/26/zfs-ora-database-fragmentation/ http://bartsjerps.wordpress.com/2013/02/26/zfs-ora-database-...
  - nabla9 12y ago
    Oracle advice on how to configure ZFS and Oracle Database http://www.oracle.com/technetwork/server-storage/solaris/config-solaris-zfs-wp-167894.pdf http://www.oracle.com/technetwork/server-storage/solaris/con... Some picks: Free space is important 10-20%. >In an environment with a high rate of data update (or churn), it is advisable to maintain a certain amount of free space. ZFS is a copy-on-write file system and relocates all writes to free disk space. Keeping a certain amount of free blocks allows ZFS to easily find space in big chunks, allows ZFS to aggregate writes and reduce the write IOPS demand on hard disks. Streaming aggregated writes can be 50-100 times less demanding on hard drives than doing small random writes. Therefore, the incentive for keeping free space is high, especially in a high churn environment, such as an active OLTP database. >The number one rule for setting up an Oracle database on ZFS is to set ZFS recordsize equal to the database block size for the file systems that contain the Oracle data files. recordsize = db_block_size
- jwildeboer 12y ago
  Who offers production Level support? Certification? No one AFAICS. I wonder why you didn't mention that.
  - ryao 12y ago
    It was not the purpose of the blog post. Anyway, you mentioned a couple of different things here. First, there are a few companies that offer production level support. SoftNAS is one. For full disclosure, my 2014 tax return will include consulting fees earned from fixing bugs for SoftNAS. It is in use at various large companies, including Netflix and Boeing: http://www.softnas.com/ http://www.softnas.com/ Another is PogoStor, which advertises that the hardware that they sell has been certified by Nexenta for use with ZFS: http://www.pogostorage.com/products/nexenta/overview/whycertified.php http://www.pogostorage.com/products/nexenta/overview/whycert... They also have their own Linux distribution based on ZoL, for which I assume that they offer commercial support: http://www.pogostorage.com/products/pogostor/pogostor-overview.php http://www.pogostorage.com/products/pogostor/pogostor-overvi... As for certification, it is not clear if you mean hardware certification or system administrator certification. I am not aware of any filesystems that have system administrator certifications. As for hardware certifications, ZFS is designed for use with commodity hardware. It probably does not make sense to try to certify it for the full range of hardware it supports (although a subset is certified by Nexenta). Specific hardware guidelines are available on the official Open ZFS wiki: http://open-zfs.org/wiki/Hardware http://open-zfs.org/wiki/Hardware
    - stock_toaster 12y ago
      ixsystems also provides support for ZFS on FreeNAS, which is based on FreeBSD.
Thaxll 12y ago
No.
- ferrantim 12y ago
  Care to elaborate?
  - valarauca1 12y ago
    Zpool is a great feature. On top of the dozen(s) of amazing features in ZFS. The problem is, and ZFS admits this: If you don't run ECC ram Zpool can "accidentally" your whole hard drive (bit of a joke there, it can corrupt your drive as it attempts to correct bit rot that never happened). This happens a lot more often then we really care to think about. (Ram corruption) For most day-to-day linux users who are just using standard consumer desktop/laptops they don't have this protection. And ZFS can do more harm then good. References: https://pthree.org/2013/12/10/zfs-administration-appendix-c-why-you-should-use-ecc-ram/ https://pthree.org/2013/12/10/zfs-administration-appendix-c-...
    - ryao 12y ago
      This is a myth started by someone who did not understand how filesystems work. Not having ECC memory is no more of a handicap for ZFS than it is for any other filesystem.
      - valarauca1 12y ago
        Care to elaborate? ZFS warned me of that while I reading its installation guide. Which is why I never installed it.
        sp332 12y ago
        If you're doing RAID, you should have ECC. All hardware RAID controllers have ECC cache, and if you're doing software RAID, you should have ECC system RAM. But this is just common sense, I guess, since it's useless to write data to redundant storage if it's already been bit-flipped in RAM.
Qantourisc 12y ago
No, it doesn't has shrink, a future you want on a desktop imo. And occasionally on server when the customer doesn't provide enough storage (on the account of being cheap) and you have move things around to get things done.
icantthinkofone 12y ago
I'm a bit confused. I always thought that proper use of ZFS required multiple drives just like RAID. Then I read an article where RAID was used with ZFS. Now I'm really confused so I'm taking the lazy way and asking here: is ZFS appropriate for a single drive system?
- Qantourisc 12y ago
  Sure, COW(copy on write), snapshots, error checksum on files, compression, support for SSD's. You just don't have redundancy.
  - FooBarWidget 12y ago
    I thought the error checking (scrubbing) only works if you have multiple drives set up in a RAID.
    - sp332 12y ago
      ZFS writes a checksum when it writes data. So you can tell whether your data is corrupted, even if you don't have the redundancy to restore the corrupted data.
    - prakashsurya 12y ago
      Error detection can always be used, but error correction may or may not be available (it depends on the type of block). Metadata blocks are redundant even on a single drive pool; so if you just have a partial failure (e.g. overwrite a metadata block) it might be able to correct the block using another redundant copy on the same drive. Data blocks will require a redundant pool configuration, though, as these are not store redundantly by default (e.g. multiple drives in a raidz or mirror).
      - mrb 12y ago
        Actually you can correct data blocks, on a single-drive ZFS pool, if you set the attribute copies=2 (ditto blocks): https://blogs.oracle.com/relling/entry/zfs_copies_and_data_protection https://blogs.oracle.com/relling/entry/zfs_copies_and_data_p... Obviously this redundancy feature makes you use twice the disk space you would normally use.
ChrisInEdmonton 12y ago
I have heard that ZFS (at least, ZFS on Linux) requires about 1 GB of memory for every 1 TB of storage. Is this an accurate statement? That's certainly a critical flaw for many use cases, though completely irrelevant for many others. If accurate, what's behind this requirement?
- dedward 12y ago
  All the extra stuff ZFS does, and does performantly, comes at a cost. That's the cost (I can't vouch for the amount.. but the principle is right. CPU power matters too.) You don't get all those features for free... but you do get them.
  - ryao 12y ago
    While this is true, ZFSOnLinux has made progress on reducing CPU overhead. Additional improvements will be in the next release. In particular, the performance of NFS exports of ZFS datasets will increase considerably (some are seeing a factor of 2 increase) while CPU utilization has dropped: https://github.com/zfsonlinux/spl/pull/369#issuecomment-58398544 https://github.com/zfsonlinux/spl/pull/369#issuecomment-5839...
- Already__Taken 12y ago
  another anecdote: I've only heard this mentioned specifically about using de-duplication.
- prakashsurya 12y ago
  No, that's flat out not true. I've seen that metric thrown around when talking about the "dedup" feature of ZFS, but honestly, don't use dedup unless you know what you're doing. It's way to easy for things to go wrong otherwise.
  - ChrisInEdmonton 12y ago
    Ah ha! Thanks. Data deduplication. Well, yeah, that makes sense to me. Thank you very much for clearing that up.
dedward 12y ago
This should be fairly obvious... ZFS has it's own requirements... to work well it needs lots of resources (cpu/ram) and EEC memory. If you have those, then sure, it gives you many advantages. If you don't, then you are opening yourself up for struggle.
- atoponce 12y ago
  ZFS doesn't need any more "resources" than any other kernel space filesystem. Deduplication on the other hand, does need a good amount of RAM, if you want to prevent the deduplication table (DDT) from spilling to slow spinning rust. However, if you have a fast, and large SSD L2ARC, the DDT spilling over to the L2ARC might not have drastic consequences. Regardless, unless using the DDT feature of ZFS, it doesn't need anymore resources than ext4 or XFS.
po 12y ago
I found an old ZFS disk laying around the other day (a few years old.. from when I was playing with zevo community edition ZFS and I had forgotten what I had stored on it) and I was interested in trying https://openzfsonosx.org https://openzfsonosx.org so I installed it and mounted my disk up. It worked like a charm. I was pretty impressed. I kind of screwed myself over though because I to do a bunch of file cleanup that I was hoping to perhaps roll back if it didn't go well so I created a snapshot and then got to work with some copying… I wasn't paying attention and I ended up running out of space on the disk. The way ZFS snapshots work is that ZFS still does what it would normally do on a copy: writes data, then verifies the write is completed before deleting the old data. When there is a snapshot, it just skips the second step and leaves the old data around. This is how snapshots have zero performance impact on data write. Turns out that I couldn't even delete the old snapshot (or rollback the new one) because I didn't have enough space to make any changes to the disk. I couldn't even delete files because they would stay in the snapshot. Aha! I can add a disk to the pool! So I added a small disk to the zpool and then I could delete the snapshot ok. However, I forgot that you can't remove a disk from a zpool. I ended up having to move the data to another disk, drop the entire zpool and recreate it, and move it back. Honestly, everything worked rather well except for forgetting a few of the limitations of ZFS. Given how often I have issues with HFS+ I really think it's pretty tempting for OS X.
- prakashsurya 12y ago
  There is work in progress to add the ability to remove a device from a pool. So, hopefully in the not too distant future, that work will land in illumos and migrate to all the downstream implementations (e.g. linux, mac, and freebsd).
- ryao 12y ago
  This issue was fixed in Illumos a few months ago, although the fix has not made its way to O3X yet: https://github.com/illumos/illumos-gate/commit/4bb73804952172060c9efb163b89c17f56804fe8 https://github.com/illumos/illumos-gate/commit/4bb7380495217... The workaround is to find a file that was written that is outside of a snapshot and truncate it. `zfs diff` can be used to do that. That allows you to delete an old snapshot.
antman 12y ago
I have not used zfs because when I read about it I found the expand-contract procedure overly complicated. (1) Caveats when planning for expansion should be addressed. (1) http://arstechnica.com/information-technology/2014/02/ars-walkthrough-using-the-zfs-next-gen-filesystem-on-linux/ http://arstechnica.com/information-technology/2014/02/ars-wa...
ryanqian 12y ago
zfs is a great FS, which add many modern concept such as zpool, and also put volume manager and fs together. storage and fs should be easy as this way. and it's in good quality on Solaris, I miss Solaris.
- snw 12y ago
  try one of the Illumos distributions like OmniOS or SmartOS. The legacy of open-solaris is well and alive.
robmccoll 12y ago
For private use, yes. I've been using ZoL for over a year on my home server with a simple ZFS raid pool and a number of volumes. Relatively stable. Snapshots every 15 minutes or so. Other than a few briefly scary kernel upgrades - nothing actually broke, just had to rebuild modules. Anytime your system comes up without pools mounted, it's scary. If only the ZFS license were compatible. Anyway, those weren't real breakages, so no problems.
deleted 12y ago
[deleted]
aidenn0 12y ago
Can anyone point me to a guide for migrating from software RAID to ZoL?
- ryao 12y ago
  I suggest emailing the zfs-discuss mailing list for this one: http://zfsonlinux.org/lists.html http://zfsonlinux.org/lists.html The answer to your question depends on your existing setup.
barrkel 12y ago
ZFS on Linux requires more memory than most other FSes, due to the Solaris compatibility layer. The cache is a bit less responsive to low memory conditions. I run it on a NAS with 4G of memory, and I've had to limit usage and flush caches occasionally, otherwise I get to the point that I can't run new programs.
higherpurpose 12y ago
I think F2FS is.
jeffdavis 12y ago
The license is pretty concerning. If anyone using ZFS sues Oracle for a patent issue, then Oracle can revoke their license to use ZFS in 60 days. (Not a lawyer, my interpretation might be wrong.) Probably OK for home use, but not much more. Even a small startup with no patents could end up getting bought by a larger company that does have patents. Then that company could be vulnerable, perhaps even if they are trying to use their patents defensively.
olavgg 12y ago
Has anyone tried ZFS together with Hadoop/HDFS? I find it to be a really interesting combination because of compression and self-healing of bad data-blocks.
erikb 12y ago
Not being an expert in this kind of area, I would really like to read a short summary. Is there one and I just have missed it, somewhere in the article? I didn't see a headline like summary or conclusions.
- ryao 12y ago
  Nearly all software works on ZoL. A small number of applications will likely need configuration of the application(s) to turn off AIO (until the next release), configuration of the application(s) to turn off DirectIO, or setting acltype=posix on the dataset to provide POSIX ACLs.

70 points by lewq 12y ago

ryao 12y ago

I wrote the above article. I will try to answer any questions that I see here.

notacoward 12y ago
Nice article. I think you understate the importance of O_DIRECT, since it's the only way many kinds of software can work around the utter lameness of Linux local filesystems' durability semantics, but other than that I think it's extremely informative and useful.
- otterley 12y ago
  O_DIRECT isn't about durability -- a broken device that falsely reports that data was committed will still be broken when O_DIRECT is used. O_DIRECT is about bypassing the kernel's buffer cache (and, presumably, zfs's ARC) so that the software can implement its own buffering.
  - notacoward 12y ago
    Bypassing the kernel's page cache (not buffer cache any more BTW) is not a complete durability story, but it's an essential part of one. Ditto for bypassing filesystems' own not-always-appropriate queuing/syncing behavior. There are other steps that are also necessary to make sure data really made it to disk, and the failure to expose the block device layer's flush behavior other than through all those other layers is part of the lameness I mentioned. A real I/O system would provide more flexible support for ordering, flushing, completion notification, etc. instead of just forcing everything into a model designed for the workloads and machine characteristics of twenty years ago.
- jfindley 12y ago
  As another commenter noted it's not a complete durability solution. However it is useful in a number of places - and not just to do with durability. AIO is also very extensively used, and bmap is pretty important. It's really important when writing comparison pages like this to not play down the weaknesses of the thing you're trying to sell - it comes across badly. O_DIRECT, AIO and to a lesser extent bmap are feature that are required of a general purpose linux filesystem. I'm sure you'll have them all working in ZoL soon enough, but please don't try to pretend they aren't important. I'm also not sure why you have NFS in there - network filesystems are not local filesystems, and aren't generally going to have identical feature-sets. It feels a bit like you just stuck NFS in there so that there'd at least be something with more missing features. The same could kind of be said of JFS - I haven't heard of anyone using JFS on linux in a serious way in a very long time.
  - ryao 12y ago
    AIO is in HEAD for the next release. I implemented it myself. As for bmap, I can say unequivocally that bmap will never be supported in ZFS. bmap is a nonstandard extension that is incompatible with filesystems that span multiple disks, incompatible with network filesystems that have no local disks and incompatible when doing writes to filesystems that support checksums. ZFS is disqualified from supporting bmap by virtue of #1 and #3. The same is true for others such as btrfs and NFS. As for O_DIRECT, there are no clearly defined semantics for O_DIRECT other than how it is implemented in XFS. My understanding of O_DIRECT is that the I/O is intended to go straight to/from the underlying disk. A true implementation of those semantics cannot be implemented in filesystem that spans multiple disks, CoW or checksums. A lesser form could probably be implemented where we try to make I/Os using O_DIRECT operate as if they were run with primarycache=metadata when primarycache=data and secondarycache=metadata when secondarycache=data. A third option would be to just ignore the O_DIRECT flag. A fourth would be to let users choose between the second and third. We might support that in the future, but I am inclined to think it will do more harm than good. As for why NFS is in the VFS API support chart, NFS is least common denominator for VFS API support that is widely deployed. Nearly all software that works on NFS can be known to work on other filesystems. Those using the chart for its intended purpose can quickly deduce whether software will likely run on ZFSOnLinux when they know that it already works on NFS. The sole exceptions are software that uses O_DIRECT or NFSv4 ACLs. The former can typically be toggled on/off in software that supports it while the latter can be dismissed with the knowledge that it runs on NFSv3 (or any other Linux filesystem at the present time).
justincormack 12y ago
I can't read the tables on my phone as can't shrink them down (Firefox Android)... So as you don't seem to spell it out is it the case that SElinux should be fine on ZFS but PAX might not be?
- ryao 12y ago
  They both work, with two caveats: 1. ZoL is currently incompatible with kernels built with either PAX_USERCOPY_SLABS or PAX_KERNEXEC_PLUGIN_METHOD_OR. The next release should fix PAX_USERCOPY_SLABS and might fix PAX_KERNEXEC_PLUGIN_METHOD_OR. The former should be fixed by a kmem-rework that the project plans to merge before the next release: https://github.com/zfsonlinux/spl/pull/369 https://github.com/zfsonlinux/spl/pull/369 https://github.com/zfsonlinux/zfs/pull/2411 https://github.com/zfsonlinux/zfs/pull/2411 The fix requires both pull requests, but I have yet to refresh the second pull request with the fix. I plan to refresh it before I leave for LinuxCon Europe 2014. The latter might be fixed by changes to drop support for pre-2.6.32 kernels that the project lead would prefer to merge for the next release: https://github.com/zfsonlinux/spl/pull/395 https://github.com/zfsonlinux/spl/pull/395 2. There is a bug specific to ZoL where unlinking a file with a directory extended attribute (Solaris-style) will trigger a full traversal of all objects (i.e. inodes) within a dataset. It manifests itself as the zfs_iput_taskq kernle thread consuming excessive CPU time. It is fixed for the next release: https://github.com/zfsonlinux/zfs/commit/0d5c500d6cf4dd9e133cbffd82440c95d89478a4 https://github.com/zfsonlinux/zfs/commit/0d5c500d6cf4dd9e133...
purplerails 12y ago
According to this article, ZFS is sub-optimal for databases due to fragmentation of the db files. Could you please comment? Thx. http://bartsjerps.wordpress.com/2013/02/26/zfs-ora-database-fragmentation/ http://bartsjerps.wordpress.com/2013/02/26/zfs-ora-database-...
- nabla9 12y ago
  Oracle advice on how to configure ZFS and Oracle Database http://www.oracle.com/technetwork/server-storage/solaris/config-solaris-zfs-wp-167894.pdf http://www.oracle.com/technetwork/server-storage/solaris/con... Some picks: Free space is important 10-20%. >In an environment with a high rate of data update (or churn), it is advisable to maintain a certain amount of free space. ZFS is a copy-on-write file system and relocates all writes to free disk space. Keeping a certain amount of free blocks allows ZFS to easily find space in big chunks, allows ZFS to aggregate writes and reduce the write IOPS demand on hard disks. Streaming aggregated writes can be 50-100 times less demanding on hard drives than doing small random writes. Therefore, the incentive for keeping free space is high, especially in a high churn environment, such as an active OLTP database. >The number one rule for setting up an Oracle database on ZFS is to set ZFS recordsize equal to the database block size for the file systems that contain the Oracle data files. recordsize = db_block_size
jwildeboer 12y ago
Who offers production Level support? Certification? No one AFAICS. I wonder why you didn't mention that.
- ryao 12y ago
  It was not the purpose of the blog post. Anyway, you mentioned a couple of different things here. First, there are a few companies that offer production level support. SoftNAS is one. For full disclosure, my 2014 tax return will include consulting fees earned from fixing bugs for SoftNAS. It is in use at various large companies, including Netflix and Boeing: http://www.softnas.com/ http://www.softnas.com/ Another is PogoStor, which advertises that the hardware that they sell has been certified by Nexenta for use with ZFS: http://www.pogostorage.com/products/nexenta/overview/whycertified.php http://www.pogostorage.com/products/nexenta/overview/whycert... They also have their own Linux distribution based on ZoL, for which I assume that they offer commercial support: http://www.pogostorage.com/products/pogostor/pogostor-overview.php http://www.pogostorage.com/products/pogostor/pogostor-overvi... As for certification, it is not clear if you mean hardware certification or system administrator certification. I am not aware of any filesystems that have system administrator certifications. As for hardware certifications, ZFS is designed for use with commodity hardware. It probably does not make sense to try to certify it for the full range of hardware it supports (although a subset is certified by Nexenta). Specific hardware guidelines are available on the official Open ZFS wiki: http://open-zfs.org/wiki/Hardware http://open-zfs.org/wiki/Hardware
  - stock_toaster 12y ago
    ixsystems also provides support for ZFS on FreeNAS, which is based on FreeBSD.

Thaxll 12y ago

No.

ferrantim 12y ago
Care to elaborate?
- valarauca1 12y ago
  Zpool is a great feature. On top of the dozen(s) of amazing features in ZFS. The problem is, and ZFS admits this: If you don't run ECC ram Zpool can "accidentally" your whole hard drive (bit of a joke there, it can corrupt your drive as it attempts to correct bit rot that never happened). This happens a lot more often then we really care to think about. (Ram corruption) For most day-to-day linux users who are just using standard consumer desktop/laptops they don't have this protection. And ZFS can do more harm then good. References: https://pthree.org/2013/12/10/zfs-administration-appendix-c-why-you-should-use-ecc-ram/ https://pthree.org/2013/12/10/zfs-administration-appendix-c-...
  - ryao 12y ago
    This is a myth started by someone who did not understand how filesystems work. Not having ECC memory is no more of a handicap for ZFS than it is for any other filesystem.
    - valarauca1 12y ago
      Care to elaborate? ZFS warned me of that while I reading its installation guide. Which is why I never installed it.
      - sp332 12y ago
        If you're doing RAID, you should have ECC. All hardware RAID controllers have ECC cache, and if you're doing software RAID, you should have ECC system RAM. But this is just common sense, I guess, since it's useless to write data to redundant storage if it's already been bit-flipped in RAM.

Qantourisc 12y ago

No, it doesn't has shrink, a future you want on a desktop imo. And occasionally on server when the customer doesn't provide enough storage (on the account of being cheap) and you have move things around to get things done.

icantthinkofone 12y ago

I'm a bit confused. I always thought that proper use of ZFS required multiple drives just like RAID. Then I read an article where RAID was used with ZFS. Now I'm really confused so I'm taking the lazy way and asking here: is ZFS appropriate for a single drive system?

Qantourisc 12y ago
Sure, COW(copy on write), snapshots, error checksum on files, compression, support for SSD's. You just don't have redundancy.
- FooBarWidget 12y ago
  I thought the error checking (scrubbing) only works if you have multiple drives set up in a RAID.
  - sp332 12y ago
    ZFS writes a checksum when it writes data. So you can tell whether your data is corrupted, even if you don't have the redundancy to restore the corrupted data.
  - prakashsurya 12y ago
    Error detection can always be used, but error correction may or may not be available (it depends on the type of block). Metadata blocks are redundant even on a single drive pool; so if you just have a partial failure (e.g. overwrite a metadata block) it might be able to correct the block using another redundant copy on the same drive. Data blocks will require a redundant pool configuration, though, as these are not store redundantly by default (e.g. multiple drives in a raidz or mirror).
    - mrb 12y ago
      Actually you can correct data blocks, on a single-drive ZFS pool, if you set the attribute copies=2 (ditto blocks): https://blogs.oracle.com/relling/entry/zfs_copies_and_data_protection https://blogs.oracle.com/relling/entry/zfs_copies_and_data_p... Obviously this redundancy feature makes you use twice the disk space you would normally use.

ChrisInEdmonton 12y ago

I have heard that ZFS (at least, ZFS on Linux) requires about 1 GB of memory for every 1 TB of storage. Is this an accurate statement? That's certainly a critical flaw for many use cases, though completely irrelevant for many others. If accurate, what's behind this requirement?

dedward 12y ago
All the extra stuff ZFS does, and does performantly, comes at a cost. That's the cost (I can't vouch for the amount.. but the principle is right. CPU power matters too.) You don't get all those features for free... but you do get them.
- ryao 12y ago
  While this is true, ZFSOnLinux has made progress on reducing CPU overhead. Additional improvements will be in the next release. In particular, the performance of NFS exports of ZFS datasets will increase considerably (some are seeing a factor of 2 increase) while CPU utilization has dropped: https://github.com/zfsonlinux/spl/pull/369#issuecomment-58398544 https://github.com/zfsonlinux/spl/pull/369#issuecomment-5839...
Already__Taken 12y ago
another anecdote: I've only heard this mentioned specifically about using de-duplication.
prakashsurya 12y ago
No, that's flat out not true. I've seen that metric thrown around when talking about the "dedup" feature of ZFS, but honestly, don't use dedup unless you know what you're doing. It's way to easy for things to go wrong otherwise.
- ChrisInEdmonton 12y ago
  Ah ha! Thanks. Data deduplication. Well, yeah, that makes sense to me. Thank you very much for clearing that up.

dedward 12y ago

This should be fairly obvious... ZFS has it's own requirements... to work well it needs lots of resources (cpu/ram) and EEC memory. If you have those, then sure, it gives you many advantages. If you don't, then you are opening yourself up for struggle.

atoponce 12y ago
ZFS doesn't need any more "resources" than any other kernel space filesystem. Deduplication on the other hand, does need a good amount of RAM, if you want to prevent the deduplication table (DDT) from spilling to slow spinning rust. However, if you have a fast, and large SSD L2ARC, the DDT spilling over to the L2ARC might not have drastic consequences. Regardless, unless using the DDT feature of ZFS, it doesn't need anymore resources than ext4 or XFS.

po 12y ago

I found an old ZFS disk laying around the other day (a few years old.. from when I was playing with zevo community edition ZFS and I had forgotten what I had stored on it) and I was interested in trying https://openzfsonosx.org https://openzfsonosx.org so I installed it and mounted my disk up. It worked like a charm. I was pretty impressed. I kind of screwed myself over though because I to do a bunch of file cleanup that I was hoping to perhaps roll back if it didn't go well so I created a snapshot and then got to work with some copying… I wasn't paying attention and I ended up running out of space on the disk. The way ZFS snapshots work is that ZFS still does what it would normally do on a copy: writes data, then verifies the write is completed before deleting the old data. When there is a snapshot, it just skips the second step and leaves the old data around. This is how snapshots have zero performance impact on data write. Turns out that I couldn't even delete the old snapshot (or rollback the new one) because I didn't have enough space to make any changes to the disk. I couldn't even delete files because they would stay in the snapshot. Aha! I can add a disk to the pool! So I added a small disk to the zpool and then I could delete the snapshot ok. However, I forgot that you can't remove a disk from a zpool. I ended up having to move the data to another disk, drop the entire zpool and recreate it, and move it back. Honestly, everything worked rather well except for forgetting a few of the limitations of ZFS. Given how often I have issues with HFS+ I really think it's pretty tempting for OS X.

prakashsurya 12y ago
There is work in progress to add the ability to remove a device from a pool. So, hopefully in the not too distant future, that work will land in illumos and migrate to all the downstream implementations (e.g. linux, mac, and freebsd).
ryao 12y ago
This issue was fixed in Illumos a few months ago, although the fix has not made its way to O3X yet: https://github.com/illumos/illumos-gate/commit/4bb73804952172060c9efb163b89c17f56804fe8 https://github.com/illumos/illumos-gate/commit/4bb7380495217... The workaround is to find a file that was written that is outside of a snapshot and truncate it. `zfs diff` can be used to do that. That allows you to delete an old snapshot.

antman 12y ago

I have not used zfs because when I read about it I found the expand-contract procedure overly complicated. (1) Caveats when planning for expansion should be addressed. (1) http://arstechnica.com/information-technology/2014/02/ars-walkthrough-using-the-zfs-next-gen-filesystem-on-linux/ http://arstechnica.com/information-technology/2014/02/ars-wa...

ryanqian 12y ago

zfs is a great FS, which add many modern concept such as zpool, and also put volume manager and fs together. storage and fs should be easy as this way. and it's in good quality on Solaris, I miss Solaris.

snw 12y ago
try one of the Illumos distributions like OmniOS or SmartOS. The legacy of open-solaris is well and alive.

robmccoll 12y ago

For private use, yes. I've been using ZoL for over a year on my home server with a simple ZFS raid pool and a number of volumes. Relatively stable. Snapshots every 15 minutes or so. Other than a few briefly scary kernel upgrades - nothing actually broke, just had to rebuild modules. Anytime your system comes up without pools mounted, it's scary. If only the ZFS license were compatible. Anyway, those weren't real breakages, so no problems.

deleted 12y ago

[deleted]

aidenn0 12y ago

Can anyone point me to a guide for migrating from software RAID to ZoL?

ryao 12y ago
I suggest emailing the zfs-discuss mailing list for this one: http://zfsonlinux.org/lists.html http://zfsonlinux.org/lists.html The answer to your question depends on your existing setup.

barrkel 12y ago

ZFS on Linux requires more memory than most other FSes, due to the Solaris compatibility layer. The cache is a bit less responsive to low memory conditions. I run it on a NAS with 4G of memory, and I've had to limit usage and flush caches occasionally, otherwise I get to the point that I can't run new programs.

higherpurpose 12y ago

I think F2FS is.

jeffdavis 12y ago

The license is pretty concerning. If anyone using ZFS sues Oracle for a patent issue, then Oracle can revoke their license to use ZFS in 60 days. (Not a lawyer, my interpretation might be wrong.) Probably OK for home use, but not much more. Even a small startup with no patents could end up getting bought by a larger company that does have patents. Then that company could be vulnerable, perhaps even if they are trying to use their patents defensively.

olavgg 12y ago

Has anyone tried ZFS together with Hadoop/HDFS? I find it to be a really interesting combination because of compression and self-healing of bad data-blocks.

erikb 12y ago

Not being an expert in this kind of area, I would really like to read a short summary. Is there one and I just have missed it, somewhere in the article? I didn't see a headline like summary or conclusions.

ryao 12y ago
Nearly all software works on ZoL. A small number of applications will likely need configuration of the application(s) to turn off AIO (until the next release), configuration of the application(s) to turn off DirectIO, or setting acltype=posix on the dataset to provide POSIX ACLs.