8 ms·

The Sorry State of CoW File Systems

91 points by louwrentius 12y ago

PaulHoule 12y ago
When it comes to file systems, I like boring. What I don't like is systems that have catastrophic unrecoverable wrecks because hypothetically a bit might get flipped at the hardware level. Or that just stop moving entirely when you run out of disk space, or whatever. High-end storage systems support multiple hard drive controllers because expensive hard drive controllers burn up like matches. Funny, I've never had a cheap hard drive controller fails, but never throw out the box an expensive hard drive controller came in because the R.M.A. is just a matter of time.
- rodgerd 12y ago
  > What I don't like is systems that have catastrophic unrecoverable wrecks because hypothetically a bit might get flipped at the hardware level. Perhaps they're hypothetical for you; that's nice. Having seen files (not filesystems, just files) wrecked by things like "power supplies that are just faulty enough to trash data but not faulty enough to stop booting the system", not to mention dodgy drives and controllers, I'm quite happy to have checksumming filesystems available.
  - johnplaynton 12y ago
    > Perhaps they're hypothetical for you; that's nice. Err... I think he's in agreement with you actually.
    - rodgerd 12y ago
      He seems to me to be suggesting that the added functions of ZFS and btrfs work are a net reliability loss that doesn't make up for the benefits.
      - PaulHoule 12y ago
        Here are a few points: (1) From the perception of most people, mainstream filesystems such as ext4 and NTFS are pretty reliable. I've certainly had mainstream filesystems get damaged but I've been able to repair them or copy data off without a lot of trouble. One reason mainstream filesystems are reliable is that they privilege reliability over performance. (2) In my experience, new file systems are dangerous. I've pretty frequently experienced data corruptions within a few days of trying a new file system. I've frequently been on projects that tried the latest thing, like the Linux filesystem that was written by a murderer, and after experiencing problems, we've gone back to mainstream file systems. (3) New file system advocates believe filesystems and disks are unreliable, so they're willing to tolerate a higher level of failures. (4) Most terrifying, look at all the discussion on this thread about options you can choose that might mitigate this problem or that problem. Every configuration choice is a decision you can make wrong, is a reason why your system can wreck in the middle of the night. If you know your stuff or hire somebody who knows his stuff, maybe you'll get good choices, otherwise you are playing Russian Roulette with Vladimir Putin.
jethro_tell 12y ago
One thing to note here, is that BTRFS use (Lack of use), is heavily based on the way RAID1 is implemented. A BTRFS RAID1 doeesn't do dual disk striping, it writes the file twice on different drives. You can do a RAID1 with 3 drives or 4 or 5. giving the same redundancy of RAID5 with a different name. If you want to double the parity, I believe you would put dual RAID1s into another RAID1. I think the functionality of BTRFS is there, though the ideas about how we build redundant data sets will need to shift a bit.
- louwrentius 12y ago
  I'm sorry but I don't think I follow you. Mirroring can never achieve the space / redundancy efficiency of parity raid, I don't see how the above works.
  - nothrabannosir 12y ago
    It looks like he's trying to say: in an array of N >= 2 drives, "BTRFS RAID 1" will still only choose 2 disks to replicate a file to. It's just 2 different ones for every file. Well, not every file, but you see the point. If that's the case, this has the advantage of having a subset of files recoverable even after >1 disk failure. Unlike RAID 5. On the flip side: read speed is limited to at most 2 concurrent disks. Unlike RAID 5, where it's N-1. But that's theoretical, what's the reality on read speeds in RAID 5? Anyway, if that's true, that would be a really weird name. Why didn't they call that "BTRFS RAID 5"? A look at the docs doesn't make this clear to me at all. I'm not sure what to believe...
    - louwrentius 12y ago
      I was not even aware of this possibility, thanks for explaining it to me. The concept fills with me with absolute horror though. Losing a part of my files is as bad as losing everything. The performance benefit with larger files of RAID1 vs RAID5/6 can be very 'substantial'. Even my old 18 TB NAS could achieve 1+ GB/s on sequential reads.
      - 12y ago
frik 12y ago
I am still waiting for a database-based file system. Some filesystems like NTFS/ReFS, BeFS and afaik ReiseFS4(?) and Btrfs(?)) could be extended. Microsoft extended the NTFS fs driver with the Cairo project in the NT4/5 era: http://en.wikipedia.org/wiki/Cairo_(operating_system) http://en.wikipedia.org/wiki/Cairo_(operating_system) All of the other functions NTFS exports for the EFS driver begin with the prefix NtOfs, which presumably stands for NT Object File System. One of the original goals of the NT 5.0 project (code-named Cairo) was to develop an object-oriented file system. Although NTFS in Win2K probably hasn't reached the level of object orientation that the Cairo planners had in mind, Microsoft has extended NTFS in several significant ways from its NT 4.0 implementation. One of those ways is NTFS's support for encrypted files via the NtOfs interfaces. -- http://windowsitpro.com/systems-management/inside-encrypting-file-system-part-1 http://windowsitpro.com/systems-management/inside-encrypting... , more info: https://books.google.at/books?id=5f3mlM-JKUAC&lpg=PA117&ots=uRVy3Jqu1x&dq=http%3A%2F%2Fmicrosoft.com%2Fntserver%2Fcairomb.htm&pg=PA117#v=onepage&q=http://microsoft.com/ntserver/cairomb.htm&f=false https://books.google.at/books?id=5f3mlM-JKUAC&lpg=PA117&ots=... Later Microsoft moved the database to user mode, the WinFS project, the MS SQL db was stored in hidden directory on the NTFS filesystem: http://en.wikipedia.org/wiki/WinFS http://en.wikipedia.org/wiki/WinFS . WinFS failed because of its over complicated Metadata-Ontology and it was too slow for a filesystem (dotNet based shell extension). Microsoft moved the ideas to "SharePoint" (2007-2016) that overs now some of the proposed features: http://en.wikipedia.org/wiki/SharePoint http://en.wikipedia.org/wiki/SharePoint . BeFS includes support for extended file attributes (metadata), with indexing and a querying language like a relational database: http://en.wikipedia.org/wiki/Be_File_System http://en.wikipedia.org/wiki/Be_File_System, BeFS book with all details: http://www.nobius.org/~dbg/practical-file-system-design.pdf http://www.nobius.org/~dbg/practical-file-system-design.pdf The IBM AS/400 and iSeries had already a database-based filesystem in the 1980s: http://en.wikipedia.org/wiki/File_system#Database_file_systems http://en.wikipedia.org/wiki/File_system#Database_file_syste...
- SigmundA 12y ago
  What is the difference between a file system and a database? From my point of view a file system is just a specific kind of database (Hierarchical key value). Then we put other kinds of databases on top of the file system database. Funny isn't it? If the file system where a complete enough database we might not need things like SQLite which purports itself as a replacement for fopen, hmm, full circle. Of course we have barely figured out how to encode text, maybe one day we will know how to store and retrieve data flexibly and consistently, one day long from now. I personally like the relational model the best, it would be interesting to see a OS and files system based on the relational model rather than the hierarchical, but what do I know.
oconnore 12y ago
What on earth are you doing with a 71 TB personal filesystem?
- louwrentius 12y ago
  First sentence: to store backups and 'media' :) . http://louwrentius.com/71-tib-diy-nas-based-on-zfs-on-linux.html http://louwrentius.com/71-tib-diy-nas-based-on-zfs-on-linux....
- protomyth 12y ago
  It is amazing once you start working with video how much space you can use with RAW formats and all the files you create. A Red EPIC in HDRx at 8:1 at 5K is almost 6GB/minute.
  - agumonkey 12y ago
    What amazes me is that not long ago this was Hollywood class specs. http://en.wikipedia.org/wiki/Monsters_vs._Aliens http://en.wikipedia.org/wiki/Monsters_vs._Aliens required 120TB of data.
    - superuser2 12y ago
      RED is Hollywood-class, though towards the lower end. A lot of big-name movies that you've heard of include scenes shot on RED cameras: http://www.red.com/shot-on-red http://www.red.com/shot-on-red. The TV show Leverage was shot entirely on a pair of RED Xs. RED's stuff is on the order of $25k. The main player for digital cinema is Arri ALEXA, on the order of $50k. It's not that far off.
      - agumonkey 12y ago
        Yeah I've read big names switching to RED (that's what made their business thrive). I was also referring to the disk space. A single person can assemble a 100TB setup. That was Pixar territory not long ago.
    - 12y ago
gnoway 12y ago
TL;DR: 1. Author dislikes ZFS because you can't grow and shrink a pool however you want. 2. Author likes BTRFS because it implements more flexible grow/shrink, but finds that in other ways it's unstable, and is unhappy that BTRFS doesn't let you have arbitrary numbers of parity drives.
- louwrentius 12y ago
  Correct TLDR. But with BTRFS it's not about 6 parity drives. It's about not support tripple-parity or even RAID60. Features supported by ZFS since eons.
  - bsder 12y ago
    As noted above by BrainInAJar, vdev removal just hit the Illumos ZFS tree. So, the ink is barely dry on it as well.
- 23david 12y ago
  The author wants zfs to support growing and shrinking pools in a specific way that it doesn't support. However it's clear from the docs that you can grow zfs pools by adding new vdevs. Shrinking a pool is not possible. In practice I haven't seen the inability to shrink pools as an issue. I simply do not create pools from entire disks and use partitions instead. For example each drive is split into 5 partitions. And if I need to grow a pool I add another partition across all my disks to the pool as a new vdev. This also allows me to have multiple pools on each system that have different Raid-z levels based on different app reqs.
  - louwrentius 12y ago
    It is clear from the article that I acknowledge this fact but it is a more convoluted, less flexible solution that wastes space. BTRFS shows how it can and should be done.
  - tw04 12y ago
    Actually, you can shrink pools now. http://blog.delphix.com/alex/2015/01/15/openzfs-device-removal/ http://blog.delphix.com/alex/2015/01/15/openzfs-device-remov...
jmbwell 12y ago
Everything that matters about vdev size and configuration is explained here, complete with charts that indicate the actual overhead of different arrangements: http://blog.delphix.com/matt/2014/06/06/zfs-stripe-width/ http://blog.delphix.com/matt/2014/06/06/zfs-stripe-width/
- louwrentius 12y ago
  That's a very good article on that topic indeed. But in some way it's sad that you even have to think about this with ZFS. It seems that this is not or way less of an issue with BTRFS (or the magic unicorn file system that is still not there).
  - mahrens 12y ago
    As the author of that article, thank you. And, I agree it would be sad if you had to think about exactly how many drives are in a RAID-Z group. The entire point of my blog post is that you do not have to think about it. "TL;DR: Choose a RAID-Z stripe width based on your IOPS needs and the amount of space you are willing to devote to parity information. Trying to optimize your RAID-Z stripe width based on exact numbers is irrelevant in nearly all cases."
legulere 12y ago
The saddest about both file systems are their licences. The CDDL prevents ZFS from being included/heavily integrated into Linux, the BSDs, OS X and Windows. Btrfs will very likely stay Linux only because of GPL. The interesting and hard parts of a file system are pretty much operating system independent, so I wish for a common file system initiative.
- justincormack 12y ago
  "The CDDL prevents ZFS from being included/heavily integrated into Linux, the BSDs, OS X and Windows" - no, it is included and integrated into FreeBSD with no issues, NetBSD has an old version that needs updating, OpenBSD thinks it is too complex. OSX could use it (they could get a commercial license anyway), but seemed to decide not to, as could Windows.
  - jethro_tell 12y ago
    I think the actual difficulty is patents. From what I remember, once Oracle bought Sun, they decided to go after netapp for infringing on ZFS. As it turned out, Oracle was infringing on NetApp's patent but NetApp had let it slide because until Oracle started being a dick, it was mutually beneficial. That is why Oracle also started BTRFS and is putting it's development behind that instead. I'm not really sure where that leaves other commercial projects (especially OSX which is badly in need of a functional file system).
    - gnoway 12y ago
      It was really the other way around. NetApp had Sun in court over ZFS infringing on WAFL patents[1] well before the buyout. I certainly don't know all the ways people use their Macs, but it seems like ZFS would not get a lot of use in OS X unless it was the only option: - It doesn't offer a whole lot over traditional filesystems in a single-device context; since Apple has basically abandoned server/enterprise, I would wager the vast majority of new OS X systems are single-drive. - At the time Apple was supposedly making a decision about this, I am not sure if ZFS handled 4k alignment (ashift); this is very important now that a lot of new Macs are shipping with SSDs. - You're still, even w/ 10.10, discouraged from using case-sensitive HFS+; at least one application[2] won't work on a case-sensitive filesystem on OS X. The above reasons probably apply to Windows as well. Also, in general, it seems like both vendors want to be the sole source for all of their core features. [1] http://www.zdnet.com/article/netapp-claims-suns-zfs-violates-its-patents/ http://www.zdnet.com/article/netapp-claims-suns-zfs-violates... [2] https://support.steampowered.com/kb_article.php?ref=8601-RYPX-5789 https://support.steampowered.com/kb_article.php?ref=8601-RYP...
mrb 12y ago
There is a very simple way to have a redundant ZFS pool that is expandable one disk at a time: use ditto blocks, see https://blogs.oracle.com/bill/entry/ditto_blocks_the_amazing_tape https://blogs.oracle.com/bill/entry/ditto_blocks_the_amazing... - start with 2 disks (not a mirror, but 2 independent vdevs) - set copies=2 to enable ditto blocks, causing data blocks to be automatically stored on 2 different disks (ZFS always tries to store ditto blocks on different vdevs when at least 2 are available) - when adding a 3rd disk, each data block will continue to be stored on 2 different disks, and you have the option to add a 4th disk, 5th disk, etc, any time The overhead is 50% (like RAID1/mirroring), so I presume this can be a downside for the hobbyist who usually cares about dollar per TB. But nonetheless this is an option. I can see why hobbyists may want to expand a pool one disk at a time, but personally I have been running ZFS for 10 years, I have a 20TB fileserver at home (I grew it from 1TB over the years), and I have never needed to add just one disk at a time. Usually when I run out of space, I replace all the disks at once with larger ones (and/or replace the server if it is more than 4-6 years old). Another point I wanted to comment on: the author makes the typical mistake of assuming ZFS on a single disk is not very interesting ("A VDEV is either a single disk (not so interesting)") but it is. To name a few features why it is still great to run on a single-disk system: end-to-end checksumming, self-healing, scrubbing, snapshots, clones, compression, ditto blocks, deduplication, CoW, zfs send/recv, simple CLI tools, etc.
- louwrentius 12y ago
  It is true that ZFS or BTRFS are relevent even without RAID/redundancy. But that is not the point of the article. It's about the redundancy / RAID features and their limitations. Maybe I've should have written 'in this context not so interesting'.
- tw04 12y ago
  For the love of god don't spread that nonsense. Ditto blocks do NOT protect against a drive failure. If you lose a drive you will lose all of your data. It is intended as a belt and suspenders function and was put in place knowing as we get larger and larger drives with higher and higher error rates, RAID alone will likely not guarantee your data is protected. http://zfs-discuss.opensolaris.narkive.com/1aGWdqth/adding-new-disks-and-ditto-block-behaviour http://zfs-discuss.opensolaris.narkive.com/1aGWdqth/adding-n...
cmurf 12y ago
Btrfs raid7 (3 parities) could be useful. But more parities really doesn't scale very well. Raid10 does scale. So does GlusterFS. A bigger scalability problem with Btrfs, is its raid10 implementation. Conventional raid1+0 says you can lose more than one devices at a time so long as you don't lose two mirror pairs. Btrfs raid10 doesn't have consistent mirror pairings, so the mirrored chunks for device A1 aren't all on B1, they will be distributed on multiple other drives thereby increasing the chances B1 chunks are lost in a 2+ device failure. Developers plain n-way raid1 Very Soon Now™ and hopefully will get around to better guarantees with raid10 to make it scale like raid10 should.
KaiserPro 12y ago
whats missing here is the documentation. ZFS has wonderful docs, and admin tools, BTRFS has terrible docs ans nasty tools that handle like mdadm.

91 points by louwrentius 12y ago

When it comes to file systems, I like boring. What I don't like is systems that have catastrophic unrecoverable wrecks because hypothetically a bit might get flipped at the hardware level. Or that just stop moving entirely when you run out of disk space, or whatever. High-end storage systems support multiple hard drive controllers because expensive hard drive controllers burn up like matches. Funny, I've never had a cheap hard drive controller fails, but never throw out the box an expensive hard drive controller came in because the R.M.A. is just a matter of time.

rodgerd 12y ago
> What I don't like is systems that have catastrophic unrecoverable wrecks because hypothetically a bit might get flipped at the hardware level. Perhaps they're hypothetical for you; that's nice. Having seen files (not filesystems, just files) wrecked by things like "power supplies that are just faulty enough to trash data but not faulty enough to stop booting the system", not to mention dodgy drives and controllers, I'm quite happy to have checksumming filesystems available.
- johnplaynton 12y ago
  > Perhaps they're hypothetical for you; that's nice. Err... I think he's in agreement with you actually.
  - rodgerd 12y ago
    He seems to me to be suggesting that the added functions of ZFS and btrfs work are a net reliability loss that doesn't make up for the benefits.
    - PaulHoule 12y ago
      Here are a few points: (1) From the perception of most people, mainstream filesystems such as ext4 and NTFS are pretty reliable. I've certainly had mainstream filesystems get damaged but I've been able to repair them or copy data off without a lot of trouble. One reason mainstream filesystems are reliable is that they privilege reliability over performance. (2) In my experience, new file systems are dangerous. I've pretty frequently experienced data corruptions within a few days of trying a new file system. I've frequently been on projects that tried the latest thing, like the Linux filesystem that was written by a murderer, and after experiencing problems, we've gone back to mainstream file systems. (3) New file system advocates believe filesystems and disks are unreliable, so they're willing to tolerate a higher level of failures. (4) Most terrifying, look at all the discussion on this thread about options you can choose that might mitigate this problem or that problem. Every configuration choice is a decision you can make wrong, is a reason why your system can wreck in the middle of the night. If you know your stuff or hire somebody who knows his stuff, maybe you'll get good choices, otherwise you are playing Russian Roulette with Vladimir Putin.

jethro_tell 12y ago

One thing to note here, is that BTRFS use (Lack of use), is heavily based on the way RAID1 is implemented. A BTRFS RAID1 doeesn't do dual disk striping, it writes the file twice on different drives. You can do a RAID1 with 3 drives or 4 or 5. giving the same redundancy of RAID5 with a different name. If you want to double the parity, I believe you would put dual RAID1s into another RAID1. I think the functionality of BTRFS is there, though the ideas about how we build redundant data sets will need to shift a bit.

louwrentius 12y ago
I'm sorry but I don't think I follow you. Mirroring can never achieve the space / redundancy efficiency of parity raid, I don't see how the above works.
- nothrabannosir 12y ago
  It looks like he's trying to say: in an array of N >= 2 drives, "BTRFS RAID 1" will still only choose 2 disks to replicate a file to. It's just 2 different ones for every file. Well, not every file, but you see the point. If that's the case, this has the advantage of having a subset of files recoverable even after >1 disk failure. Unlike RAID 5. On the flip side: read speed is limited to at most 2 concurrent disks. Unlike RAID 5, where it's N-1. But that's theoretical, what's the reality on read speeds in RAID 5? Anyway, if that's true, that would be a really weird name. Why didn't they call that "BTRFS RAID 5"? A look at the docs doesn't make this clear to me at all. I'm not sure what to believe...
  - louwrentius 12y ago
    I was not even aware of this possibility, thanks for explaining it to me. The concept fills with me with absolute horror though. Losing a part of my files is as bad as losing everything. The performance benefit with larger files of RAID1 vs RAID5/6 can be very 'substantial'. Even my old 18 TB NAS could achieve 1+ GB/s on sequential reads.
    - 12y ago

frik 12y ago

I am still waiting for a database-based file system. Some filesystems like NTFS/ReFS, BeFS and afaik ReiseFS4(?) and Btrfs(?)) could be extended. Microsoft extended the NTFS fs driver with the Cairo project in the NT4/5 era: http://en.wikipedia.org/wiki/Cairo_(operating_system) http://en.wikipedia.org/wiki/Cairo_(operating_system) All of the other functions NTFS exports for the EFS driver begin with the prefix NtOfs, which presumably stands for NT Object File System. One of the original goals of the NT 5.0 project (code-named Cairo) was to develop an object-oriented file system. Although NTFS in Win2K probably hasn't reached the level of object orientation that the Cairo planners had in mind, Microsoft has extended NTFS in several significant ways from its NT 4.0 implementation. One of those ways is NTFS's support for encrypted files via the NtOfs interfaces. -- http://windowsitpro.com/systems-management/inside-encrypting-file-system-part-1 http://windowsitpro.com/systems-management/inside-encrypting... , more info: https://books.google.at/books?id=5f3mlM-JKUAC&lpg=PA117&ots=uRVy3Jqu1x&dq=http%3A%2F%2Fmicrosoft.com%2Fntserver%2Fcairomb.htm&pg=PA117#v=onepage&q=http://microsoft.com/ntserver/cairomb.htm&f=false https://books.google.at/books?id=5f3mlM-JKUAC&lpg=PA117&ots=... Later Microsoft moved the database to user mode, the WinFS project, the MS SQL db was stored in hidden directory on the NTFS filesystem: http://en.wikipedia.org/wiki/WinFS http://en.wikipedia.org/wiki/WinFS . WinFS failed because of its over complicated Metadata-Ontology and it was too slow for a filesystem (dotNet based shell extension). Microsoft moved the ideas to "SharePoint" (2007-2016) that overs now some of the proposed features: http://en.wikipedia.org/wiki/SharePoint http://en.wikipedia.org/wiki/SharePoint . BeFS includes support for extended file attributes (metadata), with indexing and a querying language like a relational database: http://en.wikipedia.org/wiki/Be_File_System http://en.wikipedia.org/wiki/Be_File_System, BeFS book with all details: http://www.nobius.org/~dbg/practical-file-system-design.pdf http://www.nobius.org/~dbg/practical-file-system-design.pdf The IBM AS/400 and iSeries had already a database-based filesystem in the 1980s: http://en.wikipedia.org/wiki/File_system#Database_file_systems http://en.wikipedia.org/wiki/File_system#Database_file_syste...

SigmundA 12y ago
What is the difference between a file system and a database? From my point of view a file system is just a specific kind of database (Hierarchical key value). Then we put other kinds of databases on top of the file system database. Funny isn't it? If the file system where a complete enough database we might not need things like SQLite which purports itself as a replacement for fopen, hmm, full circle. Of course we have barely figured out how to encode text, maybe one day we will know how to store and retrieve data flexibly and consistently, one day long from now. I personally like the relational model the best, it would be interesting to see a OS and files system based on the relational model rather than the hierarchical, but what do I know.

oconnore 12y ago

What on earth are you doing with a 71 TB personal filesystem?

louwrentius 12y ago
First sentence: to store backups and 'media' :) . http://louwrentius.com/71-tib-diy-nas-based-on-zfs-on-linux.html http://louwrentius.com/71-tib-diy-nas-based-on-zfs-on-linux....
protomyth 12y ago
It is amazing once you start working with video how much space you can use with RAW formats and all the files you create. A Red EPIC in HDRx at 8:1 at 5K is almost 6GB/minute.
- agumonkey 12y ago
  What amazes me is that not long ago this was Hollywood class specs. http://en.wikipedia.org/wiki/Monsters_vs._Aliens http://en.wikipedia.org/wiki/Monsters_vs._Aliens required 120TB of data.
  - superuser2 12y ago
    RED is Hollywood-class, though towards the lower end. A lot of big-name movies that you've heard of include scenes shot on RED cameras: http://www.red.com/shot-on-red http://www.red.com/shot-on-red. The TV show Leverage was shot entirely on a pair of RED Xs. RED's stuff is on the order of $25k. The main player for digital cinema is Arri ALEXA, on the order of $50k. It's not that far off.
    - agumonkey 12y ago
      Yeah I've read big names switching to RED (that's what made their business thrive). I was also referring to the disk space. A single person can assemble a 100TB setup. That was Pixar territory not long ago.
  - 12y ago

gnoway 12y ago

TL;DR: 1. Author dislikes ZFS because you can't grow and shrink a pool however you want. 2. Author likes BTRFS because it implements more flexible grow/shrink, but finds that in other ways it's unstable, and is unhappy that BTRFS doesn't let you have arbitrary numbers of parity drives.

louwrentius 12y ago
Correct TLDR. But with BTRFS it's not about 6 parity drives. It's about not support tripple-parity or even RAID60. Features supported by ZFS since eons.
- bsder 12y ago
  As noted above by BrainInAJar, vdev removal just hit the Illumos ZFS tree. So, the ink is barely dry on it as well.
23david 12y ago
The author wants zfs to support growing and shrinking pools in a specific way that it doesn't support. However it's clear from the docs that you can grow zfs pools by adding new vdevs. Shrinking a pool is not possible. In practice I haven't seen the inability to shrink pools as an issue. I simply do not create pools from entire disks and use partitions instead. For example each drive is split into 5 partitions. And if I need to grow a pool I add another partition across all my disks to the pool as a new vdev. This also allows me to have multiple pools on each system that have different Raid-z levels based on different app reqs.
- louwrentius 12y ago
  It is clear from the article that I acknowledge this fact but it is a more convoluted, less flexible solution that wastes space. BTRFS shows how it can and should be done.
- tw04 12y ago
  Actually, you can shrink pools now. http://blog.delphix.com/alex/2015/01/15/openzfs-device-removal/ http://blog.delphix.com/alex/2015/01/15/openzfs-device-remov...

jmbwell 12y ago

Everything that matters about vdev size and configuration is explained here, complete with charts that indicate the actual overhead of different arrangements: http://blog.delphix.com/matt/2014/06/06/zfs-stripe-width/ http://blog.delphix.com/matt/2014/06/06/zfs-stripe-width/

louwrentius 12y ago
That's a very good article on that topic indeed. But in some way it's sad that you even have to think about this with ZFS. It seems that this is not or way less of an issue with BTRFS (or the magic unicorn file system that is still not there).
- mahrens 12y ago
  As the author of that article, thank you. And, I agree it would be sad if you had to think about exactly how many drives are in a RAID-Z group. The entire point of my blog post is that you do not have to think about it. "TL;DR: Choose a RAID-Z stripe width based on your IOPS needs and the amount of space you are willing to devote to parity information. Trying to optimize your RAID-Z stripe width based on exact numbers is irrelevant in nearly all cases."

legulere 12y ago

The saddest about both file systems are their licences. The CDDL prevents ZFS from being included/heavily integrated into Linux, the BSDs, OS X and Windows. Btrfs will very likely stay Linux only because of GPL. The interesting and hard parts of a file system are pretty much operating system independent, so I wish for a common file system initiative.

justincormack 12y ago
"The CDDL prevents ZFS from being included/heavily integrated into Linux, the BSDs, OS X and Windows" - no, it is included and integrated into FreeBSD with no issues, NetBSD has an old version that needs updating, OpenBSD thinks it is too complex. OSX could use it (they could get a commercial license anyway), but seemed to decide not to, as could Windows.
- jethro_tell 12y ago
  I think the actual difficulty is patents. From what I remember, once Oracle bought Sun, they decided to go after netapp for infringing on ZFS. As it turned out, Oracle was infringing on NetApp's patent but NetApp had let it slide because until Oracle started being a dick, it was mutually beneficial. That is why Oracle also started BTRFS and is putting it's development behind that instead. I'm not really sure where that leaves other commercial projects (especially OSX which is badly in need of a functional file system).
  - gnoway 12y ago
    It was really the other way around. NetApp had Sun in court over ZFS infringing on WAFL patents[1] well before the buyout. I certainly don't know all the ways people use their Macs, but it seems like ZFS would not get a lot of use in OS X unless it was the only option: - It doesn't offer a whole lot over traditional filesystems in a single-device context; since Apple has basically abandoned server/enterprise, I would wager the vast majority of new OS X systems are single-drive. - At the time Apple was supposedly making a decision about this, I am not sure if ZFS handled 4k alignment (ashift); this is very important now that a lot of new Macs are shipping with SSDs. - You're still, even w/ 10.10, discouraged from using case-sensitive HFS+; at least one application[2] won't work on a case-sensitive filesystem on OS X. The above reasons probably apply to Windows as well. Also, in general, it seems like both vendors want to be the sole source for all of their core features. [1] http://www.zdnet.com/article/netapp-claims-suns-zfs-violates-its-patents/ http://www.zdnet.com/article/netapp-claims-suns-zfs-violates... [2] https://support.steampowered.com/kb_article.php?ref=8601-RYPX-5789 https://support.steampowered.com/kb_article.php?ref=8601-RYP...

mrb 12y ago

There is a very simple way to have a redundant ZFS pool that is expandable one disk at a time: use ditto blocks, see https://blogs.oracle.com/bill/entry/ditto_blocks_the_amazing_tape https://blogs.oracle.com/bill/entry/ditto_blocks_the_amazing... - start with 2 disks (not a mirror, but 2 independent vdevs) - set copies=2 to enable ditto blocks, causing data blocks to be automatically stored on 2 different disks (ZFS always tries to store ditto blocks on different vdevs when at least 2 are available) - when adding a 3rd disk, each data block will continue to be stored on 2 different disks, and you have the option to add a 4th disk, 5th disk, etc, any time The overhead is 50% (like RAID1/mirroring), so I presume this can be a downside for the hobbyist who usually cares about dollar per TB. But nonetheless this is an option. I can see why hobbyists may want to expand a pool one disk at a time, but personally I have been running ZFS for 10 years, I have a 20TB fileserver at home (I grew it from 1TB over the years), and I have never needed to add just one disk at a time. Usually when I run out of space, I replace all the disks at once with larger ones (and/or replace the server if it is more than 4-6 years old). Another point I wanted to comment on: the author makes the typical mistake of assuming ZFS on a single disk is not very interesting ("A VDEV is either a single disk (not so interesting)") but it is. To name a few features why it is still great to run on a single-disk system: end-to-end checksumming, self-healing, scrubbing, snapshots, clones, compression, ditto blocks, deduplication, CoW, zfs send/recv, simple CLI tools, etc.

louwrentius 12y ago
It is true that ZFS or BTRFS are relevent even without RAID/redundancy. But that is not the point of the article. It's about the redundancy / RAID features and their limitations. Maybe I've should have written 'in this context not so interesting'.
tw04 12y ago
For the love of god don't spread that nonsense. Ditto blocks do NOT protect against a drive failure. If you lose a drive you will lose all of your data. It is intended as a belt and suspenders function and was put in place knowing as we get larger and larger drives with higher and higher error rates, RAID alone will likely not guarantee your data is protected. http://zfs-discuss.opensolaris.narkive.com/1aGWdqth/adding-new-disks-and-ditto-block-behaviour http://zfs-discuss.opensolaris.narkive.com/1aGWdqth/adding-n...

cmurf 12y ago

Btrfs raid7 (3 parities) could be useful. But more parities really doesn't scale very well. Raid10 does scale. So does GlusterFS. A bigger scalability problem with Btrfs, is its raid10 implementation. Conventional raid1+0 says you can lose more than one devices at a time so long as you don't lose two mirror pairs. Btrfs raid10 doesn't have consistent mirror pairings, so the mirrored chunks for device A1 aren't all on B1, they will be distributed on multiple other drives thereby increasing the chances B1 chunks are lost in a 2+ device failure. Developers plain n-way raid1 Very Soon Now™ and hopefully will get around to better guarantees with raid10 to make it scale like raid10 should.

KaiserPro 12y ago

whats missing here is the documentation. ZFS has wonderful docs, and admin tools, BTRFS has terrible docs ans nasty tools that handle like mdadm.