13 ms·

ZFS: Apple’s New Filesystem That Wasn’t

301 points by swills 10y ago

jhugg 10y ago
Seems like running on SSD/NVRAM may call for some new thinking. Running on watches may be even secondary to that. Not that ZFS won’t run well on SSD, but it feels like there’s a gulf between filesystems designed with SSDs in mind and those designed with spindles in mind.
- tracker1 10y ago
  It seems that COW filesystems in general are better for use with SSDs... iirc, SSDs themselves do similar things as part of their wear leveling algorithms anyway. It seems to me that it's time to break some of this down. With solid state storage reaching levels close to RAM in terms of latency, I think eventually they'll come together, and that's where it will get really interesting. On the flip side, with spinning storage approaching 10tb drives, the need for enhanced filesystems for RAID arrays with those beasts are needed as well. In general, we really need some patent releases from those companies strangling filesystem advancements (namely Oracle and MS).
  - jlgaddis 10y ago
    Out of curioisity, how do you feel that Oracle is strangling filesystem advancement?
    - dkh 10y ago
      Encryption is a big one that's been annoying everyone. Oracle baked encryption into their own ZFS implementation ages ago, but not in a way that works with the open source variety. Years later, we're only just now starting to see some solutions for this.
  - ptman 10y ago
    I'd like to see something comparable to btrfs/zfs but designed for plain nand flash. Less opaque layers to guess your way around. UBIFS?
- protomyth 10y ago
  If I remember one talk correctly ZFS was designed with persistent memory in mind, so I don't think SSDs are far out of their design thoughts. [edit: it was a BSD Now episode]
  - ahl 10y ago
    Definitely not. ZFS was primarily focused on spinning disks. Take for example the fact that the project discarded block indirection early on: the resulting random reads would be basically fine on SSDs and debilitating on HDDs. Ditto TRIM which is still not (IMO) fully baked in ZFS.
tracker1 10y ago
It's kind of a shame that ZFS hasn't seen better adoption, and that btrfs has been pretty stagnant, and that in general next generation file systems are mostly still-born. It would be nice to see some improvements in this area as 10tb hdd's are eminent, and data management is important. However, it's worth noting that RAID, especially in software on systems without ECC ram is less than ideal. Beyond this is the overhead of managing larger filesystems with ZFS. The DIY raid servers that support it have had some troublesome issues that I've experienced first hand. It's likely a lot of these advantages have been displaced by the discontinuation of Apple's server projects as well as other fears. By similar note, I've always been somewhat surprised that NTFS hasn't been more popular for external media, as it's a pretty good fit there. In the end, software at this level has been held back significantly by the patent hornets nest that exists in the world today. I truly hope that we can see some reform in this space, but with the likes of TPP and similar treaty negotiations around the world today, that is pretty unlikely. Eventually some countries will have to strike out, break these bad treaties and reign in IP law. The vast majority of software is undeserving of patent protection. Just as software really shouldn't have copyright law that lasts for decades. It's a shame all around.
- ryao 10y ago
  There is nothing special about storage that makes the absence of ECC in a computer any more of a bad idea than it already is. Building machines without ECC is a bad idea.
  - tracker1 10y ago
    When writing redundant (parity data) across multiple storage devices, you're increasing the chance of error without ECC. While there's still a chance/risk with single drive, the effects of such error are not as significant.
    - empthought 10y ago
      The effects are exactly as significant either way: corrupt data is written to disk. Whether the corrupt data is written one time or several times is immaterial.
raattgift 10y ago
Meanwhile, this works, is under active development, and is essentially at HEAD of both openzfs and zfsonlinux: https://openzfsonosx.org/ https://openzfsonosx.org/
- Analemma_ 10y ago
  Not sure what your point is. Are you saying Apple should use this? AFAICT it's still under the CDDL, which is the main reason Apple abandoned the ZFS plans in the first place.
  - cylo 10y ago
    Except for the fact DTrace is under the exact same license and ships with OS X.
    - zepto 10y ago
      DTrace is not a core technology on which the rest of the platform is built. If you read the actual article, this is discussed.
      - ahl 10y ago
        Agreed with all of this, but it's hard to imagine the licensing was truly insurmountable.
        msbarnett 10y ago
        More or less everyone I've ever talked to or seen discuss it agrees (including several people in the comments on this blog post) -- Apple Legal felt ZFS was a no-go without patent indemnification from Sun/Oracle. Sun was willing, but Oracle said no as soon as they purchased. At that point the word came down from Apple Legal to Engineering that it wasn't happening. And there's nothing more insurmountable than your own legal department.
        10y ago
rdtsc 10y ago
In general is it just me or has ZFS become more popular lately. Saw Ubuntu get behind it, even in light of Btrfs being available for many years now. https://github.com/zfsonlinux/zfs/commits/master https://github.com/zfsonlinux/zfs/commits/master is pretty active... It seems everyone at some point expected Btrfs to shoot ahead and leave other file systems in the dust, so there was no point in bothering with ZFS, "just wait a bit and Btrfs will be the default everywhere". And besides ZFS has all the legal issues with it. But it seems Btrfs progress was rather slow, so even in spite of legal issue interest in ZFS is still growing.
- barkingcat 10y ago
  the thing about ZFS is that it was an actual selling/functioning product before btrfs was in development (links at end). There always were people using it - whether through the main Sun/Oracle Solaris product or via one of the OpenSolaris / illumos / Freebsd / Bsd continuations. Btrfs was not an fs in wide production use the way xfs or ext3 / ext4 were, so it's understandable that btrfs progress is slow. The two pieces of software are just really far apart in timelines and maturity. People are paying attention to ZFS mainly because it's had a history of working and functioning in production environments and in heavy load, for a decade now. In my opinion, in order to have parity with ZFS, some big distribution (bigger than Suse, maybe Debian or Ubuntu) needs to set btrfs as its default fs, and have banks, stock exchanges, dns providers, virtual hosting companies, etc hammer the hell out of it for another 6 or 7 years and get all the bugs that come out of those experiences fixed. Then people will start paying attention to it. afaik btrfs development started 2007 (https://en.wikipedia.org/wiki/Btrfs https://en.wikipedia.org/wiki/Btrfs) ZFS was introduced as a part of Solaris in 2005, released as openZFS in 2006 (https://en.wikipedia.org/wiki/ZFS https://en.wikipedia.org/wiki/ZFS)
  - Twirrim 10y ago
    Honestly... I don't trust btrfs at the moment. It has had some nasty data losing bugs in it in the past, and admittedly its been a while since I heard of one, but it's also not often that you'll hear of someone actually using it in production. Even their wiki page is woefully lacking in examples with which to sell it: https://btrfs.wiki.kernel.org/index.php/Production_Users https://btrfs.wiki.kernel.org/index.php/Production_Users. I want to like it, we really need a nice, stable next-generation file system in Linux land (that doesn't have the licensing issues associated with ZFS)
cbsmith 10y ago
People forget that Microsoft has perennially had their Cairo/Object/Whatever FileSystem in development.
- usefulcat 10y ago
  True but NTFS is also far superior to HPFS, so arguably MSFT doesn't need a replacement for NTFS nearly as badly as Apple needs a replacement for HPFS.
  - stormbrew 10y ago
    Minor nit, but HPFS was OS/2's filesystem. Apple's is HFS (and HFS+).
    - Zardoz84 10y ago
      And NTFS was based on HPFS
  - amaks 10y ago
    Also, Microsoft has been developing ReFS as a replacement for NTFS.
- Twirrim 10y ago
  WinFS, that was an interesting concept when it was first announced. I'm still curious if it has got wings or not. Maybe some day :)
  - PeCaN 10y ago
    It even worked: https://www.youtube.com/watch?v=VW2H4sgakIA https://www.youtube.com/watch?v=VW2H4sgakIA at some point anyway. Now they're just "well, we put the ideas in SQL Server" but I still hope it comes back around. _That_ is the future of storage. ZFS/btrfs/HAMMER are all very, very cool, but WinFS was a level above that.
williesleg 10y ago
ZFS is fantastic until you try to grow a volume. Emphasis on fantastic.
- oofabz 10y ago
  If you create a zpool with '-o autoexpand=on' you can replace disks with larger ones one at a time, waiting for each one to resilver. After replacing the last disk the pool will jump to the new size.
- kinghajj 10y ago
  What difficulty did you encounter? I've had a pool running since ~2009, originally as 4x750GB, then 4x3TB in 2013, and I recall that the transition was excellent. As oofabz said, the trick is to enable "autoexpand", which can be done even after creation. In fact, I recall that I had replaced each drive, initially seeing no increase in capacity, but after enabling autoexpand the new space was immediately available.
  - Sidnicious 10y ago
    You can grow a RAID-Z “vertically” like this, but can't grow it horizontally by adding disks… which is frustrating. (I'm also frustrated that it's not possible to change redundancy within a vdev — e.g. go from RAID-Z1 to RAID-Z2.)
    - oofabz 10y ago
      You can grow it horizontally by adding a new vdev. ZFS doesn't allow you to do anything you may ever possibly want to do, but it does allow you to do more than any other RAID solution. It's not perfect but it's still pretty impressive.
    - watersb 10y ago
      I've been using the same ZFS array since 2009... At first, I assumed that RAIDZ was the obvious way to go. But I switched to a concatenation of mirrored pairs; I grow the array by adding another pair to it. I've actually had six disks fail in six months, without data loss. Was scary. But wow.
falcolas 10y ago
> this is the moral equivalent of using newspaper as insulation: it’s fine until the completely anticipated calamity destroys everything you hold dear. How... unnecessarily inflammatory. Hardlinked backups work remarkably well, and are incredibly simple to implement and understand. Of course they can be corrupted, but then again, so can every other form of backup in existence (that said, there are no protections built-in to a hardlinked backup).
- ahl 10y ago
  Hard links to directories are weird enough that many modern filesystems (including APFS) don't support them. Yes the language is perhaps unnecessarily colorful; just trying to keep things interesting.
- feelix 10y ago
  I work in backup (this has become a core part of my business: http://macdaddy.io/mac-backup-software/ http://macdaddy.io/mac-backup-software/) and part of the reason I wrote that is because of the continual reports of Time Machine backups failing to restore when people needed them. The numbers are huge. I wouldn't touch Time Machine in any conditions. I would manually backup before using that. This may be anecdotal evidence, but if you want to see some of the issues people run into with Time Machine just read through some of the posts people make about it: https://www.google.com/search?q=time+machine+backup+failure&gws_rd=cr,ssl&ei=lddhV6zQL8G7mwX3u5OABg https://www.google.com/search?q=time+machine+backup+failure&... Most of this is indeed directly related to directory backups. I also do Snapshots of sorts, but it only makes hardlinks to files, not directories. That proves to be remarkably more reliable.
  - Kluny 10y ago
    Hey, I'm interested in trying that. What's your business model? Nagware or limited time trial or what? The price is reasonable but I'd definitely like a long trial. Anyone else here using it?
    - feelix 10y ago
gribbly 10y ago
So, according to this article, Apple was making a deal with Sun to use ZFS, and later finally dropped it after an alleged discussion between Steve Jobs and Larry Ellison when Oracle owned it. My question is why did Apple think they needed to make a deal to use ZFS in the first place, and if so has Canonical (who says they'll ship openZFS with Ubuntu) made a deal with Oracle ? It's true that OpenZFS is more than Oracle's ZFS, but unless I'm mistaken, the vast majority of code in that project is still owned by Oracle. This article makes me uneasy.
- ahl 10y ago
  Apple started in 2006 with no deal or coordination with Sun. Later when the FUD started, they were looking for protection from patent lawsuits (i.e. from NetApp), for support and expertise, and, possibly, reassurances around patent clauses within the CDDL with which I'm not an expert.
- gutnor 10y ago
  Maybe the deal was to make ZFS ready for personal computer / Apple requirements that may have interested Sun but not Oracle ? In the middle of an acquisition with a guaranteed few years of uncertainty, no strong backing from the new boss meant that Jobs simply bailed out. Rumors in any case.
beedogs 10y ago
It's a real shame, because HFS is an absolute abomination. Apple will remain without a reliable, full-featured filesystem for another decade, it seems.
snarfy 10y ago
Maybe I'm being irrational, but I would never touch ZFS because Oracle.

301 points by swills 10y ago

jhugg 10y ago

Seems like running on SSD/NVRAM may call for some new thinking. Running on watches may be even secondary to that. Not that ZFS won’t run well on SSD, but it feels like there’s a gulf between filesystems designed with SSDs in mind and those designed with spindles in mind.

tracker1 10y ago
It seems that COW filesystems in general are better for use with SSDs... iirc, SSDs themselves do similar things as part of their wear leveling algorithms anyway. It seems to me that it's time to break some of this down. With solid state storage reaching levels close to RAM in terms of latency, I think eventually they'll come together, and that's where it will get really interesting. On the flip side, with spinning storage approaching 10tb drives, the need for enhanced filesystems for RAID arrays with those beasts are needed as well. In general, we really need some patent releases from those companies strangling filesystem advancements (namely Oracle and MS).
- jlgaddis 10y ago
  Out of curioisity, how do you feel that Oracle is strangling filesystem advancement?
  - dkh 10y ago
    Encryption is a big one that's been annoying everyone. Oracle baked encryption into their own ZFS implementation ages ago, but not in a way that works with the open source variety. Years later, we're only just now starting to see some solutions for this.
- ptman 10y ago
  I'd like to see something comparable to btrfs/zfs but designed for plain nand flash. Less opaque layers to guess your way around. UBIFS?
protomyth 10y ago
If I remember one talk correctly ZFS was designed with persistent memory in mind, so I don't think SSDs are far out of their design thoughts. [edit: it was a BSD Now episode]
- ahl 10y ago
  Definitely not. ZFS was primarily focused on spinning disks. Take for example the fact that the project discarded block indirection early on: the resulting random reads would be basically fine on SSDs and debilitating on HDDs. Ditto TRIM which is still not (IMO) fully baked in ZFS.

tracker1 10y ago

It's kind of a shame that ZFS hasn't seen better adoption, and that btrfs has been pretty stagnant, and that in general next generation file systems are mostly still-born. It would be nice to see some improvements in this area as 10tb hdd's are eminent, and data management is important. However, it's worth noting that RAID, especially in software on systems without ECC ram is less than ideal. Beyond this is the overhead of managing larger filesystems with ZFS. The DIY raid servers that support it have had some troublesome issues that I've experienced first hand. It's likely a lot of these advantages have been displaced by the discontinuation of Apple's server projects as well as other fears. By similar note, I've always been somewhat surprised that NTFS hasn't been more popular for external media, as it's a pretty good fit there. In the end, software at this level has been held back significantly by the patent hornets nest that exists in the world today. I truly hope that we can see some reform in this space, but with the likes of TPP and similar treaty negotiations around the world today, that is pretty unlikely. Eventually some countries will have to strike out, break these bad treaties and reign in IP law. The vast majority of software is undeserving of patent protection. Just as software really shouldn't have copyright law that lasts for decades. It's a shame all around.

ryao 10y ago
There is nothing special about storage that makes the absence of ECC in a computer any more of a bad idea than it already is. Building machines without ECC is a bad idea.
- tracker1 10y ago
  When writing redundant (parity data) across multiple storage devices, you're increasing the chance of error without ECC. While there's still a chance/risk with single drive, the effects of such error are not as significant.
  - empthought 10y ago
    The effects are exactly as significant either way: corrupt data is written to disk. Whether the corrupt data is written one time or several times is immaterial.

raattgift 10y ago

Meanwhile, this works, is under active development, and is essentially at HEAD of both openzfs and zfsonlinux: https://openzfsonosx.org/ https://openzfsonosx.org/

Analemma_ 10y ago
Not sure what your point is. Are you saying Apple should use this? AFAICT it's still under the CDDL, which is the main reason Apple abandoned the ZFS plans in the first place.
- cylo 10y ago
  Except for the fact DTrace is under the exact same license and ships with OS X.
  - zepto 10y ago
    DTrace is not a core technology on which the rest of the platform is built. If you read the actual article, this is discussed.
    - ahl 10y ago
      Agreed with all of this, but it's hard to imagine the licensing was truly insurmountable.
      - msbarnett 10y ago
        More or less everyone I've ever talked to or seen discuss it agrees (including several people in the comments on this blog post) -- Apple Legal felt ZFS was a no-go without patent indemnification from Sun/Oracle. Sun was willing, but Oracle said no as soon as they purchased. At that point the word came down from Apple Legal to Engineering that it wasn't happening. And there's nothing more insurmountable than your own legal department.
        10y ago

rdtsc 10y ago

In general is it just me or has ZFS become more popular lately. Saw Ubuntu get behind it, even in light of Btrfs being available for many years now. https://github.com/zfsonlinux/zfs/commits/master https://github.com/zfsonlinux/zfs/commits/master is pretty active... It seems everyone at some point expected Btrfs to shoot ahead and leave other file systems in the dust, so there was no point in bothering with ZFS, "just wait a bit and Btrfs will be the default everywhere". And besides ZFS has all the legal issues with it. But it seems Btrfs progress was rather slow, so even in spite of legal issue interest in ZFS is still growing.

barkingcat 10y ago
the thing about ZFS is that it was an actual selling/functioning product before btrfs was in development (links at end). There always were people using it - whether through the main Sun/Oracle Solaris product or via one of the OpenSolaris / illumos / Freebsd / Bsd continuations. Btrfs was not an fs in wide production use the way xfs or ext3 / ext4 were, so it's understandable that btrfs progress is slow. The two pieces of software are just really far apart in timelines and maturity. People are paying attention to ZFS mainly because it's had a history of working and functioning in production environments and in heavy load, for a decade now. In my opinion, in order to have parity with ZFS, some big distribution (bigger than Suse, maybe Debian or Ubuntu) needs to set btrfs as its default fs, and have banks, stock exchanges, dns providers, virtual hosting companies, etc hammer the hell out of it for another 6 or 7 years and get all the bugs that come out of those experiences fixed. Then people will start paying attention to it. afaik btrfs development started 2007 (https://en.wikipedia.org/wiki/Btrfs https://en.wikipedia.org/wiki/Btrfs) ZFS was introduced as a part of Solaris in 2005, released as openZFS in 2006 (https://en.wikipedia.org/wiki/ZFS https://en.wikipedia.org/wiki/ZFS)
- Twirrim 10y ago
  Honestly... I don't trust btrfs at the moment. It has had some nasty data losing bugs in it in the past, and admittedly its been a while since I heard of one, but it's also not often that you'll hear of someone actually using it in production. Even their wiki page is woefully lacking in examples with which to sell it: https://btrfs.wiki.kernel.org/index.php/Production_Users https://btrfs.wiki.kernel.org/index.php/Production_Users. I want to like it, we really need a nice, stable next-generation file system in Linux land (that doesn't have the licensing issues associated with ZFS)

cbsmith 10y ago

People forget that Microsoft has perennially had their Cairo/Object/Whatever FileSystem in development.

usefulcat 10y ago
True but NTFS is also far superior to HPFS, so arguably MSFT doesn't need a replacement for NTFS nearly as badly as Apple needs a replacement for HPFS.
- stormbrew 10y ago
  Minor nit, but HPFS was OS/2's filesystem. Apple's is HFS (and HFS+).
  - Zardoz84 10y ago
    And NTFS was based on HPFS
- amaks 10y ago
  Also, Microsoft has been developing ReFS as a replacement for NTFS.
Twirrim 10y ago
WinFS, that was an interesting concept when it was first announced. I'm still curious if it has got wings or not. Maybe some day :)
- PeCaN 10y ago
  It even worked: https://www.youtube.com/watch?v=VW2H4sgakIA https://www.youtube.com/watch?v=VW2H4sgakIA at some point anyway. Now they're just "well, we put the ideas in SQL Server" but I still hope it comes back around. _That_ is the future of storage. ZFS/btrfs/HAMMER are all very, very cool, but WinFS was a level above that.

williesleg 10y ago

ZFS is fantastic until you try to grow a volume. Emphasis on fantastic.

oofabz 10y ago
If you create a zpool with '-o autoexpand=on' you can replace disks with larger ones one at a time, waiting for each one to resilver. After replacing the last disk the pool will jump to the new size.
kinghajj 10y ago
What difficulty did you encounter? I've had a pool running since ~2009, originally as 4x750GB, then 4x3TB in 2013, and I recall that the transition was excellent. As oofabz said, the trick is to enable "autoexpand", which can be done even after creation. In fact, I recall that I had replaced each drive, initially seeing no increase in capacity, but after enabling autoexpand the new space was immediately available.
- Sidnicious 10y ago
  You can grow a RAID-Z “vertically” like this, but can't grow it horizontally by adding disks… which is frustrating. (I'm also frustrated that it's not possible to change redundancy within a vdev — e.g. go from RAID-Z1 to RAID-Z2.)
  - oofabz 10y ago
    You can grow it horizontally by adding a new vdev. ZFS doesn't allow you to do anything you may ever possibly want to do, but it does allow you to do more than any other RAID solution. It's not perfect but it's still pretty impressive.
  - watersb 10y ago
    I've been using the same ZFS array since 2009... At first, I assumed that RAIDZ was the obvious way to go. But I switched to a concatenation of mirrored pairs; I grow the array by adding another pair to it. I've actually had six disks fail in six months, without data loss. Was scary. But wow.

falcolas 10y ago

> this is the moral equivalent of using newspaper as insulation: it’s fine until the completely anticipated calamity destroys everything you hold dear. How... unnecessarily inflammatory. Hardlinked backups work remarkably well, and are incredibly simple to implement and understand. Of course they can be corrupted, but then again, so can every other form of backup in existence (that said, there are no protections built-in to a hardlinked backup).

ahl 10y ago
Hard links to directories are weird enough that many modern filesystems (including APFS) don't support them. Yes the language is perhaps unnecessarily colorful; just trying to keep things interesting.
feelix 10y ago
I work in backup (this has become a core part of my business: http://macdaddy.io/mac-backup-software/ http://macdaddy.io/mac-backup-software/) and part of the reason I wrote that is because of the continual reports of Time Machine backups failing to restore when people needed them. The numbers are huge. I wouldn't touch Time Machine in any conditions. I would manually backup before using that. This may be anecdotal evidence, but if you want to see some of the issues people run into with Time Machine just read through some of the posts people make about it: https://www.google.com/search?q=time+machine+backup+failure&gws_rd=cr,ssl&ei=lddhV6zQL8G7mwX3u5OABg https://www.google.com/search?q=time+machine+backup+failure&... Most of this is indeed directly related to directory backups. I also do Snapshots of sorts, but it only makes hardlinks to files, not directories. That proves to be remarkably more reliable.
- Kluny 10y ago
  Hey, I'm interested in trying that. What's your business model? Nagware or limited time trial or what? The price is reasonable but I'd definitely like a long trial. Anyone else here using it?
  - feelix 10y ago

gribbly 10y ago

So, according to this article, Apple was making a deal with Sun to use ZFS, and later finally dropped it after an alleged discussion between Steve Jobs and Larry Ellison when Oracle owned it. My question is why did Apple think they needed to make a deal to use ZFS in the first place, and if so has Canonical (who says they'll ship openZFS with Ubuntu) made a deal with Oracle ? It's true that OpenZFS is more than Oracle's ZFS, but unless I'm mistaken, the vast majority of code in that project is still owned by Oracle. This article makes me uneasy.

ahl 10y ago
Apple started in 2006 with no deal or coordination with Sun. Later when the FUD started, they were looking for protection from patent lawsuits (i.e. from NetApp), for support and expertise, and, possibly, reassurances around patent clauses within the CDDL with which I'm not an expert.
gutnor 10y ago
Maybe the deal was to make ZFS ready for personal computer / Apple requirements that may have interested Sun but not Oracle ? In the middle of an acquisition with a guaranteed few years of uncertainty, no strong backing from the new boss meant that Jobs simply bailed out. Rumors in any case.

beedogs 10y ago

It's a real shame, because HFS is an absolute abomination. Apple will remain without a reliable, full-featured filesystem for another decade, it seems.

snarfy 10y ago

Maybe I'm being irrational, but I would never touch ZFS because Oracle.