12 ms·

ZFS: Apple's new filesystem that wasn't (2016)

205 points by jitl 1y ago

jitl 1y ago
Besides the licensing issue, I wonder if optimizing ZFS for low latency + low RAM + low power on iPhone was an uphill battle or if it’s easy. My experiencing running ZFS years ago was poor latency and large RAM use with my NAS, but that hardware and drive configuration was optimized for low $ per gb stored and used parity stuff.
- alwillis 1y ago
  Apple wanted one operating system that ran on everything from a Mac Pro to an Apple Watch and there’s no way ZFS could have done that.
  - ryao 1y ago
    ZFS would be quite comfortable with the 512MB of RAM on an Apple Watch: https://iosref.com/ram-processor https://iosref.com/ram-processor People have run operating systems using ZFS on less.
  - DannyBee 1y ago
    ZFS is even used on my PLC to provide consistent snapshots. So uh, going to go with "it would probably work fine".
- zoky 1y ago
  If it were an issue it would hardly be an insurmountable one. I just can't imagine a scenario where Apple engineers go “Yep, we've eked out all of the performance we possibly can from this phone, the only thing left to do is change out the filesystem.”
  - klodolph 1y ago
    Does it matter if it’s insurmountable? At some point, the benefits of a new FS outweigh the drawbacks. This happens earlier than you might think, because of weird factors like “this lets us retain top filesystem experts on staff”.
    - karlgkk 1y ago
      It’s worth remembering that the filesystem they were looking to replace was HFS+. It was introduced in the 90s as a modernization of HFS, itself introduced in the 80s. Now, old does not necessarily mean bad, but in this case….
- twoodfin 1y ago
  This seems like an early application of the Tim Cook doctrine: Why would Apple want to surrender control of this key bit of technology for their platforms? The rollout of APFS a decade later validated this concern. There’s just no way that flawless transition happens so rapidly without a filesystem fit to order for Apple’s needs from Day 0.
  - TheNewsIsHere 1y ago
    (Edit: My comment is simply about the logistics and work involved in a very well executed filesystem migration. Not about whether ZFS is good for embedded or memory constrained devices.) What you describe hits my ear as more NIH syndrome than technical reality. Apple’s transition to APFS was managed like you’d manage any kind of mass scale filesystem migration. I can’t imagine they’d have done anything differently if they’d have adopted ZFS. Which isn’t to say they wouldn’t have modified ZFS. But with proper driver support and testing it wouldn’t have made much difference whether they wrote their own file system or adopted an existing one. They have done a fantastic job of compartmentalizing and rationalizing their OS and user data partitions and structures. It’s not like every iPhone model has a production run that has different filesystem needs that they’d have to sort out. There was an interesting talk given at WWDC a few years ago on this. The roll out of APFS came after they’d already tested the filesystem conversion for randomized groups of devices and then eventually every single device that upgraded to one of the point releases prior to iOS 10.3. The way they did this was to basically run the conversion in memory as a logic test against real data. At the end they’d have the super block for the new APFS volume, and on a successful exit they simply discarded it instead of writing it to persistent storage. If it errored it would send a trace back to Apple. Huge amounts of testing and consistency in OS and user data partitioning and directory structures is a huge part of why that migration worked so flawlessly.
  - jeroenhd 1y ago
    I don't see why ZFS wouldn't have gone over equally flawlessly. None of the features that make ZFS special were in HFS(+), so conversion wouldn't be too hard. The only challenge would be maintaining the legacy compression algorithms, but ZFS is configurable enough that Apple could've added their custom compression to it quite easily. There are probably good reasons for Apple to reinvent ZFS as APFS a decade later, but none of them technical. I also wouldn't call the rollout of APFS flawless, per se. It's still a terrible fit for (external) hard drives and their own products don't auto convert to APFS in some cases. There was also plenty of breakage when case-sensitivity flipped on people and software, but as far as I can tell Apple just never bothered to address that.
- hs86 1y ago
  While its deduplication feature clearly demands more memory, my understanding is that the ZFS ARC is treated by the kernel as a driver with a massive, persistent memory allocation that cannot be swapped out ("wired" pages). Unlike the regular file system cache, ARC's eviction is not directly managed by the kernel. Instead, ZFS itself is responsible for deciding when and how to shrink the ARC. This can lead to problems under sudden memory pressure. Because the ARC does not immediately release memory when the system needs it, userland pages might get swapped out instead. This behavior is more noticeable on personal computers, where memory usage patterns are highly dynamic (applications are constantly being started, used, and closed). On servers, where workloads are more static and predictable, the impact is usually less severe. I do wonder if this is also the case on Solaris or illumos, where there is no intermediate SPL between ZFS and the kernel. If so, I don't think that a hypothetical native integration of ZFS on macOS (or even Linux) would adopt the ARC in its current form.
  - dizhn 1y ago
    Maz arc size is configurable and it does not need the mythical 1GB per TB to function well.
  - ryao 1y ago
    The ZFS driver will release memory if the kernel requests it. The only integration level issue is that the free command does not show ARC as a buffer/cache, so it misrepresents reality, but as far as I know, this is an issue with caches used by various filesystems (e.g. extent caches). It is only obvious in the case of ZFS because the ARC can be so large. That is a feature, not a bug, since unused memory is wasted memory.
    - pseudalopex 1y ago
      > The ZFS driver will release memory if the kernel requests it. Not fast enough always.
      - teruakohatu 1y ago
        You were downvoted but I have also run into situations where it didn’t and caused a cascade of processes getting out of memory errors. In both instances I was pushing the server beyond what was reasonable.
- fweimer 1y ago
  If I recall correctly, ZFS error recovery was still “restore from backup” at the time, and iCloud acceptance was more limited. (ZFS basically gave up if an error was encountered after the checksum showed that the data was read correctly from storage media.) That's fine for deployments where the individual system does not matter (or you have dedicated staff to recover systems if necessary), but phones aren't like that. At least not from the user perspective.
  - ryao 1y ago
    ZFS has ditto blocks that allows it to self heal in the case of corrupt metadata as long as a good copy remains (and there would be at least 2 copies by default). ZFS only ever needs you to restore from backup if the damage is so severe that there is no making sense of things. Minor things like the indirect blocks being missing for a regular file only affect that file. Major things like all 3 copies of the MOS (the equivalent to a superblock) being gone for all uberblock entries would require recovery from backup. If all copies of any other filesystem’s superblock were gone too, that filesystem would be equally irrecoverable and would require restoring from backup.
    - fweimer 1y ago
      As far as I understand it, ditto blocks were only used if the corruption was detected due to checksum mismatch. If the checksum was correct, but metadata turned out to be unusable later (say because it was corrupted in memory, and the the checksum was computed after the corruption happened), that was treated as a fatal error.
      - water9 1y ago
        So the solution is to use ._ files that most people will delete because they are a nuisance?
- deleted 1y ago
  [deleted]
volemo 1y ago
It was just yesterday I relistened to the contemporary Hypercritical episode on the topic: https://hypercritical.fireside.fm/56 https://hypercritical.fireside.fm/56
- mrkstu 1y ago
  Wow, John's voice has changed a LOT from back then
randomnottaken 1y ago
[flagged]
jeroenhd 1y ago
I wonder what ZFS in the iPhone would've looked like. As far as I recall, the iPhone didn't have error correcting memory, and ZFS is notorious for corrupting itself when bit flips hit it and break the checksum on disk. ZFS' RAM-hungry nature would've also forced Apple to add more memory to their phone.
- deleted 1y ago
  [deleted]
- Dylan16807 1y ago
  > ZFS is notorious for corrupting itself when bit flips hit it and break the checksum on disk I don't think it is. I've never heard of that happening, or seen any evidence ZFS is more likely to break than any random filesystem. I've only seen people spreading paranoid rumors based on a couple pages saying ECC memory is important to fully get the benefits of ZFS.
  - thfuran 1y ago
    They also insist that you need about 10 TB RAM per TB disk space or something like that.
    - yjftsjthsd-h 1y ago
      There is a rule of thumb that you should have at least 1 GB of RAM per TB of disk when using deduplication. That's.... Different.
      - williamstein 1y ago
        Fortunately, this has significantly improved since dedup was rewritten as part of the new ZFS 2.3 release. Search for zfs “fast dedup”.
      - thfuran 1y ago
        So you've never seen the people saying you should steer clear of ZFS unless you're going to have an enormous ARC even when talking about personal media servers?
rrdharan 1y ago
Kind of odd that the blog states that "The architect for ZFS at Apple had left" and links to the LinkedIn profile of someone who doesn't have any Apple work experience listed on their resume. I assume the author linked to the wrong profile?
- nikhizzle 1y ago
  Ex-Apple File System engineer here who shared an office with the other ZFS lead at the time. Can confirm they link to the wrong profile for Don Brady. This is the correct person: https://github.com/don-brady https://github.com/don-brady Also can confirm Don is one of the kindest, nicest principal engineer level people I’ve worked with in my career. Always had time to mentor and assist.
  - ahl 1y ago
    Not sure how I fat-fingered Don's LinkedIn, but I'm updating that 9-year-old typo. Agreed that Don is a delight. In the years after this article I got to collaborate more with him, but left Delphix before he joined to work on ZFS.
  - whitepoplar 1y ago
    Given your expertise, any chance you can comment on the risk of data corruption on APFS given that it only checksums metadata?
    - nikhizzle 1y ago
      I moved out of the kernel in 2008 and never went back, so don’t have a wise opinion here which would be current.
smittywerben 1y ago
Thanks for sharing I was just looking for what happened to Sun. I like the second-hand quote comparing the IBM and HP as "garbage trucks colliding" plus the inclusion of blog posts with links to the court filings. Is it fair to say ZFS made most sense on Solaris using Solaris Containers on SPARK?
- ahl 1y ago
  ZFS was developed in Solaris, and at the time we were mostly selling SPARC systems. That changed rapidly and the biggest commercial push was in the form of the ZFS Storage Appliance that our team (known as Fishworks) built at Sun. Those systems were based on AMD servers that Sun was making at the time such as Thumper [1]. Also in 2016, Ubuntu leaned in to use of ZFS for containers [2]. There was nothing that specific about Solaris that made sense for ZFS, and even less of a connection to the SPARC architecture. [1]: https://www.theregister.com/2005/11/16/sun_thumper/ https://www.theregister.com/2005/11/16/sun_thumper/ [2]: https://ubuntu.com/blog/zfs-is-the-fs-for-containers-in-ubuntu-16-04 https://ubuntu.com/blog/zfs-is-the-fs-for-containers-in-ubun...
  - ghaff 1y ago
    Yeah I think if it hadn’t been for the combination of Oracle and CDDL, Red Hat would have been more interested in for Linux. As it was they basically went with XFS and volume management. Fedora did eventually go with btrfs but dints know if there are are any plans for copy-on-write FS for RHEL at any point.
    - m4rtink 1y ago
      Fedora Server uses XFS on LVM by default & you can do CoW with any modern filesystem on top of an LVM thin pool. And there is also the Stratis project Red Hat is involved in: https://stratis-storage.github.io/ https://stratis-storage.github.io/
      - ghaff 1y ago
        It looks like btrfs is/was the default for just Fedora Workstation. I’m less connected to Red Hat filesystem details than I used to be.
jFriedensreich 1y ago
The death of ZFS in macOS was a huge shift in the industry. This has to be seen in the context of microsoft killed their largely ambitious WinFS which felt like the death of desktop innovation in combination.
- thyristan 1y ago
  Both are imho linked to "offline desktop use cases are not important anymore". Both companies saw their future gains elsewhere, in internet-related functions and what became known as "cloud". No need to have a fancy, featurefull and expensive filesystem when it is only to be used as a cache for remote cloud stuff.
  - wenc 1y ago
    I remember eagerly anticipating ZFS for desktop hard disks. I seem to remember it never took off because memory requirements were too high and payoffs were insufficient to justify the trade off.
  - em500 1y ago
    Linux or FreeBSD developers are free to adopt ZFS as their primary file systems. But it appears that practical benefits are not really evident to most users.
    - thyristan 1y ago
      Lots of ZFS users are enthusiasts who heard about that one magic thing that does it all in one tidy box. Whereas usually you would have to known all the minutiae of LVM/mdadm/cryptsetup/nbd and mkfs.whatever to get to the same point. So while ZFS is the nicer-dicer of volume management and filesystems, the latter is your whole chef's knife set. And while you can dice with both, the user groups are not the same. And enthusiasts with the right usecases are very few. And for the thin-provisioned snapshotted subvolume usecase, btrfs is currently eating ZFS's lunch due to far better Linux integration. Think snapshots at every update, and having a/b boot to get back to a known-working config after an update. So widespread adoption through the distro route is out of the question.
ahl 1y ago
Back in 2016, Ars Technica picked up this piece from my blog [1] as well as a longer piece reviewing the newly announced APFS [2] [3]. Glad it's still finding an audience! [1]: https://arstechnica.com/gadgets/2016/06/zfs-the-other-new-apple-file-system-that-almost-was-until-it-wasnt/ https://arstechnica.com/gadgets/2016/06/zfs-the-other-new-ap... [2]: https://ahl.dtrace.org/2016/06/19/apfs-part1/ https://ahl.dtrace.org/2016/06/19/apfs-part1/ [3]: https://arstechnica.com/gadgets/2016/06/a-zfs-developers-analysis-of-the-good-and-bad-in-apples-new-apfs-file-system/ https://arstechnica.com/gadgets/2016/06/a-zfs-developers-ana...
throw0101b 1y ago
Apple and Sun couldn't agree on a 'support contract'. From Jeff Bonwick, one of the co-creators ZFS: >> Apple can currently just take the ZFS CDDL code and incorporate it (like they did with DTrace), but it may be that they wanted a "private license" from Sun (with appropriate technical support and indemnification), and the two entities couldn't come to mutually agreeable terms. > I cannot disclose details, but that is the essence of it. * https://archive.is/http://mail.opensolaris.org/pipermail/zfs-discuss/2009-October/033125.html https://archive.is/http://mail.opensolaris.org/pipermail/zfs... Apple took DTrace, licensed via CDDL—just like ZFS—and put it into the kernel without issue. Of course a file system is much more central to an operating system, so they wanted much more of a CYA for that.
- cryptonector 1y ago
  > indemnification That was the sticking point. In the context of the NetApp lawsuit Apple wanted indemnification should Sun/Oracle lose the suit.
  - DannyBee 1y ago
    That would be surprising - indemnification is usually very cheap to provide even for this sort of thing - there is always an insurer willing to sell you the policy you need to provide it at some reasonable price :)
    - cryptonector 1y ago
      There may have been more to it than that. IIRC the exact details of why Apple abandoned ZFS were never made public. Indemnification was the clearest and most likely issue, though it might also have been that leaks from the Sun side didn't help.
secabeen 1y ago
ZFS remains an excellent filesystem for bulk storage on rust, but were I Apple at the time, I would probably want to focus on something built for the coming era of flash and NVMe storage. There are a number of axioms built into ZFS that come out of the spinning disk era that still hold it back for flash-only filesystems.
- ahl 1y ago
  Certainly one would build something different starting in 2025 rather than 2001, but do you have specific examples of how ZFS’s design holds it back? I think it has been adapted extremely well for the changing ecosystem.
  - porjo 1y ago
    This presentation from 2022 covers the topic: https://m.youtube.com/watch?v=v8sl8gj9UnA https://m.youtube.com/watch?v=v8sl8gj9UnA Note: sound drops out for a couple minutes at 1:30 mark but comes back.
    - ahl 1y ago
      I didn't watch it in its entirety, but I'm familiar with Allan's work. There doesn't seem to be anything fundamental to the design of ZFS that his talk highlights as limiting, but there are many areas that have benefitted from optimization and tuning.
whartung 1y ago
As a desktop user, I am content with APFS. The only feature from ZFS that I would like, is the corruption detection. I honestly don't know how robust the image and video formats are to bit corruption. On the one hand, potentially, "very" robust. But on the other, I would think that there are some very special bits that if toggled can potentially "ruin" the entire file. But I don't know. However, I can say, every time I've tried ZFS on my iMac, it was simply a disaster. Just trying to set it up on a single USB drive, or setting it up to mirror a pair. The net effect was that it CRUSHED the performance on my machine. It became unusable. We're talking "move the mouse, watch the pointer crawl behind" unusable. "Let's type at 300 baud" unusable. Interactive performance was shot. After I remove it, all is right again.
- ryao 1y ago
  > I honestly don't know how robust the image and video formats are to bit corruption. It depends on the format. A BMP image format would limit the damage to 1 pixel, while a JPEG could propagate the damage to potentially the entire image. There is an example of a bitflip damaging a picture here: https://arstechnica.com/information-technology/2014/01/bitrot-and-atomic-cows-inside-next-gen-filesystems/ https://arstechnica.com/information-technology/2014/01/bitro... That single bit flip ruined about half of the image. As for video, that depends on how far apart I frames are. Any damage from a bit flip would likely be isolated to the section of video from the bitflip until the next I-frame occurs. As for how bad it could be, it depends on how the encoding works. > On the one hand, potentially, "very" robust. Only in uncompressed files. > But on the other, I would think that there are some very special bits that if toggled can potentially "ruin" the entire file. But I don't know. The way that image compression works means that a single bit flip prior to decompression can affect a great many pixels, as shown at Ars Technica. > However, I can say, every time I've tried ZFS on my iMac, it was simply a disaster. Did you file an issue? I am not sure what the current status of the macOS driver’s production readiness is, but it will be difficult to see it improve if people do not report issues that they have.
  - atoav 1y ago
    A thing that can destroy most files in terms of readability is a bit flip in the header section. These could be theoretically corrected with clever guesswork and a hex editor, but in practise it is going to be hard to know where the bit flip occured, when a file just can't be read anymore.
ewuhic 1y ago
What's the current state of ZFS on Macos? As far as I'm aware there's a supported fork.
- zie 1y ago
  Exactly: https://openzfsonosx.org https://openzfsonosx.org Works just fine for me.
ein0p 1y ago
ZFS sort of moved inside the NVMe controller - it also checksums and scrubs things all the time, you just don't see it. This does not, however, support multi-device redundant storage, but that is not a concern for Apple - the vast majority of their devices have only one storage device.
- DannyBee 1y ago
  Everyone seems to focus on the checksumming and scrubbing. But the real win (at least for me) - every device i have - laptops, desktops, server, even PLC's (they now use freebsd under the covers + ZFS) all backup using zfs snapshots and replication. I do not ever worry about finding an old file i accidentally deleted. Or restoring a backup to a new machine and "did it really include everything" or anything else. The machine storing backups is itself replicated to another machine in my detached garage. If i wanted even more security, i could trivially further replicate it to offsite storage in the same manner. All of this takes ~0 time to set up, and require 0 maintenance to keep working. Meanwhile, Apple has gone backwards - time machine can't even make actual full system backups anymore.
dang 1y ago
Discussed at the time: ZFS: Apple’s New Filesystem That Wasn’t - https://news.ycombinator.com/item?id=11909606 https://news.ycombinator.com/item?id=11909606 - June 2016 (128 comments)
water9 1y ago
ZFS is the king of all file systems. As someone with over a petabyte of storage across 275 drives I have never lost a single byte due to a hard drive failure or corruption thanks to ZFS
krupan 1y ago
"Still another version I’ve heard calls into question the veracity of their purported friendship, and has Steve instead suggesting that Larry go f*ck himself. Normally the iconoclast, that would, if true, represent Steve’s most mainstream opinion." LOL!! I really hope they weren't friends, that really shatters my internal narrative (mainly because I can't actually picture either of them having actual friends).
- crossroadsguy 1y ago
  [flagged]
  - nar001 1y ago
    That's exactly why so many of these can't be friends with each other though, they clash, they always have to be the most important person in the room
- ahl 1y ago
  Embarrassed to say I had forgotten writing that and LOLed myself… no more reliable audience for your sense of humor than yourself!
danudey 1y ago
Not to pointlessly nitpick, but: > HFS improved upon the Macintosh File System by adding—wait for it—hierarchy! No longer would files accumulate in a single pile; you could organize them in folders. MFS did allow you to organize your files into folders, but on-disk they were represented as a single list of files with unique filenames - meaning you could have 'resume.txt' in a folder called 'Jan's Docs' but you couldn't also have 'resume.txt' in a folder called 'Jake's Docs' - every file on the disk needed to have a unique filename. Not so much an issue in the days of 400KB floppy drives, but once people started getting 20 MB hard drives that was going to be an unacceptable limitation. The other major benefit of HFS was that it stored file data in a B-tree, which allowed directory information to be stored effectively hierarchically, meaning you could find a directory's contents very quickly. With MFS, every file being stored in a single list meant that any time you wanted to get a directory's contents you had to read through that list of every file on the disk to see which ones were stored in that directory, so every listing of any directory was O(n) for the total number of files on the disk.

205 points by jitl 1y ago

jitl 1y ago

Besides the licensing issue, I wonder if optimizing ZFS for low latency + low RAM + low power on iPhone was an uphill battle or if it’s easy. My experiencing running ZFS years ago was poor latency and large RAM use with my NAS, but that hardware and drive configuration was optimized for low $ per gb stored and used parity stuff.

alwillis 1y ago
Apple wanted one operating system that ran on everything from a Mac Pro to an Apple Watch and there’s no way ZFS could have done that.
- ryao 1y ago
  ZFS would be quite comfortable with the 512MB of RAM on an Apple Watch: https://iosref.com/ram-processor https://iosref.com/ram-processor People have run operating systems using ZFS on less.
- DannyBee 1y ago
  ZFS is even used on my PLC to provide consistent snapshots. So uh, going to go with "it would probably work fine".
zoky 1y ago
If it were an issue it would hardly be an insurmountable one. I just can't imagine a scenario where Apple engineers go “Yep, we've eked out all of the performance we possibly can from this phone, the only thing left to do is change out the filesystem.”
- klodolph 1y ago
  Does it matter if it’s insurmountable? At some point, the benefits of a new FS outweigh the drawbacks. This happens earlier than you might think, because of weird factors like “this lets us retain top filesystem experts on staff”.
  - karlgkk 1y ago
    It’s worth remembering that the filesystem they were looking to replace was HFS+. It was introduced in the 90s as a modernization of HFS, itself introduced in the 80s. Now, old does not necessarily mean bad, but in this case….
twoodfin 1y ago
This seems like an early application of the Tim Cook doctrine: Why would Apple want to surrender control of this key bit of technology for their platforms? The rollout of APFS a decade later validated this concern. There’s just no way that flawless transition happens so rapidly without a filesystem fit to order for Apple’s needs from Day 0.
- TheNewsIsHere 1y ago
  (Edit: My comment is simply about the logistics and work involved in a very well executed filesystem migration. Not about whether ZFS is good for embedded or memory constrained devices.) What you describe hits my ear as more NIH syndrome than technical reality. Apple’s transition to APFS was managed like you’d manage any kind of mass scale filesystem migration. I can’t imagine they’d have done anything differently if they’d have adopted ZFS. Which isn’t to say they wouldn’t have modified ZFS. But with proper driver support and testing it wouldn’t have made much difference whether they wrote their own file system or adopted an existing one. They have done a fantastic job of compartmentalizing and rationalizing their OS and user data partitions and structures. It’s not like every iPhone model has a production run that has different filesystem needs that they’d have to sort out. There was an interesting talk given at WWDC a few years ago on this. The roll out of APFS came after they’d already tested the filesystem conversion for randomized groups of devices and then eventually every single device that upgraded to one of the point releases prior to iOS 10.3. The way they did this was to basically run the conversion in memory as a logic test against real data. At the end they’d have the super block for the new APFS volume, and on a successful exit they simply discarded it instead of writing it to persistent storage. If it errored it would send a trace back to Apple. Huge amounts of testing and consistency in OS and user data partitioning and directory structures is a huge part of why that migration worked so flawlessly.
- jeroenhd 1y ago
  I don't see why ZFS wouldn't have gone over equally flawlessly. None of the features that make ZFS special were in HFS(+), so conversion wouldn't be too hard. The only challenge would be maintaining the legacy compression algorithms, but ZFS is configurable enough that Apple could've added their custom compression to it quite easily. There are probably good reasons for Apple to reinvent ZFS as APFS a decade later, but none of them technical. I also wouldn't call the rollout of APFS flawless, per se. It's still a terrible fit for (external) hard drives and their own products don't auto convert to APFS in some cases. There was also plenty of breakage when case-sensitivity flipped on people and software, but as far as I can tell Apple just never bothered to address that.
hs86 1y ago
While its deduplication feature clearly demands more memory, my understanding is that the ZFS ARC is treated by the kernel as a driver with a massive, persistent memory allocation that cannot be swapped out ("wired" pages). Unlike the regular file system cache, ARC's eviction is not directly managed by the kernel. Instead, ZFS itself is responsible for deciding when and how to shrink the ARC. This can lead to problems under sudden memory pressure. Because the ARC does not immediately release memory when the system needs it, userland pages might get swapped out instead. This behavior is more noticeable on personal computers, where memory usage patterns are highly dynamic (applications are constantly being started, used, and closed). On servers, where workloads are more static and predictable, the impact is usually less severe. I do wonder if this is also the case on Solaris or illumos, where there is no intermediate SPL between ZFS and the kernel. If so, I don't think that a hypothetical native integration of ZFS on macOS (or even Linux) would adopt the ARC in its current form.
- dizhn 1y ago
  Maz arc size is configurable and it does not need the mythical 1GB per TB to function well.
- ryao 1y ago
  The ZFS driver will release memory if the kernel requests it. The only integration level issue is that the free command does not show ARC as a buffer/cache, so it misrepresents reality, but as far as I know, this is an issue with caches used by various filesystems (e.g. extent caches). It is only obvious in the case of ZFS because the ARC can be so large. That is a feature, not a bug, since unused memory is wasted memory.
  - pseudalopex 1y ago
    > The ZFS driver will release memory if the kernel requests it. Not fast enough always.
    - teruakohatu 1y ago
      You were downvoted but I have also run into situations where it didn’t and caused a cascade of processes getting out of memory errors. In both instances I was pushing the server beyond what was reasonable.
fweimer 1y ago
If I recall correctly, ZFS error recovery was still “restore from backup” at the time, and iCloud acceptance was more limited. (ZFS basically gave up if an error was encountered after the checksum showed that the data was read correctly from storage media.) That's fine for deployments where the individual system does not matter (or you have dedicated staff to recover systems if necessary), but phones aren't like that. At least not from the user perspective.
- ryao 1y ago
  ZFS has ditto blocks that allows it to self heal in the case of corrupt metadata as long as a good copy remains (and there would be at least 2 copies by default). ZFS only ever needs you to restore from backup if the damage is so severe that there is no making sense of things. Minor things like the indirect blocks being missing for a regular file only affect that file. Major things like all 3 copies of the MOS (the equivalent to a superblock) being gone for all uberblock entries would require recovery from backup. If all copies of any other filesystem’s superblock were gone too, that filesystem would be equally irrecoverable and would require restoring from backup.
  - fweimer 1y ago
    As far as I understand it, ditto blocks were only used if the corruption was detected due to checksum mismatch. If the checksum was correct, but metadata turned out to be unusable later (say because it was corrupted in memory, and the the checksum was computed after the corruption happened), that was treated as a fatal error.
    - water9 1y ago
      So the solution is to use ._ files that most people will delete because they are a nuisance?
deleted 1y ago
[deleted]

volemo 1y ago

It was just yesterday I relistened to the contemporary Hypercritical episode on the topic: https://hypercritical.fireside.fm/56 https://hypercritical.fireside.fm/56

mrkstu 1y ago
Wow, John's voice has changed a LOT from back then

randomnottaken 1y ago

[flagged]

jeroenhd 1y ago

I wonder what ZFS in the iPhone would've looked like. As far as I recall, the iPhone didn't have error correcting memory, and ZFS is notorious for corrupting itself when bit flips hit it and break the checksum on disk. ZFS' RAM-hungry nature would've also forced Apple to add more memory to their phone.

deleted 1y ago
[deleted]
Dylan16807 1y ago
> ZFS is notorious for corrupting itself when bit flips hit it and break the checksum on disk I don't think it is. I've never heard of that happening, or seen any evidence ZFS is more likely to break than any random filesystem. I've only seen people spreading paranoid rumors based on a couple pages saying ECC memory is important to fully get the benefits of ZFS.
- thfuran 1y ago
  They also insist that you need about 10 TB RAM per TB disk space or something like that.
  - yjftsjthsd-h 1y ago
    There is a rule of thumb that you should have at least 1 GB of RAM per TB of disk when using deduplication. That's.... Different.
    - williamstein 1y ago
      Fortunately, this has significantly improved since dedup was rewritten as part of the new ZFS 2.3 release. Search for zfs “fast dedup”.
    - thfuran 1y ago
      So you've never seen the people saying you should steer clear of ZFS unless you're going to have an enormous ARC even when talking about personal media servers?

rrdharan 1y ago

Kind of odd that the blog states that "The architect for ZFS at Apple had left" and links to the LinkedIn profile of someone who doesn't have any Apple work experience listed on their resume. I assume the author linked to the wrong profile?

nikhizzle 1y ago
Ex-Apple File System engineer here who shared an office with the other ZFS lead at the time. Can confirm they link to the wrong profile for Don Brady. This is the correct person: https://github.com/don-brady https://github.com/don-brady Also can confirm Don is one of the kindest, nicest principal engineer level people I’ve worked with in my career. Always had time to mentor and assist.
- ahl 1y ago
  Not sure how I fat-fingered Don's LinkedIn, but I'm updating that 9-year-old typo. Agreed that Don is a delight. In the years after this article I got to collaborate more with him, but left Delphix before he joined to work on ZFS.
- whitepoplar 1y ago
  Given your expertise, any chance you can comment on the risk of data corruption on APFS given that it only checksums metadata?
  - nikhizzle 1y ago
    I moved out of the kernel in 2008 and never went back, so don’t have a wise opinion here which would be current.

smittywerben 1y ago

Thanks for sharing I was just looking for what happened to Sun. I like the second-hand quote comparing the IBM and HP as "garbage trucks colliding" plus the inclusion of blog posts with links to the court filings. Is it fair to say ZFS made most sense on Solaris using Solaris Containers on SPARK?

ahl 1y ago
ZFS was developed in Solaris, and at the time we were mostly selling SPARC systems. That changed rapidly and the biggest commercial push was in the form of the ZFS Storage Appliance that our team (known as Fishworks) built at Sun. Those systems were based on AMD servers that Sun was making at the time such as Thumper [1]. Also in 2016, Ubuntu leaned in to use of ZFS for containers [2]. There was nothing that specific about Solaris that made sense for ZFS, and even less of a connection to the SPARC architecture. [1]: https://www.theregister.com/2005/11/16/sun_thumper/ https://www.theregister.com/2005/11/16/sun_thumper/ [2]: https://ubuntu.com/blog/zfs-is-the-fs-for-containers-in-ubuntu-16-04 https://ubuntu.com/blog/zfs-is-the-fs-for-containers-in-ubun...
- ghaff 1y ago
  Yeah I think if it hadn’t been for the combination of Oracle and CDDL, Red Hat would have been more interested in for Linux. As it was they basically went with XFS and volume management. Fedora did eventually go with btrfs but dints know if there are are any plans for copy-on-write FS for RHEL at any point.
  - m4rtink 1y ago
    Fedora Server uses XFS on LVM by default & you can do CoW with any modern filesystem on top of an LVM thin pool. And there is also the Stratis project Red Hat is involved in: https://stratis-storage.github.io/ https://stratis-storage.github.io/
    - ghaff 1y ago
      It looks like btrfs is/was the default for just Fedora Workstation. I’m less connected to Red Hat filesystem details than I used to be.

jFriedensreich 1y ago

The death of ZFS in macOS was a huge shift in the industry. This has to be seen in the context of microsoft killed their largely ambitious WinFS which felt like the death of desktop innovation in combination.

thyristan 1y ago
Both are imho linked to "offline desktop use cases are not important anymore". Both companies saw their future gains elsewhere, in internet-related functions and what became known as "cloud". No need to have a fancy, featurefull and expensive filesystem when it is only to be used as a cache for remote cloud stuff.
- wenc 1y ago
  I remember eagerly anticipating ZFS for desktop hard disks. I seem to remember it never took off because memory requirements were too high and payoffs were insufficient to justify the trade off.
- em500 1y ago
  Linux or FreeBSD developers are free to adopt ZFS as their primary file systems. But it appears that practical benefits are not really evident to most users.
  - thyristan 1y ago
    Lots of ZFS users are enthusiasts who heard about that one magic thing that does it all in one tidy box. Whereas usually you would have to known all the minutiae of LVM/mdadm/cryptsetup/nbd and mkfs.whatever to get to the same point. So while ZFS is the nicer-dicer of volume management and filesystems, the latter is your whole chef's knife set. And while you can dice with both, the user groups are not the same. And enthusiasts with the right usecases are very few. And for the thin-provisioned snapshotted subvolume usecase, btrfs is currently eating ZFS's lunch due to far better Linux integration. Think snapshots at every update, and having a/b boot to get back to a known-working config after an update. So widespread adoption through the distro route is out of the question.

ahl 1y ago

Back in 2016, Ars Technica picked up this piece from my blog [1] as well as a longer piece reviewing the newly announced APFS [2] [3]. Glad it's still finding an audience! [1]: https://arstechnica.com/gadgets/2016/06/zfs-the-other-new-apple-file-system-that-almost-was-until-it-wasnt/ https://arstechnica.com/gadgets/2016/06/zfs-the-other-new-ap... [2]: https://ahl.dtrace.org/2016/06/19/apfs-part1/ https://ahl.dtrace.org/2016/06/19/apfs-part1/ [3]: https://arstechnica.com/gadgets/2016/06/a-zfs-developers-analysis-of-the-good-and-bad-in-apples-new-apfs-file-system/ https://arstechnica.com/gadgets/2016/06/a-zfs-developers-ana...

throw0101b 1y ago

Apple and Sun couldn't agree on a 'support contract'. From Jeff Bonwick, one of the co-creators ZFS: >> Apple can currently just take the ZFS CDDL code and incorporate it (like they did with DTrace), but it may be that they wanted a "private license" from Sun (with appropriate technical support and indemnification), and the two entities couldn't come to mutually agreeable terms. > I cannot disclose details, but that is the essence of it. * https://archive.is/http://mail.opensolaris.org/pipermail/zfs-discuss/2009-October/033125.html https://archive.is/http://mail.opensolaris.org/pipermail/zfs... Apple took DTrace, licensed via CDDL—just like ZFS—and put it into the kernel without issue. Of course a file system is much more central to an operating system, so they wanted much more of a CYA for that.

cryptonector 1y ago
> indemnification That was the sticking point. In the context of the NetApp lawsuit Apple wanted indemnification should Sun/Oracle lose the suit.
- DannyBee 1y ago
  That would be surprising - indemnification is usually very cheap to provide even for this sort of thing - there is always an insurer willing to sell you the policy you need to provide it at some reasonable price :)
  - cryptonector 1y ago
    There may have been more to it than that. IIRC the exact details of why Apple abandoned ZFS were never made public. Indemnification was the clearest and most likely issue, though it might also have been that leaks from the Sun side didn't help.

secabeen 1y ago

ZFS remains an excellent filesystem for bulk storage on rust, but were I Apple at the time, I would probably want to focus on something built for the coming era of flash and NVMe storage. There are a number of axioms built into ZFS that come out of the spinning disk era that still hold it back for flash-only filesystems.

ahl 1y ago
Certainly one would build something different starting in 2025 rather than 2001, but do you have specific examples of how ZFS’s design holds it back? I think it has been adapted extremely well for the changing ecosystem.
- porjo 1y ago
  This presentation from 2022 covers the topic: https://m.youtube.com/watch?v=v8sl8gj9UnA https://m.youtube.com/watch?v=v8sl8gj9UnA Note: sound drops out for a couple minutes at 1:30 mark but comes back.
  - ahl 1y ago
    I didn't watch it in its entirety, but I'm familiar with Allan's work. There doesn't seem to be anything fundamental to the design of ZFS that his talk highlights as limiting, but there are many areas that have benefitted from optimization and tuning.

whartung 1y ago

As a desktop user, I am content with APFS. The only feature from ZFS that I would like, is the corruption detection. I honestly don't know how robust the image and video formats are to bit corruption. On the one hand, potentially, "very" robust. But on the other, I would think that there are some very special bits that if toggled can potentially "ruin" the entire file. But I don't know. However, I can say, every time I've tried ZFS on my iMac, it was simply a disaster. Just trying to set it up on a single USB drive, or setting it up to mirror a pair. The net effect was that it CRUSHED the performance on my machine. It became unusable. We're talking "move the mouse, watch the pointer crawl behind" unusable. "Let's type at 300 baud" unusable. Interactive performance was shot. After I remove it, all is right again.

ryao 1y ago
> I honestly don't know how robust the image and video formats are to bit corruption. It depends on the format. A BMP image format would limit the damage to 1 pixel, while a JPEG could propagate the damage to potentially the entire image. There is an example of a bitflip damaging a picture here: https://arstechnica.com/information-technology/2014/01/bitrot-and-atomic-cows-inside-next-gen-filesystems/ https://arstechnica.com/information-technology/2014/01/bitro... That single bit flip ruined about half of the image. As for video, that depends on how far apart I frames are. Any damage from a bit flip would likely be isolated to the section of video from the bitflip until the next I-frame occurs. As for how bad it could be, it depends on how the encoding works. > On the one hand, potentially, "very" robust. Only in uncompressed files. > But on the other, I would think that there are some very special bits that if toggled can potentially "ruin" the entire file. But I don't know. The way that image compression works means that a single bit flip prior to decompression can affect a great many pixels, as shown at Ars Technica. > However, I can say, every time I've tried ZFS on my iMac, it was simply a disaster. Did you file an issue? I am not sure what the current status of the macOS driver’s production readiness is, but it will be difficult to see it improve if people do not report issues that they have.
- atoav 1y ago
  A thing that can destroy most files in terms of readability is a bit flip in the header section. These could be theoretically corrected with clever guesswork and a hex editor, but in practise it is going to be hard to know where the bit flip occured, when a file just can't be read anymore.

ewuhic 1y ago

What's the current state of ZFS on Macos? As far as I'm aware there's a supported fork.

zie 1y ago
Exactly: https://openzfsonosx.org https://openzfsonosx.org Works just fine for me.

ein0p 1y ago

ZFS sort of moved inside the NVMe controller - it also checksums and scrubs things all the time, you just don't see it. This does not, however, support multi-device redundant storage, but that is not a concern for Apple - the vast majority of their devices have only one storage device.

DannyBee 1y ago
Everyone seems to focus on the checksumming and scrubbing. But the real win (at least for me) - every device i have - laptops, desktops, server, even PLC's (they now use freebsd under the covers + ZFS) all backup using zfs snapshots and replication. I do not ever worry about finding an old file i accidentally deleted. Or restoring a backup to a new machine and "did it really include everything" or anything else. The machine storing backups is itself replicated to another machine in my detached garage. If i wanted even more security, i could trivially further replicate it to offsite storage in the same manner. All of this takes ~0 time to set up, and require 0 maintenance to keep working. Meanwhile, Apple has gone backwards - time machine can't even make actual full system backups anymore.

dang 1y ago

Discussed at the time: ZFS: Apple’s New Filesystem That Wasn’t - https://news.ycombinator.com/item?id=11909606 https://news.ycombinator.com/item?id=11909606 - June 2016 (128 comments)

water9 1y ago

ZFS is the king of all file systems. As someone with over a petabyte of storage across 275 drives I have never lost a single byte due to a hard drive failure or corruption thanks to ZFS

krupan 1y ago

"Still another version I’ve heard calls into question the veracity of their purported friendship, and has Steve instead suggesting that Larry go f*ck himself. Normally the iconoclast, that would, if true, represent Steve’s most mainstream opinion." LOL!! I really hope they weren't friends, that really shatters my internal narrative (mainly because I can't actually picture either of them having actual friends).

crossroadsguy 1y ago
[flagged]
- nar001 1y ago
  That's exactly why so many of these can't be friends with each other though, they clash, they always have to be the most important person in the room
ahl 1y ago
Embarrassed to say I had forgotten writing that and LOLed myself… no more reliable audience for your sense of humor than yourself!

danudey 1y ago

Not to pointlessly nitpick, but: > HFS improved upon the Macintosh File System by adding—wait for it—hierarchy! No longer would files accumulate in a single pile; you could organize them in folders. MFS did allow you to organize your files into folders, but on-disk they were represented as a single list of files with unique filenames - meaning you could have 'resume.txt' in a folder called 'Jan's Docs' but you couldn't also have 'resume.txt' in a folder called 'Jake's Docs' - every file on the disk needed to have a unique filename. Not so much an issue in the days of 400KB floppy drives, but once people started getting 20 MB hard drives that was going to be an unacceptable limitation. The other major benefit of HFS was that it stored file data in a B-tree, which allowed directory information to be stored effectively hierarchically, meaning you could find a directory's contents very quickly. With MFS, every file being stored in a single list meant that any time you wanted to get a directory's contents you had to read through that list of every file on the disk to see which ones were stored in that directory, so every listing of any directory was O(n) for the total number of files on the disk.