10 ms·

A brief history of APFS (2022)

138 points by tim_sw 4y ago

karlmdavis 4y ago
I don’t really understand how any file system designed in the last 20 years could be lacking data checksums. Data integrity is job #1, and yet.
- greggsy 4y ago
  Isn’t that offloaded to the underlying hardware? Presumably they have sufficient control at that level for their current lineup, but I’m not sure what it means for aftermarket disks and external disks.
- bartvk 4y ago
  Yes, it's a shame. I'm guessing Apple takes care of that with their own custom SSD firmware, but I'd love to take the additional precautions. For now, I add par2 error correction/detection to my pictures.
whartung 4y ago
Does anyone use APFS snapshots in anger? Have you ever used them at all, or do you use them routinely in any kind of workflow? What are you using them for, and how do you use them? Just curious how folks apply this day to day outside of how Apple leverages it.
sposeray 4y ago
[dead]
viraptor 4y ago
It seems to mostly work and get out of the way, but there's one thing I'd really like them to add. There's a way to explicitly compress files if you want. You can get the "afsctool" and run it on the directory with your sources for example and gain some space back. Unfortunately, this gets applied per-file only. There's no way to mark a directory to always compress everything in it. I don't think you can do it per-volume either. It would be great if those options got exposed to the user. I'm not going to hold any breath though, it may be one of those "Apple knows better" issues.
- abofh 4y ago
  It would be hard to do directory wide - how would you handle a hard link for example?
  - viraptor 4y ago
    Choose one behaviour and document it? Hard links are such a rare exception in any system that either ignoring them or compressing regardless would be fine. Or check what others have done. You can do "btrfs property set /path compress zstd", so someone's solved it before.
  - eviks 4y ago
    Compress it since it's just a regular file, doesn't matter that it exists elsewhere Or add an config option Why is that hard?
- saagarjha 4y ago
  Transparent file compression is nothing new; if anything it’s filesystem agnostic assuming what you’re running on supports resource forks. In fact in many ways it was really designed for HFS+; many of the more esoteric ways to compress files don’t make sense on APFS.
  - viraptor 4y ago
    HFS wasn't great about the compression. You got 0-size files with some extra attributes, which is not always usable. APFS is actually an improvement here - mainly because apps do not need to know about resource forks.
    - saagarjha 4y ago
      It's meant to be transparent to applications that aren't aware of how it works; the mechanism hasn't changed (you can store data in either the xattr itself or the resource fork).
Veliladon 4y ago
One of the things that is amazing about APFS is that some iOS updates prior to 10.3 would do a simulated update of the entire filesystem system as part of the update process across the entire iOS userbase. The devs would then take diagnostic data of failed mock upgrades in order to refine the process before the mostly non-event that 10.3 actually was.
- MBCook 4y ago
  I had heard of that before, that they did a mock upgrade. But a tweet yesterday said they didn’t do a mock upgrade. Every single iOS update for a year did that so they could catch and fix a ton of bugs before the big switch. Pretty genius.
steponlego 4y ago
Should have gone with ZFS. Shit Steve should have bought Sun.
- macintux 4y ago
  Given the disaster that would have unfolded had Sun bought Apple, I think Apple not buying Sun later is a fair trade for the timeline we lucked into.
  - actionfromafar 4y ago
    Disaster for Sun or for Apple? Sun vanished anyway.
    - macintux 4y ago
      Disaster for Apple (and I’d argue the industry as a whole; Apple makes things much more interesting than the Wintel duopoly would have managed without them).
      - steponlego 4y ago
        Apple could have had ZFS, which is better than AFS, and also had SPARC which is more promising than Wintel or even ARM for high performance applications.
- tyingq 4y ago
  >Steve should have bought Sun That's interesting to think about. Would have propped up the xServe idea better. A timeline where an Apple server was a somewhat credible competitor to Linux servers might have changed some things.
- denkmoon 4y ago
  One can dream :')
- mwcremer 4y ago
  They almost did: https://arstechnica.com/gadgets/2016/06/zfs-the-other-new-apple-file-system-that-almost-was-until-it-wasnt/ https://arstechnica.com/gadgets/2016/06/zfs-the-other-new-ap...
ValentineC 4y ago
Needs a (2022) in the title.
coconut08 4y ago
I never really used anything in the apple ecosystem until the m1 came out. It was such an interesting device that I wound up picking it up. I was reading about macos' history with filesystems a while back and apparently they were considering switching to zfs at some point but instead decided to design apfs as their own in house next gen filesystem. In my experience with it it seems to lack almost all of the features that would have made zfs interesting. One of the most obvious usecases would be for something like time machine. I have a parallels virtual machine that is hundreds of gigabytes in size and as far as I can tell that each time you modify even a single byte in that vm it will need to backup the entire file all over again. not only is this infeasible in the amount of space it would require it's also infeasible in that every time I do a backup i would need to spend the time transfering the whole thing all over again. this is also one of the biggest problems that next gen filesystems were designed to solve. is it really the case that apple's next generation filesystem doesn't support snapshots or block level replication when one of their most obvious usecases for it would be time machine backups?
- EricE 4y ago
  They ended up backing away from ZFS because of patents and Oracle. The irony.
- _8j50 4y ago
  It might be due to licensing, even Linux went with btrfs (not sure if they mainlined zfs yet).
  - wfriesen 4y ago
    Not mainlined now, and probably not ever given that Linus has said "considering Oracle's litigious nature, and the questions over licensing, there's no way I can feel safe in ever doing so" https://arstechnica.com/gadgets/2020/01/linus-torvalds-zfs-statements-arent-right-heres-the-straight-dope/ https://arstechnica.com/gadgets/2020/01/linus-torvalds-zfs-s...
  - deleted 4y ago
    [deleted]
eviks 4y ago
I remember "free" dir size was one of the supposed benefits. Has that materialized, are there file managers/tools that can show the size of any dir immediately just like you can have in Everything on NTFS+Windows
- the-golden-one 4y ago
  Given how long it takes an iPhone to calculate local storage usage on iOS, I assume not.
goranmoomin 4y ago
As a long-time Korean user of macOS, I hate the fact that APFS is normalization-insensitive yet normalization-preserving. As a result, this means that… if you create a Korean-named file yourself in Finder: then the file name is normalized in Unicode NFD (decomposed) form, but if you create a Korean-named file in a zsh shell, the file name is normalized in Unicode NFC (composed) form. You don’t realize the difference until you try an `ls | xxd`, but the bytes are different, even though they look the same. It’s an invisible mess that people don’t realize until they do. And a lot of programs gets subtly crazy when fopen(A/B) succeeds but listdir(A) doesn’t list B.
- p_l 4y ago
  Such issues is why ZFS has optional normalisation support, and serving files to Macs (or generally cross platform) was always one of the major examples of its use
- sneak 4y ago
  Designed by Apple in 8859-1
- sacnoradhq 4y ago
  I think the solution is to stop trying to over-manage user- and filesystem-input. Be faithful and preserve what was entered rather than trying to "second-guess" the user.
  - kalleboo 4y ago
    What is the solution here though? They forced normalization in HFS+ and everyone hated that even more since it changed your file names when you copied over files. And they can't be normalization-sensitive because then you can't type in a file name.
    - vlovich123 4y ago
      I still don’t see the problem with normalization.
ShadowBanThis01 4y ago
Remember when Apple announced that ZFS would be the new filesystem for Mac OS? Oh well.
- sneak 4y ago
  Everyone wanted that but it failed due to a failure to reach licensing terms. https://news.ycombinator.com/item?id=17852019 https://news.ycombinator.com/item?id=17852019 Apple's not big on f/oss. They maintain their hardware moat (historically) by differentiating with extremely proprietary software.
  - throw0101b 4y ago
    From one of the co-creators of ZFS: > Apple can currently just take the ZFS CDDL code and incorporate it > (like they did with DTrace), but it may be that they wanted a "private > license" from Sun (with appropriate technical support and > indemnification), and the two entities couldn't come to mutually > agreeable terms. I cannot disclose details, but that is the essence of it. * https://web.archive.org/web/20121221111757/http://mail.opensolaris.org/pipermail/zfs-discuss/2009-October/033125.html https://web.archive.org/web/20121221111757/http://mail.opens... * https://arstechnica.com/gadgets/2009/10/apple-abandons-zfs-on-mac-os-x-project-over-licensing-issues/ https://arstechnica.com/gadgets/2009/10/apple-abandons-zfs-o...
- Someone 4y ago
  > Remember when Apple announced that ZFS would be the new filesystem for Mac OS? I don’t. They worked on ZFS, even released read only support, and some people think they came close to making such an announcement (e.g. http://dtrace.org/blogs/ahl/2016/06/15/apple_and_zfs/ http://dtrace.org/blogs/ahl/2016/06/15/apple_and_zfs/) but I don’t think they ever made any statement that ZFS would be the replacement for HFS+.
  - ShadowBanThis01 4y ago
    https://www.zdnet.com/article/apple-announces-zfs-on-snow-leopard/ https://www.zdnet.com/article/apple-announces-zfs-on-snow-le... https://www.theregister.com/2007/06/07/apple_using_zfs_in_leopard/ https://www.theregister.com/2007/06/07/apple_using_zfs_in_le... https://sdtimes.com/apfs/apple-quietly-announces-new-file-system/ https://sdtimes.com/apfs/apple-quietly-announces-new-file-sy... https://osxdaily.com/2007/06/12/now-apple-wont-use-zfs-as-the-leopard-file-system/ https://osxdaily.com/2007/06/12/now-apple-wont-use-zfs-as-th...
joosters 4y ago
A brief history of APFS version numbers, more like. Doesn't seem to tell us much about APFS as a thing, other than saying 'there was a bug in this version'.
sacnoradhq 4y ago
The important Q: Even controlling a walled garden, how does one roll-out an entirely new file system that's never seen widespread commercial or retail use?
- andylynch 4y ago
  Lots of testing, up to and including as someone else mentioned doing dry run migrations on a very large number of user devices. Auto updating the file system on upgrade once that was done almost certainly simplified matters too.
flawn 4y ago
Friend of mine had randomly his home partition corrupt after upgrade to APFS on macOS ¯ \ _ ( ツ ) _ / ¯
bartvk 4y ago
When it came out on Mac, I had a Mac Mini with a tiny internal drive (120 gigs). I tried to make an APFS Fusion drive where I fused the internal plus a fast USB-attached thumbdrive together. That seemed to work but would then fail upon reboot, because macOS would renumber devices and then couldn't find the drive. Under Linux, you can simply assign /dev/sdX to a particular drive, just create an udev rule. Under macOS, that's impossible to my knowledge.

138 points by tim_sw 4y ago

I don’t really understand how any file system designed in the last 20 years could be lacking data checksums. Data integrity is job #1, and yet.

greggsy 4y ago
Isn’t that offloaded to the underlying hardware? Presumably they have sufficient control at that level for their current lineup, but I’m not sure what it means for aftermarket disks and external disks.
bartvk 4y ago
Yes, it's a shame. I'm guessing Apple takes care of that with their own custom SSD firmware, but I'd love to take the additional precautions. For now, I add par2 error correction/detection to my pictures.

whartung 4y ago

Does anyone use APFS snapshots in anger? Have you ever used them at all, or do you use them routinely in any kind of workflow? What are you using them for, and how do you use them? Just curious how folks apply this day to day outside of how Apple leverages it.

sposeray 4y ago

[dead]

viraptor 4y ago

It seems to mostly work and get out of the way, but there's one thing I'd really like them to add. There's a way to explicitly compress files if you want. You can get the "afsctool" and run it on the directory with your sources for example and gain some space back. Unfortunately, this gets applied per-file only. There's no way to mark a directory to always compress everything in it. I don't think you can do it per-volume either. It would be great if those options got exposed to the user. I'm not going to hold any breath though, it may be one of those "Apple knows better" issues.

abofh 4y ago
It would be hard to do directory wide - how would you handle a hard link for example?
- viraptor 4y ago
  Choose one behaviour and document it? Hard links are such a rare exception in any system that either ignoring them or compressing regardless would be fine. Or check what others have done. You can do "btrfs property set /path compress zstd", so someone's solved it before.
- eviks 4y ago
  Compress it since it's just a regular file, doesn't matter that it exists elsewhere Or add an config option Why is that hard?
saagarjha 4y ago
Transparent file compression is nothing new; if anything it’s filesystem agnostic assuming what you’re running on supports resource forks. In fact in many ways it was really designed for HFS+; many of the more esoteric ways to compress files don’t make sense on APFS.
- viraptor 4y ago
  HFS wasn't great about the compression. You got 0-size files with some extra attributes, which is not always usable. APFS is actually an improvement here - mainly because apps do not need to know about resource forks.
  - saagarjha 4y ago
    It's meant to be transparent to applications that aren't aware of how it works; the mechanism hasn't changed (you can store data in either the xattr itself or the resource fork).

Veliladon 4y ago

One of the things that is amazing about APFS is that some iOS updates prior to 10.3 would do a simulated update of the entire filesystem system as part of the update process across the entire iOS userbase. The devs would then take diagnostic data of failed mock upgrades in order to refine the process before the mostly non-event that 10.3 actually was.

MBCook 4y ago
I had heard of that before, that they did a mock upgrade. But a tweet yesterday said they didn’t do a mock upgrade. Every single iOS update for a year did that so they could catch and fix a ton of bugs before the big switch. Pretty genius.

steponlego 4y ago

Should have gone with ZFS. Shit Steve should have bought Sun.

macintux 4y ago
Given the disaster that would have unfolded had Sun bought Apple, I think Apple not buying Sun later is a fair trade for the timeline we lucked into.
- actionfromafar 4y ago
  Disaster for Sun or for Apple? Sun vanished anyway.
  - macintux 4y ago
    Disaster for Apple (and I’d argue the industry as a whole; Apple makes things much more interesting than the Wintel duopoly would have managed without them).
    - steponlego 4y ago
      Apple could have had ZFS, which is better than AFS, and also had SPARC which is more promising than Wintel or even ARM for high performance applications.
tyingq 4y ago
>Steve should have bought Sun That's interesting to think about. Would have propped up the xServe idea better. A timeline where an Apple server was a somewhat credible competitor to Linux servers might have changed some things.
denkmoon 4y ago
One can dream :')
mwcremer 4y ago
They almost did: https://arstechnica.com/gadgets/2016/06/zfs-the-other-new-apple-file-system-that-almost-was-until-it-wasnt/ https://arstechnica.com/gadgets/2016/06/zfs-the-other-new-ap...

ValentineC 4y ago

Needs a (2022) in the title.

coconut08 4y ago

I never really used anything in the apple ecosystem until the m1 came out. It was such an interesting device that I wound up picking it up. I was reading about macos' history with filesystems a while back and apparently they were considering switching to zfs at some point but instead decided to design apfs as their own in house next gen filesystem. In my experience with it it seems to lack almost all of the features that would have made zfs interesting. One of the most obvious usecases would be for something like time machine. I have a parallels virtual machine that is hundreds of gigabytes in size and as far as I can tell that each time you modify even a single byte in that vm it will need to backup the entire file all over again. not only is this infeasible in the amount of space it would require it's also infeasible in that every time I do a backup i would need to spend the time transfering the whole thing all over again. this is also one of the biggest problems that next gen filesystems were designed to solve. is it really the case that apple's next generation filesystem doesn't support snapshots or block level replication when one of their most obvious usecases for it would be time machine backups?

EricE 4y ago
They ended up backing away from ZFS because of patents and Oracle. The irony.
_8j50 4y ago
It might be due to licensing, even Linux went with btrfs (not sure if they mainlined zfs yet).
- wfriesen 4y ago
  Not mainlined now, and probably not ever given that Linus has said "considering Oracle's litigious nature, and the questions over licensing, there's no way I can feel safe in ever doing so" https://arstechnica.com/gadgets/2020/01/linus-torvalds-zfs-statements-arent-right-heres-the-straight-dope/ https://arstechnica.com/gadgets/2020/01/linus-torvalds-zfs-s...
- deleted 4y ago
  [deleted]

eviks 4y ago

I remember "free" dir size was one of the supposed benefits. Has that materialized, are there file managers/tools that can show the size of any dir immediately just like you can have in Everything on NTFS+Windows

the-golden-one 4y ago
Given how long it takes an iPhone to calculate local storage usage on iOS, I assume not.

goranmoomin 4y ago

As a long-time Korean user of macOS, I hate the fact that APFS is normalization-insensitive yet normalization-preserving. As a result, this means that… if you create a Korean-named file yourself in Finder: then the file name is normalized in Unicode NFD (decomposed) form, but if you create a Korean-named file in a zsh shell, the file name is normalized in Unicode NFC (composed) form. You don’t realize the difference until you try an `ls | xxd`, but the bytes are different, even though they look the same. It’s an invisible mess that people don’t realize until they do. And a lot of programs gets subtly crazy when fopen(A/B) succeeds but listdir(A) doesn’t list B.

p_l 4y ago
Such issues is why ZFS has optional normalisation support, and serving files to Macs (or generally cross platform) was always one of the major examples of its use
sneak 4y ago
Designed by Apple in 8859-1
sacnoradhq 4y ago
I think the solution is to stop trying to over-manage user- and filesystem-input. Be faithful and preserve what was entered rather than trying to "second-guess" the user.
- kalleboo 4y ago
  What is the solution here though? They forced normalization in HFS+ and everyone hated that even more since it changed your file names when you copied over files. And they can't be normalization-sensitive because then you can't type in a file name.
  - vlovich123 4y ago
    I still don’t see the problem with normalization.

ShadowBanThis01 4y ago

Remember when Apple announced that ZFS would be the new filesystem for Mac OS? Oh well.

sneak 4y ago
Everyone wanted that but it failed due to a failure to reach licensing terms. https://news.ycombinator.com/item?id=17852019 https://news.ycombinator.com/item?id=17852019 Apple's not big on f/oss. They maintain their hardware moat (historically) by differentiating with extremely proprietary software.
- throw0101b 4y ago
  From one of the co-creators of ZFS: > Apple can currently just take the ZFS CDDL code and incorporate it > (like they did with DTrace), but it may be that they wanted a "private > license" from Sun (with appropriate technical support and > indemnification), and the two entities couldn't come to mutually > agreeable terms. I cannot disclose details, but that is the essence of it. * https://web.archive.org/web/20121221111757/http://mail.opensolaris.org/pipermail/zfs-discuss/2009-October/033125.html https://web.archive.org/web/20121221111757/http://mail.opens... * https://arstechnica.com/gadgets/2009/10/apple-abandons-zfs-on-mac-os-x-project-over-licensing-issues/ https://arstechnica.com/gadgets/2009/10/apple-abandons-zfs-o...
Someone 4y ago
> Remember when Apple announced that ZFS would be the new filesystem for Mac OS? I don’t. They worked on ZFS, even released read only support, and some people think they came close to making such an announcement (e.g. http://dtrace.org/blogs/ahl/2016/06/15/apple_and_zfs/ http://dtrace.org/blogs/ahl/2016/06/15/apple_and_zfs/) but I don’t think they ever made any statement that ZFS would be the replacement for HFS+.
- ShadowBanThis01 4y ago
  https://www.zdnet.com/article/apple-announces-zfs-on-snow-leopard/ https://www.zdnet.com/article/apple-announces-zfs-on-snow-le... https://www.theregister.com/2007/06/07/apple_using_zfs_in_leopard/ https://www.theregister.com/2007/06/07/apple_using_zfs_in_le... https://sdtimes.com/apfs/apple-quietly-announces-new-file-system/ https://sdtimes.com/apfs/apple-quietly-announces-new-file-sy... https://osxdaily.com/2007/06/12/now-apple-wont-use-zfs-as-the-leopard-file-system/ https://osxdaily.com/2007/06/12/now-apple-wont-use-zfs-as-th...

joosters 4y ago

A brief history of APFS version numbers, more like. Doesn't seem to tell us much about APFS as a thing, other than saying 'there was a bug in this version'.

sacnoradhq 4y ago

The important Q: Even controlling a walled garden, how does one roll-out an entirely new file system that's never seen widespread commercial or retail use?

andylynch 4y ago
Lots of testing, up to and including as someone else mentioned doing dry run migrations on a very large number of user devices. Auto updating the file system on upgrade once that was done almost certainly simplified matters too.

flawn 4y ago

Friend of mine had randomly his home partition corrupt after upgrade to APFS on macOS ¯ \ _ ( ツ ) _ / ¯

bartvk 4y ago

When it came out on Mac, I had a Mac Mini with a tiny internal drive (120 gigs). I tried to make an APFS Fusion drive where I fused the internal plus a fast USB-attached thumbdrive together. That seemed to work but would then fail upon reboot, because macOS would renumber devices and then couldn't find the drive. Under Linux, you can simply assign /dev/sdX to a particular drive, just create an udev rule. Under macOS, that's impossible to my knowledge.