5 ms·

Happy birthday APFS, 7 years old today

99 points by ingve 3y ago

deleted 3y ago
[deleted]
timdoug 3y ago
An absolutely impressive achievement. I echo sentiments seen in other HN threads recently -- "I'd never do in-place FS upgrades!" -- yet Apple pulled it off on billions of devices with zero fanfare. Folk have scores of stores about btrfs/ZFS/etc issues, I don't recall similarly for APFS?
- detourdog 3y ago
  If you read the story you will hear about a test roll out an install. I was bitten by this and it was an awful couple of days of head scratching with Apple Support. My story is that I upgraded my mail server but the install hung never completed. I ended up with a a hard drive with no files and completely full. Diskutil and Disk Utility reported everything as good but there was a partition that could not be seen. I forget the eventual resolution. Probably starting all over again with a completely fresh install. I never understood what had happened until I saw a talk or read an interview with the APFS creaator. During the talk he spoke of how they did a complete re-format to APFS and verified everything and than reverted to HFS+ for the install. This is all from memory but I think I have the details mostly correct. The whole process was technically impressive but not without hiccups.
- Rucadi 3y ago
  I only had problems when installing Asahi linux, where APFS yeeted a little and the tools couldn't really recover the errors nicely.
- zoky 3y ago
  Ehh, when you control the hardware and software, an in-place filesystem upgrade isn’t that big of a deal. Especially when you can store everything in the cloud and pull it back down again if need be. Meanwhile, I’m expanding my ZFS single drives to mirrors and wondering why I can’t do the same with APFS. Apple should have embraced ZFS (and almost did, just like they almost embraced Ruby instead of Swift), but Apple is absolutely the worst offender when it comes to NIH syndrome.
  - 130e13a 3y ago
    > just like they almost embraced Ruby instead of Swift do you have any more information about this?
    - jonhohle 3y ago
      I don’t think it was every one or the other. Ruby wa/ hot around 2005 and Apple experimented with language bindings for AppKit (and osascript?). I did a bunch of prototyping using MacRuby and then would rewrite in Objective-C. Ruby’s object model is relatively similar to objc - no direct member access outside the object, everything is loosely coupled messages with dynamic runtime, you could get away with duck typing in some cases in objc using `id` types (at the expense of getting type checks from the compiler). I don’t think it was ever intended at the next system language, though. It was just a convenient binding and optimized runtime (many of its core types were Core Foundation types - NSString, etc. IIRC).
      - detourdog 3y ago
        Almost any object scripting language can use Objective C Bridge for scripting. https://marketsplash.com/tutorials/objective-c/objective-c-bridge-pattern/ https://marketsplash.com/tutorials/objective-c/objective-c-b...
        jonhohle 3y ago
        The difference was MacRuby was built on top of the Objective-C runtime. https://en.m.wikipedia.org/wiki/MacRuby https://en.m.wikipedia.org/wiki/MacRuby
        zoky 3y ago
        There was a brief window when it was possible to do some really cool stuff with MacRuby, such as inspecting and even monkey patching core OS X classes. Probably one major reason why Apple killed the project in the first place. Being able to interact with the operating system on a low level through macirb was an absolutely mind blowing experience. Sadly, those days are long gone, and I just don’t think Swift brings the same kind of flexibility or joy that MacRuby did.
- geerlingguy 3y ago
  I still wanted to see ZFS, would've been nice to go more cross-platform-compatible.
  - boxed 3y ago
    There are a ton of features in ZFS that would have been very nice to have. Like streaming backup. That could enable a Time Machine 2.0 with enormous efficiency/speed gains.
- steve1977 3y ago
  > Folk have scores of stores about btrfs/ZFS/etc issues, I don't recall similarly for APFS? I don't think you can compare these in terms of features (hence also not in terms of potential problems)
- madeofpalk 3y ago
  The impressive part about the APFS migration was that they performed dry runs on iOS updates prior to measure success. > We actually had this process running for earlier iOS updates … where when you updated to 10.1 or 10.2, we were trial migrating your whole file system … consistency checking it … reporting back to us whether the upgrade was 100 percent clean, then rolling it back. https://www.macobserver.com/analysis/apple-dry-run-apfs-prior-ios-10-3/ https://www.macobserver.com/analysis/apple-dry-run-apfs-prio...
  - detourdog 3y ago
    I was not impressed with the dry runs on MacOS I had mind bending issues that Apple support could not help with.
transpute 3y ago
iOS devices can read/write (but not initialize) APFS-encrypted external storage, e.g. NVME 2230 SSD.
Sweepi 3y ago
... and it was a disappointment. Apple had the one-in-30-years opportunity to ship a state of the art file system, and they choose to not have protection against user data bit rot (against meta data rot they choose yes, which is nice), which ZFS and others had for .. idk, a decade at this point?
- specialist 3y ago
  How feasible is it to add those kinds of features? Or would that have to be baked-in from the start?
  - jonhohle 3y ago
    I wrote a small utility to find and dedup[0] using APFS features and had to poke around the user exposed metadata quite a bit. The short answer is probably. Metadata is extendable and there are things that work similarly already (encryption, and clone ids, for example). 0 - https://github.com/ttkb-oss/dedup https://github.com/ttkb-oss/dedup
- postmodest 3y ago
  ...because their "disks" all have hardware ECC?
- dmitrygr 3y ago
  Why would this be needed, if every single Apple device uses flash memory which provides its own guarantees against bitrot?
  - boxed 3y ago
    Because those guarantees aren't guarantees, more like partial mitigation.
    - dmitrygr 3y ago
      modern flash ECC may not always be able to correct but it is able to DETECT almost any number of bits wrong, so that is just as good as ZFS - if the data is banged - you'll know.
GeekyBear 3y ago
The story of how ZFS was almost added to OS X (along with other Sun tech like Dtrace which did get added) as told by a Sun insider. https://arstechnica.com/gadgets/2016/06/zfs-the-other-new-apple-file-system-that-almost-was-until-it-wasnt/ https://arstechnica.com/gadgets/2016/06/zfs-the-other-new-ap...

99 points by ingve 3y ago

deleted 3y ago

[deleted]

An absolutely impressive achievement. I echo sentiments seen in other HN threads recently -- "I'd never do in-place FS upgrades!" -- yet Apple pulled it off on billions of devices with zero fanfare. Folk have scores of stores about btrfs/ZFS/etc issues, I don't recall similarly for APFS?

detourdog 3y ago
If you read the story you will hear about a test roll out an install. I was bitten by this and it was an awful couple of days of head scratching with Apple Support. My story is that I upgraded my mail server but the install hung never completed. I ended up with a a hard drive with no files and completely full. Diskutil and Disk Utility reported everything as good but there was a partition that could not be seen. I forget the eventual resolution. Probably starting all over again with a completely fresh install. I never understood what had happened until I saw a talk or read an interview with the APFS creaator. During the talk he spoke of how they did a complete re-format to APFS and verified everything and than reverted to HFS+ for the install. This is all from memory but I think I have the details mostly correct. The whole process was technically impressive but not without hiccups.
Rucadi 3y ago
I only had problems when installing Asahi linux, where APFS yeeted a little and the tools couldn't really recover the errors nicely.
zoky 3y ago
Ehh, when you control the hardware and software, an in-place filesystem upgrade isn’t that big of a deal. Especially when you can store everything in the cloud and pull it back down again if need be. Meanwhile, I’m expanding my ZFS single drives to mirrors and wondering why I can’t do the same with APFS. Apple should have embraced ZFS (and almost did, just like they almost embraced Ruby instead of Swift), but Apple is absolutely the worst offender when it comes to NIH syndrome.
- 130e13a 3y ago
  > just like they almost embraced Ruby instead of Swift do you have any more information about this?
  - jonhohle 3y ago
    I don’t think it was every one or the other. Ruby wa/ hot around 2005 and Apple experimented with language bindings for AppKit (and osascript?). I did a bunch of prototyping using MacRuby and then would rewrite in Objective-C. Ruby’s object model is relatively similar to objc - no direct member access outside the object, everything is loosely coupled messages with dynamic runtime, you could get away with duck typing in some cases in objc using `id` types (at the expense of getting type checks from the compiler). I don’t think it was ever intended at the next system language, though. It was just a convenient binding and optimized runtime (many of its core types were Core Foundation types - NSString, etc. IIRC).
    - detourdog 3y ago
      Almost any object scripting language can use Objective C Bridge for scripting. https://marketsplash.com/tutorials/objective-c/objective-c-bridge-pattern/ https://marketsplash.com/tutorials/objective-c/objective-c-b...
      - jonhohle 3y ago
        The difference was MacRuby was built on top of the Objective-C runtime. https://en.m.wikipedia.org/wiki/MacRuby https://en.m.wikipedia.org/wiki/MacRuby
        zoky 3y ago
        There was a brief window when it was possible to do some really cool stuff with MacRuby, such as inspecting and even monkey patching core OS X classes. Probably one major reason why Apple killed the project in the first place. Being able to interact with the operating system on a low level through macirb was an absolutely mind blowing experience. Sadly, those days are long gone, and I just don’t think Swift brings the same kind of flexibility or joy that MacRuby did.
geerlingguy 3y ago
I still wanted to see ZFS, would've been nice to go more cross-platform-compatible.
- boxed 3y ago
  There are a ton of features in ZFS that would have been very nice to have. Like streaming backup. That could enable a Time Machine 2.0 with enormous efficiency/speed gains.
steve1977 3y ago
> Folk have scores of stores about btrfs/ZFS/etc issues, I don't recall similarly for APFS? I don't think you can compare these in terms of features (hence also not in terms of potential problems)
madeofpalk 3y ago
The impressive part about the APFS migration was that they performed dry runs on iOS updates prior to measure success. > We actually had this process running for earlier iOS updates … where when you updated to 10.1 or 10.2, we were trial migrating your whole file system … consistency checking it … reporting back to us whether the upgrade was 100 percent clean, then rolling it back. https://www.macobserver.com/analysis/apple-dry-run-apfs-prior-ios-10-3/ https://www.macobserver.com/analysis/apple-dry-run-apfs-prio...
- detourdog 3y ago
  I was not impressed with the dry runs on MacOS I had mind bending issues that Apple support could not help with.

transpute 3y ago

iOS devices can read/write (but not initialize) APFS-encrypted external storage, e.g. NVME 2230 SSD.

Sweepi 3y ago

... and it was a disappointment. Apple had the one-in-30-years opportunity to ship a state of the art file system, and they choose to not have protection against user data bit rot (against meta data rot they choose yes, which is nice), which ZFS and others had for .. idk, a decade at this point?

specialist 3y ago
How feasible is it to add those kinds of features? Or would that have to be baked-in from the start?
- jonhohle 3y ago
  I wrote a small utility to find and dedup[0] using APFS features and had to poke around the user exposed metadata quite a bit. The short answer is probably. Metadata is extendable and there are things that work similarly already (encryption, and clone ids, for example). 0 - https://github.com/ttkb-oss/dedup https://github.com/ttkb-oss/dedup
postmodest 3y ago
...because their "disks" all have hardware ECC?
dmitrygr 3y ago
Why would this be needed, if every single Apple device uses flash memory which provides its own guarantees against bitrot?
- boxed 3y ago
  Because those guarantees aren't guarantees, more like partial mitigation.
  - dmitrygr 3y ago
    modern flash ECC may not always be able to correct but it is able to DETECT almost any number of bits wrong, so that is just as good as ZFS - if the data is banged - you'll know.

GeekyBear 3y ago

The story of how ZFS was almost added to OS X (along with other Sun tech like Dtrace which did get added) as told by a Sun insider. https://arstechnica.com/gadgets/2016/06/zfs-the-other-new-apple-file-system-that-almost-was-until-it-wasnt/ https://arstechnica.com/gadgets/2016/06/zfs-the-other-new-ap...