6 ms·

Apple's APFS Migration: A Feat of Engineering

91 points by goranmoomin 3y ago

glhaynes 3y ago
probably test their migrator for multiple months on hundreds or thousands of testing devices If I remember right, it was said at the time that they had included the migrator code in one or more of the releases before the one in which they actually did the migration. They ran it during those upgrades in a "do as much of the migration as we can without actually committing the changes" mode so that encountered errors could be reported back. So they had most of it tested on millions of devices.
- KRAKRISMOTT 3y ago
  They ought to compensate users for the excessive flash memory writes if the users are going to be guinea pigs for their beta test.
  - axoltl 3y ago
    The writes would go to /dev/null. No "excessive flash memory writes" were done, as they were testing the ingest portion.
  - orf 3y ago
    Why?
    - KRAKRISMOTT 3y ago
      Because flash memory has a limited number of write cycles and an entire file system migration will gobble up a good chunk of it.
      - orf 3y ago
        Yes, but why should users be compensated for that. And why would it use a large portion of that, rather than an additional 1 write per cell?
        KRAKRISMOTT 3y ago
        Because if you see OP's comment, they were testing their migrator on real users' devices (in what I presume is a blue green manner, without explicitly informing them except in small print) before their "official launch". If their beta test didn't go well, they will have to rollback their change, consuming more writes. https://news.ycombinator.com/item?id=40044310 https://news.ycombinator.com/item?id=40044310
  - glhaynes 3y ago
    You didn't get your check for 3 billionths of a penny when they were sending them out?
- mise_en_place 3y ago
  In addition, there is quite a bit of dogfooding that goes on at Apple.
  - JoshGlazebrook 3y ago
    I've read about those people who join a company just to fix a bug and its so tempting lol. I've been dealing with this bug for months now: https://www.reddit.com/r/MacOS/comments/17p415c/macos_sonoma_mouse_hover_bug_most_annoying_update/ https://www.reddit.com/r/MacOS/comments/17p415c/macos_sonoma... Hover just stops working on all windows unless you right click on another window, or hide windows to view desktop and come back.
    - wdb 3y ago
      I have a similar problem where the rollover region is not at the actual position of the rendered element but much higher. So you where you have to hover over is above the actual element. Really annoying. Happens a lot on Twitter or Youtube website but also at other places.
- misternugget 3y ago
  That's right, someone linked me to this article: https://www.macobserver.com/analysis/apple-dry-run-apfs-prior-ios-10-3/ https://www.macobserver.com/analysis/apple-dry-run-apfs-prio... Pretty impressive.
Night_Thastus 3y ago
It would be more impressive if I understood how big of a change this actually was. There's very little context in this post. What was the previous filesystem, and how different was APFS? If they both operated on the same core principles or underlying code, then the migration might have been trivial. Is it just a new "skin" on the prior FS? If it's a radically new filesystem based on all new concepts, then that's a much bigger feat of engineering. What did this new FS accomplish? Was it faster? More reliable? Easier to work with? Added functionality?
- SahAssar 3y ago
  The previous FS was HFS+ and APFS is quite different IIRC
- DigitalHackOp 3y ago
  It was a massive change, but one they prepared for by having the same migration happen on laptops/desktops The old filesystem was HFS+, released in 1998 and APFS was released in 2017. APFS added encryption, data integrity checksums, copy on write, snapshots, logical volume support. The migration wasn't just a "search and replace the name and ship it", but an intensive re-writing of the filesystem metadata during the migration. https://en.wikipedia.org/wiki/HFS_Plus https://en.wikipedia.org/wiki/HFS_Plus https://en.wikipedia.org/wiki/Apple_File_System https://en.wikipedia.org/wiki/Apple_File_System
  - kiwijamo 3y ago
    The APFS article you link to states that "APFS does not provide checksums for user data."
  - favorited 3y ago
    > data integrity checksums I believe it has checksums on its own metadata, but no checksums on actual user data. So you won't lose a whole folder or volume due to bitrot, but it doesn't provide protection against a flipped bit in one of your photos.
    - 3y ago
peppertree 3y ago
After seeing so many failed file system projects at vmware, I was pretty skeptical when I first heard about APFS. The fact it succeeded with so little noise just blew me away. It proves apples has the best system engineers in the world today.
- vondur 3y ago
  Here's some info from gherkin0 back from June of 2016: gherkin0 on June 26, 2016 | next [–] > Fun fact: Dominic Giampalo (who wrote the BeOS file system) is on the APFS team. His book "Practical File System Design" is an excellent description of a traditional UNIX file system design. May be out of print now but I think used copies turn up on Amazon. It looks like he has a PDF up on his website: http://www.nobius.org/~dbg/practical-file-system-design.pdf http://www.nobius.org/~dbg/practical-file-system-design.pdf
- moduspol 3y ago
  Absolutely. I doubt we'll ever see it, but it'd be really cool to see a blog post or write-up about just how many edge cases they found and had to work around--especially the more obscure ones. Obviously at Apple's scale, something that only hits one out of a million users still has the potential to be a big problem.
kimixa 3y ago
As someone who went through a MacOS APFS migration, I can certainly say it wasn't smooth. Maybe the limited capabilities that mobile device APIs give apps and user was helpful avoiding issues - can't have multiple partitions or weird time machine issues on something without partitions or time machine.
- Tagbert 3y ago
  I had no problems with the Mac OS upgrade but then I didn't have custom partitions setup and Time Machine on an external drive has always worked for me.
  - nottorp 3y ago
    Worked on my hackintosh back then. But again, just one partition. I was triple booting but always disconnected the other drives when doing OS upgrades.
- D13Fd 3y ago
  Honestly I think that's because Apple cares far more about the iOS devices than it does about Mac OS (and did even in 2017). Mac OS is a second-class citizen for Apple. The main focus is on devices, and then services.
daeros 3y ago
[flagged]
- daeros 3y ago
  [flagged]
iafisher 3y ago
Reminds me of Google flipping Nest devices from a Linux-based OS to Fuchsia: https://9to5google.com/2021/08/17/google-fuchsia-update-rolling-out-widely-nest-hub/ https://9to5google.com/2021/08/17/google-fuchsia-update-roll... A lot fewer devices than Apple, but with changing a device's entire operating system a lot more can go wrong. I wasn't on the team at the time, but maybe someone else can chime in with more details.
cglong 3y ago
This certainly was not my experience. For additional context, APFS is only supported on SSDs. My iMac didn't have an SSD, but it did have an upgraded HDD that I chose while buying the machine from Apple. While performing the OS upgrade that was supposed to determine APFS wasn't applicable, it thrashed the bootloader and wouldn't proceed (something about not finding the unmounted partition? IDK it was several years ago). I hadn't been keeping proper backups either, but was able to manually backup the most important stuff using Target Disk Mode. Then, to Apple's credit, I eventually did a full reinstallation of the broken macOS, which amazingly kept my files intact. EDIT: I originally said the migration "bricked my computer", but greedo is correct that it wasn't permanent damage.
- greedo 3y ago
  So your computer eventually was usable? That's not what "bricked" means?
  - cglong 3y ago
    I guess not, but it did render it inoperable for several months. Someone less technical wouldn't have been able to recover it at all (nor could the Apple Geniuses over my 2-3 visits there).
hoistbypetard 3y ago
I think it was also helped by the fact that iPhones and iPads are basically consoles. It's significantly easier to predict the state of the filesystem on those than on general purpose PCs. It was a much more perilous migration on Macs.
misiek08 3y ago
iOS 10, released 27.03.2017, migrated filesystem on millions of devices without major issues. iOS 17.4.1, minor update, thousands of reports with failing cellular data. IT world is going better direction every day. Congratz for those real engineers, not hype following sheeps :)
deleted 3y ago
[deleted]
forty 3y ago
It reminds me of AWS making suddenly S3 strongly consistent https://aws.amazon.com/s3/consistency/ https://aws.amazon.com/s3/consistency/ No idea what it involved under the hood, but I assume some magic
martin_drapeau 3y ago
I sometimes listen to the All in podcast. The 4 VCs recently blasted Apple saying they don't know why there are so many engineers. And they don't know why an iPhone is so expensive. They last longer now and don't need to upgrade as often. To their eyes an iPhone has become boring. My reply is Apple supports close to 2 billion devices. They have to work and never crash. They also need to get upgraded for security patches and new features. The fact they find them "boring" is a good thing. Means iPhones have become an intrisic part of their lives. This article describes one of the many things Apple has to do to make them "boring". Indeed a feat of engineering. Now, when they replicate that success with Siri, that will be a game changer. Hopefully advances in LLMs will bring that day forward.
Kwpolska 3y ago
Windows also had FAT16 -> FAT32 and FAT32 -> NTFS converters. In the 90s. But nobody produced blog posts praising them for them, this was just a Windows feature.
rising-sky 3y ago
Craig Federighi talking about how the team pulled this off https://youtu.be/IcyaadNy9Jk?t=1639 https://youtu.be/IcyaadNy9Jk?t=1639
daeros 3y ago
[flagged]
- fckgw 3y ago
  [flagged]
  - daeros 3y ago
    [flagged]
    - fckgw 3y ago
      [flagged]
      - daeros 3y ago
        [flagged]
- SpecialistK 3y ago
  That's nice. Not sure how it's relevant to this discussion about their file system migration though.
  - daeros 3y ago
    [flagged]
    - SpecialistK 3y ago
      [flagged]
- dang 3y ago
  Can you please not post in the flamewar style like this? It's not what this site is for, and destroys what it is for, so we have to ban accounts that do it repeatedly. If you'd please review https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and stick to them, that would be good. You can still make your substantive points thoughtfully. But please avoid generic tangents (e.g. $BigCo-boo-or-yay) in addition to name-calling, fulmination, and the other things the guidelines ask you not to do.

91 points by goranmoomin 3y ago

probably test their migrator for multiple months on hundreds or thousands of testing devices If I remember right, it was said at the time that they had included the migrator code in one or more of the releases before the one in which they actually did the migration. They ran it during those upgrades in a "do as much of the migration as we can without actually committing the changes" mode so that encountered errors could be reported back. So they had most of it tested on millions of devices.

KRAKRISMOTT 3y ago
They ought to compensate users for the excessive flash memory writes if the users are going to be guinea pigs for their beta test.
- axoltl 3y ago
  The writes would go to /dev/null. No "excessive flash memory writes" were done, as they were testing the ingest portion.
- orf 3y ago
  Why?
  - KRAKRISMOTT 3y ago
    Because flash memory has a limited number of write cycles and an entire file system migration will gobble up a good chunk of it.
    - orf 3y ago
      Yes, but why should users be compensated for that. And why would it use a large portion of that, rather than an additional 1 write per cell?
      - KRAKRISMOTT 3y ago
        Because if you see OP's comment, they were testing their migrator on real users' devices (in what I presume is a blue green manner, without explicitly informing them except in small print) before their "official launch". If their beta test didn't go well, they will have to rollback their change, consuming more writes. https://news.ycombinator.com/item?id=40044310 https://news.ycombinator.com/item?id=40044310
- glhaynes 3y ago
  You didn't get your check for 3 billionths of a penny when they were sending them out?
mise_en_place 3y ago
In addition, there is quite a bit of dogfooding that goes on at Apple.
- JoshGlazebrook 3y ago
  I've read about those people who join a company just to fix a bug and its so tempting lol. I've been dealing with this bug for months now: https://www.reddit.com/r/MacOS/comments/17p415c/macos_sonoma_mouse_hover_bug_most_annoying_update/ https://www.reddit.com/r/MacOS/comments/17p415c/macos_sonoma... Hover just stops working on all windows unless you right click on another window, or hide windows to view desktop and come back.
  - wdb 3y ago
    I have a similar problem where the rollover region is not at the actual position of the rendered element but much higher. So you where you have to hover over is above the actual element. Really annoying. Happens a lot on Twitter or Youtube website but also at other places.
misternugget 3y ago
That's right, someone linked me to this article: https://www.macobserver.com/analysis/apple-dry-run-apfs-prior-ios-10-3/ https://www.macobserver.com/analysis/apple-dry-run-apfs-prio... Pretty impressive.

Night_Thastus 3y ago

It would be more impressive if I understood how big of a change this actually was. There's very little context in this post. What was the previous filesystem, and how different was APFS? If they both operated on the same core principles or underlying code, then the migration might have been trivial. Is it just a new "skin" on the prior FS? If it's a radically new filesystem based on all new concepts, then that's a much bigger feat of engineering. What did this new FS accomplish? Was it faster? More reliable? Easier to work with? Added functionality?

SahAssar 3y ago
The previous FS was HFS+ and APFS is quite different IIRC
DigitalHackOp 3y ago
It was a massive change, but one they prepared for by having the same migration happen on laptops/desktops The old filesystem was HFS+, released in 1998 and APFS was released in 2017. APFS added encryption, data integrity checksums, copy on write, snapshots, logical volume support. The migration wasn't just a "search and replace the name and ship it", but an intensive re-writing of the filesystem metadata during the migration. https://en.wikipedia.org/wiki/HFS_Plus https://en.wikipedia.org/wiki/HFS_Plus https://en.wikipedia.org/wiki/Apple_File_System https://en.wikipedia.org/wiki/Apple_File_System
- kiwijamo 3y ago
  The APFS article you link to states that "APFS does not provide checksums for user data."
- favorited 3y ago
  > data integrity checksums I believe it has checksums on its own metadata, but no checksums on actual user data. So you won't lose a whole folder or volume due to bitrot, but it doesn't provide protection against a flipped bit in one of your photos.
  - 3y ago

peppertree 3y ago

After seeing so many failed file system projects at vmware, I was pretty skeptical when I first heard about APFS. The fact it succeeded with so little noise just blew me away. It proves apples has the best system engineers in the world today.

vondur 3y ago
Here's some info from gherkin0 back from June of 2016: gherkin0 on June 26, 2016 | next [–] > Fun fact: Dominic Giampalo (who wrote the BeOS file system) is on the APFS team. His book "Practical File System Design" is an excellent description of a traditional UNIX file system design. May be out of print now but I think used copies turn up on Amazon. It looks like he has a PDF up on his website: http://www.nobius.org/~dbg/practical-file-system-design.pdf http://www.nobius.org/~dbg/practical-file-system-design.pdf
moduspol 3y ago
Absolutely. I doubt we'll ever see it, but it'd be really cool to see a blog post or write-up about just how many edge cases they found and had to work around--especially the more obscure ones. Obviously at Apple's scale, something that only hits one out of a million users still has the potential to be a big problem.

kimixa 3y ago

As someone who went through a MacOS APFS migration, I can certainly say it wasn't smooth. Maybe the limited capabilities that mobile device APIs give apps and user was helpful avoiding issues - can't have multiple partitions or weird time machine issues on something without partitions or time machine.

Tagbert 3y ago
I had no problems with the Mac OS upgrade but then I didn't have custom partitions setup and Time Machine on an external drive has always worked for me.
- nottorp 3y ago
  Worked on my hackintosh back then. But again, just one partition. I was triple booting but always disconnected the other drives when doing OS upgrades.
D13Fd 3y ago
Honestly I think that's because Apple cares far more about the iOS devices than it does about Mac OS (and did even in 2017). Mac OS is a second-class citizen for Apple. The main focus is on devices, and then services.

daeros 3y ago

[flagged]

daeros 3y ago
[flagged]

iafisher 3y ago

Reminds me of Google flipping Nest devices from a Linux-based OS to Fuchsia: https://9to5google.com/2021/08/17/google-fuchsia-update-rolling-out-widely-nest-hub/ https://9to5google.com/2021/08/17/google-fuchsia-update-roll... A lot fewer devices than Apple, but with changing a device's entire operating system a lot more can go wrong. I wasn't on the team at the time, but maybe someone else can chime in with more details.

cglong 3y ago

This certainly was not my experience. For additional context, APFS is only supported on SSDs. My iMac didn't have an SSD, but it did have an upgraded HDD that I chose while buying the machine from Apple. While performing the OS upgrade that was supposed to determine APFS wasn't applicable, it thrashed the bootloader and wouldn't proceed (something about not finding the unmounted partition? IDK it was several years ago). I hadn't been keeping proper backups either, but was able to manually backup the most important stuff using Target Disk Mode. Then, to Apple's credit, I eventually did a full reinstallation of the broken macOS, which amazingly kept my files intact. EDIT: I originally said the migration "bricked my computer", but greedo is correct that it wasn't permanent damage.

greedo 3y ago
So your computer eventually was usable? That's not what "bricked" means?
- cglong 3y ago
  I guess not, but it did render it inoperable for several months. Someone less technical wouldn't have been able to recover it at all (nor could the Apple Geniuses over my 2-3 visits there).

hoistbypetard 3y ago

I think it was also helped by the fact that iPhones and iPads are basically consoles. It's significantly easier to predict the state of the filesystem on those than on general purpose PCs. It was a much more perilous migration on Macs.

misiek08 3y ago

iOS 10, released 27.03.2017, migrated filesystem on millions of devices without major issues. iOS 17.4.1, minor update, thousands of reports with failing cellular data. IT world is going better direction every day. Congratz for those real engineers, not hype following sheeps :)

deleted 3y ago

[deleted]

forty 3y ago

It reminds me of AWS making suddenly S3 strongly consistent https://aws.amazon.com/s3/consistency/ https://aws.amazon.com/s3/consistency/ No idea what it involved under the hood, but I assume some magic

martin_drapeau 3y ago

I sometimes listen to the All in podcast. The 4 VCs recently blasted Apple saying they don't know why there are so many engineers. And they don't know why an iPhone is so expensive. They last longer now and don't need to upgrade as often. To their eyes an iPhone has become boring. My reply is Apple supports close to 2 billion devices. They have to work and never crash. They also need to get upgraded for security patches and new features. The fact they find them "boring" is a good thing. Means iPhones have become an intrisic part of their lives. This article describes one of the many things Apple has to do to make them "boring". Indeed a feat of engineering. Now, when they replicate that success with Siri, that will be a game changer. Hopefully advances in LLMs will bring that day forward.

Kwpolska 3y ago

Windows also had FAT16 -> FAT32 and FAT32 -> NTFS converters. In the 90s. But nobody produced blog posts praising them for them, this was just a Windows feature.

rising-sky 3y ago

Craig Federighi talking about how the team pulled this off https://youtu.be/IcyaadNy9Jk?t=1639 https://youtu.be/IcyaadNy9Jk?t=1639

daeros 3y ago

[flagged]

fckgw 3y ago
[flagged]
- daeros 3y ago
  [flagged]
  - fckgw 3y ago
    [flagged]
    - daeros 3y ago
      [flagged]
SpecialistK 3y ago
That's nice. Not sure how it's relevant to this discussion about their file system migration though.
- daeros 3y ago
  [flagged]
  - SpecialistK 3y ago
    [flagged]
dang 3y ago
Can you please not post in the flamewar style like this? It's not what this site is for, and destroys what it is for, so we have to ban accounts that do it repeatedly. If you'd please review https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and stick to them, that would be good. You can still make your substantive points thoughtfully. But please avoid generic tangents (e.g. $BigCo-boo-or-yay) in addition to name-calling, fulmination, and the other things the guidelines ask you not to do.