27 ms·

APFS changes in Big Sur: how Time Machine backs up to APFS

167 points by angristan 6y ago

aloknnikhil 6y ago
I'm not sure I got this from the article but what exactly is the advantage of using APFS over HFS+ for Time Machine? The only advantage that was highlighted was per file encryption which seems to be an exclusive to the Apple Silicon Macs. And perhaps the logical volume management that allows you to use an external drive as both a backup disk and a general purpose disk?
- skissane 6y ago
  > And perhaps the logical volume management that allows you to use an external drive as both a backup disk and a general purpose disk? You can already can do this without any logical volume management. Time Machine backups on a disk go into the /Backups.backupdb directory. You can create other subdirectories of the root directory and put other files there (or put files directly into the volume root), and Time Machine will just ignore them completely.
- bobbylarrybobby 6y ago
  I think that because APFS uses snapshots to monitor how files change over time, and APFS uses a block-level copy-on-write mechanism for files that change, time machine may be able to create smaller diffs when backing up over APFS because it only needs to store modified blocks, not entire modified files.
  - top_sigrid 6y ago
    Yes definitely. The internals of APFS offer lots of things like these that would be very benefitial for backups. So using APFS on the backup volume should increase backup speeds big time.
  - aloknnikhil 6y ago
    That sounds great! I have a workflow that uses a network mounted HFS+ image for Time Machine backups that are then backed up to my GDrive. Anything to reduce the incremental backup size would help this.
- dnh44 6y ago
  Copying files to an external APFS is a lot faster than copying to HFS+.
ChrisMarshallNY 6y ago
Man, I love this guy. He always gives the best geek stuff (Think "Mr. Universe," in Serenity).
centimeter 6y ago
It's too bad Apple had to scrap the plans to switch to ZFS back in the late oughts. We could have been here like 10 years ago!
- yjftsjthsd-h 6y ago
  With OpenZFS, we still could (and I believe, can, just unofficially).
- throw0101a 6y ago
  "had to" could be too strong a phrase, but not entirely wrong in a way: >> Apple can currently just take the ZFS CDDL code and incorporate it (like they did with DTrace), but it may be that they wanted a "private license" from Sun (with appropriate technical support and indemnification), and the two entities couldn't come to mutually agreeable terms. > I cannot disclose details, but that is the essence of it. * https://marc.info/?l=zfs-discuss&m=125642378308127 https://marc.info/?l=zfs-discuss&m=125642378308127 * https://arstechnica.com/gadgets/2009/10/apple-abandons-zfs-on-mac-os-x-project-over-licensing-issues/ https://arstechnica.com/gadgets/2009/10/apple-abandons-zfs-o... There was a NetApp-Sun legal spat going on due to NetApp's WAFL patents (Sun won IIRC), and indemnification could have been a source of tension.
- pfranz 6y ago
  My understanding is that while ZFS might be a great fit on the desktop, but it's way too RAM hungry for phones/ipads. Maybe with Apple contribution it could work better on mobile? Maybe I'm just ignorant of different tuning or features that could be disabled? I've only used it in rather simple scenarios. APFS was pretty explicitly designed for mobile, so I wouldn't be surprised if ZFS had been adopted on desktop they still made APFS for mobile.
  - happymellon 6y ago
    I use ZFS on my RaspberryPi. It works great even in lower RAM situations.
    - 6y ago
baggy_trough 6y ago
It'd sure be nice if Time Machine learned to create reliable backups over the network one of these decades.
- toomuchtodo 6y ago
  I really wish iCloud could be a Time Machine target while we’re talking about network backups.
  - jamroom 6y ago
    I agree with you 100% - this is something I've wanted for a really long time, and I just feel like it is something really obvious that Apple is overlooking. I'm not sure if it is ever coming, but man I'd love to see it happen.
    - gumby 6y ago
      My impression is this would be more bandwidth efficient (and thus perhaps possible) with APFS than HFS+ because of block level updates. Of course I have no clue whether Apple is thinking of this or not.
  - mumblerino 6y ago
    To do that, they'd have to lower those prices. They have been the same since 2017
    - deleted 6y ago
      [deleted]
  - ksec 6y ago
    I really wish Time Capsule could do iOS Backup.
- sumedh 6y ago
  Not sure why you have been downvoted but I can never get Time Machine to back up to my Synology NAS. Google search says there are lot of people who have the same issues.
RJIb8RBYxzAMX9u 6y ago
If I read the article correctly, Time Machine's switching to something similar to overlay FS, but nested? I really hope they get it right: backup is where my tolerance for bugs is really low. APFS already supports volume snapshots but not (I think) snapshot send / receive, à la zfs & btrfs. I thought that would be the logical next step. If Apple do add this or similar feature, would Time Machine have to switch its backing store yet again? I've been generally happy with Time Machine's stability, but this is getting me a little worried. I guess now's a good time to looking into Arq (or similar) to replace my janky secondary backup scripts...
- justAnotherNET 6y ago
  Meanwhile I can’t believe Apple went to the trouble of an entirely new FS without implementing BTRFS style block-level checksums. The number of data corruption issues this could have prevented by now are too high to count. Not to mention there is a direct monetary incentive for Apple to do so, as it slightly inflates storage requirements at the cost of safety (and apple commonly stratifies their products by storage capacity). It’s 2020, we need ECC RAM and block level checksums for all new platforms going forward.
  - pram 6y ago
    DDR5 has ECC, so at least that is happening soon.
    - AnonHP 6y ago
      I looked up DDR5 on Wikipedia, but it seems like ECC is just an option, which I understand was an option with previous generations too (at a much higher cost). Please elaborate on the ECC part, especially if manufacturers have committed to making only ECC versions.
      - pram 6y ago
        DDR5 standard has on-die ECC, it's not optional.
fiddlerwoaroof 6y ago
So, I generally like APFS, especially because I can have a case-sensitive volume for git repositories without stealing space from the root volume. However, has Apple fixed the issue where filling up the hard drive makes it impossible to delete files to free up some space?
- pfranz 6y ago
  I've hit this on other filesystems and I think I ended up doing: echo "" > largefile Then you can delete your files. Not sure if it'll work on APFS.
- layoutIfNeeded 6y ago
  Nope. I'm living on the edge with my 256gb MBP, and if it fills up the only recourse is rebooting, which frees up the swap file so you have enough space to delete things...
  - post_break 6y ago
    I read that onyx can delete snapshots without needing to reboot.
- jamiek88 6y ago
  aaaah! That explains it! This happened to me recently, couldn’t get my head round it but a reboot fixed things so I didn’t investigate more. Reboot must delete the swap or something.
- curt15 6y ago
  This is a fundamental problem of all copy-on-write filesystems. With btrfs, a quick fix is to plug in a spare usb stick whose space btrfs can borrow in order to delete files on the main drive.
  - fiddlerwoaroof 6y ago
    I might be missing something, but couldn’t you reserve a couple gigs of scratch space for this?
gogopuppygogo 6y ago
Not excited about this. Using APFS on my MacBook Pro, fighting a losing battle today where a file stored in an iCloud synced folder isn’t readable. First I try to copy the file out of the iCloud sync folder to a local folder that doesn’t sync. No change. Then, I go onto iCloud.com and download the file. The downloaded file shows up and works as expected. I ctrl + c then ctrl + command + v to move it into the local folder I had created. Now, according to finder, I have two files with the same name. One is readable and the other is not. I cannot delete either. I’ll be playing with the terminal next to see if anything can be done there.
- beervirus 6y ago
  What makes you think it's an APFS bug?
caycep 6y ago
Hm...I backup to a ubuntu lts box using TM over Samba...is this solution not going to be viable going forward? Maybe it’s time to swap to an external ssd...
diekhans 6y ago
Still no data checksums .. I will stick with ZFS
- nix23 6y ago
  Being a ZFS (on FBSD) user myself, it is a bit sad that everyone seams to forget NILFS2 (checksum s for Meta and Data)
yborg 6y ago
I wish they would fix the bug in Catalina where you can't restore to Mail from TM backups, it won't access any available backups. If you go into ~/Library/Mail and manually locate the account, it will access previous versions, but restoring these doesn't seem to recover individual inboxes, likely because there is an index somewhere else. Anyway, pretty bad...

167 points by angristan 6y ago

I'm not sure I got this from the article but what exactly is the advantage of using APFS over HFS+ for Time Machine? The only advantage that was highlighted was per file encryption which seems to be an exclusive to the Apple Silicon Macs. And perhaps the logical volume management that allows you to use an external drive as both a backup disk and a general purpose disk?

skissane 6y ago
> And perhaps the logical volume management that allows you to use an external drive as both a backup disk and a general purpose disk? You can already can do this without any logical volume management. Time Machine backups on a disk go into the /Backups.backupdb directory. You can create other subdirectories of the root directory and put other files there (or put files directly into the volume root), and Time Machine will just ignore them completely.
bobbylarrybobby 6y ago
I think that because APFS uses snapshots to monitor how files change over time, and APFS uses a block-level copy-on-write mechanism for files that change, time machine may be able to create smaller diffs when backing up over APFS because it only needs to store modified blocks, not entire modified files.
- top_sigrid 6y ago
  Yes definitely. The internals of APFS offer lots of things like these that would be very benefitial for backups. So using APFS on the backup volume should increase backup speeds big time.
- aloknnikhil 6y ago
  That sounds great! I have a workflow that uses a network mounted HFS+ image for Time Machine backups that are then backed up to my GDrive. Anything to reduce the incremental backup size would help this.
dnh44 6y ago
Copying files to an external APFS is a lot faster than copying to HFS+.

ChrisMarshallNY 6y ago

Man, I love this guy. He always gives the best geek stuff (Think "Mr. Universe," in Serenity).

centimeter 6y ago

It's too bad Apple had to scrap the plans to switch to ZFS back in the late oughts. We could have been here like 10 years ago!

yjftsjthsd-h 6y ago
With OpenZFS, we still could (and I believe, can, just unofficially).
throw0101a 6y ago
"had to" could be too strong a phrase, but not entirely wrong in a way: >> Apple can currently just take the ZFS CDDL code and incorporate it (like they did with DTrace), but it may be that they wanted a "private license" from Sun (with appropriate technical support and indemnification), and the two entities couldn't come to mutually agreeable terms. > I cannot disclose details, but that is the essence of it. * https://marc.info/?l=zfs-discuss&m=125642378308127 https://marc.info/?l=zfs-discuss&m=125642378308127 * https://arstechnica.com/gadgets/2009/10/apple-abandons-zfs-on-mac-os-x-project-over-licensing-issues/ https://arstechnica.com/gadgets/2009/10/apple-abandons-zfs-o... There was a NetApp-Sun legal spat going on due to NetApp's WAFL patents (Sun won IIRC), and indemnification could have been a source of tension.
pfranz 6y ago
My understanding is that while ZFS might be a great fit on the desktop, but it's way too RAM hungry for phones/ipads. Maybe with Apple contribution it could work better on mobile? Maybe I'm just ignorant of different tuning or features that could be disabled? I've only used it in rather simple scenarios. APFS was pretty explicitly designed for mobile, so I wouldn't be surprised if ZFS had been adopted on desktop they still made APFS for mobile.
- happymellon 6y ago
  I use ZFS on my RaspberryPi. It works great even in lower RAM situations.
  - 6y ago

baggy_trough 6y ago

It'd sure be nice if Time Machine learned to create reliable backups over the network one of these decades.

toomuchtodo 6y ago
I really wish iCloud could be a Time Machine target while we’re talking about network backups.
- jamroom 6y ago
  I agree with you 100% - this is something I've wanted for a really long time, and I just feel like it is something really obvious that Apple is overlooking. I'm not sure if it is ever coming, but man I'd love to see it happen.
  - gumby 6y ago
    My impression is this would be more bandwidth efficient (and thus perhaps possible) with APFS than HFS+ because of block level updates. Of course I have no clue whether Apple is thinking of this or not.
- mumblerino 6y ago
  To do that, they'd have to lower those prices. They have been the same since 2017
  - deleted 6y ago
    [deleted]
- ksec 6y ago
  I really wish Time Capsule could do iOS Backup.
sumedh 6y ago
Not sure why you have been downvoted but I can never get Time Machine to back up to my Synology NAS. Google search says there are lot of people who have the same issues.

RJIb8RBYxzAMX9u 6y ago

If I read the article correctly, Time Machine's switching to something similar to overlay FS, but nested? I really hope they get it right: backup is where my tolerance for bugs is really low. APFS already supports volume snapshots but not (I think) snapshot send / receive, à la zfs & btrfs. I thought that would be the logical next step. If Apple do add this or similar feature, would Time Machine have to switch its backing store yet again? I've been generally happy with Time Machine's stability, but this is getting me a little worried. I guess now's a good time to looking into Arq (or similar) to replace my janky secondary backup scripts...

justAnotherNET 6y ago
Meanwhile I can’t believe Apple went to the trouble of an entirely new FS without implementing BTRFS style block-level checksums. The number of data corruption issues this could have prevented by now are too high to count. Not to mention there is a direct monetary incentive for Apple to do so, as it slightly inflates storage requirements at the cost of safety (and apple commonly stratifies their products by storage capacity). It’s 2020, we need ECC RAM and block level checksums for all new platforms going forward.
- pram 6y ago
  DDR5 has ECC, so at least that is happening soon.
  - AnonHP 6y ago
    I looked up DDR5 on Wikipedia, but it seems like ECC is just an option, which I understand was an option with previous generations too (at a much higher cost). Please elaborate on the ECC part, especially if manufacturers have committed to making only ECC versions.
    - pram 6y ago
      DDR5 standard has on-die ECC, it's not optional.

fiddlerwoaroof 6y ago

So, I generally like APFS, especially because I can have a case-sensitive volume for git repositories without stealing space from the root volume. However, has Apple fixed the issue where filling up the hard drive makes it impossible to delete files to free up some space?

pfranz 6y ago
I've hit this on other filesystems and I think I ended up doing: echo "" > largefile Then you can delete your files. Not sure if it'll work on APFS.
layoutIfNeeded 6y ago
Nope. I'm living on the edge with my 256gb MBP, and if it fills up the only recourse is rebooting, which frees up the swap file so you have enough space to delete things...
- post_break 6y ago
  I read that onyx can delete snapshots without needing to reboot.
jamiek88 6y ago
aaaah! That explains it! This happened to me recently, couldn’t get my head round it but a reboot fixed things so I didn’t investigate more. Reboot must delete the swap or something.
curt15 6y ago
This is a fundamental problem of all copy-on-write filesystems. With btrfs, a quick fix is to plug in a spare usb stick whose space btrfs can borrow in order to delete files on the main drive.
- fiddlerwoaroof 6y ago
  I might be missing something, but couldn’t you reserve a couple gigs of scratch space for this?

gogopuppygogo 6y ago

Not excited about this. Using APFS on my MacBook Pro, fighting a losing battle today where a file stored in an iCloud synced folder isn’t readable. First I try to copy the file out of the iCloud sync folder to a local folder that doesn’t sync. No change. Then, I go onto iCloud.com and download the file. The downloaded file shows up and works as expected. I ctrl + c then ctrl + command + v to move it into the local folder I had created. Now, according to finder, I have two files with the same name. One is readable and the other is not. I cannot delete either. I’ll be playing with the terminal next to see if anything can be done there.

beervirus 6y ago
What makes you think it's an APFS bug?

caycep 6y ago

Hm...I backup to a ubuntu lts box using TM over Samba...is this solution not going to be viable going forward? Maybe it’s time to swap to an external ssd...

diekhans 6y ago

Still no data checksums .. I will stick with ZFS

nix23 6y ago
Being a ZFS (on FBSD) user myself, it is a bit sad that everyone seams to forget NILFS2 (checksum s for Meta and Data)

yborg 6y ago

I wish they would fix the bug in Catalina where you can't restore to Mail from TM backups, it won't access any available backups. If you go into ~/Library/Mail and manually locate the account, it will access previous versions, but restoring these doesn't seem to recover individual inboxes, likely because there is an index somewhere else. Anyway, pretty bad...