8 ms·

APFS – A Backup Software Developer’s Perspective

168 points by 01001 9y ago

sgs1370 9y ago
thanks - this is a good heads up. I asked my team to back up anything not checked into our git repo to some disk not controlled by OSX, and btw don't rely on your time machine, who knows if some bug in the new FS might affect that also. Better safe than sorry. I finished my note with a suggestion to just wait a few months before upgrading to 10.13
- tinus_hn 9y ago
  You can also keep your Time Machine disk disconnected during the upgrade so it won’t be touched by the installer. In my opinion not a bad idea anyway.
the_common_man 9y ago
I was hoping to find atleast one technical detail of why this is an engineering challenge. But the author simply repeats the same thing over and again about how it's hard, how competitor says it's hard, how others think it's impossible. In the end it was flawless
- Girlang 9y ago
  Apple AstroTurf.
dsabanin 9y ago
"So, before installing 10.13, I started off by googling for any problems experienced by people installing and upgrading filesystems during the process using the public betas. To my surprise there were none." Well, my quick visit to developer discussions forum at Apple website related to macOs Beta quickly revealed a lot of people with the problems. It seems like the author didn't look too far: https://forums.developer.apple.com/community/beta/macos-1013-beta https://forums.developer.apple.com/community/beta/macos-1013...
- radicaldreamer 9y ago
  Yep, there are a ton of bugs in High Sierra. I think this is going to be a risky upgrade for a lot of Macs. From video drivers not being ready to ship to weird issues with permissions and APFS, there is going to be a long tail of misery and issues on the 25th.
- dpkonofa 9y ago
  That link shows discussions related to problems with the OS but it doesn't list any problems with the filesystem upgrade. If it does, they might have been bumped down by more recent posts but the only major issue (and the person posting it admits that they changed some permissions) is a read-only drive. That seems like pretty good odds.
  - dsabanin 9y ago
    If you go beyond the first page you will see more, including data loss after a few days of usage and during setup. I'm not trying to be negative, I was just pointing out the results of my own research into this question when I was considering to try High Sierra beta. It's not "zero issues" as the original author wrote.
    - dpkonofa 9y ago
      Mind linking to one? I can't find anything that has data loss in the subject and most of the other threads are not about APFS. The only one I can still see is regarding a kernel panic but it doesn't give any details, just an assumption that the issue is with APFS.
      - dsabanin 9y ago
        Sure. "I converted my Macintosh HD to the APFS and it will not let me boot it up anymore." https://forums.developer.apple.com/thread/80784?q=apfs https://forums.developer.apple.com/thread/80784?q=apfs "1TB APFS Container Corrupt, Then Reinit to Single Volume" https://forums.developer.apple.com/message/247781#247781 https://forums.developer.apple.com/message/247781#247781 "APFS Brand new filesystem corruption" https://forums.developer.apple.com/message/145039#145039 https://forums.developer.apple.com/message/145039#145039 "Re: Corrupted APFS disk map error" https://forums.developer.apple.com/message/254468#254468 https://forums.developer.apple.com/message/254468#254468 "Device does not contain a valid APFS container" https://forums.developer.apple.com/message/235888#235888 https://forums.developer.apple.com/message/235888#235888 "Re: Files & file changes disappear on APFS volume" https://forums.developer.apple.com/message/241082#241082 https://forums.developer.apple.com/message/241082#241082 "Re: MacOs High Sierra Beta corrupted my disk" https://forums.developer.apple.com/message/244120#244120 https://forums.developer.apple.com/message/244120#244120 I'm not really interested in debating this further, so if you are not satisfied by these references let's agree to disagree :)
dunham 9y ago
I'm really curious what is going wrong with Bear.app on APFS. The theming engine breaks on APFS (all the themes look like one particular theme) - but it works fine if you copy the app to a disk image formatted with any other filesystem, including Case-sensitive APFS (so it only breaks on case-insensitive APFS). Nothing stood out to me in the dtruss output.
- citruspi 9y ago
  > The theming engine breaks on APFS (all the themes look like one particular theme) I'm not sure if I'm running into the same problem as you, but I just opened up Bear on my Mac to check and applying the themes themselves works for me, however the preview of each theme is the same. This is on macOS 10.13 Beta (17A306f) and Bear 1.0.4 (3733). Does each theme look identical when applied for you? [0]: https://imgur.com/dBswvky https://imgur.com/dBswvky [1]: https://imgur.com/jpHbS4i https://imgur.com/jpHbS4i [2]: https://imgur.com/LxUWWS3 https://imgur.com/LxUWWS3
  - dunham 9y ago
    Yeah, same issue. Pretty jarring for me, because I like the white theme. Make an HFS dmg in disk utility and move the app over and it will work fine. Surprisingly, it works fine on "APFS (Case Sensitive)" too (I had thought maybe it was a case-folding issue). Edit: Oh, for me the theme works for the list of documents, just not the "edit" panel. The two halves are stored in separate files on disk for some reason. (https://i.imgur.com/ZGS3Fp9.png https://i.imgur.com/ZGS3Fp9.png)
  - danieldk 9y ago
    For me, the themes look the same in the preview pane. Interestingly enough, changing the theme changes every pane except for the editor pane (which is stuck with one theme). Restarting does not help. Bear 1.2.5 (4944) and macOS 10.13 (17A362a).
jsjohnst 9y ago
> uses Rsync under the hood In that case, what difference does the underlying block storage format really matter? If rsync didn’t work, general applications could easily be expected to have issues too.
- feelix 9y ago
  It's the metadata that's really important. Just copying a file, while faithfully preserving metadata, is actually a feat on HFS+. That is why popular tools such as BackupBouncer (to test the integrity of backups) came to be.
  - jsjohnst 9y ago
    I think you missed my point, everything rsync does is via the very same system calls any other app could use to originally get or set that metadata. As such, the kernel file system layer abstracts the block storage away from you. This is very different than say Shirt Pocket’s approach (linked to in the article) where they are directly accessing/modifying/creating low level block devices. I’d expect what they are doing to be SIGNIFICANTLY harder. The reason why OP is generally positive is because essentially the only thing they had to do to support APFS is a bunch of testing, unless there’s a part to this story they didn’t tell.
    - feelix 9y ago
      I honestly have no idea what you are talking about. Shirt Pocket would not "access/modify/create low level block devices". They would also be working at the file layer. If you mean that you think that they are implementing their own way of reading the filesystem through the POSIX node, I really dont think that they are, and also it would be crazy to try to do that.
cyphar 9y ago
If your backup tool uses rsync under the hood, which is just a normal user-space process that uses the standard filesystem APIs, why does it matter what the underlying storage is? Obviously filesystem bugs can cause issues with this assumption, but filesystem bugs can break a whole host of other things in your backup tool.
- sprayk 9y ago
  This was what I was left wondering as well. rsync for sure does not have any FS-specific code that I can find. If it uses the FS APIs provided by the OS, just like every other program that touches files, why would we expect anything other than boring old rsync doing it's thing?
  - microcolonel 9y ago
    Filenames are treated a bit differently on APFS. Not sure what else, but it seems there's a lot of other bizarre file metadata on OS X that you might want to sync. edit: not really sure though, because it doesn't seem like this program cares about any of those metadata.
    - feelix 9y ago
      It cares about those metadata a lot. Backups wont be bootable without it. And there is also disastrous data loss without it, a lot of files have their actual file data embedded in the metadata (resource forks et al). It's nuts because it's based on a 30 year old filesystem. That's one of the reasons why the smooth transition is impressive.
  - Someone 9y ago
    The Mac version has some file system specific code https://developer.apple.com/legacy/library/documentation/Darwin/Reference/ManPages/man1/rsync.1.html https://developer.apple.com/legacy/library/documentation/Dar...: ”E, --extended-attributes copy extended attributes, resource forks”
  - feelix 9y ago
swyx 9y ago
>Although too early to declare victory just yet, my anecdotal experiences combined with my research on the web does lead me to think that the mass transition from HFS+ to APFS is going to be another victory for Apple. This opinion differs from all others I have read from well informed people, many of who are predicting either catastrophe or at the very least some major stumbling blocks. In this case, time will tell, but I am bullish where most others are bearish. The only thing that bothers me is that I can’t understand how they managed to pull it off. Kudos to Apple!
realityloop 9y ago
I haven’t tried the GM seed yet, but I’ve reported a pretty annoying bug with inconsistent directory listings from host into guest with vagrant. A listing in the guest will be missing entries, but you can CD into the missing folders. This causes sites not to work as the application in the guest can’t see files it requires. The only way to resolve it to date has been backup and reinstall with without apfs.
sigi45 9y ago
Ah yeah okay tx. I started to be disappointed as soon as i read that he is using rsync. I was expecting someone who writes something like rsync and not a gui for rsync.
eps 9y ago
And this is how you do SEO-oriented content marketing, kids. Rather light on the substance, few select links (with at least one pointing out) and lots and lots of precisous keywords. -- Besides, the fact that a _file_ backup tool might need a near complete rewrite with a change of a file system should raise some eyebrows. It might be understandable for an imaging backup or if it was working directly with raw file system, but this one doesn't.
- feelix 9y ago
  I finished up the APFS compatible version of the software way quicker than I thought I would, and was so happy and energized that as part of my release I whipped up a quick blog post about the experience. Indeed it's pretty light on content, it's more just a nod to Apple for making this a painless process rather than an excruciating one, as well as an early relay of the experience to others who are going to upgrade who may be harboring some concerns like I was. The marketing for this site sucks, the software has been around since 2011 and it has nowhere near the marketshare of its competitors because it sucks. If I were really some SEO genius (and you'll see the google rankings are terrible), then this would not be the case. In short, I wish I was the evil genius that you claim, but I'm not. I'm just a dev sharing his experience with the filesystem. In regards to it not implementing the file copying tool directly - that is covered in the article. The fact that the file copying tool was compiled in 2015 and still works flawlessly is the whole premise of the article stating that it's impressive. And it being a _file_ copying tool is the part that _makes_ it impressive. You seem to have no idea how complex the metadata, built up over 30 years without a rewrite, on HFS+ is. Take a look at BackupBouncer and the associated blog posts to get an idea of the situation. It's pretty crazy.
- Confiks 9y ago
  > And this is how you do SEO-oriented content marketing, kids. And this in turn, kids, is how you assume bad faith without any reservation or doubt.
  - yjftsjthsd-h 9y ago
    Having watched the news for a while, primarily on HN and Slashdot, I've seen enough tracking, leaks, sell-outs, adware incidents, dirty marketing, astroturf campaigns, compromised software, etc. to not expect some degree of bad faith. Perhaps not this time, but it is totally reasonable to question everything.
SyneRyder 9y ago
A few people here are disappointed that this post is about a tool that uses rsync. But the "competitor" link is more interesting - it points to the makers of SuperDuper!, a tool that makes bootable external backups. A couple of highlights: "...it's important to note that Apple still hasn't released any documentation on the "proper" way to create a bootable APFS volume. An example of what they have in mind was released for the very first time when the High Sierra developer release came out a few months ago, but that's it. We basically have to make an educated guess about what they want." and "In particular, Apple has further tightened its System Integrity Protection process, and is completely denying access to some files on the startup volume, even when copying to a non-startup volume.... But since APFS is, again, basically undocumented... what could that backup be missing?" http://www.shirt-pocket.com/blog/index.php/comments/news_on_the_march http://www.shirt-pocket.com/blog/index.php/comments/news_on_...
jason_slack 9y ago
Having written a backup app several years ago, I struggled to gain foothold in the market. I sold the app off. After a few years of wanting more from my backup, I started a new backup app. I utilize a number of commands under the hood but it is an ncurses interface only right now (supports drag and drop still into a terminal window). I'd guess that if I wanted to put it out for sale I'd need to add a GUI on top that. Can anyone shed some light on why backup apps are struggling with preserving metadata?

168 points by 01001 9y ago

thanks - this is a good heads up. I asked my team to back up anything not checked into our git repo to some disk not controlled by OSX, and btw don't rely on your time machine, who knows if some bug in the new FS might affect that also. Better safe than sorry. I finished my note with a suggestion to just wait a few months before upgrading to 10.13

tinus_hn 9y ago
You can also keep your Time Machine disk disconnected during the upgrade so it won’t be touched by the installer. In my opinion not a bad idea anyway.

the_common_man 9y ago

I was hoping to find atleast one technical detail of why this is an engineering challenge. But the author simply repeats the same thing over and again about how it's hard, how competitor says it's hard, how others think it's impossible. In the end it was flawless

Girlang 9y ago
Apple AstroTurf.

dsabanin 9y ago

"So, before installing 10.13, I started off by googling for any problems experienced by people installing and upgrading filesystems during the process using the public betas. To my surprise there were none." Well, my quick visit to developer discussions forum at Apple website related to macOs Beta quickly revealed a lot of people with the problems. It seems like the author didn't look too far: https://forums.developer.apple.com/community/beta/macos-1013-beta https://forums.developer.apple.com/community/beta/macos-1013...

radicaldreamer 9y ago
Yep, there are a ton of bugs in High Sierra. I think this is going to be a risky upgrade for a lot of Macs. From video drivers not being ready to ship to weird issues with permissions and APFS, there is going to be a long tail of misery and issues on the 25th.
dpkonofa 9y ago
That link shows discussions related to problems with the OS but it doesn't list any problems with the filesystem upgrade. If it does, they might have been bumped down by more recent posts but the only major issue (and the person posting it admits that they changed some permissions) is a read-only drive. That seems like pretty good odds.
- dsabanin 9y ago
  If you go beyond the first page you will see more, including data loss after a few days of usage and during setup. I'm not trying to be negative, I was just pointing out the results of my own research into this question when I was considering to try High Sierra beta. It's not "zero issues" as the original author wrote.
  - dpkonofa 9y ago
    Mind linking to one? I can't find anything that has data loss in the subject and most of the other threads are not about APFS. The only one I can still see is regarding a kernel panic but it doesn't give any details, just an assumption that the issue is with APFS.
    - dsabanin 9y ago
      Sure. "I converted my Macintosh HD to the APFS and it will not let me boot it up anymore." https://forums.developer.apple.com/thread/80784?q=apfs https://forums.developer.apple.com/thread/80784?q=apfs "1TB APFS Container Corrupt, Then Reinit to Single Volume" https://forums.developer.apple.com/message/247781#247781 https://forums.developer.apple.com/message/247781#247781 "APFS Brand new filesystem corruption" https://forums.developer.apple.com/message/145039#145039 https://forums.developer.apple.com/message/145039#145039 "Re: Corrupted APFS disk map error" https://forums.developer.apple.com/message/254468#254468 https://forums.developer.apple.com/message/254468#254468 "Device does not contain a valid APFS container" https://forums.developer.apple.com/message/235888#235888 https://forums.developer.apple.com/message/235888#235888 "Re: Files & file changes disappear on APFS volume" https://forums.developer.apple.com/message/241082#241082 https://forums.developer.apple.com/message/241082#241082 "Re: MacOs High Sierra Beta corrupted my disk" https://forums.developer.apple.com/message/244120#244120 https://forums.developer.apple.com/message/244120#244120 I'm not really interested in debating this further, so if you are not satisfied by these references let's agree to disagree :)

dunham 9y ago

I'm really curious what is going wrong with Bear.app on APFS. The theming engine breaks on APFS (all the themes look like one particular theme) - but it works fine if you copy the app to a disk image formatted with any other filesystem, including Case-sensitive APFS (so it only breaks on case-insensitive APFS). Nothing stood out to me in the dtruss output.

citruspi 9y ago
> The theming engine breaks on APFS (all the themes look like one particular theme) I'm not sure if I'm running into the same problem as you, but I just opened up Bear on my Mac to check and applying the themes themselves works for me, however the preview of each theme is the same. This is on macOS 10.13 Beta (17A306f) and Bear 1.0.4 (3733). Does each theme look identical when applied for you? [0]: https://imgur.com/dBswvky https://imgur.com/dBswvky [1]: https://imgur.com/jpHbS4i https://imgur.com/jpHbS4i [2]: https://imgur.com/LxUWWS3 https://imgur.com/LxUWWS3
- dunham 9y ago
  Yeah, same issue. Pretty jarring for me, because I like the white theme. Make an HFS dmg in disk utility and move the app over and it will work fine. Surprisingly, it works fine on "APFS (Case Sensitive)" too (I had thought maybe it was a case-folding issue). Edit: Oh, for me the theme works for the list of documents, just not the "edit" panel. The two halves are stored in separate files on disk for some reason. (https://i.imgur.com/ZGS3Fp9.png https://i.imgur.com/ZGS3Fp9.png)
- danieldk 9y ago
  For me, the themes look the same in the preview pane. Interestingly enough, changing the theme changes every pane except for the editor pane (which is stuck with one theme). Restarting does not help. Bear 1.2.5 (4944) and macOS 10.13 (17A362a).

jsjohnst 9y ago

> uses Rsync under the hood In that case, what difference does the underlying block storage format really matter? If rsync didn’t work, general applications could easily be expected to have issues too.

feelix 9y ago
It's the metadata that's really important. Just copying a file, while faithfully preserving metadata, is actually a feat on HFS+. That is why popular tools such as BackupBouncer (to test the integrity of backups) came to be.
- jsjohnst 9y ago
  I think you missed my point, everything rsync does is via the very same system calls any other app could use to originally get or set that metadata. As such, the kernel file system layer abstracts the block storage away from you. This is very different than say Shirt Pocket’s approach (linked to in the article) where they are directly accessing/modifying/creating low level block devices. I’d expect what they are doing to be SIGNIFICANTLY harder. The reason why OP is generally positive is because essentially the only thing they had to do to support APFS is a bunch of testing, unless there’s a part to this story they didn’t tell.
  - feelix 9y ago
    I honestly have no idea what you are talking about. Shirt Pocket would not "access/modify/create low level block devices". They would also be working at the file layer. If you mean that you think that they are implementing their own way of reading the filesystem through the POSIX node, I really dont think that they are, and also it would be crazy to try to do that.

cyphar 9y ago

If your backup tool uses rsync under the hood, which is just a normal user-space process that uses the standard filesystem APIs, why does it matter what the underlying storage is? Obviously filesystem bugs can cause issues with this assumption, but filesystem bugs can break a whole host of other things in your backup tool.

sprayk 9y ago
This was what I was left wondering as well. rsync for sure does not have any FS-specific code that I can find. If it uses the FS APIs provided by the OS, just like every other program that touches files, why would we expect anything other than boring old rsync doing it's thing?
- microcolonel 9y ago
  Filenames are treated a bit differently on APFS. Not sure what else, but it seems there's a lot of other bizarre file metadata on OS X that you might want to sync. edit: not really sure though, because it doesn't seem like this program cares about any of those metadata.
  - feelix 9y ago
    It cares about those metadata a lot. Backups wont be bootable without it. And there is also disastrous data loss without it, a lot of files have their actual file data embedded in the metadata (resource forks et al). It's nuts because it's based on a 30 year old filesystem. That's one of the reasons why the smooth transition is impressive.
- Someone 9y ago
  The Mac version has some file system specific code https://developer.apple.com/legacy/library/documentation/Darwin/Reference/ManPages/man1/rsync.1.html https://developer.apple.com/legacy/library/documentation/Dar...: ”E, --extended-attributes copy extended attributes, resource forks”
- feelix 9y ago

swyx 9y ago

>Although too early to declare victory just yet, my anecdotal experiences combined with my research on the web does lead me to think that the mass transition from HFS+ to APFS is going to be another victory for Apple. This opinion differs from all others I have read from well informed people, many of who are predicting either catastrophe or at the very least some major stumbling blocks. In this case, time will tell, but I am bullish where most others are bearish. The only thing that bothers me is that I can’t understand how they managed to pull it off. Kudos to Apple!

realityloop 9y ago

I haven’t tried the GM seed yet, but I’ve reported a pretty annoying bug with inconsistent directory listings from host into guest with vagrant. A listing in the guest will be missing entries, but you can CD into the missing folders. This causes sites not to work as the application in the guest can’t see files it requires. The only way to resolve it to date has been backup and reinstall with without apfs.

sigi45 9y ago

Ah yeah okay tx. I started to be disappointed as soon as i read that he is using rsync. I was expecting someone who writes something like rsync and not a gui for rsync.

eps 9y ago

And this is how you do SEO-oriented content marketing, kids. Rather light on the substance, few select links (with at least one pointing out) and lots and lots of precisous keywords. -- Besides, the fact that a _file_ backup tool might need a near complete rewrite with a change of a file system should raise some eyebrows. It might be understandable for an imaging backup or if it was working directly with raw file system, but this one doesn't.

feelix 9y ago
I finished up the APFS compatible version of the software way quicker than I thought I would, and was so happy and energized that as part of my release I whipped up a quick blog post about the experience. Indeed it's pretty light on content, it's more just a nod to Apple for making this a painless process rather than an excruciating one, as well as an early relay of the experience to others who are going to upgrade who may be harboring some concerns like I was. The marketing for this site sucks, the software has been around since 2011 and it has nowhere near the marketshare of its competitors because it sucks. If I were really some SEO genius (and you'll see the google rankings are terrible), then this would not be the case. In short, I wish I was the evil genius that you claim, but I'm not. I'm just a dev sharing his experience with the filesystem. In regards to it not implementing the file copying tool directly - that is covered in the article. The fact that the file copying tool was compiled in 2015 and still works flawlessly is the whole premise of the article stating that it's impressive. And it being a _file_ copying tool is the part that _makes_ it impressive. You seem to have no idea how complex the metadata, built up over 30 years without a rewrite, on HFS+ is. Take a look at BackupBouncer and the associated blog posts to get an idea of the situation. It's pretty crazy.
Confiks 9y ago
> And this is how you do SEO-oriented content marketing, kids. And this in turn, kids, is how you assume bad faith without any reservation or doubt.
- yjftsjthsd-h 9y ago
  Having watched the news for a while, primarily on HN and Slashdot, I've seen enough tracking, leaks, sell-outs, adware incidents, dirty marketing, astroturf campaigns, compromised software, etc. to not expect some degree of bad faith. Perhaps not this time, but it is totally reasonable to question everything.

SyneRyder 9y ago

A few people here are disappointed that this post is about a tool that uses rsync. But the "competitor" link is more interesting - it points to the makers of SuperDuper!, a tool that makes bootable external backups. A couple of highlights: "...it's important to note that Apple still hasn't released any documentation on the "proper" way to create a bootable APFS volume. An example of what they have in mind was released for the very first time when the High Sierra developer release came out a few months ago, but that's it. We basically have to make an educated guess about what they want." and "In particular, Apple has further tightened its System Integrity Protection process, and is completely denying access to some files on the startup volume, even when copying to a non-startup volume.... But since APFS is, again, basically undocumented... what could that backup be missing?" http://www.shirt-pocket.com/blog/index.php/comments/news_on_the_march http://www.shirt-pocket.com/blog/index.php/comments/news_on_...

jason_slack 9y ago

Having written a backup app several years ago, I struggled to gain foothold in the market. I sold the app off. After a few years of wanting more from my backup, I started a new backup app. I utilize a number of commands under the hood but it is an ncurses interface only right now (supports drag and drop still into a terminal window). I'd guess that if I wanted to put it out for sale I'd need to add a GUI on top that. Can anyone shed some light on why backup apps are struggling with preserving metadata?