3 ms·

If your backup tool uses rsync under the hood, which is just a normal user-space process that uses the standard filesystem APIs, why does it matter what the und

by cyphar 9y ago

If your backup tool uses rsync under the hood, which is just a normal user-space process that uses the standard filesystem APIs, why does it matter what the underlying storage is? Obviously filesystem bugs can cause issues with this assumption, but filesystem bugs can break a whole host of other things in your backup tool.

sprayk 9y ago
This was what I was left wondering as well. rsync for sure does not have any FS-specific code that I can find. If it uses the FS APIs provided by the OS, just like every other program that touches files, why would we expect anything other than boring old rsync doing it's thing?
- microcolonel 9y ago
  Filenames are treated a bit differently on APFS. Not sure what else, but it seems there's a lot of other bizarre file metadata on OS X that you might want to sync. edit: not really sure though, because it doesn't seem like this program cares about any of those metadata.
  - feelix 9y ago
    It cares about those metadata a lot. Backups wont be bootable without it. And there is also disastrous data loss without it, a lot of files have their actual file data embedded in the metadata (resource forks et al). It's nuts because it's based on a 30 year old filesystem. That's one of the reasons why the smooth transition is impressive.
- Someone 9y ago
  The Mac version has some file system specific code https://developer.apple.com/legacy/library/documentation/Darwin/Reference/ManPages/man1/rsync.1.html https://developer.apple.com/legacy/library/documentation/Dar...: ”E, --extended-attributes copy extended attributes, resource forks”
- feelix 9y ago
  rsync has a ton of HFS+ specific code. Making a faithful duplicate of a file (specifically its metadata) is a feat on its own: http://blog.plasticsfuture.org/2006/03/05/the-state-of-backup-and-cloning-tools-under-mac-os-x/ http://blog.plasticsfuture.org/2006/03/05/the-state-of-backu...
  - sprayk 9y ago
    Upstream rsync does not seem to have any special-casing for mac/hfs. I'd wager Apple's extended-attribute handling is added in this patch[0] for the version distributed with macOS. Even here, there isn't special casing for HFS. Instead, a special library function, copyfile()[1], is used to handle copying files and their associated metadata. It seems this function was introduced in Mac OS X 10.5, which was after the article you linked. I'd wager copyfile() was introduced in response to the unwieldy file copy mechanics. After discovering how copyfile() is used in rsync, I am fairly confident that rsync works so well on the new FS as a result of Apple implementing a fairly solid copyfile() for APFS. [0] https://opensource.apple.com/source/rsync/rsync-20/patches/EA.diff.auto.html https://opensource.apple.com/source/rsync/rsync-20/patches/E... [1] http://www.manpagez.com/man/3/copyfile/ http://www.manpagez.com/man/3/copyfile/
    - trapperkeeper74 9y ago
      You're looking in the wrong place. Install the homebrew version, it has 3 patches for macOS that aren't in the upstream. https://github.com/Homebrew/homebrew-core/blob/master/Formula/rsync.rb https://github.com/Homebrew/homebrew-core/blob/master/Formul...
chrissnell 9y ago
If I understood the blog post correctly, the backup software turned out to basically work just fine on APFS. It’s the data recovery software that will take some time to port over.
djsumdog 9y ago
And if you're just using rsync under the hood, you're making money off a fancy GUI with pretty minimal actual effort as well.
- tonyedgecombe 9y ago
  Presumably his customers feel it is worth the money, they wouldn't buy it otherwise.
- feelix 9y ago
  That's the way it looks until you actually try to make a commercial piece of software by doing that. As it mentions in the blog post I come from a background of low level system programming (data recovery, even some kernel extensions, etc). It would have not added significant time to have rolled my own file copy tool. Irrespectively, it took about 1000 hours of work to make that app, as it says in the blog post. Things are nowhere as easy as they look. Doing the actual file copy is the easy part.
  - CapacitorSet 9y ago
    Let's suppose you had a fancy GUI wrapper around rsync which covered all the command line options, and a tiny output parser to display progress information. What more effort is needed to get a commercial, market-ready piece of software?
    - feelix 9y ago
      I just answered this in another response (and in addition the answer, you could answer a lot of the questions yourself by actually trying the app. It has animations built into it for example, it's clear at a glance that it's far beyond what you described): It's also not just a GUI. There is loads of code in there that does stuff. For example: • Scheduling for scheduled backups • A background daemon that looks for events to launch the UI for scheduled backups • Manipulation of disks ownership stuff in order to allow backups with permissions to happen correctly • Volume blessing, etc, in order to make bootable volumes • Trash handling & maintenance for deleted files • Deleting files (for removing old backups) And that was just off the top of my head. The app is many thousands of lines of code and it performs literally hundreds of functions. They're nearly all transparent to the user, and that is the idea.

by cyphar 9y ago

sprayk 9y ago

This was what I was left wondering as well. rsync for sure does not have any FS-specific code that I can find. If it uses the FS APIs provided by the OS, just like every other program that touches files, why would we expect anything other than boring old rsync doing it's thing?

microcolonel 9y ago
Filenames are treated a bit differently on APFS. Not sure what else, but it seems there's a lot of other bizarre file metadata on OS X that you might want to sync. edit: not really sure though, because it doesn't seem like this program cares about any of those metadata.
- feelix 9y ago
  It cares about those metadata a lot. Backups wont be bootable without it. And there is also disastrous data loss without it, a lot of files have their actual file data embedded in the metadata (resource forks et al). It's nuts because it's based on a 30 year old filesystem. That's one of the reasons why the smooth transition is impressive.
Someone 9y ago
The Mac version has some file system specific code https://developer.apple.com/legacy/library/documentation/Darwin/Reference/ManPages/man1/rsync.1.html https://developer.apple.com/legacy/library/documentation/Dar...: ”E, --extended-attributes copy extended attributes, resource forks”
feelix 9y ago
rsync has a ton of HFS+ specific code. Making a faithful duplicate of a file (specifically its metadata) is a feat on its own: http://blog.plasticsfuture.org/2006/03/05/the-state-of-backup-and-cloning-tools-under-mac-os-x/ http://blog.plasticsfuture.org/2006/03/05/the-state-of-backu...
- sprayk 9y ago
  Upstream rsync does not seem to have any special-casing for mac/hfs. I'd wager Apple's extended-attribute handling is added in this patch[0] for the version distributed with macOS. Even here, there isn't special casing for HFS. Instead, a special library function, copyfile()[1], is used to handle copying files and their associated metadata. It seems this function was introduced in Mac OS X 10.5, which was after the article you linked. I'd wager copyfile() was introduced in response to the unwieldy file copy mechanics. After discovering how copyfile() is used in rsync, I am fairly confident that rsync works so well on the new FS as a result of Apple implementing a fairly solid copyfile() for APFS. [0] https://opensource.apple.com/source/rsync/rsync-20/patches/EA.diff.auto.html https://opensource.apple.com/source/rsync/rsync-20/patches/E... [1] http://www.manpagez.com/man/3/copyfile/ http://www.manpagez.com/man/3/copyfile/
  - trapperkeeper74 9y ago
    You're looking in the wrong place. Install the homebrew version, it has 3 patches for macOS that aren't in the upstream. https://github.com/Homebrew/homebrew-core/blob/master/Formula/rsync.rb https://github.com/Homebrew/homebrew-core/blob/master/Formul...

chrissnell 9y ago

If I understood the blog post correctly, the backup software turned out to basically work just fine on APFS. It’s the data recovery software that will take some time to port over.

djsumdog 9y ago

And if you're just using rsync under the hood, you're making money off a fancy GUI with pretty minimal actual effort as well.

tonyedgecombe 9y ago
Presumably his customers feel it is worth the money, they wouldn't buy it otherwise.
feelix 9y ago
That's the way it looks until you actually try to make a commercial piece of software by doing that. As it mentions in the blog post I come from a background of low level system programming (data recovery, even some kernel extensions, etc). It would have not added significant time to have rolled my own file copy tool. Irrespectively, it took about 1000 hours of work to make that app, as it says in the blog post. Things are nowhere as easy as they look. Doing the actual file copy is the easy part.
- CapacitorSet 9y ago
  Let's suppose you had a fancy GUI wrapper around rsync which covered all the command line options, and a tiny output parser to display progress information. What more effort is needed to get a commercial, market-ready piece of software?
  - feelix 9y ago
    I just answered this in another response (and in addition the answer, you could answer a lot of the questions yourself by actually trying the app. It has animations built into it for example, it's clear at a glance that it's far beyond what you described): It's also not just a GUI. There is loads of code in there that does stuff. For example: • Scheduling for scheduled backups • A background daemon that looks for events to launch the UI for scheduled backups • Manipulation of disks ownership stuff in order to allow backups with permissions to happen correctly • Volume blessing, etc, in order to make bootable volumes • Trash handling & maintenance for deleted files • Deleting files (for removing old backups) And that was just off the top of my head. The app is many thousands of lines of code and it performs literally hundreds of functions. They're nearly all transparent to the user, and that is the idea.