5 ms·

Looking at the sparse documentation of openrsync does not create any confidence for me that it can be an acceptable substitute for rsync. In my opinion, any pr

Looking at the sparse documentation of openrsync does not create any confidence for me that it can be an acceptable substitute for rsync. In my opinion, any program that is supposed to copy files, but which is not able to make perfect copies, i.e. copies that do not lose any bit of data or metadata that was present in the original file, is just unusable garbage. Unfortunately, most copying programs available in UNIX-like operating systems (and also many archiving programs) do not make perfect file copies with their default options and many of them are never able to make perfect copies, regardless what options are used. I have not looked recently at the scp command of ssh, but at least until a few years ago it was not possible to make perfect file copies with scp, especially when the copies were done between different operating systems and file systems. That is why I never use scp, but only rsync over ssh. Rsync is the only program that I have seen, which is able (with the right options) to make perfect file copies even between different operating systems and file systems (for instance between FreeBSD with UFS and Linux with XFS), preserving also metadata like extended file attributes, access control lists and high-precision file timestamps (some copying programs and archiving programs truncate high-precision timestamps). The current documentation of openrsync does not make any guarantee that it can make complete file copies, so by default I assume that it cannot, so for now it is a program that I consider useless. Beside rsync for copying, one of the few Linux archiving programs that can archive perfect file copies is bsdtar (when using the pax file format; the ancient tar and cpio file formats cannot store all modern file metadata). (FYI: I always alias rsync to '/usr/bin/rsync --archive --xattrs --acls --hard-links --progress --rsh="ssh -p XXX -l YYYYYYY"') (With the right CLI options, "cp" from coreutils can make perfect file copies, but only if it has been compiled with appropriate options; some Linux distributions compile coreutils with wrong options, e.g. without extended file attributes support, in which case "cp" makes only partial file copies, without giving any warnings or errors.)

graemep 2y ago
This is a licensing issue for Apple, and only a small proportion of their users will care about this, and those users will just install rsync.
- adrian_b 2y ago
  You are right, but I have written my comment exactly for making those users aware of this problem. I consider this a very serious problem, because most naive users will assume automatically that when they give a file copy command they obtain a perfect duplicate of the original file. It is surprising for them to discover that this is frequently not true.
- sneak 2y ago
  And the rsync that has historically come with macOS was always way out of date, so we end up installing a newer one anyway. This doesn’t change much.
- account42 2y ago
  Is it an actual licensing issue though?
scrapheap 2y ago
What do you mean by perfect copies here? Do you mean the file content itself or are you also including the filesystem attributes related to the file in your definition?
- adrian_b 2y ago
  A file consists of data and various metadata, e.g. file name, timestamps, access rights, user-defined file attributes. By default, a file copy should include everything that is contained in the original file. Sometimes the destination file system cannot store all the original metadata, but in such cases a file copying utility must give a warning that some file metadata has been lost, e.g. like when copying to a FAT file system or to a tmpfs file system as implemented by older Linux kernels. (Many file copy or archiving utilities fail to warn the user when metadata cannot be preserved.) Some times you may no longer need some of the file metadata, but the user should be the one who chooses to loose some information, it should not be the default behavior, especially when this unexpected behavior is not advertised anywhere in the documentation. The origin of the problem is that the old UNIX file systems did not support many kinds of modern file metadata, i.e. they did not have access control lists or extended file attributes and the file timestamps had a very low resolution. When the file systems were modernized (XFS was the first Linux file system supporting such features, then slowly also the other file systems were modernized), most UNIX utilities have not been updated until many years later, and even then the additional features remained disabled by default. Copying like rsync, between different computers, creates additional problems, because even if e.g. both Windows and Linux have extended file attributes, access control lists and high-resolution file timestamps, the APIs used for accessing file metadata differ between operating systems, so a utility like rsync must contain code able to handle all such APIs, otherwise it will not be able to preserve all file metadata.
  - scrapheap 2y ago
    But what you're referring to here are the attributes that the file system stores about the file, not the file itself. By default I wouldn't expect a copy of a file to have identical file system attributes, just an identical content for the file. I would expect some of the file system attributes to be copied, but not all of them. Take the file owner for example if I take a copy of a file then by default I should be the owner of that file as it's my copy of the file, and not the original file owner's copy. An alternative way of looking at it is if I have created a file on my local machine that's owned by root and has the setuid bit set on it's file permissions then there's no way that I should be able to copy that file up to a server with my normal user account and have those atttibutes still set on the copy.
dcow 2y ago
> The current documentation of openrsync does not make any guarantee that it can make complete file copies, so by default I assume that it cannot, so for now it is a program that I consider useless. Is it possible this is just a documentation style-tone mismatch? My default assumption would be that openrsync is simply a less restrictively licensed rsync, and I wouldn’t assume it works any differently. Have you verified your strong hypothesis? Or are you just expressing skepticism. It’s hard to tell exactly. Edit: I read the openrsync readme. It says it’s compatible with rsync and points the reader to rsync’s docs. Unless extended file attributes, ACLs, and high resolution timestamps are optional at the protocol level, it must support everything modern rsync supports to be considered compatible, right? Or are you suggesting it lies and accepts the full protocol but just e.g. drops ACLs on the floor?
- wkat4242 2y ago
  From the article: > The openrsync command line tool is compatible with rsync, but as noted in the documentation openrsync accepts only a subset of rsync’s command line arguments.
  - dcow 2y ago
    Yes but that doesn't necessarily mean it is lacking the functionality to fully copy metadata. It could mean that openrsync has removed archaic and vestigial options to simplify implementation.
    - wkat4242 2y ago
      Hmm those archaic and vestigial options are probably still a pillar in many usecases :) I'm still using the multi volume support in tar for example. Which was something that stems from the time when tar was used for tape archives (hence the name tar) on actual tapes. Without that I'd be really screwed because I use a box full of harddrives as backup "tapes" (which works surprisingly well I must say, I needed a small restore only a week or two ago and it really saved my bacon). But I bet 99.9% of tar users have no idea it can even do that. Rsync is another one of those swiss army knives that people use for a lot more stuff than you might expect. Especially the remote capabilities are amazing. The problem is that when you clone something but don't provide full compatibility, you're putting your users through much headscratching and frustration. It would be better to not name it after the original then so it's clear it's something different.
inglor 2y ago
As a contrast to your take - I work for a backup company and I was really surprised to discover most of our customers (big enterprises) really do not care about 99% of metadata restored correctly and are fine with just restoring the data. (We restore everything super carefully but sometimes I feel like we're the only ones who care)
- nolok 2y ago
  I'm willing to bet a decent number "don't care" until they do care because their permissions don't work or their time based script screws up or whatever else nobody thinks about when they're in panic mode about "I lost my data".
  - ExoticPearTree 2y ago
    In case of a complete disaster recovery, the fact that a script or two might fail is super OK. That's why after recovery there's always the cleanup phase where you fix stuff that broke during recovery.
- mmcnl 2y ago
  They don't care because you care, so they never experienced the misfortune of not caring.
  - crabbone 2y ago
    Nah. Not really. A lot of the useful data out there doesn't need ACL, precise (or any dates at all) etc. Also, a lot of application-specific data formats already don't care about the "extra" attributes available in various filesystems because those aren't universally supported and implement them themselves in the file format they operate on. For example, DICOMs, or password-protected PDFs or Zip archives etc.
    - treve 2y ago
      Extended attributes (and resource forks) are mostly a liability and anti-pattern because of their non portability. It would be a huge red flag to find something important in there other than cases of backing up entire OS images.
WhyNotHugo 2y ago
OpenRsync is from the OpenBSD project. This is typically an indicator of good quality and a good focus on security. However, in this case, even the official website indicates: > We are still working on it... so please wait.
- onetom 2y ago
  OpenBSD removed ACL support though, iirc
  - SoftTalker 2y ago
    OpenBSD's filesystem doesn't have them. It just has normal unix permission bits.
- SoftTalker 2y ago
  OpenBSD often takes an approach of removing rarely-used or archaic functionality to achieve simplicity in code or configuration or improved security. They gutted a lot of openssl when they made libressl. Their OpenSMTPD is vastly simpler than something like postfix or sendmail. openrsync is very likely good code, but that doesn't mean it replicates every feature of the rsync utility.
karel-3d 2y ago
Current rsync version Apple ships is from 2006. It predates the iPhone.
- adestefan 2y ago
  That's the last GPLv2-only version of rsync.
  - _fat_santa 2y ago
    Honest question: what prevents Apple from using software that's licensed under GPL v3 vs v2?
    - deleted 2y ago
      [deleted]
    - dcow 2y ago
      GPLv3 I believe includes language that could be construed to cover your entire software distribution. IOW shipping a GPLv3 thing with the OS puts Apple at a very minor risk that a court could decide that the everything distributed with rsync must also be able to be compiled by the end user.
      - creatonez 2y ago
        This is wrong
    - blokey 2y ago
      The risk is that Apple code sign's all the executables they ship and that someone could try to use GPLv3 to force Apple to either give them their signing keys to run their own version (the anti-tivo clauses) or that it would restrict Apple from suing someone for patent infringement because they've shipped GPLv3 software. Valid or not in anyone else's opinion, it doesn't really matter, the risk that someone will attempt to use a court to enforce one of these tends to mean companies don't want to even go near it. Working in a Bank we won't touch anything GPL3, even to build our software/services or mobile app, because we don't want to even open that Pandora's box. We don't have to find out if a court would try to force us to release our signing keys if we don't use or ship any code that contains language that could in some ways be phrased to do that.
LeFantome 2y ago
Is this why bsdtar is popular, even on Linux systems that otherwise use GNU utils? I have often wondered why bsdtar is chosen. You see it in the AUR for example.
- adrian_b 2y ago
  Bsdtar certainly has additional features over GNU tar. I have switched to bsdtar many years ago precisely because I have discovered that at least at that time it was the only Linux utility that could make exact archives for the backup of my filesystems. I make extensive use of extended file attributes. For instance any file on my filesystems has a hash stored in an extended attribute for detecting errors/modifications in its content and for deduplication (the hash is updated whenever the file is modified intentionally). When I make backups, I always store at least two copies on different media and it is essential that the file hashes in extended file attributes are preserved by the archiving/backup program, so that I will be able to detect corrupted files if I try to restore them. If some file is corrupted, I can retrieve it from the other backup copy. This has saved me several times with archives stored for many years, because neither HDDs nor any other available archival media are currently reliable enough to trust them for long-term storage without errors. Like I have said, for modern backups one must use the pax file format. GNU tar, and also other "tar" programs have made some custom non-standard extensions to the standard "tar" file format, in order to be able to store some things not allowed in standard tar files, but those workarounds are inferior to what can be done in the more recent "pax" file format.
naikrovek 2y ago
I think Apple's choice here is less about functionality and more about the allergy that companies have to AGPL and GPLv3. To be clear this is an allergy that the GPL authors intended to create for this very reason. I personally dislike the GPL because I think my modifications should belong to me. I spent the effort on them, and I don't think any license which requires me to forfeit my effort is worth spending time with. Corporations agree and this is why the code licensed with GPLv2 is sparingly accepted and code licensed with GPLv3 is outright rejected by most large companies.
- thisislife2 2y ago
  What effort is "forfeited" (if you are talking about open source code)? If you use any GPL software, any modifications you make belongs to you until you choose to distribute the software with your modifications. Modifying a GPL software, for personal use, doesn't mean that you are obliged to make your modifications also open source. Moreover, GPL also means that you can never be denied access to the source code of a GPL software that is publicly distributed. This is because GPL protects a users "right to repair". For example, consider the case of a software distributed under a permissive license like MIT or BSD. If you modify and re-distribute it, anyone else can further modify the software that you patched and improved but they are not obliged to release the new source code to you. In such a scenario, you are willing "forfeiting" your effort. With GPL, at least your right to access future source codes (of publicly distributed GPL software) cannot be denied to not just you, but any user of the software.
nix0n 2y ago
I was able to find (in source-code form) the list of what arguments openrsync accepts[0]. Among permissions you use in your alias: --xattrs, --acls, --hard-links are all missing. [0]https://github.com/kristapsdz/openrsync/blob/a257c0f495af2b5ee6b41efc6724850a445f87ed/main.c#L303 https://github.com/kristapsdz/openrsync/blob/a257c0f495af2b5...
nickelpro 2y ago
I actively do not want this in a file copy utility, relying on extended file attributes is a massive anti-pattern. If you care about time stamps, they go in the file format itself. If you care about permissions, those belong in the provisioning and access systems in front of the file. The web application or other API that is providing the access. I expect file attributes of the target to be what I say they should be, not copied over from wherever the content happened to live before.
- shwouchk 2y ago
  you only ever look at files through a web application or other api?
  - nickelpro 2y ago
    Of course not, but I don't rely on the extended file attributes for anything important such that they need to be replicated during copies

graemep 2y ago

This is a licensing issue for Apple, and only a small proportion of their users will care about this, and those users will just install rsync.

adrian_b 2y ago
You are right, but I have written my comment exactly for making those users aware of this problem. I consider this a very serious problem, because most naive users will assume automatically that when they give a file copy command they obtain a perfect duplicate of the original file. It is surprising for them to discover that this is frequently not true.
sneak 2y ago
And the rsync that has historically come with macOS was always way out of date, so we end up installing a newer one anyway. This doesn’t change much.
account42 2y ago
Is it an actual licensing issue though?

scrapheap 2y ago

What do you mean by perfect copies here? Do you mean the file content itself or are you also including the filesystem attributes related to the file in your definition?

adrian_b 2y ago
A file consists of data and various metadata, e.g. file name, timestamps, access rights, user-defined file attributes. By default, a file copy should include everything that is contained in the original file. Sometimes the destination file system cannot store all the original metadata, but in such cases a file copying utility must give a warning that some file metadata has been lost, e.g. like when copying to a FAT file system or to a tmpfs file system as implemented by older Linux kernels. (Many file copy or archiving utilities fail to warn the user when metadata cannot be preserved.) Some times you may no longer need some of the file metadata, but the user should be the one who chooses to loose some information, it should not be the default behavior, especially when this unexpected behavior is not advertised anywhere in the documentation. The origin of the problem is that the old UNIX file systems did not support many kinds of modern file metadata, i.e. they did not have access control lists or extended file attributes and the file timestamps had a very low resolution. When the file systems were modernized (XFS was the first Linux file system supporting such features, then slowly also the other file systems were modernized), most UNIX utilities have not been updated until many years later, and even then the additional features remained disabled by default. Copying like rsync, between different computers, creates additional problems, because even if e.g. both Windows and Linux have extended file attributes, access control lists and high-resolution file timestamps, the APIs used for accessing file metadata differ between operating systems, so a utility like rsync must contain code able to handle all such APIs, otherwise it will not be able to preserve all file metadata.
- scrapheap 2y ago
  But what you're referring to here are the attributes that the file system stores about the file, not the file itself. By default I wouldn't expect a copy of a file to have identical file system attributes, just an identical content for the file. I would expect some of the file system attributes to be copied, but not all of them. Take the file owner for example if I take a copy of a file then by default I should be the owner of that file as it's my copy of the file, and not the original file owner's copy. An alternative way of looking at it is if I have created a file on my local machine that's owned by root and has the setuid bit set on it's file permissions then there's no way that I should be able to copy that file up to a server with my normal user account and have those atttibutes still set on the copy.

dcow 2y ago

> The current documentation of openrsync does not make any guarantee that it can make complete file copies, so by default I assume that it cannot, so for now it is a program that I consider useless. Is it possible this is just a documentation style-tone mismatch? My default assumption would be that openrsync is simply a less restrictively licensed rsync, and I wouldn’t assume it works any differently. Have you verified your strong hypothesis? Or are you just expressing skepticism. It’s hard to tell exactly. Edit: I read the openrsync readme. It says it’s compatible with rsync and points the reader to rsync’s docs. Unless extended file attributes, ACLs, and high resolution timestamps are optional at the protocol level, it must support everything modern rsync supports to be considered compatible, right? Or are you suggesting it lies and accepts the full protocol but just e.g. drops ACLs on the floor?

wkat4242 2y ago
From the article: > The openrsync command line tool is compatible with rsync, but as noted in the documentation openrsync accepts only a subset of rsync’s command line arguments.
- dcow 2y ago
  Yes but that doesn't necessarily mean it is lacking the functionality to fully copy metadata. It could mean that openrsync has removed archaic and vestigial options to simplify implementation.
  - wkat4242 2y ago
    Hmm those archaic and vestigial options are probably still a pillar in many usecases :) I'm still using the multi volume support in tar for example. Which was something that stems from the time when tar was used for tape archives (hence the name tar) on actual tapes. Without that I'd be really screwed because I use a box full of harddrives as backup "tapes" (which works surprisingly well I must say, I needed a small restore only a week or two ago and it really saved my bacon). But I bet 99.9% of tar users have no idea it can even do that. Rsync is another one of those swiss army knives that people use for a lot more stuff than you might expect. Especially the remote capabilities are amazing. The problem is that when you clone something but don't provide full compatibility, you're putting your users through much headscratching and frustration. It would be better to not name it after the original then so it's clear it's something different.

inglor 2y ago

As a contrast to your take - I work for a backup company and I was really surprised to discover most of our customers (big enterprises) really do not care about 99% of metadata restored correctly and are fine with just restoring the data. (We restore everything super carefully but sometimes I feel like we're the only ones who care)

nolok 2y ago
I'm willing to bet a decent number "don't care" until they do care because their permissions don't work or their time based script screws up or whatever else nobody thinks about when they're in panic mode about "I lost my data".
- ExoticPearTree 2y ago
  In case of a complete disaster recovery, the fact that a script or two might fail is super OK. That's why after recovery there's always the cleanup phase where you fix stuff that broke during recovery.
mmcnl 2y ago
They don't care because you care, so they never experienced the misfortune of not caring.
- crabbone 2y ago
  Nah. Not really. A lot of the useful data out there doesn't need ACL, precise (or any dates at all) etc. Also, a lot of application-specific data formats already don't care about the "extra" attributes available in various filesystems because those aren't universally supported and implement them themselves in the file format they operate on. For example, DICOMs, or password-protected PDFs or Zip archives etc.
  - treve 2y ago
    Extended attributes (and resource forks) are mostly a liability and anti-pattern because of their non portability. It would be a huge red flag to find something important in there other than cases of backing up entire OS images.

WhyNotHugo 2y ago

OpenRsync is from the OpenBSD project. This is typically an indicator of good quality and a good focus on security. However, in this case, even the official website indicates: > We are still working on it... so please wait.

onetom 2y ago
OpenBSD removed ACL support though, iirc
- SoftTalker 2y ago
  OpenBSD's filesystem doesn't have them. It just has normal unix permission bits.
SoftTalker 2y ago
OpenBSD often takes an approach of removing rarely-used or archaic functionality to achieve simplicity in code or configuration or improved security. They gutted a lot of openssl when they made libressl. Their OpenSMTPD is vastly simpler than something like postfix or sendmail. openrsync is very likely good code, but that doesn't mean it replicates every feature of the rsync utility.

karel-3d 2y ago

Current rsync version Apple ships is from 2006. It predates the iPhone.

adestefan 2y ago
That's the last GPLv2-only version of rsync.
- _fat_santa 2y ago
  Honest question: what prevents Apple from using software that's licensed under GPL v3 vs v2?
  - deleted 2y ago
    [deleted]
  - dcow 2y ago
    GPLv3 I believe includes language that could be construed to cover your entire software distribution. IOW shipping a GPLv3 thing with the OS puts Apple at a very minor risk that a court could decide that the everything distributed with rsync must also be able to be compiled by the end user.
    - creatonez 2y ago
      This is wrong
  - blokey 2y ago
    The risk is that Apple code sign's all the executables they ship and that someone could try to use GPLv3 to force Apple to either give them their signing keys to run their own version (the anti-tivo clauses) or that it would restrict Apple from suing someone for patent infringement because they've shipped GPLv3 software. Valid or not in anyone else's opinion, it doesn't really matter, the risk that someone will attempt to use a court to enforce one of these tends to mean companies don't want to even go near it. Working in a Bank we won't touch anything GPL3, even to build our software/services or mobile app, because we don't want to even open that Pandora's box. We don't have to find out if a court would try to force us to release our signing keys if we don't use or ship any code that contains language that could in some ways be phrased to do that.

LeFantome 2y ago

Is this why bsdtar is popular, even on Linux systems that otherwise use GNU utils? I have often wondered why bsdtar is chosen. You see it in the AUR for example.

adrian_b 2y ago
Bsdtar certainly has additional features over GNU tar. I have switched to bsdtar many years ago precisely because I have discovered that at least at that time it was the only Linux utility that could make exact archives for the backup of my filesystems. I make extensive use of extended file attributes. For instance any file on my filesystems has a hash stored in an extended attribute for detecting errors/modifications in its content and for deduplication (the hash is updated whenever the file is modified intentionally). When I make backups, I always store at least two copies on different media and it is essential that the file hashes in extended file attributes are preserved by the archiving/backup program, so that I will be able to detect corrupted files if I try to restore them. If some file is corrupted, I can retrieve it from the other backup copy. This has saved me several times with archives stored for many years, because neither HDDs nor any other available archival media are currently reliable enough to trust them for long-term storage without errors. Like I have said, for modern backups one must use the pax file format. GNU tar, and also other "tar" programs have made some custom non-standard extensions to the standard "tar" file format, in order to be able to store some things not allowed in standard tar files, but those workarounds are inferior to what can be done in the more recent "pax" file format.

naikrovek 2y ago

I think Apple's choice here is less about functionality and more about the allergy that companies have to AGPL and GPLv3. To be clear this is an allergy that the GPL authors intended to create for this very reason. I personally dislike the GPL because I think my modifications should belong to me. I spent the effort on them, and I don't think any license which requires me to forfeit my effort is worth spending time with. Corporations agree and this is why the code licensed with GPLv2 is sparingly accepted and code licensed with GPLv3 is outright rejected by most large companies.

thisislife2 2y ago
What effort is "forfeited" (if you are talking about open source code)? If you use any GPL software, any modifications you make belongs to you until you choose to distribute the software with your modifications. Modifying a GPL software, for personal use, doesn't mean that you are obliged to make your modifications also open source. Moreover, GPL also means that you can never be denied access to the source code of a GPL software that is publicly distributed. This is because GPL protects a users "right to repair". For example, consider the case of a software distributed under a permissive license like MIT or BSD. If you modify and re-distribute it, anyone else can further modify the software that you patched and improved but they are not obliged to release the new source code to you. In such a scenario, you are willing "forfeiting" your effort. With GPL, at least your right to access future source codes (of publicly distributed GPL software) cannot be denied to not just you, but any user of the software.

nix0n 2y ago

I was able to find (in source-code form) the list of what arguments openrsync accepts[0]. Among permissions you use in your alias: --xattrs, --acls, --hard-links are all missing. [0]https://github.com/kristapsdz/openrsync/blob/a257c0f495af2b5ee6b41efc6724850a445f87ed/main.c#L303 https://github.com/kristapsdz/openrsync/blob/a257c0f495af2b5...

nickelpro 2y ago

I actively do not want this in a file copy utility, relying on extended file attributes is a massive anti-pattern. If you care about time stamps, they go in the file format itself. If you care about permissions, those belong in the provisioning and access systems in front of the file. The web application or other API that is providing the access. I expect file attributes of the target to be what I say they should be, not copied over from wherever the content happened to live before.

shwouchk 2y ago
you only ever look at files through a web application or other api?
- nickelpro 2y ago
  Of course not, but I don't rely on the extended file attributes for anything important such that they need to be replicated during copies