4 ms·

I love rsync for what it can do, but I hate that with a passion. It's terrible design that should have never seen the light of day.

by paol 5y ago

deleted 5y ago
[deleted]
mhalle 5y ago
I end up reposting this comment whenever rsync's slash treatment gets mentioned. rsync has "weird" syntax for a reason. Unlike other unix-like commands, it treats trailing slashes as significant AND consistently. If a directory has a trailing slash, it means "contents" of the directory". No slash means "the directory itself". These are two different concepts, and a program that copies directories should take the difference into account. scp (and cp, for that matter) don't take this difference into account. That leads to gotchas with recursive (-r) copying. Most importantly, scp isn't idemopotent: scp -r fromdir todir If todir doesn't exist, scp will copy the contents of directory fromdir to a new directory named todir. Execute the same command again (now that todir exists), and scp will copy fromdir to todir/fromdir . On the other hand: rsync -a fromdir/ todir will always copy the contents of fromdir into a directory named todir (effectively, a directory rename operation), whether todir exists or not. rsync -a fromdir todir will always copy the directory fromdir into the directory todir, whether todir exists or not. These rsync operations are idempotent, which it important because rsync is designed to incrementally re-sync directories. It is expected that it will commonly be run more than once, which is why it needed to address this IMHO fundamental bug/limitation in cp and scp.
- leephillips 5y ago
  Great explanation! I prefer rsync to cp, etc., because of this consistency, but I never appreciated the idempotency implications.
- nijave 5y ago
  Isn't that problem also solved with the `dir/.` syntax? Adding the `.` signifies content versus the directly itself I think it's `cp` that uses that syntax
- bavell 5y ago
  I knew most of this but it was a great refresher and I learned a few more details! Thanks!
- lostmsu 5y ago
  I think that's a failure on SCP side. It should have never allowed scp -r dir target_dir with non-existent target_dir in the first place.
  - mhalle 5y ago
    I get what you are saying: cp/scp should have consistent behavior when the second arg is a directory. The "problem" operation you describe is analogous to copying plain files, though (copying contents to a new name). I don't think having to create a target directory first is obvious, although neither is the fact that bare "cp" doesn't copy directories. Finally, for remote copying it would be a nightmare for scp to require the destination directory to exist. To copy a directory tree to a remote server with a new name you'd need to either ssh a remote mkdir command, or add a flag to scp to specify the new name.
- adrian_b 5y ago
  I agree with you, but the cases when I want to copy the contents of a directory instead of the directory are extremely rare. Because such cases are so rare, it does not make any difference for me if I have to add "/*" instead of the slightly shorter "/" (even when I also have to add "/.*" for hidden files/directories). So I would like if rsync would have a command-line option to ignore trailing slashes, like other programs have (and which I always use with them). As it is, rsync is the only program where I have to be extra cautious before executing a command-line, because the directory names are in most cases provided by auto-complete, and most shells terminate them with a trailing slash, which I have to be very careful to always delete, which slows me a lot. An exception is when using zsh, which sometimes, but not always, deletes itself the trailing slash added by auto-complete (zsh might be confused by the aliases used by me, so it usually deletes the trailing slashes when they do not matter, but retains them when I would want them to be deleted).
  - Izkata 5y ago
    > Because such cases are so rare, it does not make any difference for me if I have to add "/*" instead of the slightly shorter "/" (even when I also have to add "/.*" for hidden files/directories). I think this is the wrong way to think of it. It's more like "rsync foo/ bar/" means "make these two directories the same" - it does change the last-modified timestamp on bar/ to match foo/. If you use "foo/*" it excludes the directory and last-modified becomes now. For all the combinations this is what's in my head, and I find it for the most part very consistent and predictable: * rsync -a foo bar/ -> put this file in this directory (a directory is a special type of file, so you end up with bar/foo/) * rsync -a foo/ bar/ -> put the contents of foo/ into bar/ (but also we're syncing the directories, which is why bar/'s last-modified is changed) * rsync -a foo bar -> put this file in here, whatever "bar" is (makes a file named bar, unless foo is a directory - you can't put a directory inside a non-directory file, so it makes directory bar and puts foo inside it) * rsync -a foo/ bar -> This is the only one I have issues with because it doesn't particularly make sense so I never have reason to even try using it - put a directory into a file? But it's the same as "foo/ bar/" case.
    - adrian_b 5y ago
- saltcured 5y ago
  Once I found the rsync --relative (-R) command mode and its /./ meta syntax, I no longer appreciated this trailing slash feature as a good idea on its own. When you are worrying about idempotent re-sync of trees, I think it is so much more important to think about where the roots of the synchronization should match up between local and remote, and to make that explicit in the calling convention. rsync -Ra /src-only/./{a,b,c} remote:/remote-only/./ I also think you should be writing this idempotent call in a script to presumably be reused over time, and this helps document the goal.
  - mhalle 5y ago
    I didn't know about this meta-syntax, thanks - it looks very useful. I would have to think about how it fits in with the philosophy of the trailing slash sometime, but I don't want to hurt my head right now.
benibela 5y ago
I wish it had some kind of rsync repositories To make backups, you clone a directory into the rsync repository, and when you later want to update the backup, you pull the newest changes from the directory into the repository. And it would be impossible to destroy the backup by forgetting a slash or syncing another directory on the wrong backup because it would remember which directory belongs to which backup repository.

I love rsync for what it can do, but I hate that with a passion. It's terrible design that should have never seen the light of day.

by paol 5y ago

I love rsync for what it can do, but I hate that with a passion. It's terrible design that should have never seen the light of day.

deleted 5y ago

[deleted]

mhalle 5y ago

I end up reposting this comment whenever rsync's slash treatment gets mentioned. rsync has "weird" syntax for a reason. Unlike other unix-like commands, it treats trailing slashes as significant AND consistently. If a directory has a trailing slash, it means "contents" of the directory". No slash means "the directory itself". These are two different concepts, and a program that copies directories should take the difference into account. scp (and cp, for that matter) don't take this difference into account. That leads to gotchas with recursive (-r) copying. Most importantly, scp isn't idemopotent: scp -r fromdir todir If todir doesn't exist, scp will copy the contents of directory fromdir to a new directory named todir. Execute the same command again (now that todir exists), and scp will copy fromdir to todir/fromdir . On the other hand: rsync -a fromdir/ todir will always copy the contents of fromdir into a directory named todir (effectively, a directory rename operation), whether todir exists or not. rsync -a fromdir todir will always copy the directory fromdir into the directory todir, whether todir exists or not. These rsync operations are idempotent, which it important because rsync is designed to incrementally re-sync directories. It is expected that it will commonly be run more than once, which is why it needed to address this IMHO fundamental bug/limitation in cp and scp.

leephillips 5y ago
Great explanation! I prefer rsync to cp, etc., because of this consistency, but I never appreciated the idempotency implications.
nijave 5y ago
Isn't that problem also solved with the `dir/.` syntax? Adding the `.` signifies content versus the directly itself I think it's `cp` that uses that syntax
bavell 5y ago
I knew most of this but it was a great refresher and I learned a few more details! Thanks!
lostmsu 5y ago
I think that's a failure on SCP side. It should have never allowed scp -r dir target_dir with non-existent target_dir in the first place.
- mhalle 5y ago
  I get what you are saying: cp/scp should have consistent behavior when the second arg is a directory. The "problem" operation you describe is analogous to copying plain files, though (copying contents to a new name). I don't think having to create a target directory first is obvious, although neither is the fact that bare "cp" doesn't copy directories. Finally, for remote copying it would be a nightmare for scp to require the destination directory to exist. To copy a directory tree to a remote server with a new name you'd need to either ssh a remote mkdir command, or add a flag to scp to specify the new name.
adrian_b 5y ago
I agree with you, but the cases when I want to copy the contents of a directory instead of the directory are extremely rare. Because such cases are so rare, it does not make any difference for me if I have to add "/*" instead of the slightly shorter "/" (even when I also have to add "/.*" for hidden files/directories). So I would like if rsync would have a command-line option to ignore trailing slashes, like other programs have (and which I always use with them). As it is, rsync is the only program where I have to be extra cautious before executing a command-line, because the directory names are in most cases provided by auto-complete, and most shells terminate them with a trailing slash, which I have to be very careful to always delete, which slows me a lot. An exception is when using zsh, which sometimes, but not always, deletes itself the trailing slash added by auto-complete (zsh might be confused by the aliases used by me, so it usually deletes the trailing slashes when they do not matter, but retains them when I would want them to be deleted).
- Izkata 5y ago
  > Because such cases are so rare, it does not make any difference for me if I have to add "/*" instead of the slightly shorter "/" (even when I also have to add "/.*" for hidden files/directories). I think this is the wrong way to think of it. It's more like "rsync foo/ bar/" means "make these two directories the same" - it does change the last-modified timestamp on bar/ to match foo/. If you use "foo/*" it excludes the directory and last-modified becomes now. For all the combinations this is what's in my head, and I find it for the most part very consistent and predictable: * rsync -a foo bar/ -> put this file in this directory (a directory is a special type of file, so you end up with bar/foo/) * rsync -a foo/ bar/ -> put the contents of foo/ into bar/ (but also we're syncing the directories, which is why bar/'s last-modified is changed) * rsync -a foo bar -> put this file in here, whatever "bar" is (makes a file named bar, unless foo is a directory - you can't put a directory inside a non-directory file, so it makes directory bar and puts foo inside it) * rsync -a foo/ bar -> This is the only one I have issues with because it doesn't particularly make sense so I never have reason to even try using it - put a directory into a file? But it's the same as "foo/ bar/" case.
  - adrian_b 5y ago
saltcured 5y ago
Once I found the rsync --relative (-R) command mode and its /./ meta syntax, I no longer appreciated this trailing slash feature as a good idea on its own. When you are worrying about idempotent re-sync of trees, I think it is so much more important to think about where the roots of the synchronization should match up between local and remote, and to make that explicit in the calling convention. rsync -Ra /src-only/./{a,b,c} remote:/remote-only/./ I also think you should be writing this idempotent call in a script to presumably be reused over time, and this helps document the goal.
- mhalle 5y ago
  I didn't know about this meta-syntax, thanks - it looks very useful. I would have to think about how it fits in with the philosophy of the trailing slash sometime, but I don't want to hurt my head right now.

benibela 5y ago

I wish it had some kind of rsync repositories To make backups, you clone a directory into the rsync repository, and when you later want to update the backup, you pull the newest changes from the directory into the repository. And it would be impossible to destroy the backup by forgetting a slash or syncing another directory on the wrong backup because it would remember which directory belongs to which backup repository.