4 ms·

> Successful UNIX tools (or any tools, for the most part) are the ones with a simple concept manifested very thoroughly, and rsync is certainly that. [...] Ther

> Successful UNIX tools (or any tools, for the most part) are the ones with a simple concept manifested very thoroughly, and rsync is certainly that. [...] There are over 100 flags in rsync that you can select to do different things. So, this is what I mean by doing something simple, and doing it very well and very thoroughly. Rsync is certainly not simple. It certainly doesn't do one thing. It's "extraordinarily versatile" to the point of madness. Here's what it does: - Copies files from A to B - Mirrors directories - Computes diffs - Applies patches - Evaluates filters - Runs as a daemon - Runs over various remote shells - Applies filters to filenames And then it doesn't give you a good progress bar or let you resume interrupted operations, or gather errors at the end. I think this is a good case where the tool may have grown too large and unwieldy, and it may be better to replace it with an API or a collection of small, UNIX-like tools. For example, if I just want to copy files from point A to B, I can do this: tar c . | ssh host tar xv If I want a progress bar, I can use pv, tar c . | pv | ssh host tar xv Usually I find this easier to use than rsync.

darkpuma 7y ago
Adding pv to a tar|ssh pipeline is simpler than just passing -P to rsync? I must disagree.
- klodolph 7y ago
  Maybe I wasn't clear about my complaint. My complaint is that rsync --progress sucks, I'm not complaining that it's hard to use.
  - darkpuma 7y ago
    I don't like the --no-h change they made to the --progress format (mentioned in another comment) because it broke my parsing once, but otherwise I don't really see a problem with it and I expect most people have never tried to parse it. What's so bad about --progress? Edit: --progress also tells you how many files rsync has been told to sync, and of those, how many are remaining. 2384908 100% 57.52MB/s 0:00:00 (xfr#1, to-chk=7/11) e.g. transfer #1 out of 11 files or directories to check, with 7 remaining.
    - klodolph 7y ago
      --progress gives you only the progress of individual file transfers.
      - sirn 7y ago
        --info=progress2 actually gives you the progress of the whole transfer (but I don't think you can do both). You can use --info=name0,progress2 if you don't care about individual files scrolling the terminal.
        klodolph 7y ago
        Seems like this user-hostile interface could have been averted if they gave rsync a "porcelain" mode, since --progress is frozen so it doesn't break scripts.
        darkpuma 7y ago
        I think you're really exaggerating the severity of what is fundamentally just a minor matter of personal taste. (I say this as somebody who's written a GUI wrapper around rsync to, among other things, implement my own progress bar. So I agree with the general principle of writing porcelain around rsync.) (Incidentally --progress has actually changed in a way that breaks (my) scripts in the past; the version that ships with MacOS and the latest version exhibit different behavior by default. It would be nice if there was a better way. If rsync followed the ffmpeg model of foremost being a library with a standard command line interface, I think that could be ideal.)
sirn 7y ago
I believe rsync can do two of your doesn't do list: > And then it doesn't give you a good progress bar rsync --progress > let you resume interrupted operations rsync --partial ...or enable both at the same time with rsync -P (These features only available on samba.org's original rsync, and not OpenBSD openrsync[1], but I don't think openrsync has seen a widespread use yet, as it's still pretty new) [1]: https://www.openrsync.org https://www.openrsync.org
- klodolph 7y ago
  rsync --progress isn't very good, which is why I worded it that way. If you use rsync --partial it has to rescan the source and destination directories again. All it does is let you resume transfers, and for some of us, transfers are not the slow part.
gchamonlive 7y ago
What is simple in rsync is the idea of synchronising stuff from A to B. That is all it does in all the cases you mentioned. It is hard to emulate what rsync does with archive mode enabled, which speeds up transfers, so tar isn't exactly a replacement for rsync. There are better ways to show progress, for example using the flag `--info=progress2`. It doesn't behave well in automated scripts though
- darkpuma 7y ago
  fyi, combining something like `--out-format="file: %f %l"` with `--progress` results in reasonably parse-able progress updates. Keep in mind however that the format of --progress has changed in the past. Depending on your version of rsync it may or may not insert thousand seperating commas into the number of bytes written, e.g. 2,384,908 vs 2384908. The former can be supressed with --no-h, an option not available on at least some older versions of rsync that default to no commas.
- klodolph 7y ago
  > It is hard to emulate what rsync does with archive mode enabled, which speeds up transfers, so tar isn't exactly a replacement for rsync. Archive mode is what tar does by default. It is just an alias for -rlptgoD. How does this speed up transfers? Are you thinking of something else?
mturmon 7y ago
Your "piped tar" doesn't cover the main use case of rsync, which is (unsurprisingly), remote file hierarchy sync'ing. In particular, when there's a hierarchy on a remote machine that I want to sync to a partially-up-to-date hierarchy on a local machine, the required rsync command is a one-liner and it's very fast because it uses a diff. Incidentally, the "tar" man page is not exactly a model of simplicity itself.
- klodolph 7y ago
  > Your "piped tar" doesn't cover the main use case of rsync, which is (unsurprisingly), remote file hierarchy sync'ing. That's exactly my point, isn't it? rsync does several different things, depending on how you invoke it, rather than being a UNIXy tool that does one thing well. Because the single tool does everything, you can't easily swap out different parts of it, and the single tool takes a hundred options.
  - deleted 7y ago
    [deleted]
aidenn0 7y ago
For copying files from A to B, I wrote a simple script[1]. rsync is not as universally available on hosts, the tar version is sometimes significantly faster than rsync when just copying, and rsync won't let me triangle-route copies. https://gist.github.com/jasom/ecdec7767b5f774cd24aebba3c2ba3d9 https://gist.github.com/jasom/ecdec7767b5f774cd24aebba3c2ba3...
somacert 7y ago
The "Does not resume interrupted operations" gripe confuses me because that is the main reason I reach for rsync, I have a large tree of files I need to be somewhere else and if the transfer fails I would like it to not transfer files already transferred. So I use rsync, which offers, as a core feature, resuming interrupted operation.
- Izkata 7y ago
  That one is badly phrased, it's about resuming a single interrupted large file. By default rsync will create a new temp file and start over, but there are options to make it resumable.

darkpuma 7y ago

Adding pv to a tar|ssh pipeline is simpler than just passing -P to rsync? I must disagree.

klodolph 7y ago
Maybe I wasn't clear about my complaint. My complaint is that rsync --progress sucks, I'm not complaining that it's hard to use.
- darkpuma 7y ago
  I don't like the --no-h change they made to the --progress format (mentioned in another comment) because it broke my parsing once, but otherwise I don't really see a problem with it and I expect most people have never tried to parse it. What's so bad about --progress? Edit: --progress also tells you how many files rsync has been told to sync, and of those, how many are remaining. 2384908 100% 57.52MB/s 0:00:00 (xfr#1, to-chk=7/11) e.g. transfer #1 out of 11 files or directories to check, with 7 remaining.
  - klodolph 7y ago
    --progress gives you only the progress of individual file transfers.
    - sirn 7y ago
      --info=progress2 actually gives you the progress of the whole transfer (but I don't think you can do both). You can use --info=name0,progress2 if you don't care about individual files scrolling the terminal.
      - klodolph 7y ago
        Seems like this user-hostile interface could have been averted if they gave rsync a "porcelain" mode, since --progress is frozen so it doesn't break scripts.
        darkpuma 7y ago
        I think you're really exaggerating the severity of what is fundamentally just a minor matter of personal taste. (I say this as somebody who's written a GUI wrapper around rsync to, among other things, implement my own progress bar. So I agree with the general principle of writing porcelain around rsync.) (Incidentally --progress has actually changed in a way that breaks (my) scripts in the past; the version that ships with MacOS and the latest version exhibit different behavior by default. It would be nice if there was a better way. If rsync followed the ffmpeg model of foremost being a library with a standard command line interface, I think that could be ideal.)

sirn 7y ago

I believe rsync can do two of your doesn't do list: > And then it doesn't give you a good progress bar rsync --progress > let you resume interrupted operations rsync --partial ...or enable both at the same time with rsync -P (These features only available on samba.org's original rsync, and not OpenBSD openrsync[1], but I don't think openrsync has seen a widespread use yet, as it's still pretty new) [1]: https://www.openrsync.org https://www.openrsync.org

klodolph 7y ago
rsync --progress isn't very good, which is why I worded it that way. If you use rsync --partial it has to rescan the source and destination directories again. All it does is let you resume transfers, and for some of us, transfers are not the slow part.

gchamonlive 7y ago

What is simple in rsync is the idea of synchronising stuff from A to B. That is all it does in all the cases you mentioned. It is hard to emulate what rsync does with archive mode enabled, which speeds up transfers, so tar isn't exactly a replacement for rsync. There are better ways to show progress, for example using the flag `--info=progress2`. It doesn't behave well in automated scripts though

darkpuma 7y ago
fyi, combining something like `--out-format="file: %f %l"` with `--progress` results in reasonably parse-able progress updates. Keep in mind however that the format of --progress has changed in the past. Depending on your version of rsync it may or may not insert thousand seperating commas into the number of bytes written, e.g. 2,384,908 vs 2384908. The former can be supressed with --no-h, an option not available on at least some older versions of rsync that default to no commas.
klodolph 7y ago
> It is hard to emulate what rsync does with archive mode enabled, which speeds up transfers, so tar isn't exactly a replacement for rsync. Archive mode is what tar does by default. It is just an alias for -rlptgoD. How does this speed up transfers? Are you thinking of something else?

mturmon 7y ago

Your "piped tar" doesn't cover the main use case of rsync, which is (unsurprisingly), remote file hierarchy sync'ing. In particular, when there's a hierarchy on a remote machine that I want to sync to a partially-up-to-date hierarchy on a local machine, the required rsync command is a one-liner and it's very fast because it uses a diff. Incidentally, the "tar" man page is not exactly a model of simplicity itself.

klodolph 7y ago
> Your "piped tar" doesn't cover the main use case of rsync, which is (unsurprisingly), remote file hierarchy sync'ing. That's exactly my point, isn't it? rsync does several different things, depending on how you invoke it, rather than being a UNIXy tool that does one thing well. Because the single tool does everything, you can't easily swap out different parts of it, and the single tool takes a hundred options.
- deleted 7y ago
  [deleted]

aidenn0 7y ago

For copying files from A to B, I wrote a simple script[1]. rsync is not as universally available on hosts, the tar version is sometimes significantly faster than rsync when just copying, and rsync won't let me triangle-route copies. https://gist.github.com/jasom/ecdec7767b5f774cd24aebba3c2ba3d9 https://gist.github.com/jasom/ecdec7767b5f774cd24aebba3c2ba3...

somacert 7y ago

The "Does not resume interrupted operations" gripe confuses me because that is the main reason I reach for rsync, I have a large tree of files I need to be somewhere else and if the transfer fails I would like it to not transfer files already transferred. So I use rsync, which offers, as a core feature, resuming interrupted operation.

Izkata 7y ago
That one is badly phrased, it's about resuming a single interrupted large file. By default rsync will create a new temp file and start over, but there are options to make it resumable.