4 ms·

How is this using rsync? Does it invoke an rsync process, or implement the rsync wire protocol? One of my gripes with rsync is I've not found another way to i

How is this using rsync? Does it invoke an rsync process, or implement the rsync wire protocol? One of my gripes with rsync is I've not found another way to integrate it into other systems. librsync doesn't implement the wire protocol and parsing the output of rsync itself is fraught with peril (for instance the formatting of --no-h has changed before.) Edit: it seems to invoke an a rsync process but doesn't parse stdout for progress. A bit disappointing but I suppose that's all which is needed for this application.

pengaru 6y ago
librsync has absolutely no connection to the rsync utility, it's a distinct implementation of the underlying differential algorithm. The rsync utility doesn't use it, and they don't even share a common data format at any level.
- bigbubba 6y ago
  Yes I'm aware, but usually when I voice my gripes with integrating rsync somebody unfamiliar with the matter quickly googles 'rsync library', finds librsync, and assumes it must be the answer to my problem. Then I have to explain to them that it's not. I thought to head that off this time by mentioning upfront that librsync is not the answer.
  - pengaru 6y ago
    I see, yeah, librsync's misleading name has clearly caused a lot of wasted time.
  - techcode 6y ago
    Just to be sure I'm getting the question - you want to use rsync(1), and you're wondering if there's a good way to guard your code that's using/calling it from underlying changes in rsync options and output, how to read status/progress and such? Here's an example of library that wraps rsync https://metacpan.org/pod/File::Rsync https://metacpan.org/pod/File::Rsync - you can provide callback functions that will be called for each line of stdout/stderr. Granted it's in Perl - and chances are your code isn't written in Perl. So if you can't find something similar for your language of choice, and if you can't be bothered with checking out implementation in Perl and rewriting it... I would suggest searching & checking out "Inline::" namespace (e.g.: https://metacpan.org/pod/distribution/Inline-Java/lib/Inline/Java.pod https://metacpan.org/pod/distribution/Inline-Java/lib/Inline..., and there's also JS, C ...etc). And next to those - Perlito can "Compile Perl to Java/JavaScript/Python/Ruby/Go/etc" https://github.com/fglock/Perlito https://github.com/fglock/Perlito. I think we had/have some Perl code running in production Hadoop/JVM.
  - danmur 6y ago
    No way to win :)
strbean 6y ago
rclone might hold some promise as an alternative, if you don't mind using Go.
rakoo 6y ago
Indeed there is no widespread documentation on how to play with rsync itself. A long time ago I played with the idea of replicating rproxy (https://rproxy.samba.org/ https://rproxy.samba.org/), which was just a cool idea. The results are here (https://rproxy.samba.org/ https://rproxy.samba.org/) but are very very basic, it was just a toy. I did have to invent my own protocol, but I did it over HTTP.

pengaru 6y ago

librsync has absolutely no connection to the rsync utility, it's a distinct implementation of the underlying differential algorithm. The rsync utility doesn't use it, and they don't even share a common data format at any level.

bigbubba 6y ago
Yes I'm aware, but usually when I voice my gripes with integrating rsync somebody unfamiliar with the matter quickly googles 'rsync library', finds librsync, and assumes it must be the answer to my problem. Then I have to explain to them that it's not. I thought to head that off this time by mentioning upfront that librsync is not the answer.
- pengaru 6y ago
  I see, yeah, librsync's misleading name has clearly caused a lot of wasted time.
- techcode 6y ago
  Just to be sure I'm getting the question - you want to use rsync(1), and you're wondering if there's a good way to guard your code that's using/calling it from underlying changes in rsync options and output, how to read status/progress and such? Here's an example of library that wraps rsync https://metacpan.org/pod/File::Rsync https://metacpan.org/pod/File::Rsync - you can provide callback functions that will be called for each line of stdout/stderr. Granted it's in Perl - and chances are your code isn't written in Perl. So if you can't find something similar for your language of choice, and if you can't be bothered with checking out implementation in Perl and rewriting it... I would suggest searching & checking out "Inline::" namespace (e.g.: https://metacpan.org/pod/distribution/Inline-Java/lib/Inline/Java.pod https://metacpan.org/pod/distribution/Inline-Java/lib/Inline..., and there's also JS, C ...etc). And next to those - Perlito can "Compile Perl to Java/JavaScript/Python/Ruby/Go/etc" https://github.com/fglock/Perlito https://github.com/fglock/Perlito. I think we had/have some Perl code running in production Hadoop/JVM.
- danmur 6y ago
  No way to win :)

strbean 6y ago

rclone might hold some promise as an alternative, if you don't mind using Go.

rakoo 6y ago

Indeed there is no widespread documentation on how to play with rsync itself. A long time ago I played with the idea of replicating rproxy (https://rproxy.samba.org/ https://rproxy.samba.org/), which was just a cool idea. The results are here (https://rproxy.samba.org/ https://rproxy.samba.org/) but are very very basic, it was just a toy. I did have to invent my own protocol, but I did it over HTTP.