2 ms·
It's more likely we'll bite the bullet and use sftp directly from libssh. rsync doesn't work for our use case (and nor does sshfs).
by rwmj 5y ago
It's more likely we'll bite the bullet and use sftp directly from libssh. rsync doesn't work for our use case (and nor does sshfs).
- tialaramex 5y agoSFTP makes sense here, that's why OpenSSH is deprecating scp because SFTP is much better defined. I actually wrote all my scp-style scripting with SFTP for years, so much so that I think projects I did this for lived their entire lifecycle and are gone (but I don't know anybody who still works at the place I wrote the earlier ones) without running into the problem I feared - it's just not clear what scp means for non-trivial cases.
- loudtieblahblah 5y agoHate to be that guy making you defend a situation you know more about than any of us here.. But I incredulously am curious as to why rsync wouldn't work for you.
- acdha 5y agoI can't speak for them but there are two big ones I've hit: 1. rsync likes to build the entire list of operations before doing anything. If you have workloads which requires scanning lots of small files and/or using something like NFS / s3fs, etc. that means a LOT of I/O delays before a single byte of payload is transferred. 2. The checksum algorithm is really cool if you have large files which only change a little but it's relatively expensive and brittle. I've run into multiple cases where naive sftp was faster because we had a high-bandwidth, high-latency network connection.
- quesera 5y agoFWIW, 1. This is no longer true in rsync-3.0.0 (Mar 2008) and later. It does build the list of course, but transfer begins after establishing just a few directories of content. This "incremental" mode is not available if you specify an option that requires the full list to begin, documented under the "--recursive" option. 2. Checksums are not computed by default. If the files match time/size, they will not be transferred or checksummed unless "--checksum" is turned on. All files which are transferred, are compared by checksum when complete but this is not meaningfully expensive since the IO is already done. Your issues with high-bandwidth, (very) high-latency links make sense. The rsync algo was designed to minimize bytes-sent over the undersea cable to Australia (low-bandwidth, moderate-latency). This still works well for most internet traffic, but not if your latency numbers are way out of balance!
- formerly_proven 5y agoI'm currently using libssh (not libssh2) in a project and I'm quite happy with it. The API is pretty nice and it behaves pretty much exactly like OpenSSH by default.
- rwmj 5y agoWe already use libssh all over the place, eg in nbdkit & qemu.