5 ms·
rsync over ssh works perfectly (i.e. with the rsync option --rsh="ssh"). Not only it works much faster than scp or sftp, but it also does not lose any file met
by adrian_b 5y ago
rsync over ssh works perfectly (i.e. with the rsync option --rsh="ssh").
Not only it works much faster than scp or sftp, but it also does not lose any file metadata.
I have not verified if newer ssh versions have corrected this, but a few years ago scp and sftp failed to copy extended attributes and on some file systems they truncated timestamps.
- chasil 5y agoI understand and sympathize with rsync advocacy, but there are two major problems here. The first is the license. Because rsync is under the GPL, it will never be bundled in the OpenSSH.com distribution. It is even missing from OpenBSD base (just confirmed, and loaded it with pkg_add - there are two versions, one with iconv). So let's take Microsoft's implementation, which will forever lack rsync (as they haven't touched it in well over a year). The file transfer agent needs to be under the same license.
- adrian_b 5y agorsync is a different program than ssh, installed as a different package, so it does not matter what license it has. In most cases, ssh will run a bash at the remote end, which also has the GPL license, so it should be obvious that it is irrelevant what license have the programs launched by ssh. When you run rsync over ssh, the local rsync starts the local ssh client, which connects to the ssh server, which runs the remote rsync, while the data is piped through the ssh connection. rsync is available for most operating systems, but it is not usually installed by default. OpenBSD might be the only exception without a rsync package, but it has openrsync. I assume that it uses the same protocol so rsync to/from OpenBSD should work as well as to/from Linux, Windows, MacOS or any *BSD. If you have the right to transfer files to a computer then normally it is very simple to install rsync, only for your own user if do not have admin rights. In most cases, you can initially just copy the rsync executable file to the target computer using scp, and then do any other file transfers with rsync over ssh. After I have abandoned both scp and sftp many years ago, I no longer had any problems with the performance or data integrity of file transfers over ssh.
- chasil 5y agoAndroid will never, ever install rsync into base. Nor will OpenBSD. The rationale for this decision is not technical, it is idealogical. A technical solution to this problem is insufficient. To be honest, I was unaware of this: https://www.openrsync.org/ https://www.openrsync.org/ For maximum utility, it should be bundled with OpenSSH.
- adrian_b 5y agoI have already written that OpenBSD has openrsync, so there are no problems with OpenBSD. Because openrsync has a BSD license, it can be used on any other system that completely avoids GPL programs.
- wumpus 5y ago> rsync over ssh works perfectly I use rsync+ssh a lot, and it doesn't exactly work perfectly on long-distance 10-100 gigabit networks: ssh has a buffering problem at high bandwidths which the maintainers won't accept patches to fix.
- adrian_b 5y agoThat is a limitation of ssh. Using scp or sftp would limit the transfer speed to a value several times lower than rsync in any case. On short distance 10 Gb/s Ethernet I did not see problems, but it does not surprise me that on long-distance links they appear. In such cases, if full speed is desired, it is likely that ssh must be replaced with either IPsec or a tunnel over TLS.
- Datagenerator 5y agoTried rclone yet? Does it all, parallel and fast.
- betaby 5y agoWhat back-end do you use? Also on top of that back-end do use type = crypt, directory_name_encryption = true and so on? I tied various options and found rclone were slow for me.
- wumpus 5y agoscp has exactly the same limitations as ssh, it's the same code. I don't know anyone in the HPC world who uses "IPsec or a tunnel over TLS".
- effie 5y ago> ssh has a buffering problem at high bandwidths This is interesting, can this be the reason for why playing 720p video in Firefox run with ssh and X11 forwarding is fine bandwidth-wise (easily 1Gb/s) but it causes responsiveness to mouse/keyboard actions to go into abysmal 5-30s territory? Any references to the ssh buffering problem out there?
- loxias 5y agoI'm probably living in my own little bubble, but I didn't know anyone used sftp for ANYTHING, at least not in the last decade. Looking at my terminal history, scp for a single file, rsync (which you don't need to pass that --rsh flag to.. during the 21st century ;P) for anything that requires the ability to be interrupted and resume gracefully (which is most cases), and, in the rare case when I need speed above all, the "tar lzop ssh bash pipe" trick (I don't know if it has a canonical name) is correct. Isn't this what we all do?!? If there's a better way, I don't see how, but would like to know.
- unanswered 5y agoNote that sftp != ftps. sftp is scp. Rsync is better if you can get it, but I'm pretty sure that whenever I transfer stuff using WinSCP and File Explorer (Android) it's using sftp.
- przemub 5y agoI use sftp every now and then I need to pull something and I don't know where exactly that is. A shortcut for finding the path via ssh and then pulling the file via scp, if you will.
- loxias 5y agoThat makes sense! In that rare situation, I typically use tab completion to find the remote file. However, I bet that would SUCK on a high latency or low bandwidth link, where sftp would be just fine.
- jcynix 5y ago> sftp for ANYTHING Solid Explorer (Android app) connects via sftp to Linux machines just fine. While I use Termux and rsync to do backups and bulk transfers, Solid Explorer allows me to rapidly browse remote files.
- lathiat 5y agoFWIW sftp vs scp is more about the protocol. Apparently new OpenSSH has an “scp” command that uses the sftp protocol. But yeah Rsync is a good choice.
- tyingq 5y agotar over ssh also does well with file metadata.