4 ms·

pv is a great tool. One of it's lesser known features is throttling; transfer a file without dominating your bandwidth: pv -L 200K < bigfile.iso | ssh somehost

pv is a great tool. One of it's lesser known features is throttling; transfer a file without dominating your bandwidth: pv -L 200K < bigfile.iso | ssh somehost 'cat > bigfile.iso' Complete with a progress bar, speed, and ETA.

smcl 4y ago
Oh damn that's neat I never thought to use `ssh` directly when transferring a file, I always used `scp bigfile.iso name@server.org:path/in/destination`
- fbergen 4y ago
  Also see `scp -l 200 bigfile.iso name@server.org:path/in/destination` from man page: -l limit Limits the used bandwidth, specified in Kbit/s.
  - MayeulC 4y ago
    Also you probably shouldn't use scp. rsync and sftp have mostly the same semantics. rsync --bwlimit=2OOK bigfile.iso name@server.org:path/in/destination sftp -l 200 bigfile.iso name@server.org:path/in/destination Although it seems that scp is becoming a wrapper around sftp these days: https://www.redhat.com/en/blog/openssh-scp-deprecation-rhel-9-what-you-need-know https://www.redhat.com/en/blog/openssh-scp-deprecation-rhel-... https://news.ycombinator.com/item?id=25005567 https://news.ycombinator.com/item?id=25005567
- tyingq 4y ago
  A similar trick that's nice is piping tar through ssh. Handy if you don't have rsync or something better around. Even handy for one file, since it preserves permissions, etc. tar -cf - some/dir | ssh remote 'cd /place/to/go && tar -xvf -'
  - Twirrim 4y ago
    I love this trick. I was dealing with some old solaris boxes something like 15 years ago when I learned you could do this. I couldn't rsync, and had started off SCP'ing hundreds of thousands of files across but it was going to take an insane length of time. Asked one of the other sysadmins if they knew a better way and they pointed out you can pipe stuff in to ssh for the other side too. Every now and then this technique proves useful in unexpected ways :)
dspillett 4y ago
Similarly, though useful less often these days, using -B/--buffer-size to increase the amount that it can buffer. If reading data from traditional hard drives, piping that data through some process, and writing the result back to the same drives, this option can increase throughput significantly by reducing head movements. It can help on other storage systems too, but usually not so much so.

smcl 4y ago

Oh damn that's neat I never thought to use `ssh` directly when transferring a file, I always used `scp bigfile.iso name@server.org:path/in/destination`

fbergen 4y ago
Also see `scp -l 200 bigfile.iso name@server.org:path/in/destination` from man page: -l limit Limits the used bandwidth, specified in Kbit/s.
- MayeulC 4y ago
  Also you probably shouldn't use scp. rsync and sftp have mostly the same semantics. rsync --bwlimit=2OOK bigfile.iso name@server.org:path/in/destination sftp -l 200 bigfile.iso name@server.org:path/in/destination Although it seems that scp is becoming a wrapper around sftp these days: https://www.redhat.com/en/blog/openssh-scp-deprecation-rhel-9-what-you-need-know https://www.redhat.com/en/blog/openssh-scp-deprecation-rhel-... https://news.ycombinator.com/item?id=25005567 https://news.ycombinator.com/item?id=25005567
tyingq 4y ago
A similar trick that's nice is piping tar through ssh. Handy if you don't have rsync or something better around. Even handy for one file, since it preserves permissions, etc. tar -cf - some/dir | ssh remote 'cd /place/to/go && tar -xvf -'
- Twirrim 4y ago
  I love this trick. I was dealing with some old solaris boxes something like 15 years ago when I learned you could do this. I couldn't rsync, and had started off SCP'ing hundreds of thousands of files across but it was going to take an insane length of time. Asked one of the other sysadmins if they knew a better way and they pointed out you can pipe stuff in to ssh for the other side too. Every now and then this technique proves useful in unexpected ways :)

dspillett 4y ago

Similarly, though useful less often these days, using -B/--buffer-size to increase the amount that it can buffer. If reading data from traditional hard drives, piping that data through some process, and writing the result back to the same drives, this option can increase throughput significantly by reducing head movements. It can help on other storage systems too, but usually not so much so.