4 ms·

It's obviously not magic. In practice, I find this feature difficult to use due to the distribution of ssh keys necessary to be practical. Actually, scp got qu

by ktpsns 7y ago

It's obviously not magic. In practice, I find this feature difficult to use due to the distribution of ssh keys necessary to be practical. Actually, scp got quite old. Rsync typically makes a better job, especially with many files or unreliable networks. Streaming tar.gz over ssh is again typically faster for many files.

mschuster91 7y ago
rsync however does not work across two remote hosts, at least not without trickery that one cannot remember. scp at leasts supports easy invocation of a two-remote-hosts scenario but not a direct host-to-host communication, you will always be limited by the client bandwidth.
yetihehe 7y ago
My favourite magic trick is dumping db on one host, gzipping on the fly, tunneling through ssh to other host, then ungzipping straight into other database. Useful when you don't have enough place on disk for local copies.
- gglitch 7y ago
  It would be super useful to me to see this with a bit more detail, in case you have/know a writeup somewhere!
  - yetihehe 7y ago
    I didn't invent it myself, you can see example at [0] or googling 'pg_dump ssh tunnel'. [0] https://coderwall.com/p/bmv0nw/dump-postgres-databases-through-ssh https://coderwall.com/p/bmv0nw/dump-postgres-databases-throu...
  - w0land 7y ago
    For copying a directory tree from host1 to host2 preserving user / permissions and compressing on the fly to reduce network usage i use: # from host1: cd $DIRECTORY && tar -jcf - . | ssh host2 "cd DESTINATION_DIR && tar jxvf - "
- UI_at_80x24 7y ago
  Yeah this is part of my workflow too.
- wang_li 7y ago
  Unless you can compress at a higher throughout than your network speed, don’t do this. I’m not familiar with too many machines that can gzip at greater than 120 Mbytes/sec, which is the network bandwidth of a switched gigabit network. There’s no point in making your task take longer with unnecessary compression steps. This also applies to encryption but that’s a much more context dependent decision.
  - m0zg 7y ago
    Just about any machine can gzip at > 120MB/s, just use lower compression settings. You can compress even faster with e.g. zstd, as well.
    - wang_li 7y ago
      That doesn't seem to be the case. A relatively fast machine can't gzip anywhere near 120 MB/s unless the data ultra compressible. Zstd is multithreaded, but single threaded it's speed isn't that fast either. $ lscpu | grep name Model name: Intel(R) Xeon(R) Gold 6154 CPU @ 3.00GHz $ time dd if=/dev/zero of=/dev/null bs=10240 count=102400 102400+0 records in 102400+0 records out 1048576000 bytes (1.0 GB) copied, 0.145168 s, 7.2 GB/s real 0m0.147s user 0m0.036s sys 0m0.111s $ time dd if=/dev/zero bs=10240 count=102400 | gzip > /dev/null 102400+0 records in 102400+0 records out 1048576000 bytes (1.0 GB) copied, 5.36061 s, 196 MB/s real 0m5.364s user 0m5.317s sys 0m0.472s $ time dd if=/dev/urandom bs=10240 count=102400 | gzip > /dev/null 102400+0 records in 102400+0 records out 1048576000 bytes (1.0 GB) copied, 27.3478 s, 38.3 MB/s real 0m27.352s user 0m27.169s sys 0m10.119s $ time dd if=biggestwords bs=10240 count=102400 | gzip > /dev/null 10642+1 records in 10642+1 records out 108981378 bytes (109 MB) copied, 6.66068 s, 16.4 MB/s real 0m6.669s user 0m6.667s sys 0m0.249s $ time dd if=biggestwords bs=10240 count=102400 | gzip -1 > /dev/null 10642+1 records in 10642+1 records out 108981378 bytes (109 MB) copied, 1.53539 s, 71.0 MB/s real 0m1.540s user 0m1.522s sys 0m0.138s $ time dd if=biggestwords of=/dev/null bs=10240 count=102400 10642+1 records in 10642+1 records out 108981378 bytes (109 MB) copied, 0.0407698 s, 2.7 GB/s real 0m0.043s user 0m0.007s sys 0m0.036s Biggestwords is simply /usr/share/dict/words concatenated to itself 22 times so as to provide ~100 MB of data. If you can get good enough compression it might still be worthwhile to compress. But it's pretty rare in my experience.
derefr 7y ago
Re: managing the SSH keys being annoying—I’d suggest setting up an OpenSSH `AuthorizedKeysCommand` in your instances (e.g. using a cloud-init script), that looks at some central repository of SSH keys. This isn’t me telling you to implement some scary arcane wire-protocol, either. All an AuthorizedKeysCommand script/binary has to do is spit out a stream of SSH pubkeys considered valid to authenticate a login attempt on a given username. The “implicit default” AuthorizedKeysCommand is `cat “/home/$1/.ssh/authorized_keys”`. Your implementation just needs to do the same thing that that command does: take “pubkeys that mean this user could log into this box” from some source, and spit them at stdout. (Don’t want that user logging in to that box? Don’t write anything to stdout.) I wrote my own little AuthorizedKeysCommand binary a while back (https://github.com/tsutsu/github-auth3 https://github.com/tsutsu/github-auth3) for using Github’s user-SSH-keys and user-org-membership APIs as such a repository. You tell it a Github org name, and it’ll stream out the username’s equivalent Github user’s SSH pubkeys, if-and-only-if that Github user is a member of the specified Github org. (The code is, like, 20 lines. Read it yourself!) —— This is also one case where specific ecosystems can really smooth things over, though. If you’re running in Google Cloud, for example, you don’t need to do any of the above. Instead, just configure “OS Login” (https://cloud.google.com/compute/docs/oslogin/ https://cloud.google.com/compute/docs/oslogin/), and then log into the gcloud(1) tool on your workstation as an IAM user that you’ve granted the “OS Login” role. Then it’s as simple as: SSH: `gcloud compute ssh instanceName` SCP: `gcloud compute scp ./f.txt instanceName:` (or vice-versa) On first use, IAM will generate a new SSH keypair for the OS Login user. Your gcloud(1) keystore will grab the private half when it needs it; Compute Engine instances will hit IAM to ask for the public half when anyone tries to log in (probably using an SSH `AuthorizedKeysCommand` stanza; I haven’t looked into it.) The user account (`me_example_com` if your IAM user is `me@example.com`) will be created on the instance if it doesn’t already exist (but you can set up your instance template such that it does already exist, if you like, or set up roaming profiles using Filestore as an NFS host, or whatever.) You can also log into your OS Login account through regular direct SSH/SCP to the instance’s external IP (this isn’t some custom proprietary version of SSH or anything), but you won’t have access to the IAM-held private key if you’re not going through gcloud(1), so you’d want to configure your IAM user with a custom authorized pubkey first (which your instances will also automatically see.)
seanc 7y ago
rsync most often uses ssh under the hood. And 'scp -c' will automatically compress and decompress your file, and 'scp -R' will walk a file tree for you.

by ktpsns 7y ago

mschuster91 7y ago

rsync however does not work across two remote hosts, at least not without trickery that one cannot remember. scp at leasts supports easy invocation of a two-remote-hosts scenario but not a direct host-to-host communication, you will always be limited by the client bandwidth.

yetihehe 7y ago

My favourite magic trick is dumping db on one host, gzipping on the fly, tunneling through ssh to other host, then ungzipping straight into other database. Useful when you don't have enough place on disk for local copies.

gglitch 7y ago
It would be super useful to me to see this with a bit more detail, in case you have/know a writeup somewhere!
- yetihehe 7y ago
  I didn't invent it myself, you can see example at [0] or googling 'pg_dump ssh tunnel'. [0] https://coderwall.com/p/bmv0nw/dump-postgres-databases-through-ssh https://coderwall.com/p/bmv0nw/dump-postgres-databases-throu...
- w0land 7y ago
  For copying a directory tree from host1 to host2 preserving user / permissions and compressing on the fly to reduce network usage i use: # from host1: cd $DIRECTORY && tar -jcf - . | ssh host2 "cd DESTINATION_DIR && tar jxvf - "
UI_at_80x24 7y ago
Yeah this is part of my workflow too.
wang_li 7y ago
Unless you can compress at a higher throughout than your network speed, don’t do this. I’m not familiar with too many machines that can gzip at greater than 120 Mbytes/sec, which is the network bandwidth of a switched gigabit network. There’s no point in making your task take longer with unnecessary compression steps. This also applies to encryption but that’s a much more context dependent decision.
- m0zg 7y ago
  Just about any machine can gzip at > 120MB/s, just use lower compression settings. You can compress even faster with e.g. zstd, as well.
  - wang_li 7y ago
    That doesn't seem to be the case. A relatively fast machine can't gzip anywhere near 120 MB/s unless the data ultra compressible. Zstd is multithreaded, but single threaded it's speed isn't that fast either. $ lscpu | grep name Model name: Intel(R) Xeon(R) Gold 6154 CPU @ 3.00GHz $ time dd if=/dev/zero of=/dev/null bs=10240 count=102400 102400+0 records in 102400+0 records out 1048576000 bytes (1.0 GB) copied, 0.145168 s, 7.2 GB/s real 0m0.147s user 0m0.036s sys 0m0.111s $ time dd if=/dev/zero bs=10240 count=102400 | gzip > /dev/null 102400+0 records in 102400+0 records out 1048576000 bytes (1.0 GB) copied, 5.36061 s, 196 MB/s real 0m5.364s user 0m5.317s sys 0m0.472s $ time dd if=/dev/urandom bs=10240 count=102400 | gzip > /dev/null 102400+0 records in 102400+0 records out 1048576000 bytes (1.0 GB) copied, 27.3478 s, 38.3 MB/s real 0m27.352s user 0m27.169s sys 0m10.119s $ time dd if=biggestwords bs=10240 count=102400 | gzip > /dev/null 10642+1 records in 10642+1 records out 108981378 bytes (109 MB) copied, 6.66068 s, 16.4 MB/s real 0m6.669s user 0m6.667s sys 0m0.249s $ time dd if=biggestwords bs=10240 count=102400 | gzip -1 > /dev/null 10642+1 records in 10642+1 records out 108981378 bytes (109 MB) copied, 1.53539 s, 71.0 MB/s real 0m1.540s user 0m1.522s sys 0m0.138s $ time dd if=biggestwords of=/dev/null bs=10240 count=102400 10642+1 records in 10642+1 records out 108981378 bytes (109 MB) copied, 0.0407698 s, 2.7 GB/s real 0m0.043s user 0m0.007s sys 0m0.036s Biggestwords is simply /usr/share/dict/words concatenated to itself 22 times so as to provide ~100 MB of data. If you can get good enough compression it might still be worthwhile to compress. But it's pretty rare in my experience.

derefr 7y ago

Re: managing the SSH keys being annoying—I’d suggest setting up an OpenSSH `AuthorizedKeysCommand` in your instances (e.g. using a cloud-init script), that looks at some central repository of SSH keys. This isn’t me telling you to implement some scary arcane wire-protocol, either. All an AuthorizedKeysCommand script/binary has to do is spit out a stream of SSH pubkeys considered valid to authenticate a login attempt on a given username. The “implicit default” AuthorizedKeysCommand is `cat “/home/$1/.ssh/authorized_keys”`. Your implementation just needs to do the same thing that that command does: take “pubkeys that mean this user could log into this box” from some source, and spit them at stdout. (Don’t want that user logging in to that box? Don’t write anything to stdout.) I wrote my own little AuthorizedKeysCommand binary a while back (https://github.com/tsutsu/github-auth3 https://github.com/tsutsu/github-auth3) for using Github’s user-SSH-keys and user-org-membership APIs as such a repository. You tell it a Github org name, and it’ll stream out the username’s equivalent Github user’s SSH pubkeys, if-and-only-if that Github user is a member of the specified Github org. (The code is, like, 20 lines. Read it yourself!) —— This is also one case where specific ecosystems can really smooth things over, though. If you’re running in Google Cloud, for example, you don’t need to do any of the above. Instead, just configure “OS Login” (https://cloud.google.com/compute/docs/oslogin/ https://cloud.google.com/compute/docs/oslogin/), and then log into the gcloud(1) tool on your workstation as an IAM user that you’ve granted the “OS Login” role. Then it’s as simple as: SSH: `gcloud compute ssh instanceName` SCP: `gcloud compute scp ./f.txt instanceName:` (or vice-versa) On first use, IAM will generate a new SSH keypair for the OS Login user. Your gcloud(1) keystore will grab the private half when it needs it; Compute Engine instances will hit IAM to ask for the public half when anyone tries to log in (probably using an SSH `AuthorizedKeysCommand` stanza; I haven’t looked into it.) The user account (`me_example_com` if your IAM user is `me@example.com`) will be created on the instance if it doesn’t already exist (but you can set up your instance template such that it does already exist, if you like, or set up roaming profiles using Filestore as an NFS host, or whatever.) You can also log into your OS Login account through regular direct SSH/SCP to the instance’s external IP (this isn’t some custom proprietary version of SSH or anything), but you won’t have access to the IAM-held private key if you’re not going through gcloud(1), so you’d want to configure your IAM user with a custom authorized pubkey first (which your instances will also automatically see.)

seanc 7y ago

rsync most often uses ssh under the hood. And 'scp -c' will automatically compress and decompress your file, and 'scp -R' will walk a file tree for you.