4 ms·

In practice many times I end up using: ssh cat /foo/bar > /bar/baz ssh tar c /foo | tar x -C /bar mostly because variants such as vagrant ssh or dock

• jcranmer 6y ago
  In practice, tar c | ssh tar x can actually be far faster than scp -r, especially when you have a lot of small files.
  • sethhochberg 6y ago
    I've found many people either forget (or never knew) that pipes have a buffer, and that buffers are very useful when you are dealing with potentially bursty throughput. In a previous team where we dealt with shipping around lots of ZFS snapshots (where both the performance of reading from disk can vary based on the ARC and filesystem metadata) and occasionally using VPN tunnels on public internet links, pv to get even larger pipe buffers than standard was often a huge performance equalizer. Readers read as fast as they can, writers write as fast as they can, and the buffer massages any variability in the middle. https://linux.die.net/man/1/pv https://linux.die.net/man/1/pv
    • saltcured 6y ago
      It's not just the buffer, it is the PIPE. Specifically, pipelining that streams one file after another, without an idle period of waiting for the round-trip acknowledgements of one file receipt prior to sending the next. Using rsync over ssh has the same massive benefit for transferring many files, even if there is no existing data to make use of its differential transfer mechanisms. (Those extra mechanisms are usually beneficial in WAN scenarios as well, except when you have high speed links, in which case blasting the full files may be faster than doing the IO to figure out comparisons and skip transfer.) Edited: adjusted confused punctuation.
• edgyquant 6y ago
  I’ve also used dd if=filename | ssh hostname dd of=remote_filename Quite a bit
  • jrumbut 6y ago
    There's something a little badass about using dd to transfer a file to a remote system.
    • schoen 6y ago
      Everything is a file!
    • asveikau 6y ago
      I was transferring files to an Android device, and adb push wasn't working for some reason. It would hit an error and kill the adb connection until I re-plugged the USB. Somehow this worked: dd if=foo | adb shell dd of=/storage/.../foo
  • a1369209993 6y ago
    Huh, pleasantly surprised to learn that dd correctly handles the truncated final block of a not-multiple-of-512-byte file; could have sworn that didn't work at some point.
    • deleted 6y ago
      [deleted]
    • dfox 6y ago
      SUS specifies that dd should do the "right thing" with input with non-integer number of blocks input, that is pad it with zeros if conv=sync, just pass through shorter otherwise.
      • a1369209993 6y ago
        > SUS specifies that dd should do the "right thing" A example the C standard could stand to learn from, then.