7 ms·
This is especially fun when combined with piping through tar, and adding pv in the mix for a transfer speed "progress" bar. Fastest way to transfer a collectio
by unilynx 6y ago
This is especially fun when combined with piping through tar, and adding pv in the mix for a transfer speed "progress" bar.
Fastest way to transfer a collection of files on a local network and doesn't require temporary storage for the archive
- enriquto 6y agopiping through tar is my "power move" when I want to steer a new recruit. I just do it casually in front of them... depending on whether they are awed or horrified by that, they'll get more hacky or more formal work.
- parliament32 6y agoYou can also pipe through gpg if you want encryption along the way!
- ahsima1 6y agoAnd through gzip/zstd/etc if you want compression!
- enriquto 6y agothat's what the -C option to ssh is for
- cellularmitosis 6y agolzop if you gotta go fast :)
- namibj 6y agozstd easily outruns a gigabit link on most somewhat-compressible data.
- 3np 6y agoAnd through tac if you want it in reverse! (...And my axe!)
- labawi 6y agoNote that gpg compresses by default. Which is thy it's so slow. You may wish to use lzop or zstd but also disable gpg's compression. I usually use ssh (e.g. ssh -L 3333:localhost:3333) and avoid gpg, though I may use lzop depending on link and CPU speed.
- kortex 6y agoWhat's an example of the syntax of this? I remember seeing a graybeard do something like this to huck some files to another machine.
- waterhouse 6y agoParaphrased from a real example: tar cf - files morefiles | ssh user@dest "tar xf -" If you want to insert, say, compression: tar cf - files morefiles | zstd | ssh user@dest "zstd -d | tar xf -"
- ashton314 6y agoNice. This to me is a great testament of the power of the Unix philosophy: small orthogonal tools that operate on streams of text.