3 ms·
Network file transfer with on-the-fly compression
- wmf 17y agoI'd like to see a comparison between this and scp with compression.
- KirinDave 17y agoThat's the first thing I thought of too. The -C flag is there specifically for this very scenario. Also, would "j" have worked better than "z" in the tar line? bzip2 is much slower than gzip, but does a better job.
- wmf 17y agoReally we should have adaptive compression so that the CPU and network usage is balanced. Too bad it may require "rampant layering violation" to implement.
- doosra 17y agoscp -C does the same thing. Both methods use gzip for compression.
- rarrrrrr 17y agoBut the tar method will generally be 5-15x faster for large collections of small files. Handing it with tar makes operations on both ends asynchronous. scp (or rsync, etc.) of multiple files does file opens on each end in lockstep, resulting in a lot of idle wait time.
- mwcremer 17y agoNote that this is not likely to help with media files (mp3, mpeg, jpeg).