3 ms·

Unless you can compress at a higher throughout than your network speed, don’t do this. I’m not familiar with too many machines that can gzip at greater than 120

Unless you can compress at a higher throughout than your network speed, don’t do this. I’m not familiar with too many machines that can gzip at greater than 120 Mbytes/sec, which is the network bandwidth of a switched gigabit network. There’s no point in making your task take longer with unnecessary compression steps. This also applies to encryption but that’s a much more context dependent decision.

m0zg 7y ago
Just about any machine can gzip at > 120MB/s, just use lower compression settings. You can compress even faster with e.g. zstd, as well.
- wang_li 7y ago
  That doesn't seem to be the case. A relatively fast machine can't gzip anywhere near 120 MB/s unless the data ultra compressible. Zstd is multithreaded, but single threaded it's speed isn't that fast either. $ lscpu | grep name Model name: Intel(R) Xeon(R) Gold 6154 CPU @ 3.00GHz $ time dd if=/dev/zero of=/dev/null bs=10240 count=102400 102400+0 records in 102400+0 records out 1048576000 bytes (1.0 GB) copied, 0.145168 s, 7.2 GB/s real 0m0.147s user 0m0.036s sys 0m0.111s $ time dd if=/dev/zero bs=10240 count=102400 | gzip > /dev/null 102400+0 records in 102400+0 records out 1048576000 bytes (1.0 GB) copied, 5.36061 s, 196 MB/s real 0m5.364s user 0m5.317s sys 0m0.472s $ time dd if=/dev/urandom bs=10240 count=102400 | gzip > /dev/null 102400+0 records in 102400+0 records out 1048576000 bytes (1.0 GB) copied, 27.3478 s, 38.3 MB/s real 0m27.352s user 0m27.169s sys 0m10.119s $ time dd if=biggestwords bs=10240 count=102400 | gzip > /dev/null 10642+1 records in 10642+1 records out 108981378 bytes (109 MB) copied, 6.66068 s, 16.4 MB/s real 0m6.669s user 0m6.667s sys 0m0.249s $ time dd if=biggestwords bs=10240 count=102400 | gzip -1 > /dev/null 10642+1 records in 10642+1 records out 108981378 bytes (109 MB) copied, 1.53539 s, 71.0 MB/s real 0m1.540s user 0m1.522s sys 0m0.138s $ time dd if=biggestwords of=/dev/null bs=10240 count=102400 10642+1 records in 10642+1 records out 108981378 bytes (109 MB) copied, 0.0407698 s, 2.7 GB/s real 0m0.043s user 0m0.007s sys 0m0.036s Biggestwords is simply /usr/share/dict/words concatenated to itself 22 times so as to provide ~100 MB of data. If you can get good enough compression it might still be worthwhile to compress. But it's pretty rare in my experience.

m0zg 7y ago

Just about any machine can gzip at > 120MB/s, just use lower compression settings. You can compress even faster with e.g. zstd, as well.

wang_li 7y ago
That doesn't seem to be the case. A relatively fast machine can't gzip anywhere near 120 MB/s unless the data ultra compressible. Zstd is multithreaded, but single threaded it's speed isn't that fast either. $ lscpu | grep name Model name: Intel(R) Xeon(R) Gold 6154 CPU @ 3.00GHz $ time dd if=/dev/zero of=/dev/null bs=10240 count=102400 102400+0 records in 102400+0 records out 1048576000 bytes (1.0 GB) copied, 0.145168 s, 7.2 GB/s real 0m0.147s user 0m0.036s sys 0m0.111s $ time dd if=/dev/zero bs=10240 count=102400 | gzip > /dev/null 102400+0 records in 102400+0 records out 1048576000 bytes (1.0 GB) copied, 5.36061 s, 196 MB/s real 0m5.364s user 0m5.317s sys 0m0.472s $ time dd if=/dev/urandom bs=10240 count=102400 | gzip > /dev/null 102400+0 records in 102400+0 records out 1048576000 bytes (1.0 GB) copied, 27.3478 s, 38.3 MB/s real 0m27.352s user 0m27.169s sys 0m10.119s $ time dd if=biggestwords bs=10240 count=102400 | gzip > /dev/null 10642+1 records in 10642+1 records out 108981378 bytes (109 MB) copied, 6.66068 s, 16.4 MB/s real 0m6.669s user 0m6.667s sys 0m0.249s $ time dd if=biggestwords bs=10240 count=102400 | gzip -1 > /dev/null 10642+1 records in 10642+1 records out 108981378 bytes (109 MB) copied, 1.53539 s, 71.0 MB/s real 0m1.540s user 0m1.522s sys 0m0.138s $ time dd if=biggestwords of=/dev/null bs=10240 count=102400 10642+1 records in 10642+1 records out 108981378 bytes (109 MB) copied, 0.0407698 s, 2.7 GB/s real 0m0.043s user 0m0.007s sys 0m0.036s Biggestwords is simply /usr/share/dict/words concatenated to itself 22 times so as to provide ~100 MB of data. If you can get good enough compression it might still be worthwhile to compress. But it's pretty rare in my experience.