6 ms·

Perhaps you should have a look at pigz, a multi-threaded implementation of gzip. It uses the same flags as gzip, so -# specifies speed (-1 runs the fastest with

Perhaps you should have a look at pigz, a multi-threaded implementation of gzip. It uses the same flags as gzip, so -# specifies speed (-1 runs the fastest with the least compression ... -9 runs the slowest with the best compression), and by default it spawns as many processes as you have processors. I've never done any formal benchmarks, so I'm not qualified to speak as to its efficiency, however. Based on your late-90's upload rate (and assuming you're working with large files that compress nicely), I'd say the more time you spend on compression, the less overall time you're looking at. That 100kb/s (theoretical) limit is really the first bottleneck you should be fixing, though. [edit]: I completely misread that as kilobits/s. Point still stands minus the 90's bit.

rogerbinns 13y ago
I usually use pbzip2 which is multi-threaded bzip2. As for efficiency, gzip -9 uses a 32kb window while bzip2 -9 uses a 900kb window. This plus other improvements is why bzip2 beats gzip. > I'd say the more time you spend on compression, the less overall time you're looking at. That turns out not to be the case. For example xz (-9 uses a 65MB window) does indeed result in better compression. However it isn't multi-threaded and uses considerably more cpu time. The extra time taken is beaten by using pbzip2 which can start the upload sooner. 7z does allow multi-threading if you can navigate it's command line options. BTW here is a quick test. I have a json file that is 8.1GB. 442MB 3m12s single threaded gzip -9 309MB 23m single threaded bzip2 -9 326MB 25m single threaded xz -9
- rogerbinns 13y ago
  As a followup, the most effective way to minimize upload time would be to start from the front of the file with gzip or something similarly quick, and simultaneously start the heavyweight compressions from the end of the file. When the two meet the more compressed content gets sent for the remainder of the file.

rogerbinns 13y ago

I usually use pbzip2 which is multi-threaded bzip2. As for efficiency, gzip -9 uses a 32kb window while bzip2 -9 uses a 900kb window. This plus other improvements is why bzip2 beats gzip. > I'd say the more time you spend on compression, the less overall time you're looking at. That turns out not to be the case. For example xz (-9 uses a 65MB window) does indeed result in better compression. However it isn't multi-threaded and uses considerably more cpu time. The extra time taken is beaten by using pbzip2 which can start the upload sooner. 7z does allow multi-threading if you can navigate it's command line options. BTW here is a quick test. I have a json file that is 8.1GB. 442MB 3m12s single threaded gzip -9 309MB 23m single threaded bzip2 -9 326MB 25m single threaded xz -9

rogerbinns 13y ago
As a followup, the most effective way to minimize upload time would be to start from the front of the file with gzip or something similarly quick, and simultaneously start the heavyweight compressions from the end of the file. When the two meet the more compressed content gets sent for the remainder of the file.