3 ms·

I misread the title as faster sending, rather than faster workflow. For dropbox we just use shared folders which work no matter what else is going on (network

I misread the title as faster sending, rather than faster workflow. For dropbox we just use shared folders which work no matter what else is going on (network connectivity, machines sleeping, reboots etc). I'd be overjoyed if someone could figure out faster sending. My DSL maxes out at 100 kbytes/sec upstream. Sending someone something like a mongodb dump involves a long wait while I locally compress it, and an even longer wait while it gets uploaded. I'd gladly have all 4 cores (plus another 4 hyperthreaded) plus 32GB of RAM dedicated to getting the files uploaded quicker. In theory some of the compression tools can do this, but some aren't multithreaded, or require a lot of research to figure out workable command line options and even then they consider 4GB to be an outrageous amount of memory. Often you have to wait for the compression to finish before you can even start the sending.

fyrabanks 13y ago
Perhaps you should have a look at pigz, a multi-threaded implementation of gzip. It uses the same flags as gzip, so -# specifies speed (-1 runs the fastest with the least compression ... -9 runs the slowest with the best compression), and by default it spawns as many processes as you have processors. I've never done any formal benchmarks, so I'm not qualified to speak as to its efficiency, however. Based on your late-90's upload rate (and assuming you're working with large files that compress nicely), I'd say the more time you spend on compression, the less overall time you're looking at. That 100kb/s (theoretical) limit is really the first bottleneck you should be fixing, though. [edit]: I completely misread that as kilobits/s. Point still stands minus the 90's bit.
- rogerbinns 13y ago
  I usually use pbzip2 which is multi-threaded bzip2. As for efficiency, gzip -9 uses a 32kb window while bzip2 -9 uses a 900kb window. This plus other improvements is why bzip2 beats gzip. > I'd say the more time you spend on compression, the less overall time you're looking at. That turns out not to be the case. For example xz (-9 uses a 65MB window) does indeed result in better compression. However it isn't multi-threaded and uses considerably more cpu time. The extra time taken is beaten by using pbzip2 which can start the upload sooner. 7z does allow multi-threading if you can navigate it's command line options. BTW here is a quick test. I have a json file that is 8.1GB. 442MB 3m12s single threaded gzip -9 309MB 23m single threaded bzip2 -9 326MB 25m single threaded xz -9
  - rogerbinns 13y ago
    As a followup, the most effective way to minimize upload time would be to start from the front of the file with gzip or something similarly quick, and simultaneously start the heavyweight compressions from the end of the file. When the two meet the more compressed content gets sent for the remainder of the file.

fyrabanks 13y ago

Perhaps you should have a look at pigz, a multi-threaded implementation of gzip. It uses the same flags as gzip, so -# specifies speed (-1 runs the fastest with the least compression ... -9 runs the slowest with the best compression), and by default it spawns as many processes as you have processors. I've never done any formal benchmarks, so I'm not qualified to speak as to its efficiency, however. Based on your late-90's upload rate (and assuming you're working with large files that compress nicely), I'd say the more time you spend on compression, the less overall time you're looking at. That 100kb/s (theoretical) limit is really the first bottleneck you should be fixing, though. [edit]: I completely misread that as kilobits/s. Point still stands minus the 90's bit.

rogerbinns 13y ago
I usually use pbzip2 which is multi-threaded bzip2. As for efficiency, gzip -9 uses a 32kb window while bzip2 -9 uses a 900kb window. This plus other improvements is why bzip2 beats gzip. > I'd say the more time you spend on compression, the less overall time you're looking at. That turns out not to be the case. For example xz (-9 uses a 65MB window) does indeed result in better compression. However it isn't multi-threaded and uses considerably more cpu time. The extra time taken is beaten by using pbzip2 which can start the upload sooner. 7z does allow multi-threading if you can navigate it's command line options. BTW here is a quick test. I have a json file that is 8.1GB. 442MB 3m12s single threaded gzip -9 309MB 23m single threaded bzip2 -9 326MB 25m single threaded xz -9
- rogerbinns 13y ago
  As a followup, the most effective way to minimize upload time would be to start from the front of the file with gzip or something similarly quick, and simultaneously start the heavyweight compressions from the end of the file. When the two meet the more compressed content gets sent for the remainder of the file.