3 ms·
> The other 95% of it is just wasted bandwidth. You can save a lot of bandwith by requesting compressed responses: $ curl -s --user-agent moo/1 -H 'Accept-E
by jwilk 8y ago
> The other 95% of it is just wasted bandwidth.
You can save a lot of bandwith by requesting compressed responses:
$ curl -s --user-agent moo/1 -H 'Accept-Encoding: gzip' "$pretty_long_url" > test.gz
$ wc -c < test.gz
63507
$ gzip -d < test.gz | wc -c
426941
(OK, that's 85% saved, not 95%, but hey.)
- dev_dull 8y agoTrading cpu for bandwidth, so optimize for what you want.
- koolba 8y agoDecompression is usually cheaper (CPU-wise) than compression so technically you’re trading their CPU for less of your bandwidth and CPU. If you do it right you can even keep the content stored compressed without re-compressing by saving the compressed byte stream directly.
- tgtweak 8y agoAlso use pigz instead of gzip so it can use multiple threads on decoding (although you're probably Limited by sending bandwidth)
- jwilk 8y agopigz doesn't help much with decompression: https://github.com/madler/pigz/issues/36#issuecomment-249041503 https://github.com/madler/pigz/issues/36#issuecomment-249041... Decompression can’t be parallelized, at least not without specially prepared deflate streams for that purpose. As a result, pigz uses a single thread (the main thread) for decompression, but will create three other threads for reading, writing, and check calculation, which can speed up decompression under some circumstances.