3 ms·
There are many more concerns to address than just compression ratio. Even the ratio one is questionable, because some people have really fast networks but we al
by jzwinck 11y ago
There are many more concerns to address than just compression ratio. Even the ratio one is questionable, because some people have really fast networks but we all have basically the same speed of computers. So a 4x CPU time and memory pressure penalty may be much worse on a system than a 2x stream size increase. Another use case is a tiny VM instance: half a gigabyte of RAM is not actually present in every machine today. Embedded, too.
Another way compression formats can win you much more than a 2x space reduction is by supporting random access within their contained files. Gzip sort of supports this if you work hard at it. Xz and bzip2 appears similar (though the details are different). I achieved a 50x speedup with this in real applications, and discussed it a bit here: http://stackoverflow.com/questions/429987/compression-formats-with-good-support-for-random-access-within-archives http://stackoverflow.com/questions/429987/compression-format...
- acqq 11y agoThanks for the random access discussion! And you are right for embedded! .xz just doesn't work there. I've also found that on the faster systems, for different uses of mine, when I want the compression to last as little as possible and the total round trip time matters (compression and decompression), gzip -1 gives the best resulting size for the reasonably short time I want to spend.
- geographomics 11y agoI've come across quite a lot of firmware on embedded Linux devices that uses LZMA (the xz compression algorithm) to compress the kernel, u-boot, and/or filesystems. One memory optimisation for these, as they are typically being decompressed straight into RAM, is for the decompressor to refer to its output as the dictionary rather than building a separate one, as would be the case in decompressing to the network or disk.
- dmit 11y agoAgreed on usefulness of random access. Here's a couple more links on seekable gzip compression: http://lh3.github.io/2014/07/05/random-access-to-zlib-compressed-files/ http://lh3.github.io/2014/07/05/random-access-to-zlib-compre... http://www.ebaytechblog.com/2015/10/09/gzinga-seekable-and-splittable-gzip/ http://www.ebaytechblog.com/2015/10/09/gzinga-seekable-and-s...
- 3minus1 11y ago> So a 4x CPU time and memory pressure penalty may be much worse on a system than a 2x stream size increase. If it's being downloaded once
- Twirrim 11y agoEven then it depends. If it takes you 60 seconds to download as gz, and 50 as xz. The decompression needs to take less than 10 seconds more for it to be comparable and you've got to be sure that your end users have more memory and sufficient processing power to throw at the task.