4 ms·
This would come with a cost of longer compression times - either multiple attempts with random shuffling or pre-compression file ordering optimization process.
by mr__y 6y ago
This would come with a cost of longer compression times - either multiple attempts with random shuffling or pre-compression file ordering optimization process. For resources that are compressed once and then distributed and decompressed multiple times this would be quite interesting solution
- m463 6y agoI got curious and looked it up. On this page [1] it looks like uncompressing with gzip vs bzip2 vs xz is: gzip bzip2 xz 1 6.771 24.23 13.251 2 6.581 24.101 12.407 3 6.39 23.955 11.975 4 6.313 24.204 11.801 5 6.153 24.513 11.08 6 6.078 24.768 10.911 7 6.057 23.199 10.781 8 6.033 25.426 10.676 9 6.026 23.486 10.623 so gzip has the fastest decompression. That said, xz is in the ballpark and can be significantly smaller. [1] https://www.rootusers.com/gzip-vs-bzip2-vs-xz-performance-comparison/ https://www.rootusers.com/gzip-vs-bzip2-vs-xz-performance-co...
- vardump 6y agoI wonder how zstd would compare.
- JoshTriplett 6y agozstd is faster and smaller. If you can choose the format, zstd beats deflate across the board, on every front except for compatibility with things that only understand deflate. Also, if you need to use deflate for compatibility, use https://github.com/zlib-ng/zlib-ng https://github.com/zlib-ng/zlib-ng , which is substantially faster than either zlib or gzip.
- joseluisq 6y agoYeah, and it is IETF standard since 2018 https://tools.ietf.org/html/rfc8478 https://tools.ietf.org/html/rfc8478
- dependenttypes 6y agoIs it though? In https://quixdb.github.io/squash-benchmark/unstable/ https://quixdb.github.io/squash-benchmark/unstable/ deflate beats zstd at least in some tests.
- joseluisq 6y agoYeah, but zstd beats deflate in others. For example the latest MySQL 8 https://dev.mysql.com/worklog/task/?id=13442 https://dev.mysql.com/worklog/task/?id=13442
- JoshTriplett 6y agoIt's universally faster if you compress threaded, and otherwise it's generally still a win.
- m463 6y agoI've never heard of zstd (thanks!), but it seems it's right under my nose. "Arch Linux added support for zstd as a package compression method in October 2019 with the release of the pacman 5.2 package manager, and in January 2020 switched from xz to zstd for the packages in the official repository. Arch uses zstd -c -T0 --ultra -20 -, the size of all compressed packages combined increased by 0.8% (compared to xz), the decompression speed is 1300% faster, decompression memory increased by 50 MiB when using multiple threads, compression memory increases but scales with the number of threads used." https://en.wikipedia.org/wiki/Zstandard https://en.wikipedia.org/wiki/Zstandard