3 ms·

How were you comparing ISA-L and libdeflate? For decompression I've found that the latest version of libdeflate is slightly faster than ISA-L.

mxmlnkn 3y ago
My extended benchmarks [0] use the rapidgzip, igzip, gzip, and pigz command line utilities and simply redirect the output to /dev/null to minimize I/O write interference. That's where I got the comparison of igzip to "zlib" as it is used in pigz. I did not compare libdeflate very often to these benchmarks, but before posting my comment, I quickly ran "time libdeflate-gzip -f -k -d 4GiB-base64.gz" inside /dev/shm, which took 20s vs. 9s for igzip. libdeflate-gzip is something I built and installed a while ago from libdeflate/programs/gzip.c. lideflate-gzip -V prints: "gzip compression program v1.18. Copyright 2016 Eric Biggers". I am aware that lots of care also has to be taken with I/O, which might make the command line utility slower than the library interface, but doing the tests in /dev/shm hopefully alleviated this. I am also aware that base64-encoded random data is a weird test case but it has its pros because it is a kind of minimal benchmark for raw Huffman decoding speed without (many) LZ references that need to be resolved. I redid the benchmark as outlined above with the three test files that I am also using for my extended benchmarks [0]: 4GiB-base64.gz -> libdeflate: 20.5 s, igzip: 9.4 s, rapidgzip: 1.5 s 20xsilesia.tar.gz -> libdeflate: 5.4 s, igzip: 6.6 s, rapidgzip: 1.8 s 10xSRR22403185_2.fastq.gz -> libdeflate: 5.8 s, igzip: 5.5 s, rapidgzip: 1.9 s File Sizes: Compressed -> Uncompressed: 4GiB-base64.gz : 4294967296 -> 3263906203 20xsilesia.tar.gz : 1364776140 -> 4239155200 10xSRR22403185_2.fastq.gz : 970458140 -> 3618153020 In conclusion, it seems that it highly depends on the test case and the one I tested to, too quickly, check my statement is one of the outliers. [0] https://github.com/mxmlnkn/rapidgzip#scaling-benchmarks-on-2xamd-epyc-cpu-7702-2x64-cores https://github.com/mxmlnkn/rapidgzip#scaling-benchmarks-on-2...
- mxmlnkn 3y ago
  I also did benchmarks with zlib and libarchivemount via their library interface here [0]. It has been a while that I have run them, so I forgot. Unfortunately, I did not add libdeflate. I did not even add ISA-l. At that point, I was already would have been glad if my custom-written gzip decompressor could match the speed of the gzip command line utility, which for some weird reason is half as fast as zlib. [0] https://github.com/mxmlnkn/rapidgzip/blob/master/src/benchmarks/benchmarkGzip.cpp https://github.com/mxmlnkn/rapidgzip/blob/master/src/benchma...

mxmlnkn 3y ago

My extended benchmarks [0] use the rapidgzip, igzip, gzip, and pigz command line utilities and simply redirect the output to /dev/null to minimize I/O write interference. That's where I got the comparison of igzip to "zlib" as it is used in pigz. I did not compare libdeflate very often to these benchmarks, but before posting my comment, I quickly ran "time libdeflate-gzip -f -k -d 4GiB-base64.gz" inside /dev/shm, which took 20s vs. 9s for igzip. libdeflate-gzip is something I built and installed a while ago from libdeflate/programs/gzip.c. lideflate-gzip -V prints: "gzip compression program v1.18. Copyright 2016 Eric Biggers". I am aware that lots of care also has to be taken with I/O, which might make the command line utility slower than the library interface, but doing the tests in /dev/shm hopefully alleviated this. I am also aware that base64-encoded random data is a weird test case but it has its pros because it is a kind of minimal benchmark for raw Huffman decoding speed without (many) LZ references that need to be resolved. I redid the benchmark as outlined above with the three test files that I am also using for my extended benchmarks [0]: 4GiB-base64.gz -> libdeflate: 20.5 s, igzip: 9.4 s, rapidgzip: 1.5 s 20xsilesia.tar.gz -> libdeflate: 5.4 s, igzip: 6.6 s, rapidgzip: 1.8 s 10xSRR22403185_2.fastq.gz -> libdeflate: 5.8 s, igzip: 5.5 s, rapidgzip: 1.9 s File Sizes: Compressed -> Uncompressed: 4GiB-base64.gz : 4294967296 -> 3263906203 20xsilesia.tar.gz : 1364776140 -> 4239155200 10xSRR22403185_2.fastq.gz : 970458140 -> 3618153020 In conclusion, it seems that it highly depends on the test case and the one I tested to, too quickly, check my statement is one of the outliers. [0] https://github.com/mxmlnkn/rapidgzip#scaling-benchmarks-on-2xamd-epyc-cpu-7702-2x64-cores https://github.com/mxmlnkn/rapidgzip#scaling-benchmarks-on-2...

mxmlnkn 3y ago
I also did benchmarks with zlib and libarchivemount via their library interface here [0]. It has been a while that I have run them, so I forgot. Unfortunately, I did not add libdeflate. I did not even add ISA-l. At that point, I was already would have been glad if my custom-written gzip decompressor could match the speed of the gzip command line utility, which for some weird reason is half as fast as zlib. [0] https://github.com/mxmlnkn/rapidgzip/blob/master/src/benchmarks/benchmarkGzip.cpp https://github.com/mxmlnkn/rapidgzip/blob/master/src/benchma...