3 ms·

What about decompression? Are we already I/O bound? Compression is irrelevant in many cases, where you can precompress static content.

by nly 11y ago

moreati 11y ago
zlib latest takes 3 seconds, zlib-ng takes 13 seconds alex@martha:~/src/zlib$ du -sh $TESTFILE 197M /home/alex/src/enwiki-20141106-pages-articles-multistream-index.txt.gz alex@martha:~/src/zlib$ LD_LIBRARY_PATH=./ time ./minigzip -d <$TESTFILE >/dev/null 3.36user 0.03system 0:03.87elapsed 87%CPU (0avgtext+0avgdata 1632maxresident)k 8inputs+0outputs (0major+90minor)pagefaults 0swaps alex@martha:~/src/zlib$ LD_LIBRARY_PATH=./ time ./minigzip -d <$TESTFILE >/dev/null 3.34user 0.04system 0:03.76elapsed 90%CPU (0avgtext+0avgdata 1704maxresident)k 0inputs+0outputs (0major+91minor)pagefaults 0swaps alex@martha:~/src/zlib$ cd ../zlib-ng/ alex@martha:~/src/zlib-ng$ LD_LIBRARY_PATH=./ time ./minigzip -d <$TESTFILE >/dev/null 12.20user 0.04system 0:13.63elapsed 89%CPU (0avgtext+0avgdata 1576maxresident)k 264inputs+0outputs (1major+86minor)pagefaults 0swaps alex@martha:~/src/zlib-ng$ LD_LIBRARY_PATH=./ time ./minigzip -d <$TESTFILE >/dev/null 12.15user 0.08system 0:13.58elapsed 90%CPU (0avgtext+0avgdata 1428maxresident)k 0inputs+0outputs (0major+84minor)pagefaults 0swaps
- gjm11 11y ago
  So the optimizations in zlib-ng enable it to decompress ... more than 4x slower than vanilla zlib? That seems worthy of some further investigation.
- pimlottc 11y ago
  I'm curious, was your decompression test file compressed with zlib? zlib-ng? Does it make a difference?
Dead2 11y ago
Unfortunately the minigzip implementation used is using a very minimal codebase to implement the functionality, among these is a problem where it will read and decompress 1 byte per call to inflate(). A patch and pull request for this has been available since yesterday at least, and I merged this today. In stock zlib this codepath is only used if you defined Z_SOLO, but in zlib-ng it is the default unless you run configure with --zlib-compat (or otherwise set the correct defines). Please retest with the updated commit, or even better with --zlib-compat if you want to get close to an apples-to-apples comparison. EDIT: PS: This goes to show that minigzip can not really be used as a reliable benchmark, since actual applications linking to the library could be using it differently (for better or worse).

What about decompression? Are we already I/O bound? Compression is irrelevant in many cases, where you can precompress static content.

by nly 11y ago

What about decompression? Are we already I/O bound? Compression is irrelevant in many cases, where you can precompress static content.

moreati 11y ago

zlib latest takes 3 seconds, zlib-ng takes 13 seconds alex@martha:~/src/zlib$ du -sh $TESTFILE 197M /home/alex/src/enwiki-20141106-pages-articles-multistream-index.txt.gz alex@martha:~/src/zlib$ LD_LIBRARY_PATH=./ time ./minigzip -d <$TESTFILE >/dev/null 3.36user 0.03system 0:03.87elapsed 87%CPU (0avgtext+0avgdata 1632maxresident)k 8inputs+0outputs (0major+90minor)pagefaults 0swaps alex@martha:~/src/zlib$ LD_LIBRARY_PATH=./ time ./minigzip -d <$TESTFILE >/dev/null 3.34user 0.04system 0:03.76elapsed 90%CPU (0avgtext+0avgdata 1704maxresident)k 0inputs+0outputs (0major+91minor)pagefaults 0swaps alex@martha:~/src/zlib$ cd ../zlib-ng/ alex@martha:~/src/zlib-ng$ LD_LIBRARY_PATH=./ time ./minigzip -d <$TESTFILE >/dev/null 12.20user 0.04system 0:13.63elapsed 89%CPU (0avgtext+0avgdata 1576maxresident)k 264inputs+0outputs (1major+86minor)pagefaults 0swaps alex@martha:~/src/zlib-ng$ LD_LIBRARY_PATH=./ time ./minigzip -d <$TESTFILE >/dev/null 12.15user 0.08system 0:13.58elapsed 90%CPU (0avgtext+0avgdata 1428maxresident)k 0inputs+0outputs (0major+84minor)pagefaults 0swaps

gjm11 11y ago
So the optimizations in zlib-ng enable it to decompress ... more than 4x slower than vanilla zlib? That seems worthy of some further investigation.
pimlottc 11y ago
I'm curious, was your decompression test file compressed with zlib? zlib-ng? Does it make a difference?

Dead2 11y ago

Unfortunately the minigzip implementation used is using a very minimal codebase to implement the functionality, among these is a problem where it will read and decompress 1 byte per call to inflate(). A patch and pull request for this has been available since yesterday at least, and I merged this today. In stock zlib this codepath is only used if you defined Z_SOLO, but in zlib-ng it is the default unless you run configure with --zlib-compat (or otherwise set the correct defines). Please retest with the updated commit, or even better with --zlib-compat if you want to get close to an apples-to-apples comparison. EDIT: PS: This goes to show that minigzip can not really be used as a reliable benchmark, since actual applications linking to the library could be using it differently (for better or worse).