5 ms·
If I'm actually doing something with my data, gzip -1 beats out lz4 for streaming, as gzip -1 can usually keep up with the slower of the in/out sides, and gzip
by aidenn0 11y ago
If I'm actually doing something with my data, gzip -1 beats out lz4 for streaming, as gzip -1 can usually keep up with the slower of the in/out sides, and gzip -1 is higher compression ratio than lz4 and faster compression (but not decompression) than lz4hc.
- dchest 11y agoFYI, this cool little project — https://code.google.com/p/miniz/ https://code.google.com/p/miniz/ — implements faster gzip compression at level 1.
- faizshah 11y agoI just tested this on my laptop, I used the first 5 million JSONLines of /u/stuck_in_the_matrix reddit dataset (~4.6GB). For compression lz4 took ~22 seconds (~210 MB/s) and I got ~30% compression, gzip -1 took ~56 seconds (~80 MB/s) and I got ~22% compression. For decompression lz4 gave me 500MB/s while gunzip gave me 300MB/s. Commands used: lz4 -cd RS_full_corpus.lz4 | pv | head -5000000 | gzip -1 > test.gz gunzip -c test.gz | pv > /dev/null lz4 -cd RS_full_corpus.lz4 | pv | head -5000000 | lz4 lz4 -cd stdin.lz4 | pv >/dev/null
- aidenn0 11y agoInteresting; on a mix of source and binaries (archived fully-built checkouts) gzip -1 outperformed lz4 in compression ratio.
- faizshah 11y agoNo you're correct, gzip -1 outperformed lz4 in my test in compression ratio. I don't know why I typed "30% compression" instead of "compression ratio of 30%." Sorry about that.