I just tested this on my laptop, I used the first 5 million JSONLines of /u/stuck_in_the_matrix reddit dataset (~4.6GB). For compression lz4 took ~22 seconds (

I just tested this on my laptop, I used the first 5 million JSONLines of /u/stuck_in_the_matrix reddit dataset (~4.6GB). For compression lz4 took ~22 seconds (~210 MB/s) and I got ~30% compression, gzip -1 took ~56 seconds (~80 MB/s) and I got ~22% compression. For decompression lz4 gave me 500MB/s while gunzip gave me 300MB/s. Commands used: lz4 -cd RS_full_corpus.lz4 | pv | head -5000000 | gzip -1 > test.gz gunzip -c test.gz | pv > /dev/null lz4 -cd RS_full_corpus.lz4 | pv | head -5000000 | lz4 lz4 -cd stdin.lz4 | pv >/dev/null