5 ms·

I found lz4 to be the best for general purpose analysis, it increased the throughput of my processing 10x compared to bz2. Then if you're working with very lar

I found lz4 to be the best for general purpose analysis, it increased the throughput of my processing 10x compared to bz2. Then if you're working with very large files you can use the splittable version of lz4, 4mc, which also works as a Hadoop InputFormat. I just wish they would switch the Common Crawl archives to lz4. I should probably mention the compression ratio was slightly worse than bz2 (maybe 15% larger archive) but for the 10x increase in throughput I didn't really mind that much. I could actually analyze my data from my laptop!

rshm 11y ago
Last time i checked lz4 did not had a streaming decompression support on their python lib. It will be problem for larger files like common crawl if you are not planning to pre-decompress before processing.
- faizshah 11y ago
  It's not a problem for me since I mostly use Java. However, you can probably just pipe in your data from the lz4 CLI then use that InputStream for whatever python parser you're using and you should be fine. The biggest problem is using a parser that can do 600MB/s streaming parsing. If you use a command line parser don't try jq even with gnu parallel.
aidenn0 11y ago
If I'm actually doing something with my data, gzip -1 beats out lz4 for streaming, as gzip -1 can usually keep up with the slower of the in/out sides, and gzip -1 is higher compression ratio than lz4 and faster compression (but not decompression) than lz4hc.
- dchest 11y ago
  FYI, this cool little project — https://code.google.com/p/miniz/ https://code.google.com/p/miniz/ — implements faster gzip compression at level 1.
- faizshah 11y ago
  I just tested this on my laptop, I used the first 5 million JSONLines of /u/stuck_in_the_matrix reddit dataset (~4.6GB). For compression lz4 took ~22 seconds (~210 MB/s) and I got ~30% compression, gzip -1 took ~56 seconds (~80 MB/s) and I got ~22% compression. For decompression lz4 gave me 500MB/s while gunzip gave me 300MB/s. Commands used: lz4 -cd RS_full_corpus.lz4 | pv | head -5000000 | gzip -1 > test.gz gunzip -c test.gz | pv > /dev/null lz4 -cd RS_full_corpus.lz4 | pv | head -5000000 | lz4 lz4 -cd stdin.lz4 | pv >/dev/null
  - aidenn0 11y ago
    Interesting; on a mix of source and binaries (archived fully-built checkouts) gzip -1 outperformed lz4 in compression ratio.
    - faizshah 11y ago
      No you're correct, gzip -1 outperformed lz4 in my test in compression ratio. I don't know why I typed "30% compression" instead of "compression ratio of 30%." Sorry about that.

rshm 11y ago

Last time i checked lz4 did not had a streaming decompression support on their python lib. It will be problem for larger files like common crawl if you are not planning to pre-decompress before processing.

faizshah 11y ago
It's not a problem for me since I mostly use Java. However, you can probably just pipe in your data from the lz4 CLI then use that InputStream for whatever python parser you're using and you should be fine. The biggest problem is using a parser that can do 600MB/s streaming parsing. If you use a command line parser don't try jq even with gnu parallel.

aidenn0 11y ago

If I'm actually doing something with my data, gzip -1 beats out lz4 for streaming, as gzip -1 can usually keep up with the slower of the in/out sides, and gzip -1 is higher compression ratio than lz4 and faster compression (but not decompression) than lz4hc.

dchest 11y ago
FYI, this cool little project — https://code.google.com/p/miniz/ https://code.google.com/p/miniz/ — implements faster gzip compression at level 1.
faizshah 11y ago
I just tested this on my laptop, I used the first 5 million JSONLines of /u/stuck_in_the_matrix reddit dataset (~4.6GB). For compression lz4 took ~22 seconds (~210 MB/s) and I got ~30% compression, gzip -1 took ~56 seconds (~80 MB/s) and I got ~22% compression. For decompression lz4 gave me 500MB/s while gunzip gave me 300MB/s. Commands used: lz4 -cd RS_full_corpus.lz4 | pv | head -5000000 | gzip -1 > test.gz gunzip -c test.gz | pv > /dev/null lz4 -cd RS_full_corpus.lz4 | pv | head -5000000 | lz4 lz4 -cd stdin.lz4 | pv >/dev/null
- aidenn0 11y ago
  Interesting; on a mix of source and binaries (archived fully-built checkouts) gzip -1 outperformed lz4 in compression ratio.
  - faizshah 11y ago
    No you're correct, gzip -1 outperformed lz4 in my test in compression ratio. I don't know why I typed "30% compression" instead of "compression ratio of 30%." Sorry about that.