3 ms·
How did I miss zstd? Here are my benchmarks for 2.3 GB of jsonl, on a laptop. Compressed size, compress time, decompress time; using defaults. gzip 7.3%
by 8organicbits 19d ago
How did I miss zstd?
Here are my benchmarks for 2.3 GB of jsonl, on a laptop. Compressed size, compress time, decompress time; using defaults.
gzip 7.3% 21s 9s
bzip2 4.6% 251s 50s
bzip3 3.3% 82s 69s
zstd 6.9% 2s 3s
lzma 4.7% 51s 3s
- vlovich123 19d agoAt what levels? There’s no guarantee that the default compression level is comparable. You have to normalize by time spent compressing.
- out_of_protocol 19d agozstd with better compression level would be nice - these numbers are not really comparable since both time and compression level are too different
- kccqzy 19d agoBut zstd is super tunable. Where gzip gives you compression levels from 1 to 9, zstd gives you up to 22 for ultra compression and negative compression levels for ultra fast. The ultra fast options so fast that they are great as a substitute for memcpy if your CPU is already waiting for other things, like DRAM.
- handsome_jack_ 19d ago[flagged]
- kccqzy 19d agoNo it’s not. The pace of improvement of CPU compute speed is far greater than that of DRAM throughput. And in fact compression algorithms geared towards speed aims to outperform memcpy (on suitable machines).
- praseodym 19d agozstd has a built-in benchmark mode to compare different compression levels, e.g. `zstd -b1 -e9 [FILE]` to test levels 1 to 9 (try up to 22 if you have enough spare time)