4 ms·

What's surprising to me is that zstd beat out lz4 on compress/decompress speed. LZ4 is supposed to be purpose built to optimize on those metrics. Great job zs

• ctur 4y ago
  The lz4 developers are the zstd developers :) In this case I suspect the IO reductions on the output are what matter -- while lz4 is faster, it also produces more output (of course) which you then have to write to disk. This can make the wall time take longer... basically you become IO bound on output, be it network or disk. zstd also can tune itself and adjust ratios to saturate output bandwidth, which is pretty cool (--adapt).
  • marceloaltmann 4y ago
    That is exactly it. The difference comes from compression ratio. ZSTD is compressing the data more, so you need to write less back to disk. Also when talking about streaming, the difference is even more as it has to go over wan to S3 (provider used on blog test).