5 ms·

Last I checked you can't get much better than 1.5GB/s per core with LZ4 (from RAM), up to a maximum ratio < 3:1, and multicore decompression is not really possi

Last I checked you can't get much better than 1.5GB/s per core with LZ4 (from RAM), up to a maximum ratio < 3:1, and multicore decompression is not really possible unless you manually tweak the compression. The benchmarks above that are usually misleading, because they assume no dependence between blocks, which is nuts. In real scenarios, blocks need to be parsed, depend on their previous blocks, and you need to carry around that context. My RAM can deliver close to 20GB/s, and my SSD 7GB/s, and that is all commodity hardware. Meaning unless you have quite slow disks, you're better off without compression.

riku_iki 2y ago
> Last I checked you can't get much better than 1.5GB/s per core with LZ4 you can partition your dataset and process each partition on separate core, which will produce some massive XX or even XXX GB/s? > up to a maximum ratio < 3:1 this is obviously depends on your data pattern. If it is some low cardinality IDs, they can be compressed by ratio 100 easily.
- Galanwe 2y ago
  > you can partition your dataset and process each partition on separate core, which will produce some massive XX or even XXX GB/s? Yes, but as I mentioned: > multicore decompression is not really possible unless you manually tweak the compression That is, there is no stable implementation out there that does it. You will have to do that manually and painfully. In which case, you're opening the doors for exotic/niche compression/decompression, and there are better alternatives than LZ4 if you're in the niche market. > this is obviously depends on your data pattern. If it is some low cardinality IDs, they can be compressed by ratio 100 easily. Everything is possible in theory. Yet we have to agree on what is a reasonable expectation. A compression factor of around 3:1 is, from my experience, what you would get from a reasonable compression speed on reasonably distributed data.
  - riku_iki 2y ago
    > Yes, but as I mentioned > multicore decompression is not really possible unless you manually tweak the compression I don't understand your point. Decompression will be applied on separate partitions using separate cores the same way as compression.. > Yet we have to agree on what is a reasonable expectation. A compression factor of around 3:1 is, from my experience well, my prod database is compressed by ratio 7 (many hundreds billions IDs).

riku_iki 2y ago

> Last I checked you can't get much better than 1.5GB/s per core with LZ4 you can partition your dataset and process each partition on separate core, which will produce some massive XX or even XXX GB/s? > up to a maximum ratio < 3:1 this is obviously depends on your data pattern. If it is some low cardinality IDs, they can be compressed by ratio 100 easily.

Galanwe 2y ago
> you can partition your dataset and process each partition on separate core, which will produce some massive XX or even XXX GB/s? Yes, but as I mentioned: > multicore decompression is not really possible unless you manually tweak the compression That is, there is no stable implementation out there that does it. You will have to do that manually and painfully. In which case, you're opening the doors for exotic/niche compression/decompression, and there are better alternatives than LZ4 if you're in the niche market. > this is obviously depends on your data pattern. If it is some low cardinality IDs, they can be compressed by ratio 100 easily. Everything is possible in theory. Yet we have to agree on what is a reasonable expectation. A compression factor of around 3:1 is, from my experience, what you would get from a reasonable compression speed on reasonably distributed data.
- riku_iki 2y ago
  > Yes, but as I mentioned > multicore decompression is not really possible unless you manually tweak the compression I don't understand your point. Decompression will be applied on separate partitions using separate cores the same way as compression.. > Yet we have to agree on what is a reasonable expectation. A compression factor of around 3:1 is, from my experience well, my prod database is compressed by ratio 7 (many hundreds billions IDs).