3 ms·

What do people use these for?

deleted 3y ago
[deleted]
hmottestad 3y ago
Someone else brought up the recent podcast episode of CoRecursive. One benefit that was brought up was compression of the cache close to the user’s physical location. Facebook has a lot of machines around the world dedicated to this sort of cache. They need a balance between compression ratio and compression/decompression throughput. There’s no point using a great compression algorithm to achieve an amazing compression ratio if you end up being CPU-bound when you need to decompress it later on.
bugboy73 3y ago
DISCLAIMER: I'm one of the Sneller developers Compression is required to make more efficient use of the available bandwidth and RAM. Although AWS provides up to 100Gb/s networking, we improve the net throughput using compression. Also we cache data on the computing nodes, so storing it in compressed form allows us to store more data on the node. Our query engine is often limited by memory bandwidth when using uncompressed data. So it needs a lot of data and it needs it fast. Because we cache data in compressed form, decompression is always needed, so it should be as fast as possible. That's the primary reason why we developed Iguana, because a faster decompression results in a faster query engine.
marginalia_nu 3y ago
I don't use Iguana in particular, but zstd and lz4 in Marginalia Search. First, I use it to store and retrieve crawl data faster. Mechanical hard drives are much slower than these decompression algorithms. So for sequential access, you can make a cheap mechanical drive faster and bigger as big as long as you've got some CPU cycles to spare to decode the data. (This is also true for network, basically anything outside of RAM, as long as it's nice and sequential, there's probably an argument for compressing it) I also use them to keep more stuff in RAM than what should be possible. I do transparent compression/decompression of some large strings e.g. document HTML that typically compresses with a factor 10x. These algorithms are so fast the performance hit of decompressing the strings as they're used is like a 10% pessimization of the run time for significantly slashing the RAM usage.
zX41ZdbW 3y ago
You can check my presentation on this topic. It covers the case when compression has to be faster than memory bandwidth: https://presentations.clickhouse.com/meetup53/optimizations/ https://presentations.clickhouse.com/meetup53/optimizations/

deleted 3y ago

[deleted]

Someone else brought up the recent podcast episode of CoRecursive. One benefit that was brought up was compression of the cache close to the user’s physical location. Facebook has a lot of machines around the world dedicated to this sort of cache. They need a balance between compression ratio and compression/decompression throughput. There’s no point using a great compression algorithm to achieve an amazing compression ratio if you end up being CPU-bound when you need to decompress it later on.

bugboy73 3y ago

DISCLAIMER: I'm one of the Sneller developers Compression is required to make more efficient use of the available bandwidth and RAM. Although AWS provides up to 100Gb/s networking, we improve the net throughput using compression. Also we cache data on the computing nodes, so storing it in compressed form allows us to store more data on the node. Our query engine is often limited by memory bandwidth when using uncompressed data. So it needs a lot of data and it needs it fast. Because we cache data in compressed form, decompression is always needed, so it should be as fast as possible. That's the primary reason why we developed Iguana, because a faster decompression results in a faster query engine.

marginalia_nu 3y ago

I don't use Iguana in particular, but zstd and lz4 in Marginalia Search. First, I use it to store and retrieve crawl data faster. Mechanical hard drives are much slower than these decompression algorithms. So for sequential access, you can make a cheap mechanical drive faster and bigger as big as long as you've got some CPU cycles to spare to decode the data. (This is also true for network, basically anything outside of RAM, as long as it's nice and sequential, there's probably an argument for compressing it) I also use them to keep more stuff in RAM than what should be possible. I do transparent compression/decompression of some large strings e.g. document HTML that typically compresses with a factor 10x. These algorithms are so fast the performance hit of decompressing the strings as they're used is like a 10% pessimization of the run time for significantly slashing the RAM usage.

zX41ZdbW 3y ago

You can check my presentation on this topic. It covers the case when compression has to be faster than memory bandwidth: https://presentations.clickhouse.com/meetup53/optimizations/ https://presentations.clickhouse.com/meetup53/optimizations/