4 ms·

Decompression speed looks good, but in my experience once you get past a certain point (~X000 MB/s) performance gains become pretty marginal in real world appli

by aydyn 3y ago

Decompression speed looks good, but in my experience once you get past a certain point (~X000 MB/s) performance gains become pretty marginal in real world applications. I'd like to see compression speeds and performance on AVX if AVX-512 is not available.

2h 3y ago
I don't agree with this. The whole point of tools like this, is that we don't live in age of dial up internet anymore, and haven't for a long time. Back then you wanted maximum compression like XZ, because the slowest part of the process was the download. Now especially with XZ awful decompress rate, the slowest part is extract. In many cases it's better to have 9% larger size, for 500% increase in extract, like you get with ZST.
- scaramanga 3y ago
  Definitely there's a lot of workloads that are pure data-munging, where you don't want to store the whole bloated JSON (or protobuf) mess on eg. S3 where you're paying by the byte, but at the same time, you're looking at key parts of your infra which are spending double digit percentages of their CPU time just decompressing stuff. Sure, you can say the problem is JSON, protobuf, whatever. But in a huge organization with hundreds or thousands of types of documents, it's a needless friction to require custom optimized formats for each one. So a faster decompressor is often a no-brainer of a win and can have substantial dollar values attached.
gopalv 3y ago
> in my experience once you get past a certain point (~X000 MB/s) performance gains become pretty marginal in real world applications There might be no latency benefits to that, but there's always a throughput gain from using fewer cpu cycles over all - the cpu cycles you would have spent decompressing can be used for other things. For that reason single-core performance would still be nice to have and always great to have open source code which is performance centric so that we all get to find out "How the hell did they do it?" (it is very readable assembly [1]) > I'd like to see compression speeds and performance on AVX if AVX-512 is not available. The ARM is probably more interesting in the elastic compute world right now, the AVX world is unlikely to get cheaper. And unlike before, I don't need to wait for a vendor to sell me an SoC, Gravitons are already here. [1] - https://github.com/SnellerInc/sneller/blob/master/ion/zion/iguana/decompress_iguana_amd64.s#L34 https://github.com/SnellerInc/sneller/blob/master/ion/zion/i...
marginalia_nu 3y ago
Yeah, I think often these benchmarks don't capture the constant factor. The rate at which 4 Kb strings can be decompressed probably matter more than rate at which 400 Mb files can be decompressed. Benchmarks are hard though.

by aydyn 3y ago

2h 3y ago

I don't agree with this. The whole point of tools like this, is that we don't live in age of dial up internet anymore, and haven't for a long time. Back then you wanted maximum compression like XZ, because the slowest part of the process was the download. Now especially with XZ awful decompress rate, the slowest part is extract. In many cases it's better to have 9% larger size, for 500% increase in extract, like you get with ZST.

scaramanga 3y ago
Definitely there's a lot of workloads that are pure data-munging, where you don't want to store the whole bloated JSON (or protobuf) mess on eg. S3 where you're paying by the byte, but at the same time, you're looking at key parts of your infra which are spending double digit percentages of their CPU time just decompressing stuff. Sure, you can say the problem is JSON, protobuf, whatever. But in a huge organization with hundreds or thousands of types of documents, it's a needless friction to require custom optimized formats for each one. So a faster decompressor is often a no-brainer of a win and can have substantial dollar values attached.

gopalv 3y ago

> in my experience once you get past a certain point (~X000 MB/s) performance gains become pretty marginal in real world applications There might be no latency benefits to that, but there's always a throughput gain from using fewer cpu cycles over all - the cpu cycles you would have spent decompressing can be used for other things. For that reason single-core performance would still be nice to have and always great to have open source code which is performance centric so that we all get to find out "How the hell did they do it?" (it is very readable assembly [1]) > I'd like to see compression speeds and performance on AVX if AVX-512 is not available. The ARM is probably more interesting in the elastic compute world right now, the AVX world is unlikely to get cheaper. And unlike before, I don't need to wait for a vendor to sell me an SoC, Gravitons are already here. [1] - https://github.com/SnellerInc/sneller/blob/master/ion/zion/iguana/decompress_iguana_amd64.s#L34 https://github.com/SnellerInc/sneller/blob/master/ion/zion/i...

marginalia_nu 3y ago

Yeah, I think often these benchmarks don't capture the constant factor. The rate at which 4 Kb strings can be decompressed probably matter more than rate at which 400 Mb files can be decompressed. Benchmarks are hard though.