3 ms·
> Are there other dimensions that might be important for a particular use case so a different algo may be best? Choosing a compression/decompression algo is an
by Galanwe 2y ago
> Are there other dimensions that might be important for a particular use case so a different algo may be best?
Choosing a compression/decompression algo is an optimization problem essentially. It mainly depends on your IO for input.
For instance, if you have very fast IO, say a local SSD at 7GB/s, then you probably don't want compression, as even the fastest decompression will not go beyond 4GB/s.
If you have moderately fast IO, say 4GB/s, then lz4 is likely a good candidate, you will get maybe a 0.5 compression ratio, meaning you will spend half the time reading your input, while decompressing at nearly the same same speed that you would read uncompressed data.
If you have very slow IO, say 100MB/s, then the compression ratio becomes the main factor, overriding decompression speed.
So, essentially, you want to solve an optimization problem of lowering the time to read the compressed file and decompressing it, conditioned by your reading speed, writing speed, and CPU to spare, under a time constraint of compression speed.
There are solvers on internet doe this problem, but I would love to see someone write it down here in plain math.