3 ms·
I believe zstd can detect incompressible data quickly and falls back to a low-overhead passthrough mode in that case. So you can just run everything through zst
by infogulch 4y ago
I believe zstd can detect incompressible data quickly and falls back to a low-overhead passthrough mode in that case. So you can just run everything through zstd and it handles those cases pretty optimally already.
- le-mark 4y agoHmm this made me chuckle, is that not equivalent to a test for randomness, which is very very hard indeed?
- Cyberdog 4y agoI think a useful test for this case could be pretty simple. Compress maybe 10% of the file (up to some reasonable maximum, like maybe a few megabytes) and check the compression ratio. If it's not something better than maybe 1.1x, give up, throw out the compressed data, and just pass through the file. Otherwise keep going.
- hexxagone 4y agoIt is not. Just look at the frequencies in the data (histogram) and compute the order 0 entropy. Then you know if your data is compressible.