5 ms·
What's the memory footprint for Zstandard (vs. zlib) for compression and decompression? The article goes into CPU usage (which seems awesome...) but I don't se
by koolba 10y ago
What's the memory footprint for Zstandard (vs. zlib) for compression and decompression?
The article goes into CPU usage (which seems awesome...) but I don't see anything about the buffer sizes for either. It just mentions them as knobs that can be twiddled.
- rdtsc 10y agoThere is mention in the blog of the creator about window size. It can be configured down to 4KB to it can operate in low memory environments: http://fastcompression.blogspot.com/ http://fastcompression.blogspot.com/ see under "Custom window sizes" section
- indygreg2 10y agoIt depends on the "window size" used by the compressor. According to https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md#window_descriptor https://github.com/facebook/zstd/blob/dev/doc/zstd_compressi..., the window size can be anywhere from 1KB to 1.875TB. The table mapping compression levels to window sizes is at https://github.com/facebook/zstd/blob/15a7a99653c78a57d1ccbf5c5b4571e62183bf4f/lib/compress/zstd_compress.c#L3250 https://github.com/facebook/zstd/blob/15a7a99653c78a57d1ccbf.... The first element is the window size. I'm pretty sure you raise 2 to that power to get the window size. As you can see, level 1 is 2^18, which is 256KB - far larger than the minimum of 1KB. Lowering it will make compression faster at the expense of compression ratio. For comparison, zlib's max window size is 32KB. That's one reason its compression ratio is so limited.