3 ms·
> zstd at compression level 1 can do ~2GB/s per core So unless you can multithread that workload, it's already behind by a factor of 2. > Computers are plenty
by magicalhippo 5y ago
> zstd at compression level 1 can do ~2GB/s per core
So unless you can multithread that workload, it's already behind by a factor of 2.
> Computers are plenty fast.
My point was you can no longer assume the disk is significantly slower, at least for streaming workloads. You can often still win by spending CPU cycles doing clever stuff, but it's not several orders of magnitude difference like it used to be.
- willis936 5y agoMost zstd implementations are multithreaded for single compression and decompression tasks. My home server has 24 cores and is using 16x PCIe 3.0 for storage (16 GB/s). Through benchmarking I found that I am storage bound from zstd-3+ and compute bound from zstd-2 and zstd-1. I run zstd-3.
- 1_player 5y agoDumb question, but shouldn't it be the opposite? The lower the compression level, the faster the CPU can process the data, making storage the bottleneck. Higher compression level -> CPU becomes the bottleneck.
- willis936 5y agoI believe in the TrueNAS interface I use these are negative compression levels. I could be wrong. It was just some empirical observations. I may flesh out my matrix more to paint a better picture.