2 ms·

On three test machines, HDD and SSD, the single decompression thread ranges from 9% to 15% CPU, and has maxed out the read+write capacity for my storage. But ma

by cmurf 8y ago

On three test machines, HDD and SSD, the single decompression thread ranges from 9% to 15% CPU, and has maxed out the read+write capacity for my storage. But maybe you have super fast source and target storage?

usefulcat 8y ago
In my scenario, I'm decompressing from HDD and piping the decompressed data directly to another process. I may be limited by disk sometimes, but often the data is already in the filesystem cache. To give a concrete example: zstd -d < compressed.zst | pv > /dev/null == ~330 MB/s For comparison, pixz with the same data using 32 cores: pixz -d -p 32 < compressed.xz | pv > /dev/null == 1.15 GB/s Granted, zstd is far, far more efficient per core, but there are plenty of workloads where I can afford to use a lot of cores for decompression. Also pixz still compresses slightly better than zstd -19, but I'd be willing to trade that for more efficient decompression if I could still have the option of really fast decompression using multiple threads. Note also that with this particular data, I'm seeing a compression ratio of only about 4.3:1 using zstd -19. I can imagine that zstd would use less CPU when decompressing if the ratio was higher.
- cmurf 8y ago
  pzstd does support -d (decompression) option with a default of 4 processes, which can be set with -p. It's part of the zstd package, but I guess it's separate because it's experimental? Not sure what the difference is between zstd -T4 for compressing, and pzstd -p4 for compressing. Anyway, for a test file I get ~125% CPU with pzstd -d, so it is able to do more work, and slightly decreases time. Decompress zstd -d real 0m19.125s user 0m10.888s sys 0m2.575s Decompress pzstd -d real 0m14.819s user 0m14.017s sys 0m4.163s
  - vbtechguy 8y ago
    should be the same though zstd defaults to 1 cpu thread AFAIK so probably why you seeing differences ?

by cmurf 8y ago

usefulcat 8y ago

In my scenario, I'm decompressing from HDD and piping the decompressed data directly to another process. I may be limited by disk sometimes, but often the data is already in the filesystem cache. To give a concrete example: zstd -d < compressed.zst | pv > /dev/null == ~330 MB/s For comparison, pixz with the same data using 32 cores: pixz -d -p 32 < compressed.xz | pv > /dev/null == 1.15 GB/s Granted, zstd is far, far more efficient per core, but there are plenty of workloads where I can afford to use a lot of cores for decompression. Also pixz still compresses slightly better than zstd -19, but I'd be willing to trade that for more efficient decompression if I could still have the option of really fast decompression using multiple threads. Note also that with this particular data, I'm seeing a compression ratio of only about 4.3:1 using zstd -19. I can imagine that zstd would use less CPU when decompressing if the ratio was higher.

cmurf 8y ago
pzstd does support -d (decompression) option with a default of 4 processes, which can be set with -p. It's part of the zstd package, but I guess it's separate because it's experimental? Not sure what the difference is between zstd -T4 for compressing, and pzstd -p4 for compressing. Anyway, for a test file I get ~125% CPU with pzstd -d, so it is able to do more work, and slightly decreases time. Decompress zstd -d real 0m19.125s user 0m10.888s sys 0m2.575s Decompress pzstd -d real 0m14.819s user 0m14.017s sys 0m4.163s
- vbtechguy 8y ago
  should be the same though zstd defaults to 1 cpu thread AFAIK so probably why you seeing differences ?