3 ms·

We have many terabytes of large files that are currently compressed using xz (well, pixz actually). In terms of compression speed and ratio, zstd is pretty comp

We have many terabytes of large files that are currently compressed using xz (well, pixz actually). In terms of compression speed and ratio, zstd is pretty comparable, and for single-threaded decompression it's faster. At this point the only thing stopping us from using it instead of xz/pixz is the fact that multi-threaded pixz decompression is faster. Are there any plans to add MT decompression to zstd?

cmurf 8y ago
On three test machines, HDD and SSD, the single decompression thread ranges from 9% to 15% CPU, and has maxed out the read+write capacity for my storage. But maybe you have super fast source and target storage?
- usefulcat 8y ago
  In my scenario, I'm decompressing from HDD and piping the decompressed data directly to another process. I may be limited by disk sometimes, but often the data is already in the filesystem cache. To give a concrete example: zstd -d < compressed.zst | pv > /dev/null == ~330 MB/s For comparison, pixz with the same data using 32 cores: pixz -d -p 32 < compressed.xz | pv > /dev/null == 1.15 GB/s Granted, zstd is far, far more efficient per core, but there are plenty of workloads where I can afford to use a lot of cores for decompression. Also pixz still compresses slightly better than zstd -19, but I'd be willing to trade that for more efficient decompression if I could still have the option of really fast decompression using multiple threads. Note also that with this particular data, I'm seeing a compression ratio of only about 4.3:1 using zstd -19. I can imagine that zstd would use less CPU when decompressing if the ratio was higher.
  - cmurf 8y ago
    pzstd does support -d (decompression) option with a default of 4 processes, which can be set with -p. It's part of the zstd package, but I guess it's separate because it's experimental? Not sure what the difference is between zstd -T4 for compressing, and pzstd -p4 for compressing. Anyway, for a test file I get ~125% CPU with pzstd -d, so it is able to do more work, and slightly decreases time. Decompress zstd -d real 0m19.125s user 0m10.888s sys 0m2.575s Decompress pzstd -d real 0m14.819s user 0m14.017s sys 0m4.163s
    - vbtechguy 8y ago
      should be the same though zstd defaults to 1 cpu thread AFAIK so probably why you seeing differences ?
terrelln 8y ago
There aren't any technical limitations to adding multithreaded decompression zstd. We just need a compelling enough use case to justify the work it would take to add it. pzstd is now obsoleted by zstd -T0, but it offers multithreaded decompression for files compressed by pzstd (it will still be single threaded for files compressed by zstd).
- usefulcat 8y ago
  Thanks, I didn't know about pzstd, will check it out. See my reply below for more details of my particular use case.

cmurf 8y ago

On three test machines, HDD and SSD, the single decompression thread ranges from 9% to 15% CPU, and has maxed out the read+write capacity for my storage. But maybe you have super fast source and target storage?

usefulcat 8y ago
In my scenario, I'm decompressing from HDD and piping the decompressed data directly to another process. I may be limited by disk sometimes, but often the data is already in the filesystem cache. To give a concrete example: zstd -d < compressed.zst | pv > /dev/null == ~330 MB/s For comparison, pixz with the same data using 32 cores: pixz -d -p 32 < compressed.xz | pv > /dev/null == 1.15 GB/s Granted, zstd is far, far more efficient per core, but there are plenty of workloads where I can afford to use a lot of cores for decompression. Also pixz still compresses slightly better than zstd -19, but I'd be willing to trade that for more efficient decompression if I could still have the option of really fast decompression using multiple threads. Note also that with this particular data, I'm seeing a compression ratio of only about 4.3:1 using zstd -19. I can imagine that zstd would use less CPU when decompressing if the ratio was higher.
- cmurf 8y ago
  pzstd does support -d (decompression) option with a default of 4 processes, which can be set with -p. It's part of the zstd package, but I guess it's separate because it's experimental? Not sure what the difference is between zstd -T4 for compressing, and pzstd -p4 for compressing. Anyway, for a test file I get ~125% CPU with pzstd -d, so it is able to do more work, and slightly decreases time. Decompress zstd -d real 0m19.125s user 0m10.888s sys 0m2.575s Decompress pzstd -d real 0m14.819s user 0m14.017s sys 0m4.163s
  - vbtechguy 8y ago
    should be the same though zstd defaults to 1 cpu thread AFAIK so probably why you seeing differences ?

terrelln 8y ago

There aren't any technical limitations to adding multithreaded decompression zstd. We just need a compelling enough use case to justify the work it would take to add it. pzstd is now obsoleted by zstd -T0, but it offers multithreaded decompression for files compressed by pzstd (it will still be single threaded for files compressed by zstd).

usefulcat 8y ago
Thanks, I didn't know about pzstd, will check it out. See my reply below for more details of my particular use case.