4 ms·

zstd is NOT single-threaded by default. It's dual threaded. You have to pass the --single-thread option to make it single threaded. I think you need to try p

• lifthrasiir 5y ago
  --single-thread probably doesn't do what you think, it forces zstd to serialize I/O with the compression job and it doesn't mean compression itself is multi-threaded with -T1 [EDIT]. I can't try that anyway because my zstd version is slightly lower (1.3.3), but I can try pigz: 3,664,854,169 0:42 pigz -1 -p4 3,664,854,169 0:28 pigz -1 -p8 (default, also probably the max possible with my box) Now I'm very curious where you got your copy of zstdmt. (I'm using stock Ubuntu packages for all of them.) [EDIT] Was "it doesn't make compression itself multi-threaded", which can falsely imply that --single-thread seemingly enables multi-threaded compression but it doesn't ;)
  • axiolite 5y ago
    Using stock RHEL7/EPEL packages: zstd-1.5.0-1.el7.x86_64 On an old Athlon II X4 615e right now. The difference in our results certainly is curious.
    • lifthrasiir 5y ago
      Very interesting. It might be the case that zstd optimizes more for recent machines; zstd famously uses four different compression streams to maximize instruction-level parallelism and that might not work well in older machines. I haven't seen any machine where zstd is significantly slower than it should, but those machines I could test came from 2013 or later. Or either the RHEL package might have been optimized for recent machines. It would be interesting to test a binary optimized for the current machine (-march=native -mtune=native).
• hexxagone 5y ago
  pigz is no match for zstd...