4 ms·

Sounds like you're comparing multi-threaded zstd to other utilities that are single-threaded (e.g. gzip instead of pigz). I duped /usr/share/dict/words into an

Sounds like you're comparing multi-threaded zstd to other utilities that are single-threaded (e.g. gzip instead of pigz). I duped /usr/share/dict/words into an 8GB file and did a couple tests on the old system I'm on: 8589934592 original 3075315128 pigz -1 1m0.44s 2926825272 zstdmt -3 2m37.52s 2877549999 pigz -3 1m28.94s

lifthrasiir 5y ago
Zstd by default is single-threaded. Just in case though... 8,826,654,133 original 3,033,695,892 0:22 zstd 1.3.3 -3 -T4 3,025,746,073 1:21 zstd 1.3.3 -3 -T1 (default) 3,017,972,162 1:05 zstd 1.5.0 -3 -T1 (default) 3,013,860,663 1:23 zstd 1.5.0 -3 --single-thread It is just no match. EDIT: axiolite wanted to see --single-thread and while my Linux box only has an older zstd (1.3.3) without that option I realized I do have a Windows executable for recent zstd (1.5.0). Both executables have run in the same machine but I can't guarantee anything about the 1.5.0 binary.
- axiolite 5y ago
  zstd is NOT single-threaded by default. It's dual threaded. You have to pass the --single-thread option to make it single threaded. I think you need to try pigz...
  - lifthrasiir 5y ago
    --single-thread probably doesn't do what you think, it forces zstd to serialize I/O with the compression job and it doesn't mean compression itself is multi-threaded with -T1 [EDIT]. I can't try that anyway because my zstd version is slightly lower (1.3.3), but I can try pigz: 3,664,854,169 0:42 pigz -1 -p4 3,664,854,169 0:28 pigz -1 -p8 (default, also probably the max possible with my box) Now I'm very curious where you got your copy of zstdmt. (I'm using stock Ubuntu packages for all of them.) [EDIT] Was "it doesn't make compression itself multi-threaded", which can falsely imply that --single-thread seemingly enables multi-threaded compression but it doesn't ;)
    - axiolite 5y ago
      Using stock RHEL7/EPEL packages: zstd-1.5.0-1.el7.x86_64 On an old Athlon II X4 615e right now. The difference in our results certainly is curious.
      - lifthrasiir 5y ago
        Very interesting. It might be the case that zstd optimizes more for recent machines; zstd famously uses four different compression streams to maximize instruction-level parallelism and that might not work well in older machines. I haven't seen any machine where zstd is significantly slower than it should, but those machines I could test came from 2013 or later. Or either the RHEL package might have been optimized for recent machines. It would be interesting to test a binary optimized for the current machine (-march=native -mtune=native).
  - hexxagone 5y ago
    pigz is no match for zstd...

lifthrasiir 5y ago

Zstd by default is single-threaded. Just in case though... 8,826,654,133 original 3,033,695,892 0:22 zstd 1.3.3 -3 -T4 3,025,746,073 1:21 zstd 1.3.3 -3 -T1 (default) 3,017,972,162 1:05 zstd 1.5.0 -3 -T1 (default) 3,013,860,663 1:23 zstd 1.5.0 -3 --single-thread It is just no match. EDIT: axiolite wanted to see --single-thread and while my Linux box only has an older zstd (1.3.3) without that option I realized I do have a Windows executable for recent zstd (1.5.0). Both executables have run in the same machine but I can't guarantee anything about the 1.5.0 binary.

axiolite 5y ago
zstd is NOT single-threaded by default. It's dual threaded. You have to pass the --single-thread option to make it single threaded. I think you need to try pigz...
- lifthrasiir 5y ago
  --single-thread probably doesn't do what you think, it forces zstd to serialize I/O with the compression job and it doesn't mean compression itself is multi-threaded with -T1 [EDIT]. I can't try that anyway because my zstd version is slightly lower (1.3.3), but I can try pigz: 3,664,854,169 0:42 pigz -1 -p4 3,664,854,169 0:28 pigz -1 -p8 (default, also probably the max possible with my box) Now I'm very curious where you got your copy of zstdmt. (I'm using stock Ubuntu packages for all of them.) [EDIT] Was "it doesn't make compression itself multi-threaded", which can falsely imply that --single-thread seemingly enables multi-threaded compression but it doesn't ;)
  - axiolite 5y ago
    Using stock RHEL7/EPEL packages: zstd-1.5.0-1.el7.x86_64 On an old Athlon II X4 615e right now. The difference in our results certainly is curious.
    - lifthrasiir 5y ago
      Very interesting. It might be the case that zstd optimizes more for recent machines; zstd famously uses four different compression streams to maximize instruction-level parallelism and that might not work well in older machines. I haven't seen any machine where zstd is significantly slower than it should, but those machines I could test came from 2013 or later. Or either the RHEL package might have been optimized for recent machines. It would be interesting to test a binary optimized for the current machine (-march=native -mtune=native).
- hexxagone 5y ago
  pigz is no match for zstd...