3 ms·

Interesting, in general I've noticed SHA-256 hashing is relatively CPU intensive and slow.. is the 2024 Xeon CPU in my server too old to include the hardware im

Interesting, in general I've noticed SHA-256 hashing is relatively CPU intensive and slow.. is the 2024 Xeon CPU in my server too old to include the hardware implementation?

adrian_b 4y ago
I think that you might have mistyped the Xeon model number. Nevertheless, most existing Xeon CPUs are too old to have SHA hardware. In Intel server CPUs SHA was introduced many years later than in AMD server CPUs or Intel desktop CPUs, i.e. only in "the 3rd generation Xeon Scalable" based on the Ice Lake Server cores, in Q2 of 2021 (Xeon model numbers 83xx, 63xx and 53xx).
tedunangst 4y ago
Is your software using SHA instructions?
- metadat 4y ago
  I imagine the sha256sum program would use the best available technique. Reasonable assumption?
  - adrian_b 4y ago
    Last time when I have checked what sha256sum (from coreutils) does was one year ago. At that time, on x86 CPUs (I have not tested on ARM) neither sha256sum nor sha224sum nor sha1sum used the hardware instructions (and I have compiled them from sources, with the appropriate flags). Because I have a Zen CPU (but that would also be true on Atom CPUs or Core CPUs since Ice Lake), I have to use "openssl dgst -r -sha256" instead of sha256sum and "openssl dgst -r -sha1" instead of sha1sum. The openssl implementation uses the hardware instructions where available and in that case it is several times faster. (The "-r" option flag in "openssl dgst" selects an output format that is more similar to the sha???sum tools, i.e. hash value followed by file name, but openssl still outputs an extra character, which must be accounted in scripts.)
    - metadat 4y ago
      I also see similarly slow performance when using the Go built-in Sha256 hashing library. Thanks for the info! Fortunately for my purposes of integrity checking, slow is okay.
      - adrian_b 4y ago
        I also use them for file integrity checking, but I frequently check large files, e.g. 50 GB files, and in that case the difference in time until finish between "sha256sum" and "openssl dgst -r -sha256" is extremely noticeable on a Zen CPU. On the other hand, on older Skylake/Kaby Lake/Coffee Lake etc. CPUs, both programs have the same speed.
metadat 4y ago
*2014, yes it was a typo

Interesting, in general I've noticed SHA-256 hashing is relatively CPU intensive and slow.. is the 2024 Xeon CPU in my server too old to include the hardware implementation?

adrian_b 4y ago

I think that you might have mistyped the Xeon model number. Nevertheless, most existing Xeon CPUs are too old to have SHA hardware. In Intel server CPUs SHA was introduced many years later than in AMD server CPUs or Intel desktop CPUs, i.e. only in "the 3rd generation Xeon Scalable" based on the Ice Lake Server cores, in Q2 of 2021 (Xeon model numbers 83xx, 63xx and 53xx).

tedunangst 4y ago

Is your software using SHA instructions?

metadat 4y ago
I imagine the sha256sum program would use the best available technique. Reasonable assumption?
- adrian_b 4y ago
  Last time when I have checked what sha256sum (from coreutils) does was one year ago. At that time, on x86 CPUs (I have not tested on ARM) neither sha256sum nor sha224sum nor sha1sum used the hardware instructions (and I have compiled them from sources, with the appropriate flags). Because I have a Zen CPU (but that would also be true on Atom CPUs or Core CPUs since Ice Lake), I have to use "openssl dgst -r -sha256" instead of sha256sum and "openssl dgst -r -sha1" instead of sha1sum. The openssl implementation uses the hardware instructions where available and in that case it is several times faster. (The "-r" option flag in "openssl dgst" selects an output format that is more similar to the sha???sum tools, i.e. hash value followed by file name, but openssl still outputs an extra character, which must be accounted in scripts.)
  - metadat 4y ago
    I also see similarly slow performance when using the Go built-in Sha256 hashing library. Thanks for the info! Fortunately for my purposes of integrity checking, slow is okay.
    - adrian_b 4y ago
      I also use them for file integrity checking, but I frequently check large files, e.g. 50 GB files, and in that case the difference in time until finish between "sha256sum" and "openssl dgst -r -sha256" is extremely noticeable on a Zen CPU. On the other hand, on older Skylake/Kaby Lake/Coffee Lake etc. CPUs, both programs have the same speed.

metadat 4y ago

*2014, yes it was a typo