3 ms·

Fast hash functions are really important, and SHA256 is really slow. Switching the hash function where you can is enough to result in user-visible speedups for

Fast hash functions are really important, and SHA256 is really slow. Switching the hash function where you can is enough to result in user-visible speedups for common hashing use cases; verifying build artifacts, seeing if on-disk files changed, etc. I was writing something to produce OCI container images a few months ago, and the 3x SHA256 required by the spec for layers actually takes on the order of seconds. (.5s to sha256 a 50MB file, on my 2019-era Threadripper!) I was shocked to discover this. (gzip is also very slow, like shockingly slow, but fortunately the OCI spec lets you use Zstd, which is significantly faster.)

coppsilgold 3y ago
sha256 is not slow on modern hardware. openssl doesn't have blake3, but here is blake2: type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes BLAKE2s256 75697.37k 308777.40k 479373.40k 567875.81k 592687.09k 591254.18k BLAKE2b512 63478.11k 243125.73k 671822.08k 922093.51k 1047833.51k 1048959.57k sha256 129376.82k 416316.32k 1041909.33k 1664480.49k 2018678.67k 2043838.46k This is with the x86 sha256 instructions: sha256msg1, sha256msg2, sha256rnds2
- dralley 3y ago
  "modern hardware" deserves some caveats. AMD has supported those extensions since the original Zen, but Intel CPUs generally lacked them until only about 2 years ago.
  - adrian_b 3y ago
    For many years, starting in 2016, Intel has supported SHA-256 only in their Atom CPUs. The reason seems to be that the Atom CPUs were compared in Geekbench with ARM CPUs, and without hardware SHA the Intel CPUs would have obtained worst benchmark scores. In their big cores, SHA has been added in 2019, in Ice Lake (while Comet Lake still lacked it, being a Skylake derivative), and since then all newer Intel CPUs have it. So except for the Intel Core CPUs, the x86 and ARM CPUs have had hardware SHA for at least 7 years, while the Intel Core CPUs have had it for the last 4 years.
    - dralley 3y ago
      >SHA has been added in 2019, in Ice Lake (while Comet Lake still lacked it, being a Skylake derivative) Ice Lake was effectively a paper launch with low volume, repeated delays and mediocre performance. The server CPUs weren't released until 2021. In terms of relevant quantities and relevant markets (e.g. not Atom or gaming laptops), Intel CPUs have only been "shipping" with those extensions for around 2.5 years, not 4.
richardwhiuk 3y ago
If you want a fast hash function (and don't care about it's cryptographic properties), don't use a cryptographic hash function.
- dralley 3y ago
  BLAKE3 is actually competitive with non-cryptographic hashes like crc32.
  - oconnor663 3y ago
    To be fair, it really depends on the platform. There's an argument to be made that platforms where you care about the difference are specifically the ones where BLAKE3 is slower (no SIMD, no threads).
    - dralley 3y ago
      Fair
  - RaisingSpear 3y ago
    Not even close. CRC32 can easily run at >50GB/s single thread on this i7-12700K CPU (VPCLMULQDQ implementation). The BLAKE3 page claims around 7GB/s single thread. Fudging the figures a bit to cater to CPU differences, BLAKE3 is still a far cry from CRC32.
    - dralley 3y ago
      You're referring to the implementation that was on HN a month or two ago? There's nothing wrong with that, it's a fair comparison, but I'm talking about the existing common implementations.
adrian_b 3y ago
SHA256 is very fast on most modern CPUs, i.e. all AMD Zen, all Intel Atom since 2016, Intel Core Ice Lake or newer, Armv8 and Armv9. I use every day both SHA-256 and BLAKE3. BLAKE3 is faster only because it is computed by multiple threads using all available CPU cores. When restricted to a single thread, it is slower on CPUs with fast hardware SHA-256. The extra speed of BLAKE3 is not always desirable. The fact that it uses all cores can slow down other concurrent activities, without decreasing the overall execution time of the application. There are cases when the computation of a hash like SHA-256 can be done as a background concurrent activity, or when the speed of hashing is limited by the streaming speed of data from the main memory or from a SSD, so spawning multiple threads does not gain anything and it only gets in the way of other activities. So the right choice between SHA-256 and BLAKE3 depends on the application. Both can be useful. SHA-256 has the additional advantage that it needs negligible additional code, only a few lines are necessary to write a loop that computes the hash using the hardware instructions.
- dralley 3y ago
  >I use every day both SHA-256 and BLAKE3. BLAKE3 is faster only because it is computed by multiple threads using all available CPU cores. When restricted to a single thread, it is slower on CPUs with fast hardware SHA-256. That's not actually my experience. Last I tested, BLAKE3 was about twice as fast, single-threaded, as SHA256 on a Zen 3 CPU, which has the extensions. Lower down in the thread someone else did a comparison, and came out with a similar result.

coppsilgold 3y ago

sha256 is not slow on modern hardware. openssl doesn't have blake3, but here is blake2: type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes BLAKE2s256 75697.37k 308777.40k 479373.40k 567875.81k 592687.09k 591254.18k BLAKE2b512 63478.11k 243125.73k 671822.08k 922093.51k 1047833.51k 1048959.57k sha256 129376.82k 416316.32k 1041909.33k 1664480.49k 2018678.67k 2043838.46k This is with the x86 sha256 instructions: sha256msg1, sha256msg2, sha256rnds2

dralley 3y ago
"modern hardware" deserves some caveats. AMD has supported those extensions since the original Zen, but Intel CPUs generally lacked them until only about 2 years ago.
- adrian_b 3y ago
  For many years, starting in 2016, Intel has supported SHA-256 only in their Atom CPUs. The reason seems to be that the Atom CPUs were compared in Geekbench with ARM CPUs, and without hardware SHA the Intel CPUs would have obtained worst benchmark scores. In their big cores, SHA has been added in 2019, in Ice Lake (while Comet Lake still lacked it, being a Skylake derivative), and since then all newer Intel CPUs have it. So except for the Intel Core CPUs, the x86 and ARM CPUs have had hardware SHA for at least 7 years, while the Intel Core CPUs have had it for the last 4 years.
  - dralley 3y ago
    >SHA has been added in 2019, in Ice Lake (while Comet Lake still lacked it, being a Skylake derivative) Ice Lake was effectively a paper launch with low volume, repeated delays and mediocre performance. The server CPUs weren't released until 2021. In terms of relevant quantities and relevant markets (e.g. not Atom or gaming laptops), Intel CPUs have only been "shipping" with those extensions for around 2.5 years, not 4.

richardwhiuk 3y ago

If you want a fast hash function (and don't care about it's cryptographic properties), don't use a cryptographic hash function.

dralley 3y ago
BLAKE3 is actually competitive with non-cryptographic hashes like crc32.
- oconnor663 3y ago
  To be fair, it really depends on the platform. There's an argument to be made that platforms where you care about the difference are specifically the ones where BLAKE3 is slower (no SIMD, no threads).
  - dralley 3y ago
    Fair
- RaisingSpear 3y ago
  Not even close. CRC32 can easily run at >50GB/s single thread on this i7-12700K CPU (VPCLMULQDQ implementation). The BLAKE3 page claims around 7GB/s single thread. Fudging the figures a bit to cater to CPU differences, BLAKE3 is still a far cry from CRC32.
  - dralley 3y ago
    You're referring to the implementation that was on HN a month or two ago? There's nothing wrong with that, it's a fair comparison, but I'm talking about the existing common implementations.

adrian_b 3y ago

SHA256 is very fast on most modern CPUs, i.e. all AMD Zen, all Intel Atom since 2016, Intel Core Ice Lake or newer, Armv8 and Armv9. I use every day both SHA-256 and BLAKE3. BLAKE3 is faster only because it is computed by multiple threads using all available CPU cores. When restricted to a single thread, it is slower on CPUs with fast hardware SHA-256. The extra speed of BLAKE3 is not always desirable. The fact that it uses all cores can slow down other concurrent activities, without decreasing the overall execution time of the application. There are cases when the computation of a hash like SHA-256 can be done as a background concurrent activity, or when the speed of hashing is limited by the streaming speed of data from the main memory or from a SSD, so spawning multiple threads does not gain anything and it only gets in the way of other activities. So the right choice between SHA-256 and BLAKE3 depends on the application. Both can be useful. SHA-256 has the additional advantage that it needs negligible additional code, only a few lines are necessary to write a loop that computes the hash using the hardware instructions.

dralley 3y ago
>I use every day both SHA-256 and BLAKE3. BLAKE3 is faster only because it is computed by multiple threads using all available CPU cores. When restricted to a single thread, it is slower on CPUs with fast hardware SHA-256. That's not actually my experience. Last I tested, BLAKE3 was about twice as fast, single-threaded, as SHA256 on a Zen 3 CPU, which has the extensions. Lower down in the thread someone else did a comparison, and came out with a similar result.