5 ms·

These are some of my favorite tricks. SHA2-256 is never “fast” but it can be really slow if you’re using a non-SIMD implementation (openssl uses SIMD). A trick

These are some of my favorite tricks. SHA2-256 is never “fast” but it can be really slow if you’re using a non-SIMD implementation (openssl uses SIMD). A trick not mentioned here: for Python devs, import “orjson” instead of the json standard library; it is usually a drop-in replacement.

scaramanga 4y ago
orjson is great, it's not quite a drop-in replacement, but in a good way... With orjson, encoding produces a bytes object instead of a string object, and when you're writing to a file that avoids a bunch of extra memory management overhead and a str.encode() when the text-file wrapper converts that string to a bytes behind the scenes. So that interface change is quite a big performance win over and above just having the faster JSON encoder.
adrian_b 4y ago
On all sufficiently new CPUs (all AMD Zen, most Intel Atom, Intel Core since Ice Lake, almost all 64-bit ARM), both SHA-1 and SHA2-256 are extremely fast, because there are special hardware instructions implementing them. They may be slow only in programs which do not check for hardware support and which do not use the dedicated hardware instructions (which is the case in many programs, because Intel Skylake derivatives have been the most popular CPUs during many years, and they were the only modern CPUs without hardware support for SHA-1/SHA2-256, so most developers did not bother to optimize their programs for the other CPUs). OpenSSL is one of the few programs which use the hardware instructions when available. This makes, e.g., "openssl dgst -sha256" much faster than "sha256sum" on recent CPUs. When using the hardware instructions, SHA-1 and SHA2-256 are faster than many non-cryptographic hashes and only 1 cryptographic hash is faster: BLAKE3. However, it must be kept in mind that BLAKE3 is much faster than any other cryptographic hash only because it distributes the computation on all CPU cores. So the much higher speed is accompanied by a much higher CPU utilization. If you have something else that should be done in parallel with the hash computation, using BLAKE3 does not necessarily reduce the total execution time for your entire application, even if the hash is computed much faster, because other concurrent activities may be stalled until the hash computation is completed.
- oconnor663 4y ago
  > However, it must be kept in mind that BLAKE3 is much faster than any other cryptographic hash only because it distributes the computation on all CPU cores. Surprisingly, this is incorrect. Here are some single-threaded measurements on a CPU with SHA-NI: https://bench.cr.yp.to/results-hash.html#amd64-icelake https://bench.cr.yp.to/results-hash.html#amd64-icelake. What you see there is that BLAKE3 can take better advantage of SIMD parallelism than other hashes, and the C and Rust library implementations do this by default. Multithreading isn't enabled by default in the library APIs, but if you do use it (and you have enough input to feed it) the benefits are multiplicative. The b3sum CLI does use multithreading by default. > only 1 cryptographic hash is faster: BLAKE3 SIMD implementations of KangarooTwelve are also about as fast as BLAKE3, given enough input.
  - adrian_b 4y ago
    Those benchmark results might be caused by a very poor hardware implementation on Intel Ice Lake (the first Intel Core with SHA, neglecting Cannon Lake). In my tests on AMD Zen 3, BLAKE3 is much faster only when multi-threaded and that should be true on any CPU where the SHA instructions are well implemented. You are right that it is a little exaggerated to say that BLAKE3 is the only hash with these properties. Using a similar construction with BLAKE3 to allow parallel computation, instead of using the traditional Merkle–Damgård iterated construction, like SHA-1, SHA-2 and many others, it is possible to design many other fast hash algorithms and there already are many such experimental hashes. What I have meant was that BLAKE3 is, for now, the only one that has both a freely available and easy to install high-quality implementation, and it is based on a theory that has been studied long enough to have confidence in it. There are many other hashes that are candidates for being useful fast cryptographic hashes, but it is likely that a few more years are needed to trust them enough.
- jonstewart 4y ago
  Yes, SHA-NI helps a lot. There are a lot of Skylake and even Haswell systems still out there, though, depending on deployment intent and, as you note, several old and slow implementations using the reference code. Skylake, Haswell, and even Ivy/Sandy Bridge benefit a lot from SSE4/AVX implementations, with such SHA-1 implementations still faster than MD5 (if not SHA-256). The author talks about Java a lot and I honestly don’t know whether OpenJDK et al use for crypto primitive implementations (hopefully OpenSSL!). BLAKE3 is not multithreaded by default.
Aachen 4y ago
s/import/install/ It's not in the standard library, can't just import it and expect it to always be there

scaramanga 4y ago

orjson is great, it's not quite a drop-in replacement, but in a good way... With orjson, encoding produces a bytes object instead of a string object, and when you're writing to a file that avoids a bunch of extra memory management overhead and a str.encode() when the text-file wrapper converts that string to a bytes behind the scenes. So that interface change is quite a big performance win over and above just having the faster JSON encoder.

adrian_b 4y ago

On all sufficiently new CPUs (all AMD Zen, most Intel Atom, Intel Core since Ice Lake, almost all 64-bit ARM), both SHA-1 and SHA2-256 are extremely fast, because there are special hardware instructions implementing them. They may be slow only in programs which do not check for hardware support and which do not use the dedicated hardware instructions (which is the case in many programs, because Intel Skylake derivatives have been the most popular CPUs during many years, and they were the only modern CPUs without hardware support for SHA-1/SHA2-256, so most developers did not bother to optimize their programs for the other CPUs). OpenSSL is one of the few programs which use the hardware instructions when available. This makes, e.g., "openssl dgst -sha256" much faster than "sha256sum" on recent CPUs. When using the hardware instructions, SHA-1 and SHA2-256 are faster than many non-cryptographic hashes and only 1 cryptographic hash is faster: BLAKE3. However, it must be kept in mind that BLAKE3 is much faster than any other cryptographic hash only because it distributes the computation on all CPU cores. So the much higher speed is accompanied by a much higher CPU utilization. If you have something else that should be done in parallel with the hash computation, using BLAKE3 does not necessarily reduce the total execution time for your entire application, even if the hash is computed much faster, because other concurrent activities may be stalled until the hash computation is completed.

oconnor663 4y ago
> However, it must be kept in mind that BLAKE3 is much faster than any other cryptographic hash only because it distributes the computation on all CPU cores. Surprisingly, this is incorrect. Here are some single-threaded measurements on a CPU with SHA-NI: https://bench.cr.yp.to/results-hash.html#amd64-icelake https://bench.cr.yp.to/results-hash.html#amd64-icelake. What you see there is that BLAKE3 can take better advantage of SIMD parallelism than other hashes, and the C and Rust library implementations do this by default. Multithreading isn't enabled by default in the library APIs, but if you do use it (and you have enough input to feed it) the benefits are multiplicative. The b3sum CLI does use multithreading by default. > only 1 cryptographic hash is faster: BLAKE3 SIMD implementations of KangarooTwelve are also about as fast as BLAKE3, given enough input.
- adrian_b 4y ago
  Those benchmark results might be caused by a very poor hardware implementation on Intel Ice Lake (the first Intel Core with SHA, neglecting Cannon Lake). In my tests on AMD Zen 3, BLAKE3 is much faster only when multi-threaded and that should be true on any CPU where the SHA instructions are well implemented. You are right that it is a little exaggerated to say that BLAKE3 is the only hash with these properties. Using a similar construction with BLAKE3 to allow parallel computation, instead of using the traditional Merkle–Damgård iterated construction, like SHA-1, SHA-2 and many others, it is possible to design many other fast hash algorithms and there already are many such experimental hashes. What I have meant was that BLAKE3 is, for now, the only one that has both a freely available and easy to install high-quality implementation, and it is based on a theory that has been studied long enough to have confidence in it. There are many other hashes that are candidates for being useful fast cryptographic hashes, but it is likely that a few more years are needed to trust them enough.
jonstewart 4y ago
Yes, SHA-NI helps a lot. There are a lot of Skylake and even Haswell systems still out there, though, depending on deployment intent and, as you note, several old and slow implementations using the reference code. Skylake, Haswell, and even Ivy/Sandy Bridge benefit a lot from SSE4/AVX implementations, with such SHA-1 implementations still faster than MD5 (if not SHA-256). The author talks about Java a lot and I honestly don’t know whether OpenJDK et al use for crypto primitive implementations (hopefully OpenSSL!). BLAKE3 is not multithreaded by default.

Aachen 4y ago

s/import/install/ It's not in the standard library, can't just import it and expect it to always be there