14 ms·

BLAKE2: “Harder, Better, Faster, Stronger” Than MD5

122 points by lebek 13y ago

kastnerkyle 13y ago
Can someone who knows crypto shed some light on the "gravity" of this algorithm? Is this going to out and out replace MD5, or will it fall among the many "awesome, but not widely used" hashes out there? It sounds pretty awesome... almost too awesome!
- borando 13y ago
  With serious protocol designers now largely side-stepping the IETF due to BULLRUN infiltration, BLAKE2 and other non-standard primitives have a better shot than before of seeing mass adoption. I hope to see BLAKE2 widely used in the future. I also hope to see more non-standard crypto and protocols, where "the market" leads the way, and standards groups try to keep up in order to appear legitimate.
  - tptacek 13y ago
    The IETF doesn't standardize cryptography. If you think Joan Daemen is an NSA plant, you've got bigger problems than hash functions.
    - borando 13y ago
      Maybe we're using different definitions of standardization. The IETF writes RFCs which developers are expected to follow, and (mostly) do. This is a standardization of sorts, but it's beside the point I was making. I'm not talking about Joan Daemen wrt BULLRUN. I'm referring to secure protocols that offer RC4 but not Salsa20, TLS without a single constant time cipher, 112-bit security, secure protocols that aren't even encrypted, null ciphers, Dragonfly, cipher suites so complex that they're expected to be implemented wrongly, secure protocols made so complex they won't be used at all, crypto advisory groups run by NSA employees, etc.
      - tptacek 13y ago
        TLS offers RC4 and not Salsa20 because RC4 existed when SSL was first defined by Netscape, and Salsa20 didn't, and wouldn't for over a decade. It's worth mentioning here that the "encrypted by default" Internet that was the dream of the 1990s was a government project, and TLS more or less thwarted it. Different ciphers can be more or less straightforward to implement without timing leaks, but "constant time" is a property of an implementation, not of a cipher. Dragonfly is exceedingly lame, but it's also completely inside-baseball. Even if Dragonfly had been "standardized" by the TLS WG, nobody ever would have used it, because nobody ever used SRP either, and SRP was better. The ciphersuites in TLS aren't complicated. They're very simple. The problem with TLS ciphersuites isn't that they're complicated but that they're wrong. Which is unsurprising, because they were designed before anything like Bellare and Namprempre; in fact, they were designed in an era where many practitioners believed that message authentication was unnecessary for cryptosystems at all. I'm not sure which complex secure protocols you're referring to. TLS and SSH are so widely used it seems fair to call them universal. As for the CFRG chair, well, I won't repeat myself: https://news.ycombinator.com/item?id=6942145 https://news.ycombinator.com/item?id=6942145 In the end, though, the real issue I have with your comment is that the IETF has nothing at all to do with BLAKE2's standards- friendliness. The IETF will soon standardize ChaCha20-Poly1305 for TLS, for instance, despite the fact that no NIST standard will ever do the same.
  - mpyne 13y ago
    > I also hope to see more non-standard crypto and protocols, where "the market" leads the way This is super-dangerous, unless the amorphous "market" is also paying for cryptanalysts to bang on the crypto primitives as a public service to all competitors in the market. After all, RSA adopting Dual EC DRBG was a business decision, and one which the market didn't reverse despite Dual EC DRBG being publically known to have a probable backdoor since 2007.
- tptacek 13y ago
  MD5 has already been out-and-out replaced. For applications that require collision resistance, MD5 is already dangerous. Even in a setting where MD5 is unlikely to cause real problems, like in an HMAC authenticator, its use would still be flagged in an audit. No competent new design in 2014 would use SHA1 either. In the world we live in now, there are two "mainstream" options for hash functions: SHA2 (ie, SHA2/256, SHA2/512, &c) and SHA3. SHA3 is actually not at all mainstream (very few applications use it), but (a) that will change over time and (b) nobody will really bat an eyelid if you choose to use it. What Zooko is advocating for is a third mainstream option: BLAKE2. BLAKE2 is derived from a well-regarded SHA3 finalist; the preponderance of evidence suggests that BLAKE2 is safe, and arguably a more conservative choice than SHA3 (a/k/a Keccak, pronounced "Ketchup"), because SHA3's novel sponge function design hasn't been banged on the way SHA2's design, which BLAKE sort of shares, has. The reality is that for most software designers, this isn't an important issue; most software isn't constrained by the speed of its hashing algorithms (the common bottleneck in crypto code is the bignum work public key algorithms have to do --- if you're worried about bottlenecked on hashing, you probably have a much bigger problem with I/O).
  - dfc 13y ago
    I was wondering how the new "optional parallel mode" affects the long history of professional scrutiny. Is adding an optional parallel mode something that a professional cryptographer can do without significantly changing the security model?[^1] Are there certain families of hash designs amenable slight tweaks that sacrifice a little security for a relatively large speed improvement or is this something that could happen with the majority of hash schemes? As always, thanks for playing "HN asks tptacek about security." [^1]: This is a genuine question. I do not have any reason to doubt zooko et al's judgement. I guess another way of saying this is "For someone smarter than me is it easy to know when this is a good/bad idea?" Strange segway: I was browsing Plos and saw this article about egg washing and shell characteristics. After reading the title for some reason I remembered reading a somewhat lengthy comment you wrote about the safety of eggs and different egg washing regimes in the us and the uk. I did not check too see if my recollection was accurate but in case it is and the comment was indicative of your interest in the subject you might get a kick out of: Effect of Egg Washing and Correlation between Eggshell Characteristics and Egg Penetration by Various Salmonella Typhimurium Strains. PLoS ONE 9(3): e90987. doi:10.1371/journal.pone.0090987 http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0090987 http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjourna...
NathanKP 13y ago
For those using Node.js who are interesting in trying out BLAKE2, I found this module: https://github.com/sekitaka/node-blake2 https://github.com/sekitaka/node-blake2 I hasn't been touched in a year, but it looks fairly good. I may try it out soon to see how it performs myself.
bwooce 13y ago
Go impl: https://github.com/dchest/blake2b https://github.com/dchest/blake2b
- nly 13y ago
  The lack of a preprocessor looks to be really hurting the impl of the rounds there.
kzrdude 13y ago
BLAKE2 seems like a good fit for btrfs checksums -- I'm not sure why they haven't implemented alternatives beyond their default crc32c yet.
- arielweisberg 13y ago
  Cryptographic hashes and checksums serve different purposes. CRC32c has a hardware implementation on Intel that can do between 3-4 gigabytes a second per core (depends on if you pipeline instructions) with software implementations doing 800 megabytes a second per core. Fast cryptographic hashes are more like 100 megabytes a second per core or slower. You can use cryptographic hashes in performance sensitive scenarios like git to get a unique fixed size handle on some bytes, but CRC32c is faster if all you want to detect is random changes. I wouldn't expect a filesystem to need collision resistant cryptographic hashes. There is hardware support for SHA-1 and SHA-2 now, but is recent and I haven't seen an implementation or benchmark. I doubt it's as fast as the hardware CRC32c implementation.
  - pbsd 13y ago
    FWIW, BLAKE2 does over 1 gigabyte per second on recent Intel chips. With the parallel modes it should be able to reach 3-4 gigabytes per second with multiple cores.
  - kzrdude 13y ago
    Your estimation of BLAKE2's data/second is indeed off by an order of magnitude <https://blake2.net/ https://blake2.net/ >, and the demand is certainly there to have better checksum algorithms in filesystems, both for integrity and deduplication. ZFS for example uses SHA-2. BLAKE2's authors do say they designed a fast algorithm envisioned for storage, so that's why I think it fits btrfs.
    - arielweisberg 13y ago
      Replying to all the helpful posters here. Thanks, that's very information. I checked out the link and it is very cool how blake2 can be tuned for different roles. I hadn't though of how newer filesystems are doing deduplication, my head is stuck in the ext4 era. I still think performance matters. Even at 800 megabytes a second you are talking about committing two entire cores to checksums on 10-gig E if you need to move data around. An entire core if you are talking about sequentially scanning an SSD. I suppose this will stop mattering as we get more cores. If a filesystem is using CRC32 for something, it doesn't need the properties of a cryptographic hash or they are doing it wrong. I can see how you can argue against CRC for reliability. I am not sure whether you actually risk a corrupt block every 1 in 2^32 blocks. Most blocks won't be corrupt so the CRC only has to detect a much smaller number of errors. Assuming every block had an error needing detection you will miss an error every 16 terabytes (assuming other things as well). Assuming 1% of blocks are corrupt you would miss an error every 1.6 petabytes? Maybe I am thinking about this wrong, and I recall other factors like block size effecting CRC's reliability.
Goopplesoft 13y ago
Excuse my lack of knowledge on the subject. I know MD5 has it cryptographic security issues [1] but I almost always assumed that the main reason it was a 'weak' hash was due to it's speed and how, consequently, rainbow tables we're readily available. If this premise is correct, wouldn't that inherently make Blake2 less strong/secure since its faster? [1] http://en.wikipedia.org/wiki/MD5#Security http://en.wikipedia.org/wiki/MD5#Security
- ftwinnovations 13y ago
  It sounds like you are referring to password hashing. Neither BLAKE nor MD5 should be used for password hashing. At the bottom of the article make sure to read his section "P.S. this isn't about hashing passwords"
  - zooko 13y ago
    You win a prize for reading to the end before posting!
- JoshTriplett 13y ago
  > I know MD5 has it cryptographic security issues [1] but I almost always assumed that the main reason it was a 'weak' hash was due to it's speed No, for hashes, you definitely want speed; that's not the problem with MD5. For applications where speed is not appropriate, such as password hashing, you want an iterated hash of some kind. (Disclaimer: as with most crypto, you want one of the existing implementations, not something custom.) The cyptographic security issues with MD5 concern breaks to the hash algorithm that make it possible to find hash collisions more easily than the number of bits in the hash would imply. A 128-bit hash should require 2128 hashing operations to find a collision against an existing fixed value. (Finding two independently chosen values with the same hash is far easier, due to the birthday problem: https://en.wikipedia.org/wiki/Birthday_problem https://en.wikipedia.org/wiki/Birthday_problem ) MD5 has algorithmic weaknesses that make it far easier to find collisions, to the point where a current system can find some types of collisions in seconds.
- dredmorbius 13y ago
bbanyc 13y ago
For non-cryptographic applications that don't require top-grade security, but do want a short reasonably-unique identifier for a file to detect duplicates, MD5's speed and ubiquity made it a natural choice. This is the first thing I've seen aimed at restoring some security to that particular niche.
- joeblau 13y ago
  Security and detecting duplicates are two different things. You can detect duplicates using a Bloom filter and use a lot less memory than using MD5s.
  - Dylan16807 13y ago
    A bloom filter is built on top of hash functions, not instead of them.
- mindslight 13y ago
  Until someone gives you two different files with the same md5sum, which are quite easy to create, and violation of your program's assumptions cause it to misbehave in unexpected ways. Most applications of hash functions probably actually do want cryptographic strength, as they will end up being used in ways you did not forsee (and thus exploited).
orasis 13y ago
Than MD5??? Weak marketing.
- dfc 13y ago
  Faster than MD5 = strong marketing
dionyziz 13y ago
These reasons don't matter. What matters is that you use a widely used hash function through a widely used software library. Most security issues are implementation-related, not cryptography-related. And if there are problems with a scheme, more people should look for them. SHA2/3 are better options under this light.
- rjzzleep 13y ago
  sha2 based, and faster than md5? do you really not see the value in this? sure you wouldn't want to use this your own coin, because of say asic, and gpu miners, and your saltless password hash for bruteforcing issues. but databases, filesystems, checksums for packages, git(to be fair, they're all some sort of storage systems) by the definition you said above we would still be using md5 in everything.
  - vbuterin 13y ago
    Actually, even in coins fast hashes have their place (specifically, in Merkle trees; we're using SHA3 in Ethereum precisely because computers will have hardware modules for it), and for mining you don't want to be using anything that's classified as a hash function anyway.
  - dionyziz 13y ago
    md5 is fine for filesystems and databases. I'm talking about using hashes for cryptographic purposes.
- Pacabel 13y ago
  How do you propose that new algorithms become "widely used" if nobody should use them until they are "widely used"?
  - dionyziz 13y ago
    Standardization is a way. SHA2 and SHA3 are standardized. Unfortunately, with the recent RSA shortcomings, this may no longer be trustable.
shin_lao 13y ago
Because SHA3 is the standard, we' re going to see hardware implementations of it and a lot of research will go into its software implementation. If you care about performance, stick to the standard. Another commenter also rightly pointed out that SHA3 implementations will be more scrutinized, and therefore more secure. Exploitable security issues lie rarely in the algorithm. Long story short: stick to the standard.
- floody-berry 13y ago
  SHA-1 intrinsics are going to be out in 2015-2016. You're going to be waiting a very long time before SHA-3 intrinsics hit. If you care about software performance, you will not be using SHA-3.
  - shin_lao 13y ago
    Accelerated SHA1 has been available for a long time, this was necessary for embedded systems. The NIST is currently modifying SHA3 to make it faster, with some controversy. You are right in the sense we will have to wait for the definitive standard to get out before we see some heavily optimized implementations. If you want immediate performance, use SHA2. If you want more "future proof" security, use SHA3 (and one could say that SHA2 is still very good in terms of security). Stick to the standard, unless you know a lot about crypto.
- cyphunk 13y ago
  > Long story short: stick to the standard. ahem... http://en.wikipedia.org/wiki/Dual_EC_DRBG http://en.wikipedia.org/wiki/Dual_EC_DRBG when you can no longer trust the organizations setting the standards (NIST, CFRG) this argument looses water.
  - mpyne 13y ago
    Even there the standardization process worked: The flaw in Dual EC DRBG was pointed out very quickly, precisely because there was so much cryptanalytic attention paid to it. The fact that RSA (and only RSA, AFAICT) didn't pay attention to the crypto break in exchange for $10MM doesn't change the fact that the vulnerability had been quickly discovered. If the algorithm had been left unstandardized and was simply "Foo Corp's Custom Wonderful Bit Generator™" the public may never have known of the vulnerability, while the NSA would still have the resources to have discovered the flaw on their own and use TAO to recover the priv key.
ankitml 13y ago
Harder, Better, Faster, Stronger Sounds like MD5 on viagra.
- ZenoArrow 13y ago
  Or Daft Punk MD5... www.youtube.com/watch?v=gAjR4_CbPpQ
haberman 13y ago
Glad I came across this -- I have a need for a hash function with cryptographic properties but that I don't actually mean to use for crypto, but just to uniquely identify a file by its contents. Speed is really important to me. This sounds perfect for my application.
- gojomo 13y ago
  If the secure-hash properties are truly a non-factor for your application, you could also consider Ciyhash128/CityHash256: http://en.wikipedia.org/wiki/CityHash http://en.wikipedia.org/wiki/CityHash
  - haberman 13y ago
    Good point, I had forgotten it could output 128/256 bits.
keeperofdakeys 13y ago
This author almost completely misses the point. The basic architecture behind SHA-2 is similar to MD5 and SHA-1, and the fact that attacks are possible on MD5 and SHA-1 means an attack could be possible on SHA-2. Then if SHA-3 used the same architecture again, a weakness in SHA-2 might lead to a weakness SHA-3. Then both SHA-2 and SHA-3 are insecure. Since SHA-3 is now based on a totally different architecture, any weakness in SHA-2 has no effect for SHA-3. At the same time, if someone does find an attack against SHA-3, SHA-2 is still secure.

122 points by lebek 13y ago

Can someone who knows crypto shed some light on the "gravity" of this algorithm? Is this going to out and out replace MD5, or will it fall among the many "awesome, but not widely used" hashes out there? It sounds pretty awesome... almost too awesome!

borando 13y ago
With serious protocol designers now largely side-stepping the IETF due to BULLRUN infiltration, BLAKE2 and other non-standard primitives have a better shot than before of seeing mass adoption. I hope to see BLAKE2 widely used in the future. I also hope to see more non-standard crypto and protocols, where "the market" leads the way, and standards groups try to keep up in order to appear legitimate.
- tptacek 13y ago
  The IETF doesn't standardize cryptography. If you think Joan Daemen is an NSA plant, you've got bigger problems than hash functions.
  - borando 13y ago
    Maybe we're using different definitions of standardization. The IETF writes RFCs which developers are expected to follow, and (mostly) do. This is a standardization of sorts, but it's beside the point I was making. I'm not talking about Joan Daemen wrt BULLRUN. I'm referring to secure protocols that offer RC4 but not Salsa20, TLS without a single constant time cipher, 112-bit security, secure protocols that aren't even encrypted, null ciphers, Dragonfly, cipher suites so complex that they're expected to be implemented wrongly, secure protocols made so complex they won't be used at all, crypto advisory groups run by NSA employees, etc.
    - tptacek 13y ago
      TLS offers RC4 and not Salsa20 because RC4 existed when SSL was first defined by Netscape, and Salsa20 didn't, and wouldn't for over a decade. It's worth mentioning here that the "encrypted by default" Internet that was the dream of the 1990s was a government project, and TLS more or less thwarted it. Different ciphers can be more or less straightforward to implement without timing leaks, but "constant time" is a property of an implementation, not of a cipher. Dragonfly is exceedingly lame, but it's also completely inside-baseball. Even if Dragonfly had been "standardized" by the TLS WG, nobody ever would have used it, because nobody ever used SRP either, and SRP was better. The ciphersuites in TLS aren't complicated. They're very simple. The problem with TLS ciphersuites isn't that they're complicated but that they're wrong. Which is unsurprising, because they were designed before anything like Bellare and Namprempre; in fact, they were designed in an era where many practitioners believed that message authentication was unnecessary for cryptosystems at all. I'm not sure which complex secure protocols you're referring to. TLS and SSH are so widely used it seems fair to call them universal. As for the CFRG chair, well, I won't repeat myself: https://news.ycombinator.com/item?id=6942145 https://news.ycombinator.com/item?id=6942145 In the end, though, the real issue I have with your comment is that the IETF has nothing at all to do with BLAKE2's standards- friendliness. The IETF will soon standardize ChaCha20-Poly1305 for TLS, for instance, despite the fact that no NIST standard will ever do the same.
- mpyne 13y ago
  > I also hope to see more non-standard crypto and protocols, where "the market" leads the way This is super-dangerous, unless the amorphous "market" is also paying for cryptanalysts to bang on the crypto primitives as a public service to all competitors in the market. After all, RSA adopting Dual EC DRBG was a business decision, and one which the market didn't reverse despite Dual EC DRBG being publically known to have a probable backdoor since 2007.
tptacek 13y ago
MD5 has already been out-and-out replaced. For applications that require collision resistance, MD5 is already dangerous. Even in a setting where MD5 is unlikely to cause real problems, like in an HMAC authenticator, its use would still be flagged in an audit. No competent new design in 2014 would use SHA1 either. In the world we live in now, there are two "mainstream" options for hash functions: SHA2 (ie, SHA2/256, SHA2/512, &c) and SHA3. SHA3 is actually not at all mainstream (very few applications use it), but (a) that will change over time and (b) nobody will really bat an eyelid if you choose to use it. What Zooko is advocating for is a third mainstream option: BLAKE2. BLAKE2 is derived from a well-regarded SHA3 finalist; the preponderance of evidence suggests that BLAKE2 is safe, and arguably a more conservative choice than SHA3 (a/k/a Keccak, pronounced "Ketchup"), because SHA3's novel sponge function design hasn't been banged on the way SHA2's design, which BLAKE sort of shares, has. The reality is that for most software designers, this isn't an important issue; most software isn't constrained by the speed of its hashing algorithms (the common bottleneck in crypto code is the bignum work public key algorithms have to do --- if you're worried about bottlenecked on hashing, you probably have a much bigger problem with I/O).
- dfc 13y ago
  I was wondering how the new "optional parallel mode" affects the long history of professional scrutiny. Is adding an optional parallel mode something that a professional cryptographer can do without significantly changing the security model?[^1] Are there certain families of hash designs amenable slight tweaks that sacrifice a little security for a relatively large speed improvement or is this something that could happen with the majority of hash schemes? As always, thanks for playing "HN asks tptacek about security." [^1]: This is a genuine question. I do not have any reason to doubt zooko et al's judgement. I guess another way of saying this is "For someone smarter than me is it easy to know when this is a good/bad idea?" Strange segway: I was browsing Plos and saw this article about egg washing and shell characteristics. After reading the title for some reason I remembered reading a somewhat lengthy comment you wrote about the safety of eggs and different egg washing regimes in the us and the uk. I did not check too see if my recollection was accurate but in case it is and the comment was indicative of your interest in the subject you might get a kick out of: Effect of Egg Washing and Correlation between Eggshell Characteristics and Egg Penetration by Various Salmonella Typhimurium Strains. PLoS ONE 9(3): e90987. doi:10.1371/journal.pone.0090987 http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0090987 http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjourna...

NathanKP 13y ago

For those using Node.js who are interesting in trying out BLAKE2, I found this module: https://github.com/sekitaka/node-blake2 https://github.com/sekitaka/node-blake2 I hasn't been touched in a year, but it looks fairly good. I may try it out soon to see how it performs myself.

bwooce 13y ago

Go impl: https://github.com/dchest/blake2b https://github.com/dchest/blake2b

nly 13y ago
The lack of a preprocessor looks to be really hurting the impl of the rounds there.

kzrdude 13y ago

BLAKE2 seems like a good fit for btrfs checksums -- I'm not sure why they haven't implemented alternatives beyond their default crc32c yet.

arielweisberg 13y ago
Cryptographic hashes and checksums serve different purposes. CRC32c has a hardware implementation on Intel that can do between 3-4 gigabytes a second per core (depends on if you pipeline instructions) with software implementations doing 800 megabytes a second per core. Fast cryptographic hashes are more like 100 megabytes a second per core or slower. You can use cryptographic hashes in performance sensitive scenarios like git to get a unique fixed size handle on some bytes, but CRC32c is faster if all you want to detect is random changes. I wouldn't expect a filesystem to need collision resistant cryptographic hashes. There is hardware support for SHA-1 and SHA-2 now, but is recent and I haven't seen an implementation or benchmark. I doubt it's as fast as the hardware CRC32c implementation.
- pbsd 13y ago
  FWIW, BLAKE2 does over 1 gigabyte per second on recent Intel chips. With the parallel modes it should be able to reach 3-4 gigabytes per second with multiple cores.
- kzrdude 13y ago
  Your estimation of BLAKE2's data/second is indeed off by an order of magnitude <https://blake2.net/ https://blake2.net/ >, and the demand is certainly there to have better checksum algorithms in filesystems, both for integrity and deduplication. ZFS for example uses SHA-2. BLAKE2's authors do say they designed a fast algorithm envisioned for storage, so that's why I think it fits btrfs.
  - arielweisberg 13y ago
    Replying to all the helpful posters here. Thanks, that's very information. I checked out the link and it is very cool how blake2 can be tuned for different roles. I hadn't though of how newer filesystems are doing deduplication, my head is stuck in the ext4 era. I still think performance matters. Even at 800 megabytes a second you are talking about committing two entire cores to checksums on 10-gig E if you need to move data around. An entire core if you are talking about sequentially scanning an SSD. I suppose this will stop mattering as we get more cores. If a filesystem is using CRC32 for something, it doesn't need the properties of a cryptographic hash or they are doing it wrong. I can see how you can argue against CRC for reliability. I am not sure whether you actually risk a corrupt block every 1 in 2^32 blocks. Most blocks won't be corrupt so the CRC only has to detect a much smaller number of errors. Assuming every block had an error needing detection you will miss an error every 16 terabytes (assuming other things as well). Assuming 1% of blocks are corrupt you would miss an error every 1.6 petabytes? Maybe I am thinking about this wrong, and I recall other factors like block size effecting CRC's reliability.

Goopplesoft 13y ago

Excuse my lack of knowledge on the subject. I know MD5 has it cryptographic security issues [1] but I almost always assumed that the main reason it was a 'weak' hash was due to it's speed and how, consequently, rainbow tables we're readily available. If this premise is correct, wouldn't that inherently make Blake2 less strong/secure since its faster? [1] http://en.wikipedia.org/wiki/MD5#Security http://en.wikipedia.org/wiki/MD5#Security

ftwinnovations 13y ago
It sounds like you are referring to password hashing. Neither BLAKE nor MD5 should be used for password hashing. At the bottom of the article make sure to read his section "P.S. this isn't about hashing passwords"
- zooko 13y ago
  You win a prize for reading to the end before posting!
JoshTriplett 13y ago
> I know MD5 has it cryptographic security issues [1] but I almost always assumed that the main reason it was a 'weak' hash was due to it's speed No, for hashes, you definitely want speed; that's not the problem with MD5. For applications where speed is not appropriate, such as password hashing, you want an iterated hash of some kind. (Disclaimer: as with most crypto, you want one of the existing implementations, not something custom.) The cyptographic security issues with MD5 concern breaks to the hash algorithm that make it possible to find hash collisions more easily than the number of bits in the hash would imply. A 128-bit hash should require 2128 hashing operations to find a collision against an existing fixed value. (Finding two independently chosen values with the same hash is far easier, due to the birthday problem: https://en.wikipedia.org/wiki/Birthday_problem https://en.wikipedia.org/wiki/Birthday_problem ) MD5 has algorithmic weaknesses that make it far easier to find collisions, to the point where a current system can find some types of collisions in seconds.
dredmorbius 13y ago

bbanyc 13y ago

For non-cryptographic applications that don't require top-grade security, but do want a short reasonably-unique identifier for a file to detect duplicates, MD5's speed and ubiquity made it a natural choice. This is the first thing I've seen aimed at restoring some security to that particular niche.

joeblau 13y ago
Security and detecting duplicates are two different things. You can detect duplicates using a Bloom filter and use a lot less memory than using MD5s.
- Dylan16807 13y ago
  A bloom filter is built on top of hash functions, not instead of them.
mindslight 13y ago
Until someone gives you two different files with the same md5sum, which are quite easy to create, and violation of your program's assumptions cause it to misbehave in unexpected ways. Most applications of hash functions probably actually do want cryptographic strength, as they will end up being used in ways you did not forsee (and thus exploited).

orasis 13y ago

Than MD5??? Weak marketing.

dfc 13y ago
Faster than MD5 = strong marketing

dionyziz 13y ago

These reasons don't matter. What matters is that you use a widely used hash function through a widely used software library. Most security issues are implementation-related, not cryptography-related. And if there are problems with a scheme, more people should look for them. SHA2/3 are better options under this light.

rjzzleep 13y ago
sha2 based, and faster than md5? do you really not see the value in this? sure you wouldn't want to use this your own coin, because of say asic, and gpu miners, and your saltless password hash for bruteforcing issues. but databases, filesystems, checksums for packages, git(to be fair, they're all some sort of storage systems) by the definition you said above we would still be using md5 in everything.
- vbuterin 13y ago
  Actually, even in coins fast hashes have their place (specifically, in Merkle trees; we're using SHA3 in Ethereum precisely because computers will have hardware modules for it), and for mining you don't want to be using anything that's classified as a hash function anyway.
- dionyziz 13y ago
  md5 is fine for filesystems and databases. I'm talking about using hashes for cryptographic purposes.
Pacabel 13y ago
How do you propose that new algorithms become "widely used" if nobody should use them until they are "widely used"?
- dionyziz 13y ago
  Standardization is a way. SHA2 and SHA3 are standardized. Unfortunately, with the recent RSA shortcomings, this may no longer be trustable.

shin_lao 13y ago

Because SHA3 is the standard, we' re going to see hardware implementations of it and a lot of research will go into its software implementation. If you care about performance, stick to the standard. Another commenter also rightly pointed out that SHA3 implementations will be more scrutinized, and therefore more secure. Exploitable security issues lie rarely in the algorithm. Long story short: stick to the standard.

floody-berry 13y ago
SHA-1 intrinsics are going to be out in 2015-2016. You're going to be waiting a very long time before SHA-3 intrinsics hit. If you care about software performance, you will not be using SHA-3.
- shin_lao 13y ago
  Accelerated SHA1 has been available for a long time, this was necessary for embedded systems. The NIST is currently modifying SHA3 to make it faster, with some controversy. You are right in the sense we will have to wait for the definitive standard to get out before we see some heavily optimized implementations. If you want immediate performance, use SHA2. If you want more "future proof" security, use SHA3 (and one could say that SHA2 is still very good in terms of security). Stick to the standard, unless you know a lot about crypto.
cyphunk 13y ago
> Long story short: stick to the standard. ahem... http://en.wikipedia.org/wiki/Dual_EC_DRBG http://en.wikipedia.org/wiki/Dual_EC_DRBG when you can no longer trust the organizations setting the standards (NIST, CFRG) this argument looses water.
- mpyne 13y ago
  Even there the standardization process worked: The flaw in Dual EC DRBG was pointed out very quickly, precisely because there was so much cryptanalytic attention paid to it. The fact that RSA (and only RSA, AFAICT) didn't pay attention to the crypto break in exchange for $10MM doesn't change the fact that the vulnerability had been quickly discovered. If the algorithm had been left unstandardized and was simply "Foo Corp's Custom Wonderful Bit Generator™" the public may never have known of the vulnerability, while the NSA would still have the resources to have discovered the flaw on their own and use TAO to recover the priv key.

ankitml 13y ago

Harder, Better, Faster, Stronger Sounds like MD5 on viagra.

ZenoArrow 13y ago
Or Daft Punk MD5... www.youtube.com/watch?v=gAjR4_CbPpQ

haberman 13y ago

Glad I came across this -- I have a need for a hash function with cryptographic properties but that I don't actually mean to use for crypto, but just to uniquely identify a file by its contents. Speed is really important to me. This sounds perfect for my application.

gojomo 13y ago
If the secure-hash properties are truly a non-factor for your application, you could also consider Ciyhash128/CityHash256: http://en.wikipedia.org/wiki/CityHash http://en.wikipedia.org/wiki/CityHash
- haberman 13y ago
  Good point, I had forgotten it could output 128/256 bits.

keeperofdakeys 13y ago

This author almost completely misses the point. The basic architecture behind SHA-2 is similar to MD5 and SHA-1, and the fact that attacks are possible on MD5 and SHA-1 means an attack could be possible on SHA-2. Then if SHA-3 used the same architecture again, a weakness in SHA-2 might lead to a weakness SHA-3. Then both SHA-2 and SHA-3 are insecure. Since SHA-3 is now based on a totally different architecture, any weakness in SHA-2 has no effect for SHA-3. At the same time, if someone does find an attack against SHA-3, SHA-2 is still secure.