6 ms·

MD5 also remains quite popular for file hashing/hash set analysis. I.e., is this file a member of a known set? For example, NIST's NSRL still includes MD5 hashe

MD5 also remains quite popular for file hashing/hash set analysis. I.e., is this file a member of a known set? For example, NIST's NSRL still includes MD5 hashes. * This is not an endorsement of MD5 for such use cases.

smackfu 12y ago
Do the security problems with MD5 make it bad as a hashing algorithm?
- tkmcc 12y ago
  Not necessarily, but if that's your use case (and the hashing speed isn't extremely critical), why not use a better algorithm with a larger output?
- derobert 12y ago
  Depends. Can an attacker introduce files? If so, he can probably introduce collisions. In most uses of a hash, that's an issue—a performance (DoS) issue if nothing else. If the application was just looking for a longer CRC/Adler32/etc. and not depending on the hash to be strong against attack, then MD5 wasn't an appropriate choice in the first place, as its much slower than need be. But the security problems are irrelevant.
- ChuckMcM 12y ago
  Generally the tuple (<length>,<hash>) is unique since the attacks on MD5 all involve changing the number of octets in the hashed source. That said, it isn't secure if you can change the length independently of changing the hash. Hence the challenge of using it cryptographically. If you're confident you know the correct 'length' value (acts as a sort of openly shared secret in this case) then you trust the hash.
  - heinrich5991 12y ago
    I believe this is wrong, and in this case it's very dangerous information. See for example the Wikipedia article for two chunks of data that only differ in a few bits (not length!) and hash the to same MD5 hash: https://en.wikipedia.org/wiki/MD5#Collision_vulnerabilities https://en.wikipedia.org/wiki/MD5#Collision_vulnerabilities
- ketralnis 12y ago
  Not unless hash collisions are also a security issue (see http://bugs.python.org/issue13703 http://bugs.python.org/issue13703). But it's much slower than a traditional hashtable hashing function and not much slower than a sha family hash. Maybe that's the sweet spot for your project, but I think most non-security uses of MD5 are because it already exists in most languages and is pretty universal
- dredmorbius 12y ago
  Do you know the contexts your hashing requirement is likely to be used in, and/or may be adapted to? The worst and most persistent security problems emerge when someone defends half-assery as acceptable because, take your pick, it's a quick hack / it's a personal project / it's a small project / it's not used for critical infrastructure. Until eventually is one or more of the above are violated. The primary advantage of MD5 is that the hashes are (slightly) shorter than those of other checksum methods. This makes it slightly more convenient to manually compare or transmit hashes. My problem is that I happen to have used md5sum so often and for so long that it's wired into my own wetware and muscle-memory, and I'm not sure which of the alternative SHA sums I should use, and which of those are widely available. I honestly don't know the answer to that off the top of my head, and "what to use instead of md5sum" as a DDG or Google search doesn't turn up a clearly useful answer. "sha1 sha256 sha384 sha512 which to use" does better. And locally I've got utilities for SHA 1, 224, 256, and 512 installed, that I can tell. Looks as if SHA-2 and SHA-3, which include keylengths of 224-512, are considered secure: http://en.wikipedia.org/wiki/Secure_Hash_Algorithm http://en.wikipedia.org/wiki/Secure_Hash_Algorithm Then there are some openssl utilities. But let's not go there.
MagicWishMonkey 12y ago
Are collisions really that much of a concern (other than for security reasons)? IIRC they are still incredibly rare even with older hashing algorithms like MD5.
- nly 12y ago
  Collisions render hash functions useless as a component of most (but not all) digital signature algorithms
- ekimekim 12y ago
  Yes, if you don't care about collisions* or being able to reverse the hash (a pre-image attack) then there's nothing wrong with MD5 per se. However, MD5 was designed as a secure algorithm, and makes tradeoffs, mainly in speed, to that effect. If you don't need security, you should use something designed for that, like CRC. * note that even if it isn't immediately obvious, in most situations you really don't want collisions to be an easy thing to cause. For example, many algorithms and structures involving hashes (even a simple in-memory hash map) suffer massively degraded performance when there's a large number of collisions. If your user can craft an input that causes operations to take orders of magnitude longer, that's a denial of service attack.
stubborndude 12y ago
Was going to make this point. But lets instead talk about the endorsement aspect, because lots of people in the forensic community take the "its popular so i use them, but i neither endorse nor oopose" position. There is certainly risk of hash collisions in files, but most attacks require you to generate two different files with the same hash? Do people think a pre-image attack is feasible? If feasible, at what scale? What type of actor can give me a file that has different content, but the same hash as winword.exe. If they can, what is the likelihood that said file will also be executable or contain a linkable library *similarly, and hypocritically, not an endorsement of md5

smackfu 12y ago

Do the security problems with MD5 make it bad as a hashing algorithm?

tkmcc 12y ago
Not necessarily, but if that's your use case (and the hashing speed isn't extremely critical), why not use a better algorithm with a larger output?
derobert 12y ago
Depends. Can an attacker introduce files? If so, he can probably introduce collisions. In most uses of a hash, that's an issue—a performance (DoS) issue if nothing else. If the application was just looking for a longer CRC/Adler32/etc. and not depending on the hash to be strong against attack, then MD5 wasn't an appropriate choice in the first place, as its much slower than need be. But the security problems are irrelevant.
ChuckMcM 12y ago
Generally the tuple (<length>,<hash>) is unique since the attacks on MD5 all involve changing the number of octets in the hashed source. That said, it isn't secure if you can change the length independently of changing the hash. Hence the challenge of using it cryptographically. If you're confident you know the correct 'length' value (acts as a sort of openly shared secret in this case) then you trust the hash.
- heinrich5991 12y ago
  I believe this is wrong, and in this case it's very dangerous information. See for example the Wikipedia article for two chunks of data that only differ in a few bits (not length!) and hash the to same MD5 hash: https://en.wikipedia.org/wiki/MD5#Collision_vulnerabilities https://en.wikipedia.org/wiki/MD5#Collision_vulnerabilities
ketralnis 12y ago
Not unless hash collisions are also a security issue (see http://bugs.python.org/issue13703 http://bugs.python.org/issue13703). But it's much slower than a traditional hashtable hashing function and not much slower than a sha family hash. Maybe that's the sweet spot for your project, but I think most non-security uses of MD5 are because it already exists in most languages and is pretty universal
dredmorbius 12y ago
Do you know the contexts your hashing requirement is likely to be used in, and/or may be adapted to? The worst and most persistent security problems emerge when someone defends half-assery as acceptable because, take your pick, it's a quick hack / it's a personal project / it's a small project / it's not used for critical infrastructure. Until eventually is one or more of the above are violated. The primary advantage of MD5 is that the hashes are (slightly) shorter than those of other checksum methods. This makes it slightly more convenient to manually compare or transmit hashes. My problem is that I happen to have used md5sum so often and for so long that it's wired into my own wetware and muscle-memory, and I'm not sure which of the alternative SHA sums I should use, and which of those are widely available. I honestly don't know the answer to that off the top of my head, and "what to use instead of md5sum" as a DDG or Google search doesn't turn up a clearly useful answer. "sha1 sha256 sha384 sha512 which to use" does better. And locally I've got utilities for SHA 1, 224, 256, and 512 installed, that I can tell. Looks as if SHA-2 and SHA-3, which include keylengths of 224-512, are considered secure: http://en.wikipedia.org/wiki/Secure_Hash_Algorithm http://en.wikipedia.org/wiki/Secure_Hash_Algorithm Then there are some openssl utilities. But let's not go there.

MagicWishMonkey 12y ago

Are collisions really that much of a concern (other than for security reasons)? IIRC they are still incredibly rare even with older hashing algorithms like MD5.

nly 12y ago
Collisions render hash functions useless as a component of most (but not all) digital signature algorithms
ekimekim 12y ago
Yes, if you don't care about collisions* or being able to reverse the hash (a pre-image attack) then there's nothing wrong with MD5 per se. However, MD5 was designed as a secure algorithm, and makes tradeoffs, mainly in speed, to that effect. If you don't need security, you should use something designed for that, like CRC. * note that even if it isn't immediately obvious, in most situations you really don't want collisions to be an easy thing to cause. For example, many algorithms and structures involving hashes (even a simple in-memory hash map) suffer massively degraded performance when there's a large number of collisions. If your user can craft an input that causes operations to take orders of magnitude longer, that's a denial of service attack.

stubborndude 12y ago

Was going to make this point. But lets instead talk about the endorsement aspect, because lots of people in the forensic community take the "its popular so i use them, but i neither endorse nor oopose" position. There is certainly risk of hash collisions in files, but most attacks require you to generate two different files with the same hash? Do people think a pre-image attack is feasible? If feasible, at what scale? What type of actor can give me a file that has different content, but the same hash as winword.exe. If they can, what is the likelihood that said file will also be executable or contain a linkable library *similarly, and hypocritically, not an endorsement of md5