4 ms·
Of course, algorithms like md5 are still useful when not used as cryptographic functions. For example, they can be used for duplicate detection across a large d
by barrkel 12y ago
Of course, algorithms like md5 are still useful when not used as cryptographic functions. For example, they can be used for duplicate detection across a large data set. Unless the data set is under control by an attacker, and they have something to gain by creating a collision, there's little risk for this use.
- afro88 12y agoThat's the issue though isn't it - that an attacker would have control over the dataset and have something to gain by providing a modified file under a publicly known "safe" md5. For example, compromised Linux install iso's.
- emillon 12y agoIn that case that's second pre-image resistance though, not collision resistance. TTBOMK that's still not possible with MD5 (though it will probably be in a few years).
- brohee 12y agoOnly if the attacker had no part in making the "safe" ISO image in the first place. You just cannot know. Using MD5 makes no sense whatsoever, if you don't need collision resistance, there are faster options, and if you do, it's obviously completely unsuitable.
- deleted 12y ago[deleted]
- VLM 12y ago"For example, they can be used for duplicate detection across a large data set." From extensive experimentation with this exact topic, you have to be really careful because there's a lot of handwavey arguments or "just use openssl" but just because md5 is 33% faster than sha1 on 32 bit openssl in i386 doesn't mean whatever random platform and library combo that you're actually using will be exactly 33% by switching from sha1 to md5. Probably, someone out there has a pathological example where md5 is somehow actually slower than sha1. MD5 is almost too good of a hash function for mere de-duping, but I have to use it because its near universal. On mysql I have the function "md5()". In clojure its "(digest/md5 "something")". On the AS/400 thankfully thats not my area of responsibility but I know its compatible with all of the above. In perl its "use Digest::MD5". The most important part is a md5 calculated on an engineering data record will always be the same hash result number no matter the source, which is very important for de-duping. The world of programmers has some very peculiar ideas about hashes such that there is usually only one library for hashing and it often only has stuff like md5 or sha1 which is not good enough for crypto and overkill for de-duping and ID creation. The world would be a better place if as a profession we split our hashing libraries on all machines and languages into "really fast insecure de-dupe hash functions" and "really secure glacially slow crypto hash functions"
- gliptic 12y agoYou can also use e.g. BLAKE2 which is at least as fast as MD5, while still being cryptographically strong. Win-win.
- _almosnow 12y agoThanks for BLAKE2
- smt88 12y agoFor hashing passwords, you want something slow, not fast.
- gliptic 12y agoWho said anything about hashing passwords? But this is a misconception. You don't want a primitive that is slow. You want one that gives you a lot of security margin in few cycles, then you make it slow by, for instance, iterating it many times. There are many password hashing schemes based on BLAKE2, 6 different ones submitted to PHC (https://password-hashing.net/ https://password-hashing.net/).
- smt88 12y agoIt's sadly common for inexperienced devs to use MD5 and call it a day (if they hash at all). I just wanted to point out that, for situations where user input and security are important, you want the algorithm to be slow. I didn't say anything about how to implement it or whether you should use BLAKE2 or what. There's a lot more to it than I could put in a reply here, and even quick Googling would turn up info about salting/iterating/etc.
- TheGrassyKnoll 12y agoNice BLAKE2 article from HN in March, 2014 https://leastauthority.com/blog/BLAKE2-harder-better-faster-stronger-than-MD5.html https://leastauthority.com/blog/BLAKE2-harder-better-faster-...
- wazoox 12y ago> For example, they can be used for duplicate detection across a large data set. And that's how my colleague sees the photos of some unknown woman in his Dropbox (and supposedly, she sees his pictures too).
- Eiriksmal 12y agoCan you elaborate on what mechanism of Dropbox lets this work?
- wazoox 12y agoAlmost certainly by using some form of fragile hashing mechanism such as MD5 to identify users, instead of UUID or something similar.
- habitue 12y agoNo, almost certainly not. Md5 being broken in this way doesn't mean the chances of two random 160 bit numbers are any more likely to collide by chance. It requires an active adversary. There are a million ways dropbox's software could be buggy that would result in this, that are more likely than a chance md5 collision Furthermore, the kind of weakness md5 has now would require that that the attacker generate both usernames. And the usernames would be really really long because they had gibberish tacked onto the end to make the hashes collide. In the jpeg format you can make that part invisible, but not in a username.
- GhotiFish 12y agoIf you arn't concerned about an attacker, then Murmur and fnv-1a are attractive hashes http://programmers.stackexchange.com/questions/49550/which-hashing-algorithm-is-best-for-uniqueness-and-speed/145633#145633 http://programmers.stackexchange.com/questions/49550/which-h...