"For example, they can be used for duplicate detection across a large data set." From extensive experimentation with this exact topic, you have to be really ca

"For example, they can be used for duplicate detection across a large data set." From extensive experimentation with this exact topic, you have to be really careful because there's a lot of handwavey arguments or "just use openssl" but just because md5 is 33% faster than sha1 on 32 bit openssl in i386 doesn't mean whatever random platform and library combo that you're actually using will be exactly 33% by switching from sha1 to md5. Probably, someone out there has a pathological example where md5 is somehow actually slower than sha1. MD5 is almost too good of a hash function for mere de-duping, but I have to use it because its near universal. On mysql I have the function "md5()". In clojure its "(digest/md5 "something")". On the AS/400 thankfully thats not my area of responsibility but I know its compatible with all of the above. In perl its "use Digest::MD5". The most important part is a md5 calculated on an engineering data record will always be the same hash result number no matter the source, which is very important for de-duping. The world of programmers has some very peculiar ideas about hashes such that there is usually only one library for hashing and it often only has stuff like md5 or sha1 which is not good enough for crypto and overkill for de-duping and ID creation. The world would be a better place if as a profession we split our hashing libraries on all machines and languages into "really fast insecure de-dupe hash functions" and "really secure glacially slow crypto hash functions"