3 ms·

And yet MD5 is still recommended in Applied Cryptography.

traceroute66 4y ago
> And yet MD5 is still recommended in Applied Cryptography. I'm guessing this is the usual HN dig at Schneier's book ? Fact is that Schneier was already warning against MD5 as far back as 1996. And if you look on more recent blog posts, he is not exactly silent on dissuading people against MD5. Fact is that like it or not, even today, MD5 still sees usage in the crypto landscape (see AWS S3 Content-MD5, for example). Therefore being able to understand how it works is still relevant, and in that respect Schneier's book is as good as any other.
- stevewatson301 4y ago
  Further, SHA-256 operates on the same principle as well (repeated iterations of a one-way compression function, called a Merkle-Damgard iterative construction[1]). Learning how MD5 works is quite instructive in that regard. [1] https://en.wikipedia.org/wiki/Merkle%E2%80%93Damg%C3%A5rd_construction https://en.wikipedia.org/wiki/Merkle%E2%80%93Damg%C3%A5rd_co...
- dylan604 4y ago
  Like everything else in the world, knowing your audience, intended use, etc plays a key role. If you're protecting data against state actors, then yes, MD5 is not good enough. If you're just trying to decide if a transfer of a file went across a network without munging the data, then MD5 is good enough. If you're trying to post that file onto something that is publicly facing and want to provide the person downloading the file that it has not been modified, MD5 is probably not the best method.
  - dingosity 4y ago
    The point the OP made is it's easy to construct a second preimage with MD5. So if a bad guy, even a bad guy with meager resources, modifies the hash or the data in transit, you won't be able to identify the change the MITM made in either the data or the hash while it was transmitted. What you're probably thinking is a digital signature, which combines a hash and an asymmetric cipher (or a keyed hash if you keep that key secret.) But even then a bad actor can construct a second pre-image with MD5 which would cause the digital signature validate as authentic even though it's been changed. And an adversary with meager computational resources could perform this hack. This is the OP's point. You don't have to be a state actor to do this.
    - dylan604 4y ago
      Essentially, I was agreeing that MD5 is worthless for anything "secure" from 3rd party manipulation. However, it is fine for verification of non-adversarial purposes. It's much faster to run MD5 than SHA256 over a 60-120GB MOV file. Copying camera original footage from the card that is still warm from the camera and making copies to client deliverable storage before telling camera dept that it is okay to wipe the card and reuse is a thing I have done a lot. Speed is important. Security is not a concern at all. Copying a large MOV file from one storage pool to another, retrieving large media assets from tape archive, etc are also common situations where you just want to know that the copy is actually what you think it is from I/O errors during whatever transfer process. In these situations, I have zero concern that someone wearing a blackhat maliciously manipulated the data during the transfer.
      - dingosity 4y ago
        The cost of the calculation of the hash is dwarfed by the cost of copying to/from an SD card (or even nvme devices.) If you hash the file as you transfer it, you'll not notice the difference between MD5 and SHA256.
- retrocryptid 4y ago
  It's more of a dig on people who claim MD5 is okay because it's in Schneier's book. Not so much a dig on Schneier himself. And yes. I understand MD5 etag headers are common. I even understand that MD5 still passes the avalanche test -- Robshaw's observations on MD5 in 1996 are just as valid today. But according to Sasaki & Aoki, //both// collisions and second pre-image calculation are faster than brute force. Maybe MD5 //shouldn't// be used if all you need is avalanche. It turns out that SHA256 has both collision and second pre-image calculation resistance //as well as// avalanche effect. Maybe use that one instead.
  - fluoridation 4y ago
    I don't remember the last time I saw anyone use MD5 for cryptography. It's still used a lot for data integrity, because it still works perfectly well for that and it's faster than the SHA family (when there's no hardware implementation).
    - jabl 4y ago
      Based on a quick test here on a 417MB .tar.gz file I had lying around, best of 5 times for various coreutils (8.32) implementations: - md5sum: 0.570s - sha1sum: 0.667s - sha256sum: 1.662s - sha512sum: 1.011s - b2sum: 0.486s - cksum: 0.982s In conclusion: b2sum is both the fastest and AFAIU considered secure.
      - retrocryptid 4y ago
        Moving from MD5 to SHA256 means we just about triple the amount of time it takes to generate a hash. But I suggest there are few applications where this speed improvement justifies the confusion I've seen in junior engineers who believe MD5 is okay because a. they use MD5SUM and b. Bruce Schneier said it was okay. Don't get me wrong. I trust that //YOU// will know not depend on a MD5 hash for anything where a bad guy can modify content over the wire. But... I'm going to go out on a limb and guess you're somewhat experienced. I worry about the kids who without the benefit of experience re-enact scenes from the cryptography edition of Lord of the Flies. But... if you know what you're doing... sure... use MD5... There's certainly no way your code will ever be used by a less experienced engineer, right?

> And yet MD5 is still recommended in Applied Cryptography. I'm guessing this is the usual HN dig at Schneier's book ? Fact is that Schneier was already warning against MD5 as far back as 1996. And if you look on more recent blog posts, he is not exactly silent on dissuading people against MD5. Fact is that like it or not, even today, MD5 still sees usage in the crypto landscape (see AWS S3 Content-MD5, for example). Therefore being able to understand how it works is still relevant, and in that respect Schneier's book is as good as any other.

stevewatson301 4y ago
Further, SHA-256 operates on the same principle as well (repeated iterations of a one-way compression function, called a Merkle-Damgard iterative construction[1]). Learning how MD5 works is quite instructive in that regard. [1] https://en.wikipedia.org/wiki/Merkle%E2%80%93Damg%C3%A5rd_construction https://en.wikipedia.org/wiki/Merkle%E2%80%93Damg%C3%A5rd_co...
dylan604 4y ago
Like everything else in the world, knowing your audience, intended use, etc plays a key role. If you're protecting data against state actors, then yes, MD5 is not good enough. If you're just trying to decide if a transfer of a file went across a network without munging the data, then MD5 is good enough. If you're trying to post that file onto something that is publicly facing and want to provide the person downloading the file that it has not been modified, MD5 is probably not the best method.
- dingosity 4y ago
  The point the OP made is it's easy to construct a second preimage with MD5. So if a bad guy, even a bad guy with meager resources, modifies the hash or the data in transit, you won't be able to identify the change the MITM made in either the data or the hash while it was transmitted. What you're probably thinking is a digital signature, which combines a hash and an asymmetric cipher (or a keyed hash if you keep that key secret.) But even then a bad actor can construct a second pre-image with MD5 which would cause the digital signature validate as authentic even though it's been changed. And an adversary with meager computational resources could perform this hack. This is the OP's point. You don't have to be a state actor to do this.
  - dylan604 4y ago
    Essentially, I was agreeing that MD5 is worthless for anything "secure" from 3rd party manipulation. However, it is fine for verification of non-adversarial purposes. It's much faster to run MD5 than SHA256 over a 60-120GB MOV file. Copying camera original footage from the card that is still warm from the camera and making copies to client deliverable storage before telling camera dept that it is okay to wipe the card and reuse is a thing I have done a lot. Speed is important. Security is not a concern at all. Copying a large MOV file from one storage pool to another, retrieving large media assets from tape archive, etc are also common situations where you just want to know that the copy is actually what you think it is from I/O errors during whatever transfer process. In these situations, I have zero concern that someone wearing a blackhat maliciously manipulated the data during the transfer.
    - dingosity 4y ago
      The cost of the calculation of the hash is dwarfed by the cost of copying to/from an SD card (or even nvme devices.) If you hash the file as you transfer it, you'll not notice the difference between MD5 and SHA256.
retrocryptid 4y ago
It's more of a dig on people who claim MD5 is okay because it's in Schneier's book. Not so much a dig on Schneier himself. And yes. I understand MD5 etag headers are common. I even understand that MD5 still passes the avalanche test -- Robshaw's observations on MD5 in 1996 are just as valid today. But according to Sasaki & Aoki, //both// collisions and second pre-image calculation are faster than brute force. Maybe MD5 //shouldn't// be used if all you need is avalanche. It turns out that SHA256 has both collision and second pre-image calculation resistance //as well as// avalanche effect. Maybe use that one instead.
- fluoridation 4y ago
  I don't remember the last time I saw anyone use MD5 for cryptography. It's still used a lot for data integrity, because it still works perfectly well for that and it's faster than the SHA family (when there's no hardware implementation).
  - jabl 4y ago
    Based on a quick test here on a 417MB .tar.gz file I had lying around, best of 5 times for various coreutils (8.32) implementations: - md5sum: 0.570s - sha1sum: 0.667s - sha256sum: 1.662s - sha512sum: 1.011s - b2sum: 0.486s - cksum: 0.982s In conclusion: b2sum is both the fastest and AFAIU considered secure.
    - retrocryptid 4y ago
      Moving from MD5 to SHA256 means we just about triple the amount of time it takes to generate a hash. But I suggest there are few applications where this speed improvement justifies the confusion I've seen in junior engineers who believe MD5 is okay because a. they use MD5SUM and b. Bruce Schneier said it was okay. Don't get me wrong. I trust that //YOU// will know not depend on a MD5 hash for anything where a bad guy can modify content over the wire. But... I'm going to go out on a limb and guess you're somewhat experienced. I worry about the kids who without the benefit of experience re-enact scenes from the cryptography edition of Lord of the Flies. But... if you know what you're doing... sure... use MD5... There's certainly no way your code will ever be used by a less experienced engineer, right?