11 ms·
In forensics work, MD5 is still used for checksums. It's used for consistency, not uniqueness, so it will likely be usable for years to come.
by WrtCdEvrydy 6y ago
In forensics work, MD5 is still used for checksums. It's used for consistency, not uniqueness, so it will likely be usable for years to come.
- eli 6y agoStill seems less than ideal to be able to generate a second document that hashes to the same value.
- tialaramex 6y agoYour phrasing here is unclear. Being able to construct B such that hash(B) = hash(A) for some document A is Second Pre-image and there is no viable Second Pre-image attack for SHA-1 or even MD5. What does exist is collision, where you construct documents C and C' such that hash(C) = hash(C'). This means bad guys can persuade humans (and more importantly in most cases, machines) to sign seemingly innocuous document C; and then apparently produce proof document C' was signed even though the human / machine has never seen C' and would not have signed it.
- im3w1l 6y agoCollision combined with length extension could be a neat way of attacking deduping. Basically first a prefix, then a collision block, then a suffix that is encrypted with a key derived from one of the collision blocks. If you can arrange for the deduper to find the document with the wrong key first, then it will skip over the super secret stuff.
- formerly_proven 6y agoFor deduplicated storage you really want A=B <=> H(A)=H(B) to hold for all data you are going to see, so accessible constructions for collisions should already bar an algorithm from this application.
- im3w1l 6y agoI 100% agree, but some people seem to go "eh it's not a problem in practice" for as long as they possibly can.
- bawolff 6y agoWhat does consistency mean in this context?
- bumblebritches5 6y agoVerifying your FLAC decoded to the original content.
- deleted 6y ago[deleted]
- WrtCdEvrydy 6y agoGiven an input image of a hard disk or other storage medium, validate that the hash when given to you and after your analysis are the same.
- bawolff 6y agoIn the presence of someone malicious swapping out drives for a malicious end? Or as a guard against accidentally swapping out the wrong hard drive due to an accident? For the former, md5 isnt secure (although its not really an attack you can pull off after the fact). For the latter, i guess so, but it'd be hilarious if whatever attacker is causing you to do forensics made all the hard drives have the same hash to make your life more complicated.
- WrtCdEvrydy 6y agoForensics work mostly comprises of people doing criminal or civil things. Network intrusions, someone sending a fucked up email at work would require you to retrieve data whether it be some logs, or a full hard disk copy. The idea is that I do analysis and can prove that what I started with (hash1) is the same as when I ended (hash2). If the hash changed, either there was a hardware failure (on write blockers) or a mistake was made when analysing the dataa.
- ReactiveJelly 6y agoIf the authors of Blake3 are to be believed, MD5 is slower than Blake3, Blake2, and SHA1. And on par with SHA512. (On a given architecture) https://raw.githubusercontent.com/BLAKE3-team/BLAKE3/master/media/speed.svg https://raw.githubusercontent.com/BLAKE3-team/BLAKE3/master/... I could understand if hashing is a large portion of their CPU time. But if it's not, why not compute MD5 and also something good and store both hashes?
- oconnor663 6y ago> (On a given architecture) Yes this is a good caveat. When I've tried the OpenSSL implementations on older ARM chips (a Raspberry Pi Zero), I've seen MD5 come out on top. I'm not sure exactly why it switches.
- perihelions 6y agoI think they're just inverted on x86 because it provides hardware acceleration for SHA-1 and SHA-2. https://en.wikipedia.org/wiki/Intel_SHA_extensions https://en.wikipedia.org/wiki/Intel_SHA_extensions https://github.com/openssl/openssl/blob/master/crypto/sha/asm/sha1-x86_64.pl#L412 https://github.com/openssl/openssl/blob/master/crypto/sha/as...
- oconnor663 6y agoI find SHA-1 to be faster than MD5 on Skylake, which does not have the SHA extensions.
- jcrawfordor 6y agoI think a lot of semi-delusion happens around whether or not cryptographic hash functions are used for integrity and what kind of integrity. I would argue against this two ways: First, if the only concern was bit-error detection, forensic tools would use a simple checksum. CRC, for example, is far faster to compute than any of these cryptographic functions. Second, one of the functions of evidential custody is to provide some defense against malicious tampering with evidence, even if unlikely. And for this purpose MD5 is clearly an inferior choice to a SHA2 function. The real reason that forensics tools stick to MD5, I would contend, is a combination of the state of the art in filesystem forensics tools being surprisingly bad and bureaucratic lockin related to documentation and interoperability (the chain of custody form asks for MD5, after all, not SHA512). This kind of factors into one of my hills to die on, which is that careless or cargo-cult selection of hash functions causes problems. When you select a hash or integrity function for files, you need to make a clear decision about whether your goal is to prevent collision or simply to protect bit errors. You should then choose a function that is state-of-the-art for one of those two categories. The whole line of reasoning behind "MD5 is good enough because we're not really using it for security" is just asking for trouble. When you split the middle you either have a bad integrity check, a bad security control, or more likely both (MD5).
- userbinator 6y agoCRC, for example, is far faster to compute than any of these cryptographic functions. ...and offers far less protection even from random corruption; the biggest common CRC is 64 bits, which is really small in comparison to 128-bit MD5 or 160-bit SHA1.
- cyphar 6y agoI assume GP just used CRC as an example everyone is familiar with. A more modern non-cryptographic hash function like FNV or MurmurHash can produce more output bits (up to 1024 and 128 respectively), reducing random collision probability to a comparable level. In addition, CRCs are actually quite good at detecting common errors in data transfer (not just single bit flips but burst errors as well). In fact you can prove that certain kinds of errors will always be detected by an n-bit CRC -- which is something you can't prove with modern cryptographic hash functions. Yeah they're useless against attackers but that's not the threat model if you're "not using MD5 for security".
- dreamcompiler 6y agoI use SHA1 for duplicate detection on hard drives. It's more robust than MD5 and fast on modern processors. Nothing wrong with that. But I would not use it for crypto.
- hyko 6y agoYikes. Surely in forensics you’d want the most collision resistant hashing tool available? Doesn’t the uniqueness guarantee the consistency?
- WrtCdEvrydy 6y agoThat's kinda the issue, you only want to guarantee that the image didn't change (ie, noone planted anything and the chain of evidence from the crime scene to your nice forensics lab is untampered). I think once someone makes a tool that allows you to add a file to an image and have it resolve to the same hash (anti-forensics as a field), it will probably change. Most tools do offer dual MD5 and SHA256 support though.
- wbl 6y agoThat's a second preimage attack not a collision attack.
- hyko 6y agoI thought a second preimage attach implied a collision attack?
- SAI_Peregrinus 6y agoYes but the reverse isn't true. Collisions don't imply second-preimage attacks, at least not at any practical difficulty.
- hyko 6y agoyou only want to guarantee that the image didn’t change But if the algorithm allows you to find hash collisions, you can’t guarantee that the image didn’t change based on the MD5 hash value? e.g. https://natmchugh.blogspot.com/2014/11/three-way-md5-collision.html?m=1 https://natmchugh.blogspot.com/2014/11/three-way-md5-collisi... Obviously that example is a chosen prefix collision, but this data is coming from an untrusted source after all, so there’s really nothing to stop the attacker choosing the prefix in advance and then publicly shredding trust in the hash at a later date. In practice it sounds like you’d also have a hash of the complete file system, but at this point you’d have to question what advantage there is to using MD5 at all. Attacks never get worse, only ever get better, and the last thing you’d want is for the dam to burst during a lengthy and important investigation.
- kzrdude 6y agoIf they are already good at keeping multiple points of view in mind: that MD5 can be broken for one application, but it still works for others; then they should also be able to reason that, MD5 might work passably for this application, but it is time to move on.