9 ms·

SHA-1 is still perfectly fine for some applications like detecting duplicate files on a storage medium (and it's less likely to produce a false positive than MD

SHA-1 is still perfectly fine for some applications like detecting duplicate files on a storage medium (and it's less likely to produce a false positive than MD5) but it's been a bad idea for anything security related for a decade. The biggest issue is that git still uses it, which presents a problem if you want to protect a repo from active integrity attacks.

dragontamer 4y ago
I feel like the properties of CRC make them superior for that task in most cases though. (CRC8, CRC16, CRC32 and CRC64, maybe CRC128 if anyone ever bothered going that far) In particular, CRC guarantees detection on all bursts of the given length. CRC32 protects vs all bursts of length 32 bits.
- AlotOfReading 4y ago
  There are only a few examples of anything larger than CRC64 being characterized and they're not very useful. For the sake of the next person who has to maintain your code though, please choose algorithms that adequately communicate your intentions. Choose CRCs only if you need to detect random errors in a noisy channel with a small number of bits and use a length appropriate to the intended usage (i.e. almost certainly not CRC64).
  - dragontamer 4y ago
    And when you choose SHA1, does it mean you understood that it's no longer secure? Or is it chosen because it was secure 20 years ago but the code is old and needs to be updated? CRC says that you never intended security from the start. It's timeless, aimed to prevent burst errors and random errors. -------- BTW, what is the guaranteed Hamming distance between SHA1? How good is SHA1 vs burst errors? What about random errors? Because the Hamming distances of CRC have been calculated and analyzed. We actually can determine, to an exact level, how good CRC codes are.
    - dagenix 4y ago
      If you are choosing between a CRC and SHA1, you probably need to reconsider your understanding of the problem you are trying to solve. Those algorithms solve different use cases.
      - dragontamer 4y ago
        If you are choosing SHA1, now that it is retired, you probably should think harder about the problem in general. CRC should be better for any error detection code issue. Faster to calculate, more studied guaranteed detection modes, and so forth. SHA1 has no error detection studies. It's designed as a cryptographic hash, to look random. As it so happens, it is more efficient to use other algorithms and do better than random if you have a better idea of how your error looks like. Real world errors are either random or bursty. CRC is designed for these cases. CRC detects the longest burst possible for it's bitsize.
    - AlotOfReading 4y ago
      You shouldn't choose SHA-1, that's the point of this announcement. Seeing it indicates both that there was the potential for malicious input and that the code is old. The appropriate mitigation is to move to a secure hash, not CRCs. You may not know the bounds and distances exactly, but you know them probabilistically. Bit errors almost always map to a different hash. The same is true of CRCs over a large enough input as an aside.
    - benlivengood 4y ago
      https://eprint.iacr.org/2019/078 https://eprint.iacr.org/2019/078 is a good overview of the statistical properties of SHA hashes. Basically after ~10 rounds the output is always indistinguishable from randomness which means hamming distance is what you'd expect (about half the bits differ) between the hashes of any two bitstreams.
      - dragontamer 4y ago
        CRC does better than random vs burst errors. And vs random errors is indistinguishable from any other methodology. If you are only worried about random errors, might as well use chksum.
- IncRnd 4y ago
  > I feel like the properties of CRC make them superior for that task in most cases though. THIS IS FALSE. Please do not ever do this. Why not? For example, by controlling any four contiguous bytes in a file, the resultant 32bit CRC can be forced to take on any value. A CRC is meant to detect errors due to noise - not changes due to a malicious actor. Program design should not be done based upon one's feelings. CRCs absolutely do not have the required properties to detect duplication or to preserve integrity of a stored file that an attacker can modify.
  - dragontamer 4y ago
    > THIS IS FALSE. Please do not ever do this. Why not? For example, by controlling any four contiguous bytes in a file, the resultant 32bit CRC can be forced to take on any value. And SHA1 is now broken like this, with collisions and so forth. Perhaps it's not as simple as just 4 bytes, but the ability to create collisions is forcing this retirement. If adversarial collisions are an issue, then MD5 and SHA1 are fully obsolete now. If you don't care for an adversary, might as well use the cheaper, faster CRC check. ------ CRC is now more valid use case than SHA1. That's the point of this announcement.
    - IncRnd 4y ago
      > CRC is now more valid use case than SHA1. That's the point of this announcement. No. That isn't the point of this announcement. This announcement codifies a transition time for SHA-1 use in SBU and has nothing to do with a CRC.
      - Dylan16807 4y ago
        The point of the announcement is to give a timeline to start treating SHA1 as having no real security. This is relevant to when you'd choose CRC, because it also has no real security.
        IncRnd 4y ago
        > The point of the announcement is to give a timeline to start treating SHA1 as having no real security. That's also false. There is a large body of knowledge here that you aren't expressing in your comments. That leads me to see that you are unfamiliar with the purposes of hash functions and their utility in real world situations. The announcement refers to the transition timeline to stop using SHA-1, preferring the SHA-2 and SHA-3 families. However, the recommendations for years from NIST have been not to use SHA-1. For example SP 800-131Ar2 discusses not to use SHA-1 for digital sig gen and that digital sig ver is only acceptable for legacy uses. The recommendation would have been for years to not use SHA-1 at all, except for this carve-out to handle already stored data that uses SHA-1. The remaining use cases cover protocol use, such as TLS, where SHA-1 is used as a component in constructs and not solely as a primitive.
oconnore 4y ago
If you need a shorter hash just truncate a modern hash algorithm down to 160 or 128 bits. Obviously the standard lengths were chosen for a reason, but SHA2-256/160 or SHA2-256/128 are better hash functions than SHA1 or MD5, respectively. Blake2b/160 is even faster than SHA1! (I suspect this would be a good compromise for git, since so much tooling assumes a 160 bit hash, and yet we don't want to continue using SHA1)
- GTP 4y ago
  Or use Keccak (which is the permutation that was chosen for SHA3), which lets you pick the length that you need.
- kzrdude 4y ago
  Git has already implemented a solution based on SHA-2 with 256 bit output so that's unlikely to be changed for the time being. (But it has not really been launched in earnest, only as a preview feature.)
- danbruc 4y ago
  Just as a note, the primary reason for the truncated variants is not to get a shorter hash but to prevent extension attacks. For variants without truncation, the final hash is the entire internal state, therefore an attacker can calculated the hash of any message that starts with the original message and then has additional content without knowing the original message. Truncating the hash denies access to the complete internal state and makes this impossible.
  - cesarb 4y ago
    Another way to prevent extension attacks is to make the internal state different whenever the current block is the last block, as done for instance in BLAKE3 (which has as an additional input on each block a set of flags, and one of the flags says "this is the last block").
umvi 4y ago
What kind of attack against a git repo can be realistically carried out today?
- kadoban 4y ago
  Given a git repo with signed commits I can give you the same repo with some file(s) changed with the signatures intact. Pretty sure you can anyway, I haven't thought deeply about the git file formats involved and such. Note that this attack isn't _that_ serious. There's not a lot of cases where this would make sense.
  - AnssiH 4y ago
    I don't think so, unless you utilize some not-yet-public vulnerability. As far as I know, with current public SHA-1 vulnerabilities, you can create two new objects with the same hash (collision attack), but cannot create a second object that has the same hash as some already existing object (preimage attack).
    - kadoban 4y ago
      My bad, yep you're right. So you could only either give 2 people different git repos that should be the same or I guess you could submit a collided file into a repo you can submit changes to (eg a public one that accepts PRs) and give someone else the other version. Given the limitations, really not too practical.
- deleted 4y ago
  [deleted]
EGreg 4y ago
Actually SHA1-HMAC is fine still. Although I had this question: https://crypto.stackexchange.com/questions/102990/what-is-more-secure-hmac-with-sha-1-or-with-sha-256-and-take-a-substring https://crypto.stackexchange.com/questions/102990/what-is-mo...
- oconnore 4y ago
  Since I'm on a roll with truncated hashes: SHA2-512/384, and SHA2-512/256 are not vulnerable to a Merkle-Damgard length extension attack. Thus, a construct like hash(key + message) can be used similar to SHA3 [1] [1]: https://keccak.team/keccak_strengths.html https://keccak.team/keccak_strengths.html
- classichasclass 4y ago
  This is worth emphasizing since HOTP (and thus TOTP) are based on it.
  - deleted 4y ago
    [deleted]
amluto 4y ago
> SHA-1 is still perfectly fine for some applications like detecting duplicate files on a storage medium If by “perfectly fine” you mean “subject to attacks that generate somewhat targeted collisions that are practical enough that people do them for amusement and excuses to write blog posts and cute Twitter threads”, then maybe I agree. Snark aside, SHA-1 is not fine for deduplication in any context where an attacker could control any inputs. Do not use it for new designs. Try to get rid of it in old designs.
- orev 4y ago
  By “perfectly fine” they mean detecting duplicate image or document files on your local storage, which it’s still perfectly fine for, and a frequent mode of usage for these types of tools. Not every tool needs to be completely resilient to an entire Internets’ worth of attacks.
  - dagenix 4y ago
    Why would you pick SHA1 which has significant attacks you need to consider as opposed to something like SHA2 that doesn't?
    - dylan604 4y ago
      who is attacking you in this situation though? you're scanning the files on your local system and storing their hashes. you then look for duplicate hashes, and compare the files that created them. if the files are truly duplicates, you can now decide what to do about that. if they are not truly the same, then you claim to have found another case of collisions, write your blog post/twitthread and move on, but does that constitute being attacked? sometimes, i really feel like people in crypto just can't detach themselves enough to see that just because they have a hammer, not everything in the world is a nail.
      - dagenix 4y ago
        Why would you pick a function that is known to have issues when there are other functions that do the same thing but don't have known issues? Your comparison is flawed. It's more like if you have a nail and next to it a workbench with two hammers - a good hammer and a not as good hammer. This isn't a hard choice. But for reasons that are unclear to me, people in this thread are insisting on picking the less good hammer and rationalizing why for this specific nail it isn't all that much worse. Just pick the better hammer!
dwheeler 4y ago
Git no longer uses SHA-1. It instead uses a variant called SHA-1DC that detects some known problems, and in those cases returns a different answer. More info: <https://github.com/cr-marcstevens/sha1collisiondetection https://github.com/cr-marcstevens/sha1collisiondetection>. Git switched to SHA-1DC in its version 2.13 release in 2017. It's a decent stopgap but not a grrat long term solution. There is also work to support SHA-256, though that seems to have stalled: https://lwn.net/Articles/898522/ https://lwn.net/Articles/898522/ The fundamental problem is that get developers assumed that hash algorithms would never be changed, and that was a ridiculous assumption. It's much wiser to implement crypto agility.
- woodruffw 4y ago
  > The fundamental problem is that get developers assumed that hash algorithms would never be changed, and that was a ridiculous assumption. It's much wiser to implement crypto agility. Cryptographic agility makes this problem worse, not better: instead of having a "flag day" (or release) where `git`'s digest choice reflects the State of the Art, agility ensures that every future version of `git` can be downgraded to a broken digest.
  - est31 4y ago
    That's the general anti-agility argument wielded against git, but note that git's use cases require it to process historic data. E.g. you will want to be able to read some sha-1-only repo from disk that was last touched a decade ago. That's a different thing than some protocol which requires both parties to be on-line, say wireguard, in which instance it's easier to switch both to a new version that uses a different cryptographic algorithm. Git has such protocols as well, and maybe it can deprecate sha-1 support there eventually, but even there it has to support both sha-1 and sha-2 for a while because not everyone is using the latest and greatest version of git, and no sysadmin wants the absolute horror of flag days.
    - miga 4y ago
      It would be safer to forbid broken hashes after certain date, and consider only those earlier hashes that have been counter-signed by new algorithms.
hannob 4y ago
> SHA-1 is still perfectly fine for some applications like detecting duplicate files on a storage medium That's what the developers of subversion thought, but they didn't anticipate that once colliding files were available people would commit them to SVN repos as test cases. And then everything broke: https://www.bleepingcomputer.com/news/security/sha1-collision-attack-makes-its-first-victim-subversion-repositories/ https://www.bleepingcomputer.com/news/security/sha1-collisio...
- TheRealPomax 4y ago
  That changes the parameters quite a bit though. For local digests, like image deduplication of your own content, on your own computers, sha-1 is still perfectly fine. Heck, even MD5 is still workable (although more prone to collide). Nowhere in that process is the internet, or "users" or anything else like that involved =) You use digests to quickly detect potential collisions, then you verify each collision report, then you delete the actual duplicates. Human involvement still very much required because you're curating your own data.
  - dylan604 4y ago
    If we're talking specifically image deduplication, then a hash comparison is only going to find you exact matches. what about the image deduplication of trying to find alt versions of things like scaling, different codecs, etc? if you want to dedupe images, some sort of phashing would be much better so that the actual image is considered vs just the specific bits to generate the image.
    - dreamcompiler 4y ago
      Very true but if the hash matches the images are guaranteed to match too. That's my first pass when deduping my drives. My second pass is looking for "logically equivalent" images.
    - TheRealPomax 4y ago
      Depends on the images. For photographs, a digest is enough. For "random images downloaded from the web", or when you're deduplicating lots of user's data, sure, you want data appropriate digests, like SIFT prints. But then we're back to "you had to bring network content back into this" =)
kiratp 4y ago
As an industry we need to get over this pattern of scoping down usage of something that has failed it’s prime directive. People still use MD5 in secure related things because it’s been allowed to stick around without huge deprecation warnings in libraries and tools. SHA1 (and MD5) need to be treated the same way you would treat O(n^2) sorting in a code review for a PR written by a newbie.
0xbadcafebee 4y ago
If you're only using the hash for non-cryptographic applications, there are much faster hashes: https://github.com/Cyan4973/xxHash https://github.com/Cyan4973/xxHash SHA1 and MD5 are the most widely accessible, though, and I agree it's fine to use them if you don't care about security.
bawolff 4y ago
But sha-1 is now both slow and insecure. If you dont care about security, use a faster hash. If you care about security use sha256 (which is about the same speed anyways). The only valid reason to still use it in non-security critical roles is backwards compat.
VLM 4y ago
“We recommend that anyone relying on SHA-1 for security migrate to SHA-2 or SHA-3 as soon as possible.” —Chris Celi, NIST computer scientist The emphasis being on "for security" I've also used SHA-1 over the years for binning and verifying file transfer success, none of those are security related. Sometimes, if you make a great big pile of different systems, what's held in common across them can be weird, SHA-1 popped out of the list so we used it. I'm well aware its possible to write or automate the writing of dedicated specialized "perfect" hashing algos to match the incoming data, to bin the data more perfectlyier, but sometimes its nice if wildly separate systems all bin incoming data the same highly predictable way thats "good enough" and "fast enough".
- jraph 4y ago
  > I've also used SHA-1 over the years for binning and verifying file transfer success, none of those are security related. It could. If you want to verify that the file has not been tempered by someone, it is security related.
  - VLM 4y ago
    Prob a poor choice of words on my part. Verified as in "is this file completely transferred or not?" non-security critical data, I just want a general idea if its valid or the file transfer failed half way thru or the thing sending it went bonkers and just sent trash to us. Another funny file transfer use: Send me a file of data every hour. Is the non-crypto-hash new or the same old hash? If its the same old hash, those clowns sent me the same file twice, I'm supposed to get a new one. Yes I know I can dedupe "easily" but not as "easily" as sha-1. And some application layer software like MySQL can directly generate SHA1 as a function in the query. Its really quite handy sometimes!
LinuxBender 4y ago
SHA-1 is still perfectly fine for some applications like detecting duplicate files on a storage medium Absolutely agree, especially when speed is a workable trade-off and accepting real world hash collisions are unlikely and perhaps an acceptable risk. For financial data, especially files not belonging to me I would have md5+sha1+sha256 checksums and maybe even GPG sign a manifest of the checksums ... because why not. For my own files md5 has always been sufficient. I have yet to run into a real world collision. FWIW anyone using `rsync --checksum` is still using md5. Not that long ago I think 2014 it was using md4. I would be surprised if rsync started using anything beyond md5 any time soon. I would love to see all the checksum algorithms become CPU instruction sets. Optimizations: no SIMD-roll, no asm-roll, no openssl-crypto, asm-MD5 Checksum list: md5 md4 none Compress list: zstd lz4 zlibx zlib none Daemon auth list: md5 md4
- tptacek 4y ago
  You're never going to _accidentally_ run into an MD5 collision.
pornel 4y ago
Even if git didn't have protection against the known attack, it's still safe in practice. The SHA-1 collision attack can only work if you take a specially-crafted file from the attacker and commit it to your repository. The file needs to have a specific structure, and will contain binary data that looks like junk. It can't look like innocent source code. If you execute unintelligible binary blobs from strangers, you're in trouble anyway. There is no preimage weakness in SHA-1, so nobody is able to change or inject new data to an arbitrary repo/commit that doesn't already contain their colliding file.

dragontamer 4y ago

I feel like the properties of CRC make them superior for that task in most cases though. (CRC8, CRC16, CRC32 and CRC64, maybe CRC128 if anyone ever bothered going that far) In particular, CRC guarantees detection on all bursts of the given length. CRC32 protects vs all bursts of length 32 bits.

AlotOfReading 4y ago
There are only a few examples of anything larger than CRC64 being characterized and they're not very useful. For the sake of the next person who has to maintain your code though, please choose algorithms that adequately communicate your intentions. Choose CRCs only if you need to detect random errors in a noisy channel with a small number of bits and use a length appropriate to the intended usage (i.e. almost certainly not CRC64).
- dragontamer 4y ago
  And when you choose SHA1, does it mean you understood that it's no longer secure? Or is it chosen because it was secure 20 years ago but the code is old and needs to be updated? CRC says that you never intended security from the start. It's timeless, aimed to prevent burst errors and random errors. -------- BTW, what is the guaranteed Hamming distance between SHA1? How good is SHA1 vs burst errors? What about random errors? Because the Hamming distances of CRC have been calculated and analyzed. We actually can determine, to an exact level, how good CRC codes are.
  - dagenix 4y ago
    If you are choosing between a CRC and SHA1, you probably need to reconsider your understanding of the problem you are trying to solve. Those algorithms solve different use cases.
    - dragontamer 4y ago
      If you are choosing SHA1, now that it is retired, you probably should think harder about the problem in general. CRC should be better for any error detection code issue. Faster to calculate, more studied guaranteed detection modes, and so forth. SHA1 has no error detection studies. It's designed as a cryptographic hash, to look random. As it so happens, it is more efficient to use other algorithms and do better than random if you have a better idea of how your error looks like. Real world errors are either random or bursty. CRC is designed for these cases. CRC detects the longest burst possible for it's bitsize.
  - AlotOfReading 4y ago
    You shouldn't choose SHA-1, that's the point of this announcement. Seeing it indicates both that there was the potential for malicious input and that the code is old. The appropriate mitigation is to move to a secure hash, not CRCs. You may not know the bounds and distances exactly, but you know them probabilistically. Bit errors almost always map to a different hash. The same is true of CRCs over a large enough input as an aside.
  - benlivengood 4y ago
    https://eprint.iacr.org/2019/078 https://eprint.iacr.org/2019/078 is a good overview of the statistical properties of SHA hashes. Basically after ~10 rounds the output is always indistinguishable from randomness which means hamming distance is what you'd expect (about half the bits differ) between the hashes of any two bitstreams.
    - dragontamer 4y ago
      CRC does better than random vs burst errors. And vs random errors is indistinguishable from any other methodology. If you are only worried about random errors, might as well use chksum.
IncRnd 4y ago
> I feel like the properties of CRC make them superior for that task in most cases though. THIS IS FALSE. Please do not ever do this. Why not? For example, by controlling any four contiguous bytes in a file, the resultant 32bit CRC can be forced to take on any value. A CRC is meant to detect errors due to noise - not changes due to a malicious actor. Program design should not be done based upon one's feelings. CRCs absolutely do not have the required properties to detect duplication or to preserve integrity of a stored file that an attacker can modify.
- dragontamer 4y ago
  > THIS IS FALSE. Please do not ever do this. Why not? For example, by controlling any four contiguous bytes in a file, the resultant 32bit CRC can be forced to take on any value. And SHA1 is now broken like this, with collisions and so forth. Perhaps it's not as simple as just 4 bytes, but the ability to create collisions is forcing this retirement. If adversarial collisions are an issue, then MD5 and SHA1 are fully obsolete now. If you don't care for an adversary, might as well use the cheaper, faster CRC check. ------ CRC is now more valid use case than SHA1. That's the point of this announcement.
  - IncRnd 4y ago
    > CRC is now more valid use case than SHA1. That's the point of this announcement. No. That isn't the point of this announcement. This announcement codifies a transition time for SHA-1 use in SBU and has nothing to do with a CRC.
    - Dylan16807 4y ago
      The point of the announcement is to give a timeline to start treating SHA1 as having no real security. This is relevant to when you'd choose CRC, because it also has no real security.
      - IncRnd 4y ago
        > The point of the announcement is to give a timeline to start treating SHA1 as having no real security. That's also false. There is a large body of knowledge here that you aren't expressing in your comments. That leads me to see that you are unfamiliar with the purposes of hash functions and their utility in real world situations. The announcement refers to the transition timeline to stop using SHA-1, preferring the SHA-2 and SHA-3 families. However, the recommendations for years from NIST have been not to use SHA-1. For example SP 800-131Ar2 discusses not to use SHA-1 for digital sig gen and that digital sig ver is only acceptable for legacy uses. The recommendation would have been for years to not use SHA-1 at all, except for this carve-out to handle already stored data that uses SHA-1. The remaining use cases cover protocol use, such as TLS, where SHA-1 is used as a component in constructs and not solely as a primitive.

oconnore 4y ago

If you need a shorter hash just truncate a modern hash algorithm down to 160 or 128 bits. Obviously the standard lengths were chosen for a reason, but SHA2-256/160 or SHA2-256/128 are better hash functions than SHA1 or MD5, respectively. Blake2b/160 is even faster than SHA1! (I suspect this would be a good compromise for git, since so much tooling assumes a 160 bit hash, and yet we don't want to continue using SHA1)

GTP 4y ago
Or use Keccak (which is the permutation that was chosen for SHA3), which lets you pick the length that you need.
kzrdude 4y ago
Git has already implemented a solution based on SHA-2 with 256 bit output so that's unlikely to be changed for the time being. (But it has not really been launched in earnest, only as a preview feature.)
danbruc 4y ago
Just as a note, the primary reason for the truncated variants is not to get a shorter hash but to prevent extension attacks. For variants without truncation, the final hash is the entire internal state, therefore an attacker can calculated the hash of any message that starts with the original message and then has additional content without knowing the original message. Truncating the hash denies access to the complete internal state and makes this impossible.
- cesarb 4y ago
  Another way to prevent extension attacks is to make the internal state different whenever the current block is the last block, as done for instance in BLAKE3 (which has as an additional input on each block a set of flags, and one of the flags says "this is the last block").

umvi 4y ago

What kind of attack against a git repo can be realistically carried out today?

kadoban 4y ago
Given a git repo with signed commits I can give you the same repo with some file(s) changed with the signatures intact. Pretty sure you can anyway, I haven't thought deeply about the git file formats involved and such. Note that this attack isn't _that_ serious. There's not a lot of cases where this would make sense.
- AnssiH 4y ago
  I don't think so, unless you utilize some not-yet-public vulnerability. As far as I know, with current public SHA-1 vulnerabilities, you can create two new objects with the same hash (collision attack), but cannot create a second object that has the same hash as some already existing object (preimage attack).
  - kadoban 4y ago
    My bad, yep you're right. So you could only either give 2 people different git repos that should be the same or I guess you could submit a collided file into a repo you can submit changes to (eg a public one that accepts PRs) and give someone else the other version. Given the limitations, really not too practical.
deleted 4y ago
[deleted]

EGreg 4y ago

Actually SHA1-HMAC is fine still. Although I had this question: https://crypto.stackexchange.com/questions/102990/what-is-more-secure-hmac-with-sha-1-or-with-sha-256-and-take-a-substring https://crypto.stackexchange.com/questions/102990/what-is-mo...

oconnore 4y ago
Since I'm on a roll with truncated hashes: SHA2-512/384, and SHA2-512/256 are not vulnerable to a Merkle-Damgard length extension attack. Thus, a construct like hash(key + message) can be used similar to SHA3 [1] [1]: https://keccak.team/keccak_strengths.html https://keccak.team/keccak_strengths.html
classichasclass 4y ago
This is worth emphasizing since HOTP (and thus TOTP) are based on it.
- deleted 4y ago
  [deleted]

amluto 4y ago

> SHA-1 is still perfectly fine for some applications like detecting duplicate files on a storage medium If by “perfectly fine” you mean “subject to attacks that generate somewhat targeted collisions that are practical enough that people do them for amusement and excuses to write blog posts and cute Twitter threads”, then maybe I agree. Snark aside, SHA-1 is not fine for deduplication in any context where an attacker could control any inputs. Do not use it for new designs. Try to get rid of it in old designs.

orev 4y ago
By “perfectly fine” they mean detecting duplicate image or document files on your local storage, which it’s still perfectly fine for, and a frequent mode of usage for these types of tools. Not every tool needs to be completely resilient to an entire Internets’ worth of attacks.
- dagenix 4y ago
  Why would you pick SHA1 which has significant attacks you need to consider as opposed to something like SHA2 that doesn't?
  - dylan604 4y ago
    who is attacking you in this situation though? you're scanning the files on your local system and storing their hashes. you then look for duplicate hashes, and compare the files that created them. if the files are truly duplicates, you can now decide what to do about that. if they are not truly the same, then you claim to have found another case of collisions, write your blog post/twitthread and move on, but does that constitute being attacked? sometimes, i really feel like people in crypto just can't detach themselves enough to see that just because they have a hammer, not everything in the world is a nail.
    - dagenix 4y ago
      Why would you pick a function that is known to have issues when there are other functions that do the same thing but don't have known issues? Your comparison is flawed. It's more like if you have a nail and next to it a workbench with two hammers - a good hammer and a not as good hammer. This isn't a hard choice. But for reasons that are unclear to me, people in this thread are insisting on picking the less good hammer and rationalizing why for this specific nail it isn't all that much worse. Just pick the better hammer!

dwheeler 4y ago

Git no longer uses SHA-1. It instead uses a variant called SHA-1DC that detects some known problems, and in those cases returns a different answer. More info: <https://github.com/cr-marcstevens/sha1collisiondetection https://github.com/cr-marcstevens/sha1collisiondetection>. Git switched to SHA-1DC in its version 2.13 release in 2017. It's a decent stopgap but not a grrat long term solution. There is also work to support SHA-256, though that seems to have stalled: https://lwn.net/Articles/898522/ https://lwn.net/Articles/898522/ The fundamental problem is that get developers assumed that hash algorithms would never be changed, and that was a ridiculous assumption. It's much wiser to implement crypto agility.

woodruffw 4y ago
> The fundamental problem is that get developers assumed that hash algorithms would never be changed, and that was a ridiculous assumption. It's much wiser to implement crypto agility. Cryptographic agility makes this problem worse, not better: instead of having a "flag day" (or release) where `git`'s digest choice reflects the State of the Art, agility ensures that every future version of `git` can be downgraded to a broken digest.
- est31 4y ago
  That's the general anti-agility argument wielded against git, but note that git's use cases require it to process historic data. E.g. you will want to be able to read some sha-1-only repo from disk that was last touched a decade ago. That's a different thing than some protocol which requires both parties to be on-line, say wireguard, in which instance it's easier to switch both to a new version that uses a different cryptographic algorithm. Git has such protocols as well, and maybe it can deprecate sha-1 support there eventually, but even there it has to support both sha-1 and sha-2 for a while because not everyone is using the latest and greatest version of git, and no sysadmin wants the absolute horror of flag days.
  - miga 4y ago
    It would be safer to forbid broken hashes after certain date, and consider only those earlier hashes that have been counter-signed by new algorithms.

hannob 4y ago

> SHA-1 is still perfectly fine for some applications like detecting duplicate files on a storage medium That's what the developers of subversion thought, but they didn't anticipate that once colliding files were available people would commit them to SVN repos as test cases. And then everything broke: https://www.bleepingcomputer.com/news/security/sha1-collision-attack-makes-its-first-victim-subversion-repositories/ https://www.bleepingcomputer.com/news/security/sha1-collisio...

TheRealPomax 4y ago
That changes the parameters quite a bit though. For local digests, like image deduplication of your own content, on your own computers, sha-1 is still perfectly fine. Heck, even MD5 is still workable (although more prone to collide). Nowhere in that process is the internet, or "users" or anything else like that involved =) You use digests to quickly detect potential collisions, then you verify each collision report, then you delete the actual duplicates. Human involvement still very much required because you're curating your own data.
- dylan604 4y ago
  If we're talking specifically image deduplication, then a hash comparison is only going to find you exact matches. what about the image deduplication of trying to find alt versions of things like scaling, different codecs, etc? if you want to dedupe images, some sort of phashing would be much better so that the actual image is considered vs just the specific bits to generate the image.
  - dreamcompiler 4y ago
    Very true but if the hash matches the images are guaranteed to match too. That's my first pass when deduping my drives. My second pass is looking for "logically equivalent" images.
  - TheRealPomax 4y ago
    Depends on the images. For photographs, a digest is enough. For "random images downloaded from the web", or when you're deduplicating lots of user's data, sure, you want data appropriate digests, like SIFT prints. But then we're back to "you had to bring network content back into this" =)

kiratp 4y ago

As an industry we need to get over this pattern of scoping down usage of something that has failed it’s prime directive. People still use MD5 in secure related things because it’s been allowed to stick around without huge deprecation warnings in libraries and tools. SHA1 (and MD5) need to be treated the same way you would treat O(n^2) sorting in a code review for a PR written by a newbie.

0xbadcafebee 4y ago

If you're only using the hash for non-cryptographic applications, there are much faster hashes: https://github.com/Cyan4973/xxHash https://github.com/Cyan4973/xxHash SHA1 and MD5 are the most widely accessible, though, and I agree it's fine to use them if you don't care about security.

bawolff 4y ago

But sha-1 is now both slow and insecure. If you dont care about security, use a faster hash. If you care about security use sha256 (which is about the same speed anyways). The only valid reason to still use it in non-security critical roles is backwards compat.

VLM 4y ago

“We recommend that anyone relying on SHA-1 for security migrate to SHA-2 or SHA-3 as soon as possible.” —Chris Celi, NIST computer scientist The emphasis being on "for security" I've also used SHA-1 over the years for binning and verifying file transfer success, none of those are security related. Sometimes, if you make a great big pile of different systems, what's held in common across them can be weird, SHA-1 popped out of the list so we used it. I'm well aware its possible to write or automate the writing of dedicated specialized "perfect" hashing algos to match the incoming data, to bin the data more perfectlyier, but sometimes its nice if wildly separate systems all bin incoming data the same highly predictable way thats "good enough" and "fast enough".

jraph 4y ago
> I've also used SHA-1 over the years for binning and verifying file transfer success, none of those are security related. It could. If you want to verify that the file has not been tempered by someone, it is security related.
- VLM 4y ago
  Prob a poor choice of words on my part. Verified as in "is this file completely transferred or not?" non-security critical data, I just want a general idea if its valid or the file transfer failed half way thru or the thing sending it went bonkers and just sent trash to us. Another funny file transfer use: Send me a file of data every hour. Is the non-crypto-hash new or the same old hash? If its the same old hash, those clowns sent me the same file twice, I'm supposed to get a new one. Yes I know I can dedupe "easily" but not as "easily" as sha-1. And some application layer software like MySQL can directly generate SHA1 as a function in the query. Its really quite handy sometimes!

LinuxBender 4y ago

SHA-1 is still perfectly fine for some applications like detecting duplicate files on a storage medium Absolutely agree, especially when speed is a workable trade-off and accepting real world hash collisions are unlikely and perhaps an acceptable risk. For financial data, especially files not belonging to me I would have md5+sha1+sha256 checksums and maybe even GPG sign a manifest of the checksums ... because why not. For my own files md5 has always been sufficient. I have yet to run into a real world collision. FWIW anyone using `rsync --checksum` is still using md5. Not that long ago I think 2014 it was using md4. I would be surprised if rsync started using anything beyond md5 any time soon. I would love to see all the checksum algorithms become CPU instruction sets. Optimizations: no SIMD-roll, no asm-roll, no openssl-crypto, asm-MD5 Checksum list: md5 md4 none Compress list: zstd lz4 zlibx zlib none Daemon auth list: md5 md4

tptacek 4y ago
You're never going to _accidentally_ run into an MD5 collision.

pornel 4y ago

Even if git didn't have protection against the known attack, it's still safe in practice. The SHA-1 collision attack can only work if you take a specially-crafted file from the attacker and commit it to your repository. The file needs to have a specific structure, and will contain binary data that looks like junk. It can't look like innocent source code. If you execute unintelligible binary blobs from strangers, you're in trouble anyway. There is no preimage weakness in SHA-1, so nobody is able to change or inject new data to an arbitrary repo/commit that doesn't already contain their colliding file.