5 ms·

I feel like the properties of CRC make them superior for that task in most cases though. (CRC8, CRC16, CRC32 and CRC64, maybe CRC128 if anyone ever bothered goi

I feel like the properties of CRC make them superior for that task in most cases though. (CRC8, CRC16, CRC32 and CRC64, maybe CRC128 if anyone ever bothered going that far) In particular, CRC guarantees detection on all bursts of the given length. CRC32 protects vs all bursts of length 32 bits.

AlotOfReading 4y ago
There are only a few examples of anything larger than CRC64 being characterized and they're not very useful. For the sake of the next person who has to maintain your code though, please choose algorithms that adequately communicate your intentions. Choose CRCs only if you need to detect random errors in a noisy channel with a small number of bits and use a length appropriate to the intended usage (i.e. almost certainly not CRC64).
- dragontamer 4y ago
  And when you choose SHA1, does it mean you understood that it's no longer secure? Or is it chosen because it was secure 20 years ago but the code is old and needs to be updated? CRC says that you never intended security from the start. It's timeless, aimed to prevent burst errors and random errors. -------- BTW, what is the guaranteed Hamming distance between SHA1? How good is SHA1 vs burst errors? What about random errors? Because the Hamming distances of CRC have been calculated and analyzed. We actually can determine, to an exact level, how good CRC codes are.
  - dagenix 4y ago
    If you are choosing between a CRC and SHA1, you probably need to reconsider your understanding of the problem you are trying to solve. Those algorithms solve different use cases.
    - dragontamer 4y ago
      If you are choosing SHA1, now that it is retired, you probably should think harder about the problem in general. CRC should be better for any error detection code issue. Faster to calculate, more studied guaranteed detection modes, and so forth. SHA1 has no error detection studies. It's designed as a cryptographic hash, to look random. As it so happens, it is more efficient to use other algorithms and do better than random if you have a better idea of how your error looks like. Real world errors are either random or bursty. CRC is designed for these cases. CRC detects the longest burst possible for it's bitsize.
  - AlotOfReading 4y ago
    You shouldn't choose SHA-1, that's the point of this announcement. Seeing it indicates both that there was the potential for malicious input and that the code is old. The appropriate mitigation is to move to a secure hash, not CRCs. You may not know the bounds and distances exactly, but you know them probabilistically. Bit errors almost always map to a different hash. The same is true of CRCs over a large enough input as an aside.
  - benlivengood 4y ago
    https://eprint.iacr.org/2019/078 https://eprint.iacr.org/2019/078 is a good overview of the statistical properties of SHA hashes. Basically after ~10 rounds the output is always indistinguishable from randomness which means hamming distance is what you'd expect (about half the bits differ) between the hashes of any two bitstreams.
    - dragontamer 4y ago
      CRC does better than random vs burst errors. And vs random errors is indistinguishable from any other methodology. If you are only worried about random errors, might as well use chksum.
IncRnd 4y ago
> I feel like the properties of CRC make them superior for that task in most cases though. THIS IS FALSE. Please do not ever do this. Why not? For example, by controlling any four contiguous bytes in a file, the resultant 32bit CRC can be forced to take on any value. A CRC is meant to detect errors due to noise - not changes due to a malicious actor. Program design should not be done based upon one's feelings. CRCs absolutely do not have the required properties to detect duplication or to preserve integrity of a stored file that an attacker can modify.
- dragontamer 4y ago
  > THIS IS FALSE. Please do not ever do this. Why not? For example, by controlling any four contiguous bytes in a file, the resultant 32bit CRC can be forced to take on any value. And SHA1 is now broken like this, with collisions and so forth. Perhaps it's not as simple as just 4 bytes, but the ability to create collisions is forcing this retirement. If adversarial collisions are an issue, then MD5 and SHA1 are fully obsolete now. If you don't care for an adversary, might as well use the cheaper, faster CRC check. ------ CRC is now more valid use case than SHA1. That's the point of this announcement.
  - IncRnd 4y ago
    > CRC is now more valid use case than SHA1. That's the point of this announcement. No. That isn't the point of this announcement. This announcement codifies a transition time for SHA-1 use in SBU and has nothing to do with a CRC.
    - Dylan16807 4y ago
      The point of the announcement is to give a timeline to start treating SHA1 as having no real security. This is relevant to when you'd choose CRC, because it also has no real security.
      - IncRnd 4y ago
        > The point of the announcement is to give a timeline to start treating SHA1 as having no real security. That's also false. There is a large body of knowledge here that you aren't expressing in your comments. That leads me to see that you are unfamiliar with the purposes of hash functions and their utility in real world situations. The announcement refers to the transition timeline to stop using SHA-1, preferring the SHA-2 and SHA-3 families. However, the recommendations for years from NIST have been not to use SHA-1. For example SP 800-131Ar2 discusses not to use SHA-1 for digital sig gen and that digital sig ver is only acceptable for legacy uses. The recommendation would have been for years to not use SHA-1 at all, except for this carve-out to handle already stored data that uses SHA-1. The remaining use cases cover protocol use, such as TLS, where SHA-1 is used as a component in constructs and not solely as a primitive.

AlotOfReading 4y ago

There are only a few examples of anything larger than CRC64 being characterized and they're not very useful. For the sake of the next person who has to maintain your code though, please choose algorithms that adequately communicate your intentions. Choose CRCs only if you need to detect random errors in a noisy channel with a small number of bits and use a length appropriate to the intended usage (i.e. almost certainly not CRC64).

dragontamer 4y ago
And when you choose SHA1, does it mean you understood that it's no longer secure? Or is it chosen because it was secure 20 years ago but the code is old and needs to be updated? CRC says that you never intended security from the start. It's timeless, aimed to prevent burst errors and random errors. -------- BTW, what is the guaranteed Hamming distance between SHA1? How good is SHA1 vs burst errors? What about random errors? Because the Hamming distances of CRC have been calculated and analyzed. We actually can determine, to an exact level, how good CRC codes are.
- dagenix 4y ago
  If you are choosing between a CRC and SHA1, you probably need to reconsider your understanding of the problem you are trying to solve. Those algorithms solve different use cases.
  - dragontamer 4y ago
    If you are choosing SHA1, now that it is retired, you probably should think harder about the problem in general. CRC should be better for any error detection code issue. Faster to calculate, more studied guaranteed detection modes, and so forth. SHA1 has no error detection studies. It's designed as a cryptographic hash, to look random. As it so happens, it is more efficient to use other algorithms and do better than random if you have a better idea of how your error looks like. Real world errors are either random or bursty. CRC is designed for these cases. CRC detects the longest burst possible for it's bitsize.
- AlotOfReading 4y ago
  You shouldn't choose SHA-1, that's the point of this announcement. Seeing it indicates both that there was the potential for malicious input and that the code is old. The appropriate mitigation is to move to a secure hash, not CRCs. You may not know the bounds and distances exactly, but you know them probabilistically. Bit errors almost always map to a different hash. The same is true of CRCs over a large enough input as an aside.
- benlivengood 4y ago
  https://eprint.iacr.org/2019/078 https://eprint.iacr.org/2019/078 is a good overview of the statistical properties of SHA hashes. Basically after ~10 rounds the output is always indistinguishable from randomness which means hamming distance is what you'd expect (about half the bits differ) between the hashes of any two bitstreams.
  - dragontamer 4y ago
    CRC does better than random vs burst errors. And vs random errors is indistinguishable from any other methodology. If you are only worried about random errors, might as well use chksum.

IncRnd 4y ago

> I feel like the properties of CRC make them superior for that task in most cases though. THIS IS FALSE. Please do not ever do this. Why not? For example, by controlling any four contiguous bytes in a file, the resultant 32bit CRC can be forced to take on any value. A CRC is meant to detect errors due to noise - not changes due to a malicious actor. Program design should not be done based upon one's feelings. CRCs absolutely do not have the required properties to detect duplication or to preserve integrity of a stored file that an attacker can modify.

dragontamer 4y ago
> THIS IS FALSE. Please do not ever do this. Why not? For example, by controlling any four contiguous bytes in a file, the resultant 32bit CRC can be forced to take on any value. And SHA1 is now broken like this, with collisions and so forth. Perhaps it's not as simple as just 4 bytes, but the ability to create collisions is forcing this retirement. If adversarial collisions are an issue, then MD5 and SHA1 are fully obsolete now. If you don't care for an adversary, might as well use the cheaper, faster CRC check. ------ CRC is now more valid use case than SHA1. That's the point of this announcement.
- IncRnd 4y ago
  > CRC is now more valid use case than SHA1. That's the point of this announcement. No. That isn't the point of this announcement. This announcement codifies a transition time for SHA-1 use in SBU and has nothing to do with a CRC.
  - Dylan16807 4y ago
    The point of the announcement is to give a timeline to start treating SHA1 as having no real security. This is relevant to when you'd choose CRC, because it also has no real security.
    - IncRnd 4y ago
      > The point of the announcement is to give a timeline to start treating SHA1 as having no real security. That's also false. There is a large body of knowledge here that you aren't expressing in your comments. That leads me to see that you are unfamiliar with the purposes of hash functions and their utility in real world situations. The announcement refers to the transition timeline to stop using SHA-1, preferring the SHA-2 and SHA-3 families. However, the recommendations for years from NIST have been not to use SHA-1. For example SP 800-131Ar2 discusses not to use SHA-1 for digital sig gen and that digital sig ver is only acceptable for legacy uses. The recommendation would have been for years to not use SHA-1 at all, except for this carve-out to handle already stored data that uses SHA-1. The remaining use cases cover protocol use, such as TLS, where SHA-1 is used as a component in constructs and not solely as a primitive.