4 ms·
SHA1 is still OK for identifying files. The probability of random collision is still very low. The only problem is for forged collisions. The object store coul
by chmike 10y ago
SHA1 is still OK for identifying files. The probability of random collision is still very low. The only problem is for forged collisions.
The object store could be modified to support file collision. One way to disambiguate collision is to use a randomly generated byte sequence as SHA1 seed or hashed before the file data. This random byte sequence would behave like a salt and disable any forged collisions. A single seed for the whole repository would be enough. It should remain secret to prevent forging a collision with the two hashes. It's harder but not impossible.
To test if a given file is in the object store, one first compute the SHA1 key to use as file name. If no collisions ever occured with an object a file with that SHA1 name will be present in the store. That file contains the usual data plus the second hash computed with the random seed. This second hash could be added as needed to keep backward compatibility and provide silent automatic upgrade.
When one need to test if the file is present in the store, one computes the normal SHA1 key and the secondary hash with the seed. We locate the object in the store uisng the first SHA and test for file equality with the randomly seeded hash.
Using a faster hash like blacke2 to compute the random seeded hash could mitigate the price to compute two hashes. It should be parameterized this time and the hash size should be variable.
If a collision is detected, that is the secondary hash differ, the file is replaced by a directory with the common SHA1 as name. The colliding files would be stored in the directory using the secondary hash as name. Or the files could be packed in a single tar like file with the secondary hash used as file identifier.
This should be enough to protect against forged collisions which is the only real problem. The required change to git would be limited. The only serious disadvantage is the need to compute the randomly seeded hash.