4 ms·

Excuse my ignorance, but couldn’t they just add a SHA256 hash to commit objects (or some new commit-verify object) of the entire trees current concatenated cont

by donatj 7y ago

Excuse my ignorance, but couldn’t they just add a SHA256 hash to commit objects (or some new commit-verify object) of the entire trees current concatenated content, leave everything else SHA1 and get the same benefit without rewriting the entire thing from the ground up? Git could even do that as part of the git gc step slowly over time - tag commits with a secondary hash. Rewriting the whole thing including every git repos history seems like throwing the baby out with the bathwater, when you could just add a secondary transparent verification instead. Just seems like there has to be a better way.

kzrdude 7y ago
Hashing everything in one go doesn't scale well. When making a new commit you want to only hash a proportional part of the repository, and the tree structure of git allows that, only the files and "tree" objects (directory listings) that change are hashed again.
wongarsu 7y ago
You can't change past commits to add that hash (without changing all commit hashes), so this method could only protect new commits. For any existing repo this would lead to a very weird security model: We admit that sha1 hashes are broken, and only guarantee that commits made by git versions newer than git x.x.x are safe from after-the-fact modification (or alternatively only commits made after date X).
- deleted 7y ago
  [deleted]
- arve0 7y ago
  I’m not known with the internal data structure of git, but couldn’t you add the new hash as a commit in a new format “on the side”, leaving the original commit as is?
  - WorldMaker 7y ago
    Git does have an commit-related object called a note that you can attach as a separate object. [1] Presumably the proposed "hash translation store" could use an approach similar to notes, and include the hash translations as objects in the git database (hopefully in a way that could be signed by a tag). [1] http://alblue.bandlem.com/2011/11/git-tip-of-week-git-notes.html http://alblue.bandlem.com/2011/11/git-tip-of-week-git-notes....
  - loeg 7y ago
    This is kind of what rewriting the repo is. Yes, you could leave the SHA1 commit tree around afterwards (i.e., for convenience of existing URLs), but you wouldn't want to keep SHA1 around as the authoritative hashname.
- gregmac 7y ago
  My inclination is that protecting only new commits might be enough, but it gets me thinking: What would a practical attack on this look like, assuming sha1 was broken? Let's say I'm trying to insert a line of code that does something nefarious, and that it's now trivial to generate "magic text" you can stick anywhere in a file (eg, inside a comment at the end of a line) to get any desired sha1 hash. Are all the other future commits still valid, or am I going to suddenly get conflicts or garbled text? Depending on where the modification is done, that code might have gone through much more churn -- especially if there are a bunch of sha-256 commits after it (which I can't attack). I don't know enough about how git stores content blobs to answer this. Second problem: Can I push my replacement commit to another repository (eg, github)? Would even force push work? Do I have to delete branches and re-push my own? If I already have enough permission on the repository to do this, it means I can already push whatever I want -- so does this attack even matter at all? Assuming that's successful (or I can trick people into using my own repository), what will happen to someone that already has a clone and does a pull? Will they get my change (and will it work or be a pile of conflicts or garbled text)? Even if only fresh clones will get the changes it could still be quite devastating -- especially if using CI -- but I'm just not clear if this attack is even theoretically possible.
  - OJFord 7y ago
    > My inclination is that protecting only new commits might be enough Why? It's not the same as saying 'versions after vX are safe', it's the same as saying 'any unsafety after vX was there before, not introduced since' (both with 'as a result of SHA-1 collision' qualifiers of course). > Can I push my replacement commit to another repository (eg, github)? Would even force push work? Implementation dependent I suppose, but I wouldn't have thought so - I don't see why they'd actually check the content when the hash is supposed to indicate whether it differs or not. > Do I have to delete branches and re-push my own? If I already have enough permission on the repository to do this, it means I can already push whatever I want -- so does this attack even matter at all? I think an attack would look more like: 1. Create hostile commit that collides with extant commit SHA 2. Infiltrate a package repository, or GitHub, or corporate network, or ... 3. Insert hostile commit in place of real one Of course it's a problem if 2 & 3 happen alone anyway, but the problem with the collision commit is that it makes it so much less detectable.
- tzs 7y ago
  Couldn't they make a table that contains a list of all the old objects by SHA1 hash, for for each contains the new SHA256 hash of that object, and then commit this table in the repository?
  - loeg 7y ago
    Yep.
- jayd16 7y ago
  What if we use the exploit to add the new data but keep the sha1 the same? :)
- deleted 7y ago
  [deleted]

by donatj 7y ago

kzrdude 7y ago

Hashing everything in one go doesn't scale well. When making a new commit you want to only hash a proportional part of the repository, and the tree structure of git allows that, only the files and "tree" objects (directory listings) that change are hashed again.

wongarsu 7y ago

You can't change past commits to add that hash (without changing all commit hashes), so this method could only protect new commits. For any existing repo this would lead to a very weird security model: We admit that sha1 hashes are broken, and only guarantee that commits made by git versions newer than git x.x.x are safe from after-the-fact modification (or alternatively only commits made after date X).

deleted 7y ago
[deleted]
arve0 7y ago
I’m not known with the internal data structure of git, but couldn’t you add the new hash as a commit in a new format “on the side”, leaving the original commit as is?
- WorldMaker 7y ago
  Git does have an commit-related object called a note that you can attach as a separate object. [1] Presumably the proposed "hash translation store" could use an approach similar to notes, and include the hash translations as objects in the git database (hopefully in a way that could be signed by a tag). [1] http://alblue.bandlem.com/2011/11/git-tip-of-week-git-notes.html http://alblue.bandlem.com/2011/11/git-tip-of-week-git-notes....
- loeg 7y ago
  This is kind of what rewriting the repo is. Yes, you could leave the SHA1 commit tree around afterwards (i.e., for convenience of existing URLs), but you wouldn't want to keep SHA1 around as the authoritative hashname.
gregmac 7y ago
My inclination is that protecting only new commits might be enough, but it gets me thinking: What would a practical attack on this look like, assuming sha1 was broken? Let's say I'm trying to insert a line of code that does something nefarious, and that it's now trivial to generate "magic text" you can stick anywhere in a file (eg, inside a comment at the end of a line) to get any desired sha1 hash. Are all the other future commits still valid, or am I going to suddenly get conflicts or garbled text? Depending on where the modification is done, that code might have gone through much more churn -- especially if there are a bunch of sha-256 commits after it (which I can't attack). I don't know enough about how git stores content blobs to answer this. Second problem: Can I push my replacement commit to another repository (eg, github)? Would even force push work? Do I have to delete branches and re-push my own? If I already have enough permission on the repository to do this, it means I can already push whatever I want -- so does this attack even matter at all? Assuming that's successful (or I can trick people into using my own repository), what will happen to someone that already has a clone and does a pull? Will they get my change (and will it work or be a pile of conflicts or garbled text)? Even if only fresh clones will get the changes it could still be quite devastating -- especially if using CI -- but I'm just not clear if this attack is even theoretically possible.
- OJFord 7y ago
  > My inclination is that protecting only new commits might be enough Why? It's not the same as saying 'versions after vX are safe', it's the same as saying 'any unsafety after vX was there before, not introduced since' (both with 'as a result of SHA-1 collision' qualifiers of course). > Can I push my replacement commit to another repository (eg, github)? Would even force push work? Implementation dependent I suppose, but I wouldn't have thought so - I don't see why they'd actually check the content when the hash is supposed to indicate whether it differs or not. > Do I have to delete branches and re-push my own? If I already have enough permission on the repository to do this, it means I can already push whatever I want -- so does this attack even matter at all? I think an attack would look more like: 1. Create hostile commit that collides with extant commit SHA 2. Infiltrate a package repository, or GitHub, or corporate network, or ... 3. Insert hostile commit in place of real one Of course it's a problem if 2 & 3 happen alone anyway, but the problem with the collision commit is that it makes it so much less detectable.
tzs 7y ago
Couldn't they make a table that contains a list of all the old objects by SHA1 hash, for for each contains the new SHA256 hash of that object, and then commit this table in the repository?
- loeg 7y ago
  Yep.
jayd16 7y ago
What if we use the exploit to add the new data but keep the sha1 the same? :)
deleted 7y ago
[deleted]