5 ms·

I'm the person and Git developer (Ævar) quoted in the article. I didn't expect this to end up on LWN. I'm happy to answer any questions here that people might h

by avar 4y ago

I'm the person and Git developer (Ævar) quoted in the article. I didn't expect this to end up on LWN. I'm happy to answer any questions here that people might have. I don't think the LWN article can be said to take anything out of context. But I think it's worth empathizing that this is a thread on the Git ML in response to a user who's asking if Git/SHA-256 is something "that users should start changing over to[?]". I stand by the comments that I think the current state of Git is that we shouldn't be recommending to users that they use SHA-256 repositories without explaining some major caveats, mainly to do with third party software support, particularly the lack of support from the big online "forges". But I don't think there's any disagreement in the Git development community (and certainly not from me) that Git should be moving towards migrating away from SHA-1.

lalaland1125 4y ago
Have you considered moving over to a combined SHA-1, SHA-256 model where both hashes are calculated, with SHA-1 shown to the user and SHA-256 only used in the background to prevent collisions? There is a compute cost for that, but it should be minimal relative to the security benefits?
- pmarreck 4y ago
  This is what I would have recommended... A transition period where both are used
- avar 4y ago
  Someone probably brought it up at some point, I can't remember. But I'm not aware of any known scenario where the SHA1DC library Git uses doesn't give you the benefits of that and more. "And more" because to detect a collision with a background SHA-256 you'll need both objects, whereas SHA1DC detects attempts to spoof SHA1 in a way that leads to collisions. So it won't pass along an object that collides with another one, even though it only has 1/2 objects. That distinction is something that's generally considered important, e.g. there's been past exploits in Git where you could trick a client into doing something bad by e.g. a crafted .gitmodules file. The fix has not only been to patch clients, but also to patch "git fsck" to detect and reject such bad contents, so that e.g. the forges can't be used to relay a repository exploit to users running older versions. A viable hash collision exploit in the wild might likewise want to make use of such an attack scenarios, so having servers capable of detecting collisions without having both sides is preferable to doing so by re-hashing with SHA-256.
  - bostik 4y ago
    > SHA1DC detects attempts to spoof SHA1 in a way that leads to collisions This is the kind of detail I would have loved to see quoted directly in the article. Sure enough, it's prominently displayed on the project's README::about section, but your very succinct explanation here made it clear in immediate context. The idea of counter-cryptanalysis is eye-opening.
michaelt 4y ago
Thanks for your work on Git! > I'm happy to answer any questions here that people might have. Is there any way to achieve a gradual, staged rollout of SHA256? What's the impact of converting a repo to SHA256 - will old commit IDs become invalid? Would signed commits' signatures be invalidated?
- avar 4y ago
  The answer is somewhat hand-waivy, because this code doesn't exist as anything except out-of-tree WIP code (and even in that case, incomplete). But yes, the plan is definitely to support a gradual, hopefully mostly seamless rollout. The design document for that is shipped as part of git.git, and available online. Here's the relevant part: https://git-scm.com/docs/hash-function-transition/#_translation_table https://git-scm.com/docs/hash-function-transition/#_translat... Basically the idea is that you'd have a say a SHA-256 local repository, and talk to a SHA-1 upstream server. Each time you'd "pull" or "push" we'd "rehash" the content (which we do anyway, even when using just one hash). The interop-specific magic (covered in that documentation) is that we'd use a translation table, so you could e.g. "git show" on a SHA-1 object ID, and we'd be able to serve up the locally packed SHA-256 content as a result. But the hard parts of this still need to be worked out, and problems shaken out. E.g. for hosting providers what you get when you "git clone" is an already-hashed *.pack file that's mostly served up as-is from disk. For simultaneously serving clients of both hash formats you'd essentially need to double your storage space. There's also been past in-person developer meet-up discussion (the last one being before Covid, the next one in fall this year) about the gritty details of how such a translation table will function exactly. E.g. if linux.git switches they'd probably want a "flag day" where they'd transition 100% to SHA-256, but many clients would still probably want the SHA-1<->SHA-256 translation table kept around for older commits, to e.g. look up hash references from something like the mailing list archive, or old comments in ticketing systems. Currently the answer to how that'll work exactly is that we'll see when someone submits completed patches for that sort of functionality, and doubtless issues & edge cases will emerge that we didn't or couldn't expect until the rubber hits the road.
  - dwheeler 4y ago
    Has anyone considered doing "add16" on the first character of the sha-256 hash, e.g., so the SHA-256 hash 1d06... becomes hd06... ? Then you could see, from the first character, if it is SHA-1 or SHA-256. Having a clear distinction on the first character would make it clear which hash is being used (without needing lots of chars).
tux3 4y ago
Has there been any feedback/communication with forges happening, on or off-list? I'm curious how closely (if at all) they've been following this effort
- avar 4y ago
  The forges are keenly aware of this effort, e.g. the person who's by far done most of the work on the SHA-256 transition (brian m. carlson) has I believe mostly or entirely been done so on behalf of GitHub. I myself do some work on upstream Git on behalf of GitLab, although none of it's been on anything related to the SHA-256 transition. As to why no big forge has SHA-256 support, I think it's a bit of a chicken & egg problem (and these comments are entirely my own, and not on behalf of anyone). I think it's safe to say that all of the forges are expecting the transition, e.g. I don't think there's anyone creating CHAR(40) database tables for Git hashes anymore (or if they are, someone is planning to deal with it). Another is that for a successful transition for anything except entirely new repository networks (which already use SHA-1) you really need the "git" client to play along, see my other comment discussing hash interop plans. Some of that same code then needs to run on the server-side. That code isn't part of git yet, and it's really needed for any sort of viable migration plan. I mean, it's not really needed, at some point a lot of people reading this migrated from CVS and/or SVN to Git. But a full export/import with a lot of users is painful. We really want it to suck less for Git, to the point that it should Just Work for most or all users. And a major one is the human factor. For things to happen in free software development someone needs to submit patches, brian m. carlson has been performing a heroic amount of effort over the year on the transition over the years. As he notes in the linked ML thread he's had life reasons for why he hasn't been able to work on it as actively recently as he did in the past.
  - rurban 4y ago
    brian moved from Texas to Canada, but mostly his employee, GitHub, is not priotizing the remaining sha256 patches. someone needs to finish up the transition patches, and forges need to double their disc space.
    - 4y ago

by avar 4y ago

lalaland1125 4y ago

Have you considered moving over to a combined SHA-1, SHA-256 model where both hashes are calculated, with SHA-1 shown to the user and SHA-256 only used in the background to prevent collisions? There is a compute cost for that, but it should be minimal relative to the security benefits?

pmarreck 4y ago
This is what I would have recommended... A transition period where both are used
avar 4y ago
Someone probably brought it up at some point, I can't remember. But I'm not aware of any known scenario where the SHA1DC library Git uses doesn't give you the benefits of that and more. "And more" because to detect a collision with a background SHA-256 you'll need both objects, whereas SHA1DC detects attempts to spoof SHA1 in a way that leads to collisions. So it won't pass along an object that collides with another one, even though it only has 1/2 objects. That distinction is something that's generally considered important, e.g. there's been past exploits in Git where you could trick a client into doing something bad by e.g. a crafted .gitmodules file. The fix has not only been to patch clients, but also to patch "git fsck" to detect and reject such bad contents, so that e.g. the forges can't be used to relay a repository exploit to users running older versions. A viable hash collision exploit in the wild might likewise want to make use of such an attack scenarios, so having servers capable of detecting collisions without having both sides is preferable to doing so by re-hashing with SHA-256.
- bostik 4y ago
  > SHA1DC detects attempts to spoof SHA1 in a way that leads to collisions This is the kind of detail I would have loved to see quoted directly in the article. Sure enough, it's prominently displayed on the project's README::about section, but your very succinct explanation here made it clear in immediate context. The idea of counter-cryptanalysis is eye-opening.

michaelt 4y ago

Thanks for your work on Git! > I'm happy to answer any questions here that people might have. Is there any way to achieve a gradual, staged rollout of SHA256? What's the impact of converting a repo to SHA256 - will old commit IDs become invalid? Would signed commits' signatures be invalidated?

avar 4y ago
The answer is somewhat hand-waivy, because this code doesn't exist as anything except out-of-tree WIP code (and even in that case, incomplete). But yes, the plan is definitely to support a gradual, hopefully mostly seamless rollout. The design document for that is shipped as part of git.git, and available online. Here's the relevant part: https://git-scm.com/docs/hash-function-transition/#_translation_table https://git-scm.com/docs/hash-function-transition/#_translat... Basically the idea is that you'd have a say a SHA-256 local repository, and talk to a SHA-1 upstream server. Each time you'd "pull" or "push" we'd "rehash" the content (which we do anyway, even when using just one hash). The interop-specific magic (covered in that documentation) is that we'd use a translation table, so you could e.g. "git show" on a SHA-1 object ID, and we'd be able to serve up the locally packed SHA-256 content as a result. But the hard parts of this still need to be worked out, and problems shaken out. E.g. for hosting providers what you get when you "git clone" is an already-hashed *.pack file that's mostly served up as-is from disk. For simultaneously serving clients of both hash formats you'd essentially need to double your storage space. There's also been past in-person developer meet-up discussion (the last one being before Covid, the next one in fall this year) about the gritty details of how such a translation table will function exactly. E.g. if linux.git switches they'd probably want a "flag day" where they'd transition 100% to SHA-256, but many clients would still probably want the SHA-1<->SHA-256 translation table kept around for older commits, to e.g. look up hash references from something like the mailing list archive, or old comments in ticketing systems. Currently the answer to how that'll work exactly is that we'll see when someone submits completed patches for that sort of functionality, and doubtless issues & edge cases will emerge that we didn't or couldn't expect until the rubber hits the road.
- dwheeler 4y ago
  Has anyone considered doing "add16" on the first character of the sha-256 hash, e.g., so the SHA-256 hash 1d06... becomes hd06... ? Then you could see, from the first character, if it is SHA-1 or SHA-256. Having a clear distinction on the first character would make it clear which hash is being used (without needing lots of chars).

tux3 4y ago

Has there been any feedback/communication with forges happening, on or off-list? I'm curious how closely (if at all) they've been following this effort

avar 4y ago
The forges are keenly aware of this effort, e.g. the person who's by far done most of the work on the SHA-256 transition (brian m. carlson) has I believe mostly or entirely been done so on behalf of GitHub. I myself do some work on upstream Git on behalf of GitLab, although none of it's been on anything related to the SHA-256 transition. As to why no big forge has SHA-256 support, I think it's a bit of a chicken & egg problem (and these comments are entirely my own, and not on behalf of anyone). I think it's safe to say that all of the forges are expecting the transition, e.g. I don't think there's anyone creating CHAR(40) database tables for Git hashes anymore (or if they are, someone is planning to deal with it). Another is that for a successful transition for anything except entirely new repository networks (which already use SHA-1) you really need the "git" client to play along, see my other comment discussing hash interop plans. Some of that same code then needs to run on the server-side. That code isn't part of git yet, and it's really needed for any sort of viable migration plan. I mean, it's not really needed, at some point a lot of people reading this migrated from CVS and/or SVN to Git. But a full export/import with a lot of users is painful. We really want it to suck less for Git, to the point that it should Just Work for most or all users. And a major one is the human factor. For things to happen in free software development someone needs to submit patches, brian m. carlson has been performing a heroic amount of effort over the year on the transition over the years. As he notes in the linked ML thread he's had life reasons for why he hasn't been able to work on it as actively recently as he did in the past.
- rurban 4y ago
  brian moved from Texas to Canada, but mostly his employee, GitHub, is not priotizing the remaining sha256 patches. someone needs to finish up the transition patches, and forges need to double their disc space.
  - 4y ago