20 ms·

Whatever happened to SHA-256 support in Git?

455 points by chmaynard 4y ago

kerblang 4y ago
The part of software engineering they don't teach in college is migration. Some of the most creative work you'll do is figuring out how to get from X to Y without bringing everything crashing down around you (or at least only a couple things crashing down at a time).
TillE 4y ago
Bjarmason has a good response about the practicalities of an attack; it explains why a "broken" hash is rarely a running-around-with-your-hair-on-fire level emergency. It would clearly be better to use a better hash, but is it actually urgent for anyone? Probably not.
corbet 4y ago
It's nice to see LWN on HN for the second time in one day, but please remember: it is only LWN subscribers that make this kind of writing possible. If you are enjoying it, please consider becoming a subscriber yourself — or, even better, getting your employer to subscribe.
- jra_samba 4y ago
  Just want to second this ! Please subscribe to lwn. I learn new things from lwn every week. It's really worth the money.
- O__________O 4y ago
  For easy of reference, here is the link to subscribe, which includes a description of the benefits: https://lwn.net/subscribe/Info https://lwn.net/subscribe/Info And the Wikipedia page for LWN, if you’re not familiar with it: https://en.m.wikipedia.org/wiki/LWN.net https://en.m.wikipedia.org/wiki/LWN.net
- cockhole_desu 4y ago
- williadc 4y ago
  Googlers can subscribe through work by visiting go/lwn and following the instructions.
- bloopernova 4y ago
  subbed, thanks for the reminder!
bradhe 4y ago
Does the usage of SHA-1 in Git actually have security implications, though? It's basically only used to generate addresses for refs and hunks and all that.
- saghm 4y ago
  I don't think it does; sure, someone could potentially craft a malicious commit that causes a SHA1 collision in your repo, but I think if you are merging commits from malicious authors, you've got way bigger problems than that.
  - corbet 4y ago
    ...and if you're merging commits from a developer who, unknown to either of you, had their laptop compromised and their repo corrupted? Remember that the compromise of kernel.org happened via a developer's laptop, and it was only the security of the hash chains that preserved confidence in the repositories stored there. As noted in the article, an SHA-1 collision attack does not appear practical now, but that is a situation that can change.
  - shakna 4y ago
    GitHub actually makes pull requests available as an unlisted part of the original repository under refs/pull/$PR/head and refs/pull/$PR/merge, which allows a malicious author to add themselves to your index, without your involvement. Not to say that this attack is in any way practical, yet. Just that some providers don't require active involvement to try and attempt it.
- mmastrac 4y ago
  If a repo accepts third-party contributions, you can create a split brain where half the people see one set of contents and the others see a different set, but the same hashes are available. I don't know if this would survive additional commits on top as I'm not familiar enough with git's internals.
  - progval 4y ago
yjftsjthsd-h 4y ago
> In his view, the only "defensible" reason to use SHA-1 at this point is interoperability with the Git forge providers. Okay, but that's a pretty big reason! A git repo that can't be pushed to github/lab is... not always useless, but certainly extremely impaired.
- kragen 4y ago
  In case anyone has forgotten, the process for pushing it to your own server is three shell commands. You run, on the server: git init --bare public_html/mything.git cd public_html/mything.git/hooks/ mv post-update.sample post-update # runs git update-server-info on push (This assumes that your public_html directory exists and is mapped into webspace, as with the usual configuration of Apache, NCSA httpd, and CERN httpd. If you don't have an account on such a thing you can get such PHP shared hosting accounts with shell access anywhere in the world for a dollar or two a month.) And then on your dev machine, it's precisely the same as for pushing to Gitlab or whatever, except that you use your own username instead of git@: git remote add someremotename user@myserver:public_html/mything.git git push -u someremotename master # assuming you want it to be your upstream Then anyone can clone from your repo with a command like this: git clone https://myserver/~user/mything.git They can also add the URL as a remote for pulls. If you want them to be able to push, you'll need to give them an account on the same server and either set umasks and group ownerships and permissions appropriately or set a POSIX ACL. Alternatively they can do the same thing on their server and you can pull from it. There are reportedly permission bugs in recent versions of Git (the last five years) that prevent this from being safe with people you don't trust (https://www.spinics.net/lists/git/msg298544.html https://www.spinics.net/lists/git/msg298544.html). Of course source control is only part of the overall development project workflow, so for many purposes adding SHA-256 support to Gogs or Gitlab or Gitea or sr.ht is probably pretty important: you want a Wiki and CI integration and bug tracking and merge requests. But the git repo still works fine with a bog-standard ssh and HTTP server, though slightly less efficiently. It's easier than setting up a new repo on GitLab etc. Running a git repack -an && git update-server-info in the repo on the server can help a lot with the efficiency, and for having a browseable tree on the server as well as a clonable repo I put this script at http://canonical.org/~kragen/sw/dev3.git/hooks/post-update http://canonical.org/~kragen/sw/dev3.git/hooks/post-update: #!/bin/sh set -e echo -n 'updating... ' git update-server-info echo 'done. going to dev3' cd /home/kragen/public_html/sw/dev3 echo -n 'pulling... ' env -u GIT_DIR git pull echo -n 'updating... ' env -u GIT_DIR git update-server-info echo 'done.' That's very far from being GitLab (contrast http://canonical.org/~kragen/sw/dev3 http://canonical.org/~kragen/sw/dev3 with any GitHub tree view), and it's potentially dangerously powerful: if you're doing this in a repo where you pull from other people, and the server is configured to run PHP files or server-side includes in your webspace (mine isn't!) or CGI scripts (mine is!), then just dropping a file in the repo can run programs on the server with your account privileges. This is great if that's what you want, and it's a hell of a lot better than updating your PHP site over FTP, but that code has full authority to, for example, rewrite your Git history. In theory you can do other things from your post-update hook as well, like rebuild a Jekyll site, send a message on IRC or some other message queueing system, or fire off a CI build in a Docker container. (Some of these would run afoul of guardrails common in cheap PHP shared hosting providers and you'd have to upgrade to a US$5/month VPS.)
armada651 4y ago
> Adding my own 0.02, what some of us are facing is resistance to adopting git in our or client organizations because of the presence of SHA-1. There are organizations where SHA-1 is blanket banned across the board - regardless of its use. [...] Getting around this blanket ban is a serious amount of work and I have very recently seen customers move to older much less functional (or useful) VCS platforms just because of SHA-1. Seems like this company could just use the current SHA-256 support then? Especially if it's the type of company that does all its development in-house and there's no need for SHA-1 interoperability.
- cratermoon 4y ago
  Except said company likely uses one of the Git forge providers, either in-house or as a SaaS, as the (oxymoronic for git) central repo. Until they support SHA-256, or the company goes with a its own git repo solution that is set up for it, companies won't make the move.
  - wepple 4y ago
    Not just git forge but probably the myriad other ancillary tools that assume SHA1
- gorkish 4y ago
  > There are organizations where SHA-1 is blanket banned across the board - regardless of its use. > I have very recently seen customers move to older much less functional (or useful) VCS platforms just because of SHA-1. A company this dysfunctional has problems far beyond their choice of revision control system.
  - bostik 4y ago
    I can name a couple of industries where compliance (and their enforcement arm, security[0]) teams require N+1 different monitoring and enforcement agents on all systems because Compliance[TM]. Due to these agents the systems' IDLE load is approaching 1.00 - on a good day. On a less good you need four cores to have one of them available for workload processing. 0: I use the word "security" only because the teams themselves are named like that. You can probably infer my opinion from the tone.
WorldMaker 4y ago
Almost feels like by the time git finally transitions to SHA-256 some bitcoin miner somewhere will have a solved preimage weakness on SHA-256.
- jagger27 4y ago
  Thankfully existing Bitcoin ASICs don’t pose much of a threat because they’re only good for sha256(sha256(Bitcoin block)). If a practical pre-image attack on SHA-256 comes around we have bigger problems than git.
  - WorldMaker 4y ago
    Obviously the concern is not the ASICs themselves but the ASIC designers. (Using miners here in the colloquial sense of human collectives/corporations backing the machines than than the specific sense of the raw machines themselves.) Yes, a practical preimage weakness in SHA-256 is a nightmare scenario with huge implications to the rest of internet security beyond just get. It's why I sometimes can't sleep at night knowing how much energy bitcoin spends daily on a continuous massively distributed partial preimage attack on SHA-256.
    - marktangotango 4y ago
      > how much energy bitcoin spends daily on a continuous massively distributed partial preimage attack on SHA-256. I would not be concerned about this. The way the asics operate is they discard the results. Also, the hashes are random strings which don't compress very well, so storing trillions upon trillions of them (for later analysis) is not practical.
      - WorldMaker 4y ago
        Again, the point of the fear is not the specifics of current operations (ASIC details; which y'all are talking about as if all of the miners are using the same hardware), but the fear of future operations and that there's an enormous industrial preimage attack effort at all. One that we can see in real time, in global energy consumption graphs. Maybe you find "cold comfort" that because we can watch it in real time if someone discovers a weakness we will also watch its repercussions and the subsequent horrifying fall in real time, too, but I certainly don't.
mdavidn 4y ago
I don’t depend on the collision resistance of SHA-1 for the security of my git repos because I don’t accept pushes from people I don’t trust. If I did, objects with hash collisions would not be transferred or (I hope) accepted. Am I missing something? Granted, signed tags do depend on this collision resistance, but I don’t use that feature. Signing entire releases from a trusted repo seems like a better approach.
- teraflop 4y ago
  It's not just the pushes themselves; anyone who can create commits or blobs that eventually get merged into your repository, directly or indirectly, can potentially engage in a collision attack. Sure, if you use git with a very closed development model, this doesn't necessarily affect you much. But it's (potentially) a big problem for collaborative open-source projects, because it requires trust in every single contributor. And the trust requirement can't necessarily be mitigated using ordinary means like code reviews.
  - pornel 4y ago
    Collision isn't a spooky action at distance. Even if they tricked the victim into accepting a file they have a collision for, they still can't do anything nefarious. Attack requires an opportunity to replace the colliding file with its evil twin, and that requires write access to victim's repository or tricking the victim into re-fetching their files from an attacker-controlled repository. Besides, the known collision attack generates files with blocks of binary garbage, which makes it difficult to trick someone into accepting. It won't look like source code, and if someone accepts binary blobs of executable code, you don't need collisions to pwn them.
    - staticassertion 4y ago
      > Besides, the known collision attack generates files with blocks of binary garbage, which makes it difficult to trick someone into accepting. It won't look like source code, and if someone accepts binary blobs of executable code, you don't need collisions to pwn them. IDK, I could see this happening in multiple ways. 1. Images / media artifacts stored for display purposes 2. Cached files - 'zero install' config for yarn comes to mind, where every dependency has its file cached in git. Plus binary files aren't displayed in git diffs so it seems somewhat easy to sneak in. Otherwise, yeah, agree. Most people don't rely on Git's security model, they rely on Github's.
ainar-g 4y ago
The article mentions that “none of the Git hosting providers appear to be supporting SHA-256”, but what about self-hosted solutions? In particular, sr.ht. Seems to be nothing[1] in their issue tracker. [1]: https://todo.sr.ht/~sircmpwn/git.sr.ht?search=sha-256 https://todo.sr.ht/~sircmpwn/git.sr.ht?search=sha-256
- oynqr 4y ago
  How about https://todo.sr.ht/~sircmpwn/git.sr.ht?search=sha256 https://todo.sr.ht/~sircmpwn/git.sr.ht?search=sha256
  - ainar-g 4y ago
    Hah! I guess there should be one about smarter search as well, heh. Thanks!
donatj 4y ago
Potentially stupid question, would it be reasonable to use SHA-256 truncated to the first 40 digits? It seems like that could ease much of the migration problems if it's not a problem?
- stingraycharles 4y ago
  I found this, which says that the SHA algorithm allows for truncation: https://csrc.nist.gov/publications/detail/sp/800-107/rev-1/final https://csrc.nist.gov/publications/detail/sp/800-107/rev-1/f...
  - Dylan16807 4y ago
    Not just allows, it becomes more secure when you truncate.
    - neon_electro 4y ago
      Care to elaborate? This is not something I would've intuited.
      - bawolff 4y ago
        Presumably they are referring to length extension attacks. You can't pull them off if you truncate. https://en.m.wikipedia.org/wiki/Length_extension_attack https://en.m.wikipedia.org/wiki/Length_extension_attack Generally though length ext attacks have a solution - HMAC, which is much more secure than truncate. The more you truncate, the more vulnerable you are to birthday attacks (practically speaking you would have to truncate quite a lot)
        kzrdude 4y ago
        The canonical solution is SHA-512/256 i.e 512 truncated to 256 bits where "nothing is lost" compared to SHA-256 and something is gained. It might even be faster (due to the 64-bit word formulation of SHA-512) in some implementations.
simias 4y ago
It's frankly amateurish for the git dev to delay this. The longer this lasts, the more painful it'll be whenever the switch will finally take place. Linus shouldn't have used SHA-1 in the first place, it was already being deprecated by the time git got its original release. Then every time a new milestone is reached to break SHA-1 we see the same rationalization about how it's not a big deal and it's not a direct threat to git and blablabla. It'll keep not mattering until it matters and the longer their wait the more churn it'll create. Let's rip the bandaid that's been hanging there for over 15 years now.
- runeks 4y ago
  > Linus shouldn't have used SHA-1 in the first place, it was already being deprecated by the time git got its original release. Using SHA-1 to begin with was fine. However, commit hashes should have been prepended with a version byte to make it easier to transition to the next hash algorithm. This would mean an old Git client could report an error to the user of the nature “please upgrade your software to support cloning from this Git server” instead of failing with an error that’s inseparable from “the Git server is broken” when trying to clone a Git repo using SHA-256.
  - jackweirdy 4y ago
    There’s already a version byte: if it’s [0-9a-f], that’s version 1 ;)
    - LeifCarrotson 4y ago
      That's a 4-bit nibble, the version byte is 0x00 to 0xFF.
      - Arnavion 4y ago
        They're talking about the hex representation. It doesn't make sense to think they were referring to the nibble as the version, given that all 16 values of that nibble are already in use.
  - 4y ago
Zamicol 4y ago
This is one of the reasons why Go has its own versioning system. From a project's `go.sum`: example.com/example v0.0.0-20171218180944-5ea4d0ddac55 h1:jbGlDKdzAZ92NzK65hUP98ri0/r50vVVvmZsFP/nIqo= Where "h1" is an upgradeable hash (h1 is SHA-256). If there's ever a problem with h1, the hash can be simply upgraded. Git's documentation describes how to sign a git commit: $ git commit -a -S -m 'signed commit' When signing a git commit using the built in gpg function the project is not rehashed with a secure hash function, like SHA-256 or SHA3-256. Instead gpg signs the SHA-1 commit digest directly. It's not signing the result of a secure hash algorithm. SHA-1 has been considered weak for a long time (about 17 years). Bruce Schneier warned in February 2005 that SHA-1 needed to be replaced. Git development didn't start until April 2005. Before git started development, SHA-1 was identified as needing deprecation.
- er4hn 4y ago
  Just to nit on your portion of signing: wouldn't you need to rehash all prior commits as well so that they used the better hash function? Otherwise someone could find a collision for a prior commit hashed with sha-1, slip that in, and the final commit being hashed with sha256 wouldn't matter. This then makes the signing code use its own form of hashing that is different from the rest of git's commmit hashing, and seems like a novel way to introduce tooling issues / bugs / etc.
  - chimeracoder 4y ago
    > and the final commit being hashed with sha256 wouldn't matter. Git stores content, not diffs. So the signature verifies all content stored in that commit. It does t verify anything that came before it, unless those are specifically signed as well.
    - ElectricalUnion 4y ago
      > Git stores content, not diffs. But the "contents" is just pointers to tree roots with a trusted hash. If the hash is no longer secure, you can't garantee that any such trees are your content, or safe.
mjw1007 4y ago
> All that is left is the hard work of making the transition to a new hash easy for users — what could be thought of as "the other 90%" of the job. If that was all that was left, we could at least be using sha256 for new repositories. It seems to me the big missing piece is support in libgit2, which is at least showing signs of progress: https://github.com/libgit2/libgit2/pull/6191 https://github.com/libgit2/libgit2/pull/6191
- xyzzy_plugh 4y ago
  libgit2 isn't an official library, and even if it did support sha256 dependents would still need to update, so I really don't perceive this as a missing piece. If everyone started using sha256 then all these problems would be addressed practically overnight.
ivoras 4y ago
Is there an explanation of what would go wrong with the naive approach? E.g.: - Change the binary file format in repos to support arbitrary hash algorithms, in a way which unambigously makes old software fail. - Increment the Git major version number to 3.0 - Make the new version support both the old version repos and the new ones. Make it a per-repo config item that allows/disallows old/new hash formats. In theory, there's nothing wrong with having objects hashed with mixed algorithms as long as the software knows how to deal with that. - The old format will probably have to be supported forever because of Linux. Most user-facing utilities don't care what the hash algo actually is, they just use the hash as an opaque string.
- deleted 4y ago
  [deleted]
- runeks 4y ago
  Releasing new software is the simple part. The problem is that versioning is lacking in the old software, and therefore it doesn’t know how to talk to the new software. So for the old software there’s no difference between “invalid data” and “I’m too old, please upgrade me”.
  - dingleberry420 4y ago
    > So for the old software there’s no difference between “invalid data” and “I’m too old, please upgrade me”. And why is this an issue? Release the new version that can read new repo formats, but doesn't write them yet. Wait a year. Release new version that can write new repo formats and encourage users to upgrade. Anyone who hasn't upgraded in the past year probably doesn't care about security and should be left behind. Besides, once they google the error message they'll figure it out soon enough. It's not like git is known for its great UX anyway.
    - hamilyon2 4y ago
      I might be mistaken, but github could be using their own version of git and accompanying tools. So, unless they implement uprade themselves, no amount of waiting will make git interoperable with them.
wepple 4y ago
I was curious about the “sha1dc” that git uses and reportedly helps protect against collision attacks. Here’s the paper: https://marc-stevens.nl/research/papers/C13-S.pdf https://marc-stevens.nl/research/papers/C13-S.pdf
O__________O 4y ago
Anyone aware of any exploits tied the SHA-1 weakness in the wild? (I have seen proofs of concept [1], but never actually heard of an exploit in the wild using it; for example, on: digital certificate signatures, email PGP/GPG signatures, software vendor signatures, software updates, ISO checksums, backup systems, deduplication systems, Git, etc.) [1] https://shattered.io/ https://shattered.io/
- password4321 4y ago
  Applications of that collision: https://twitter.com/rauchg/status/834770508633694208 https://twitter.com/rauchg/status/834770508633694208 > a SHA-1 "Pinata" [...] claimed https://news.ycombinator.com/item?id=13723892 https://news.ycombinator.com/item?id=13723892 > Make your own colliding PDFs https://news.ycombinator.com/item?id=13917990 https://news.ycombinator.com/item?id=13917990 > Collision Detection
- bawolff 4y ago
  Most security critical systems have switched to sha256 at this point, and making a fresh collision still costs tens of thousands, so people arent really doing it for kicks (that said, once you have one collision you can reuse it for free as long as you keep the same prefix, so the proof of concept can be repurposed with certain constraints). The most in the wild one i have ever heard of was when webkit accidentally broke their svn repo by checking in a collision. However you can look at the history of md5 which had a similar flaw which was exploited by the flame malware.
  - O__________O 4y ago
    Thanks, agree the Flame’s use of a collision attack was both comparable and notable: https://en.m.wikipedia.org/wiki/Flame_(malware) https://en.m.wikipedia.org/wiki/Flame_(malware)
encryptluks2 4y ago
It seems like something more modern, like b3sum would be better... no? What about b2sum?
- kortex 4y ago
  I love the performance of blake3 but my understanding is it's still a bit of the new kid on the block. Blake2 is a SHA-3 finalist so should be perfectly sufficient, plus it has variable digest sizes, reasonably fast, and other nice features. Either way, anything relying on hashes for data integrity should at least be flexible to the option of multiple hash algos. But with git, it's going to be hard enough as is to change to SHA-256, and I don't know how parametric it'll be.
  - notfed 4y ago
    Actually, BLAKE2 was not a SHA-3 finalist, BLAKE was. Also, BLAKE3 is so significantly faster than it's probably worth waiting for (waiting for it to get vetted cryptanalysis-wise).
- MrStonedOne 4y ago
- rockdoe 4y ago
  I mean, SHA-3?
  - encryptluks2 4y ago
    Blake3 is faster and more secure. I also really don't trust NIST to be honest when they have a history of promoting solutions with backdoors.
    - oconnor663 4y ago
      I'm not aware of any reason SHA-3 would be less secure than BLAKE3. Quite the opposite: SHA-3 has a very large security margin, so large that the authors cut the number of rounds in half for KangarooTwelve.
chmaynard 4y ago
Previous articles on this topic: A new hash algorithm for Git https://lwn.net/Articles/811068/ https://lwn.net/Articles/811068/ Updating the Git protocol for SHA-256 https://lwn.net/Articles/823352/ https://lwn.net/Articles/823352/
kazinator 4y ago
> Given the threat that the SHA-1 hash poses I give -3 flying ducks about this, and don't want the Git storage format to be diddled with in any way. Git in 2122 should read and write a git repo made in 2010. Git is not a public crypto system. If you think a commit is important and needs to be signed, you need to sign the files and add the signature to the commit.
- kzrdude 4y ago
  Migrating to a non broken hash is the better chance at being a century long strong repository format.
pornel 4y ago
The worst thing about the SHA-1 collision is the tedium of explaining the difference between a collision attack and a preimage attack.
heynowheynow 4y ago
It might be wiser to keep SHA1 and use SHA2, SHA3, etc. and GPG as overlays for compatibility and simplicity reasons.
harryvederci 4y ago
Relevant quote from the Fossil website[0]: "Fossil started out using 160-bit SHA-1 hashes to identify check-ins, just as in Git. That changed in early 2017 when news of the SHAttered attack broke, demonstrating that SHA-1 collisions were now practical to create. Two weeks later, the creator of Fossil delivered a new release allowing a clean migration to 256-bit SHA-3 with full backwards compatibility to old SHA-1 based repositories. [...] Meanwhile, the Git community took until August 2018 to publish their first plan for solving the same problem by moving to SHA-256, a variant of the older SHA-2 algorithm. As of this writing in February 2020, that plan hasn't been implemented, as far as this author is aware, but there is now a competing SHA-256 based plan which requires complete repository conversion from SHA-1 to SHA-256, breaking all public hashes in the repo." [0]: https://fossil-scm.org/home/doc/trunk/www/fossil-v-git.wiki#hash https://fossil-scm.org/home/doc/trunk/www/fossil-v-git.wiki#...
- ludwigvan 4y ago
  Migrations are easier when you are the only one using your software. :p Joking aside, expected from a developer whose work is the recommended storage format for Library of Congress.
- tremon 4y ago
  This raises an interesting point: given that git has been dragging its feet for so long on the transition to SHA-256, it's better if they were to drag their feet a bit longer and move directly to SHA3-256 too, and never let the current SHA-256 implementation get widely deployed.
YesThatTom2 4y ago
GitHub won’t feel any heat about this until Microsoft salespeople start demanding it. I’ve added to my todo list a reminder to raise this issue with mine. In fact, I’m going to give them a deadline for when we will start evaluating competitors that do support SHA256. I suspect that most people on HN do not interact with their MS account team. That relationship is probably managed by your CIO or IT department. They probably have monthly or quarterly “business review” meetings. You should get this issue on the agenda of that meeting.
- sdfhdhjdw3 4y ago
  Thank you.
- codazoda 4y ago
  Is there something special about GitHub on this? This seems like a Git issue and not a GitHub issue to me; unless I'm missing something.
  - lucb1e 4y ago
    They don't accept pushes of repositories in that format. The article says "none of the Git hosting providers appear to be supporting SHA-256", and while GH is not mentioned by name (and I applaud them for indeed not strengthening this "git == github-the-brand" trap), I can't imagine GH was left out of scope when checking the major hosting providers.
  - evil-olive 4y ago
    as the article says, you can create a local git repository with SHA-256 hashes today, and it should work fine...but the moment you try to push your repo up to Github, you'll hit a brick wall. Gitlab also appears to be lacking support [0], and the same with Gitea [1]. so it's a grey area where Git itself supports SHA-256-based repos, but without the major Git hosting services also supporting them, the support in core Git is somewhat useless. 0: https://gitlab.com/groups/gitlab-org/-/epics/794 https://gitlab.com/groups/gitlab-org/-/epics/794 1: https://github.com/go-gitea/gitea/issues/13794 https://github.com/go-gitea/gitea/issues/13794
slim 4y ago
Why didn't Linux migrate their repo to SHA-256 ? is it that difficult to migrate a repo ?
sdfhdhjdw3 4y ago
> Even if creating a collision were feasible for an attacker, Bjarmason pointed out, that is only the first step in the development of a successful attack. Finding a collision of any type is hard; finding one that is still working code, that has the functionality the attacker is after, and that looks reasonable to both humans and compilers is quite a bit harder — if it is possible at all. Sounds like there's money in this.
jiggawatts 4y ago
If you’re going to “fix” the hash algorithm, do it properly! Sha256 can only be computed in a single sequential stream (thread) by definition. For large files this is increasingly becoming a performance limitation. A Merkle tree based on SHA512 would have significant benefits. SHA512 is faster than SHA256 on modern CPUs because processes 64 bits per internal register instead of 32 bits. A tree-structured hash can be parallelised across all cores. For repositories with files over 100MB in them on an SSD this would make a noticeable difference…
- dchest 4y ago
  Most git objects are tiny files, so internal tree-based parallelization won't bring much compared to file parallelization (git is a hash tree itself, with variable-length leaves). SHA256 is actually a lot faster on modern CPUs due to https://en.wikipedia.org/wiki/Intel_SHA_extensions https://en.wikipedia.org/wiki/Intel_SHA_extensions (and similar on Arm), which are implemented for SHA-256 but not for SHA-512, e.g. openssl speed sha256 sha512 on M1: type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes sha256 89474.97k 283341.15k 901724.41k 1730980.24k 2339109.86k sha512 66160.19k 262139.03k 365675.96k 487572.26k 545142.91k
  - jiggawatts 4y ago
    A fair point about the instruction sets, and it is also true that “most” files are small. But again, due precisely to their size, large files take a disproportionate amount of time to process. Don’t confuse the typical use-case with the fundamental concept: versioning. Git could be a general purpose versioning system with many more use-cases, but limitations like this hold it back unnecessarily…
    - dchest 4y ago
      Hashing is not the only thing that stops git from being useful for large file versioning. For this purpose, splitting files into chunks using a rolling hash (similar to how git packs, rsync, tarsnap or IPFS) would work better. This again doesn't require "internal" tree hashing, since each chunk would be hashed separately.
avar 4y ago
I'm the person and Git developer (Ævar) quoted in the article. I didn't expect this to end up on LWN. I'm happy to answer any questions here that people might have. I don't think the LWN article can be said to take anything out of context. But I think it's worth empathizing that this is a thread on the Git ML in response to a user who's asking if Git/SHA-256 is something "that users should start changing over to[?]". I stand by the comments that I think the current state of Git is that we shouldn't be recommending to users that they use SHA-256 repositories without explaining some major caveats, mainly to do with third party software support, particularly the lack of support from the big online "forges". But I don't think there's any disagreement in the Git development community (and certainly not from me) that Git should be moving towards migrating away from SHA-1.
- lalaland1125 4y ago
  Have you considered moving over to a combined SHA-1, SHA-256 model where both hashes are calculated, with SHA-1 shown to the user and SHA-256 only used in the background to prevent collisions? There is a compute cost for that, but it should be minimal relative to the security benefits?
  - pmarreck 4y ago
    This is what I would have recommended... A transition period where both are used
  - avar 4y ago
    Someone probably brought it up at some point, I can't remember. But I'm not aware of any known scenario where the SHA1DC library Git uses doesn't give you the benefits of that and more. "And more" because to detect a collision with a background SHA-256 you'll need both objects, whereas SHA1DC detects attempts to spoof SHA1 in a way that leads to collisions. So it won't pass along an object that collides with another one, even though it only has 1/2 objects. That distinction is something that's generally considered important, e.g. there's been past exploits in Git where you could trick a client into doing something bad by e.g. a crafted .gitmodules file. The fix has not only been to patch clients, but also to patch "git fsck" to detect and reject such bad contents, so that e.g. the forges can't be used to relay a repository exploit to users running older versions. A viable hash collision exploit in the wild might likewise want to make use of such an attack scenarios, so having servers capable of detecting collisions without having both sides is preferable to doing so by re-hashing with SHA-256.
hrgiger 4y ago
Probably performance impact is little but still noticeable for a large project ,maybe even with modern hardware but wouldnt a portable approach would be better? Not the security but performance perspective maybe one would prefer cheap calculation, i.e. ext4 pre-calculate already crc32c , just found out google added alternative highwayhash recommends and claims it is faster. https://github.com/google/highwayhash https://github.com/google/highwayhash
- janwas 4y ago
  Author here. HighwayHash is indeed reasonably fast for a PRF/MAC/fingerprint, but our security claims are not strong enough for it to serve as a cryptographic hash. No collision nor preimage attacks are known to us, but it would not be appropriate to use HighwayHash in this context.
  - hrgiger 4y ago
    Thanks for the response, yes I can understand the preference in security context and dont want to drop 'why cryptographic hash' bomb into discussion, I was just thinking maybe keeping sha256/512 as default but having switchable options through the configuration for alternatives like your implementation or proven implementations or using filesystem features. No matter what that is laziness from my side, just throwing question rather than spending time on source.
pmarreck 4y ago
Couldn't there be a transition period where both are supported?
hmsimha 4y ago
Regarding the difficulty of generating a collision with working code, wouldn't it be as "trivial" as generating a SHA-1 collision in the first place? Just have the malicious code in a file and append a multiline comment block, then have your collision generator insert random junk into that comment block
- ed25519FUUU 4y ago
  Generating the “junk” is the extraordinarily challenging part, though it’s been proved possible.
  - hmsimha 4y ago
    Sure, I had the word "trivial" in quotes for that reason. I meant that if you're able to generate a collision, the rest of this doesn't follow (from the article) > Even if creating a collision were feasible for an attacker, Bjarmason pointed out, that is only the first step in the development of a successful attack. Finding a collision of any type is hard; finding one that is still working code, that has the functionality the attacker is after, and that looks reasonable to both humans and compilers is quite a bit harder — if it is possible at all.
phlogisticfugu 4y ago
at the risk of sounding like a heretic: if backwards compatibility is so hard, why not break it? replace git with git2 in all places (command line, URLs, protocol, etc) and make a git1to2 utility, and you are golden
londons_explore 4y ago
Normally migrations of this type are done in two phases: * Add support for new hash * Migrate all data to new hash, dropping support for clients that don't support it. It appears we're in the waiting phase between the two bullet points. I imagine that could be many years, because many people don't update their git clients often.
userbinator 4y ago
MD5 is still strong to preimage attacks although you can generate collisions in seconds now. Thought experiment: What if Git had used MD5? The news has been proclaimed loudly and often: the SHA-1 hash algorithm is terminally broken and should not be used in any situation where security matters. There are organizations where SHA-1 is blanket banned across the board - regardless of its use I think it's about time people understood the different types of attacks and what they mean for various uses of hash functions instead of blindly cargo-culting as the security (aka paranoia) industry seems to want to do. Then again, the people in those same organizations who promote this sort of anti-thinking would probably suddenly not care if you just renamed all occurrences of "SHA1" in everything they see to something else, because they are such incompetent idiots anyway --- true stories from experience...
iraqmtpizza 4y ago
Curious why SHA-256 and not SHA-512 or SHA-512/256 which have higher throughput. Interoperability? Hardware acceleration?
rnhmjoj 4y ago
I hope this doesn't turn into an IPv4/IPv6 situation, but it looks like it's directly headed that way. Without a proper transition mechanism companies won't implement SHA-256 because there's no demand for it and users can't switch until companies support SHA-256.
red_admiral 4y ago
Is SHA-1 being used in a security-critical way in git, though? I tend to agree with the points made towards the end of the article. If someone malicious can make commits to your repo, you have a much worse problem than a possible hash collision. Most languages these days have a built-in hashtable type, and they use some kind of non-security-critical hash function, albeit usually with a random seed, that's a lot weaker than SHA-1 - often you only need something like 32-64 bits of output in the first place as no-one allocates more than 2^64 buckets as far as I know. From a security perspective that's even worse than MD5! I'd argue (and so does the article, in part) that the use of SHA-1 in git is closer to the use of a hash in a hashtable than to the use of a cryptographic checksum - using SHA-1 for subresource integrity on the web is an obvious security risk (luckily, it's not allowed there) but I can't see an obvious attack on git's use that doesn't also assume you can do much worse things.
- haileys 4y ago
  > If someone malicious can make commits to your repo, you have a much worse problem than a possible hash collision. This is exactly how GitHub works though: all repositories in a fork network share a single underlying repository, only the ref namespaces are separated.
  - wildfire 4y ago
    I am skeptical that is actually the case though, there are so many use-cases where you have to think through the consequences too carefully: - what happens when the source repository rewrites all the commits (to remove one); should the forks also be re-written? - what happens when the source repository disappears? _All_ of the problems go away when you actually do a "git clone" of the source repository and have your own objects. More storage, yes, but way fewer corner cases to think through.
- deleted 4y ago
  [deleted]
billpg 4y ago
I'm about to start a new repo. Should I use the SHA256 option?

455 points by chmaynard 4y ago

The part of software engineering they don't teach in college is migration. Some of the most creative work you'll do is figuring out how to get from X to Y without bringing everything crashing down around you (or at least only a couple things crashing down at a time).

TillE 4y ago

Bjarmason has a good response about the practicalities of an attack; it explains why a "broken" hash is rarely a running-around-with-your-hair-on-fire level emergency. It would clearly be better to use a better hash, but is it actually urgent for anyone? Probably not.

corbet 4y ago

It's nice to see LWN on HN for the second time in one day, but please remember: it is only LWN subscribers that make this kind of writing possible. If you are enjoying it, please consider becoming a subscriber yourself — or, even better, getting your employer to subscribe.

jra_samba 4y ago
Just want to second this ! Please subscribe to lwn. I learn new things from lwn every week. It's really worth the money.
O__________O 4y ago
For easy of reference, here is the link to subscribe, which includes a description of the benefits: https://lwn.net/subscribe/Info https://lwn.net/subscribe/Info And the Wikipedia page for LWN, if you’re not familiar with it: https://en.m.wikipedia.org/wiki/LWN.net https://en.m.wikipedia.org/wiki/LWN.net
cockhole_desu 4y ago
williadc 4y ago
Googlers can subscribe through work by visiting go/lwn and following the instructions.
bloopernova 4y ago
subbed, thanks for the reminder!

bradhe 4y ago

Does the usage of SHA-1 in Git actually have security implications, though? It's basically only used to generate addresses for refs and hunks and all that.

saghm 4y ago
I don't think it does; sure, someone could potentially craft a malicious commit that causes a SHA1 collision in your repo, but I think if you are merging commits from malicious authors, you've got way bigger problems than that.
- corbet 4y ago
  ...and if you're merging commits from a developer who, unknown to either of you, had their laptop compromised and their repo corrupted? Remember that the compromise of kernel.org happened via a developer's laptop, and it was only the security of the hash chains that preserved confidence in the repositories stored there. As noted in the article, an SHA-1 collision attack does not appear practical now, but that is a situation that can change.
- shakna 4y ago
  GitHub actually makes pull requests available as an unlisted part of the original repository under refs/pull/$PR/head and refs/pull/$PR/merge, which allows a malicious author to add themselves to your index, without your involvement. Not to say that this attack is in any way practical, yet. Just that some providers don't require active involvement to try and attempt it.
mmastrac 4y ago
If a repo accepts third-party contributions, you can create a split brain where half the people see one set of contents and the others see a different set, but the same hashes are available. I don't know if this would survive additional commits on top as I'm not familiar enough with git's internals.
- progval 4y ago

yjftsjthsd-h 4y ago

> In his view, the only "defensible" reason to use SHA-1 at this point is interoperability with the Git forge providers. Okay, but that's a pretty big reason! A git repo that can't be pushed to github/lab is... not always useless, but certainly extremely impaired.

kragen 4y ago
In case anyone has forgotten, the process for pushing it to your own server is three shell commands. You run, on the server: git init --bare public_html/mything.git cd public_html/mything.git/hooks/ mv post-update.sample post-update # runs git update-server-info on push (This assumes that your public_html directory exists and is mapped into webspace, as with the usual configuration of Apache, NCSA httpd, and CERN httpd. If you don't have an account on such a thing you can get such PHP shared hosting accounts with shell access anywhere in the world for a dollar or two a month.) And then on your dev machine, it's precisely the same as for pushing to Gitlab or whatever, except that you use your own username instead of git@: git remote add someremotename user@myserver:public_html/mything.git git push -u someremotename master # assuming you want it to be your upstream Then anyone can clone from your repo with a command like this: git clone https://myserver/~user/mything.git They can also add the URL as a remote for pulls. If you want them to be able to push, you'll need to give them an account on the same server and either set umasks and group ownerships and permissions appropriately or set a POSIX ACL. Alternatively they can do the same thing on their server and you can pull from it. There are reportedly permission bugs in recent versions of Git (the last five years) that prevent this from being safe with people you don't trust (https://www.spinics.net/lists/git/msg298544.html https://www.spinics.net/lists/git/msg298544.html). Of course source control is only part of the overall development project workflow, so for many purposes adding SHA-256 support to Gogs or Gitlab or Gitea or sr.ht is probably pretty important: you want a Wiki and CI integration and bug tracking and merge requests. But the git repo still works fine with a bog-standard ssh and HTTP server, though slightly less efficiently. It's easier than setting up a new repo on GitLab etc. Running a git repack -an && git update-server-info in the repo on the server can help a lot with the efficiency, and for having a browseable tree on the server as well as a clonable repo I put this script at http://canonical.org/~kragen/sw/dev3.git/hooks/post-update http://canonical.org/~kragen/sw/dev3.git/hooks/post-update: #!/bin/sh set -e echo -n 'updating... ' git update-server-info echo 'done. going to dev3' cd /home/kragen/public_html/sw/dev3 echo -n 'pulling... ' env -u GIT_DIR git pull echo -n 'updating... ' env -u GIT_DIR git update-server-info echo 'done.' That's very far from being GitLab (contrast http://canonical.org/~kragen/sw/dev3 http://canonical.org/~kragen/sw/dev3 with any GitHub tree view), and it's potentially dangerously powerful: if you're doing this in a repo where you pull from other people, and the server is configured to run PHP files or server-side includes in your webspace (mine isn't!) or CGI scripts (mine is!), then just dropping a file in the repo can run programs on the server with your account privileges. This is great if that's what you want, and it's a hell of a lot better than updating your PHP site over FTP, but that code has full authority to, for example, rewrite your Git history. In theory you can do other things from your post-update hook as well, like rebuild a Jekyll site, send a message on IRC or some other message queueing system, or fire off a CI build in a Docker container. (Some of these would run afoul of guardrails common in cheap PHP shared hosting providers and you'd have to upgrade to a US$5/month VPS.)

armada651 4y ago

> Adding my own 0.02, what some of us are facing is resistance to adopting git in our or client organizations because of the presence of SHA-1. There are organizations where SHA-1 is blanket banned across the board - regardless of its use. [...] Getting around this blanket ban is a serious amount of work and I have very recently seen customers move to older much less functional (or useful) VCS platforms just because of SHA-1. Seems like this company could just use the current SHA-256 support then? Especially if it's the type of company that does all its development in-house and there's no need for SHA-1 interoperability.

cratermoon 4y ago
Except said company likely uses one of the Git forge providers, either in-house or as a SaaS, as the (oxymoronic for git) central repo. Until they support SHA-256, or the company goes with a its own git repo solution that is set up for it, companies won't make the move.
- wepple 4y ago
  Not just git forge but probably the myriad other ancillary tools that assume SHA1
gorkish 4y ago
> There are organizations where SHA-1 is blanket banned across the board - regardless of its use. > I have very recently seen customers move to older much less functional (or useful) VCS platforms just because of SHA-1. A company this dysfunctional has problems far beyond their choice of revision control system.
- bostik 4y ago
  I can name a couple of industries where compliance (and their enforcement arm, security[0]) teams require N+1 different monitoring and enforcement agents on all systems because Compliance[TM]. Due to these agents the systems' IDLE load is approaching 1.00 - on a good day. On a less good you need four cores to have one of them available for workload processing. 0: I use the word "security" only because the teams themselves are named like that. You can probably infer my opinion from the tone.

WorldMaker 4y ago

Almost feels like by the time git finally transitions to SHA-256 some bitcoin miner somewhere will have a solved preimage weakness on SHA-256.

jagger27 4y ago
Thankfully existing Bitcoin ASICs don’t pose much of a threat because they’re only good for sha256(sha256(Bitcoin block)). If a practical pre-image attack on SHA-256 comes around we have bigger problems than git.
- WorldMaker 4y ago
  Obviously the concern is not the ASICs themselves but the ASIC designers. (Using miners here in the colloquial sense of human collectives/corporations backing the machines than than the specific sense of the raw machines themselves.) Yes, a practical preimage weakness in SHA-256 is a nightmare scenario with huge implications to the rest of internet security beyond just get. It's why I sometimes can't sleep at night knowing how much energy bitcoin spends daily on a continuous massively distributed partial preimage attack on SHA-256.
  - marktangotango 4y ago
    > how much energy bitcoin spends daily on a continuous massively distributed partial preimage attack on SHA-256. I would not be concerned about this. The way the asics operate is they discard the results. Also, the hashes are random strings which don't compress very well, so storing trillions upon trillions of them (for later analysis) is not practical.
    - WorldMaker 4y ago
      Again, the point of the fear is not the specifics of current operations (ASIC details; which y'all are talking about as if all of the miners are using the same hardware), but the fear of future operations and that there's an enormous industrial preimage attack effort at all. One that we can see in real time, in global energy consumption graphs. Maybe you find "cold comfort" that because we can watch it in real time if someone discovers a weakness we will also watch its repercussions and the subsequent horrifying fall in real time, too, but I certainly don't.

mdavidn 4y ago

I don’t depend on the collision resistance of SHA-1 for the security of my git repos because I don’t accept pushes from people I don’t trust. If I did, objects with hash collisions would not be transferred or (I hope) accepted. Am I missing something? Granted, signed tags do depend on this collision resistance, but I don’t use that feature. Signing entire releases from a trusted repo seems like a better approach.

teraflop 4y ago
It's not just the pushes themselves; anyone who can create commits or blobs that eventually get merged into your repository, directly or indirectly, can potentially engage in a collision attack. Sure, if you use git with a very closed development model, this doesn't necessarily affect you much. But it's (potentially) a big problem for collaborative open-source projects, because it requires trust in every single contributor. And the trust requirement can't necessarily be mitigated using ordinary means like code reviews.
- pornel 4y ago
  Collision isn't a spooky action at distance. Even if they tricked the victim into accepting a file they have a collision for, they still can't do anything nefarious. Attack requires an opportunity to replace the colliding file with its evil twin, and that requires write access to victim's repository or tricking the victim into re-fetching their files from an attacker-controlled repository. Besides, the known collision attack generates files with blocks of binary garbage, which makes it difficult to trick someone into accepting. It won't look like source code, and if someone accepts binary blobs of executable code, you don't need collisions to pwn them.
  - staticassertion 4y ago
    > Besides, the known collision attack generates files with blocks of binary garbage, which makes it difficult to trick someone into accepting. It won't look like source code, and if someone accepts binary blobs of executable code, you don't need collisions to pwn them. IDK, I could see this happening in multiple ways. 1. Images / media artifacts stored for display purposes 2. Cached files - 'zero install' config for yarn comes to mind, where every dependency has its file cached in git. Plus binary files aren't displayed in git diffs so it seems somewhat easy to sneak in. Otherwise, yeah, agree. Most people don't rely on Git's security model, they rely on Github's.

ainar-g 4y ago

The article mentions that “none of the Git hosting providers appear to be supporting SHA-256”, but what about self-hosted solutions? In particular, sr.ht. Seems to be nothing[1] in their issue tracker. [1]: https://todo.sr.ht/~sircmpwn/git.sr.ht?search=sha-256 https://todo.sr.ht/~sircmpwn/git.sr.ht?search=sha-256

oynqr 4y ago
How about https://todo.sr.ht/~sircmpwn/git.sr.ht?search=sha256 https://todo.sr.ht/~sircmpwn/git.sr.ht?search=sha256
- ainar-g 4y ago
  Hah! I guess there should be one about smarter search as well, heh. Thanks!

donatj 4y ago

Potentially stupid question, would it be reasonable to use SHA-256 truncated to the first 40 digits? It seems like that could ease much of the migration problems if it's not a problem?

stingraycharles 4y ago
I found this, which says that the SHA algorithm allows for truncation: https://csrc.nist.gov/publications/detail/sp/800-107/rev-1/final https://csrc.nist.gov/publications/detail/sp/800-107/rev-1/f...
- Dylan16807 4y ago
  Not just allows, it becomes more secure when you truncate.
  - neon_electro 4y ago
    Care to elaborate? This is not something I would've intuited.
    - bawolff 4y ago
      Presumably they are referring to length extension attacks. You can't pull them off if you truncate. https://en.m.wikipedia.org/wiki/Length_extension_attack https://en.m.wikipedia.org/wiki/Length_extension_attack Generally though length ext attacks have a solution - HMAC, which is much more secure than truncate. The more you truncate, the more vulnerable you are to birthday attacks (practically speaking you would have to truncate quite a lot)
      - kzrdude 4y ago
        The canonical solution is SHA-512/256 i.e 512 truncated to 256 bits where "nothing is lost" compared to SHA-256 and something is gained. It might even be faster (due to the 64-bit word formulation of SHA-512) in some implementations.

simias 4y ago

It's frankly amateurish for the git dev to delay this. The longer this lasts, the more painful it'll be whenever the switch will finally take place. Linus shouldn't have used SHA-1 in the first place, it was already being deprecated by the time git got its original release. Then every time a new milestone is reached to break SHA-1 we see the same rationalization about how it's not a big deal and it's not a direct threat to git and blablabla. It'll keep not mattering until it matters and the longer their wait the more churn it'll create. Let's rip the bandaid that's been hanging there for over 15 years now.

runeks 4y ago
> Linus shouldn't have used SHA-1 in the first place, it was already being deprecated by the time git got its original release. Using SHA-1 to begin with was fine. However, commit hashes should have been prepended with a version byte to make it easier to transition to the next hash algorithm. This would mean an old Git client could report an error to the user of the nature “please upgrade your software to support cloning from this Git server” instead of failing with an error that’s inseparable from “the Git server is broken” when trying to clone a Git repo using SHA-256.
- jackweirdy 4y ago
  There’s already a version byte: if it’s [0-9a-f], that’s version 1 ;)
  - LeifCarrotson 4y ago
    That's a 4-bit nibble, the version byte is 0x00 to 0xFF.
    - Arnavion 4y ago
      They're talking about the hex representation. It doesn't make sense to think they were referring to the nibble as the version, given that all 16 values of that nibble are already in use.
- 4y ago

Zamicol 4y ago

This is one of the reasons why Go has its own versioning system. From a project's `go.sum`: example.com/example v0.0.0-20171218180944-5ea4d0ddac55 h1:jbGlDKdzAZ92NzK65hUP98ri0/r50vVVvmZsFP/nIqo= Where "h1" is an upgradeable hash (h1 is SHA-256). If there's ever a problem with h1, the hash can be simply upgraded. Git's documentation describes how to sign a git commit: $ git commit -a -S -m 'signed commit' When signing a git commit using the built in gpg function the project is not rehashed with a secure hash function, like SHA-256 or SHA3-256. Instead gpg signs the SHA-1 commit digest directly. It's not signing the result of a secure hash algorithm. SHA-1 has been considered weak for a long time (about 17 years). Bruce Schneier warned in February 2005 that SHA-1 needed to be replaced. Git development didn't start until April 2005. Before git started development, SHA-1 was identified as needing deprecation.

er4hn 4y ago
Just to nit on your portion of signing: wouldn't you need to rehash all prior commits as well so that they used the better hash function? Otherwise someone could find a collision for a prior commit hashed with sha-1, slip that in, and the final commit being hashed with sha256 wouldn't matter. This then makes the signing code use its own form of hashing that is different from the rest of git's commmit hashing, and seems like a novel way to introduce tooling issues / bugs / etc.
- chimeracoder 4y ago
  > and the final commit being hashed with sha256 wouldn't matter. Git stores content, not diffs. So the signature verifies all content stored in that commit. It does t verify anything that came before it, unless those are specifically signed as well.
  - ElectricalUnion 4y ago
    > Git stores content, not diffs. But the "contents" is just pointers to tree roots with a trusted hash. If the hash is no longer secure, you can't garantee that any such trees are your content, or safe.

mjw1007 4y ago

> All that is left is the hard work of making the transition to a new hash easy for users — what could be thought of as "the other 90%" of the job. If that was all that was left, we could at least be using sha256 for new repositories. It seems to me the big missing piece is support in libgit2, which is at least showing signs of progress: https://github.com/libgit2/libgit2/pull/6191 https://github.com/libgit2/libgit2/pull/6191

xyzzy_plugh 4y ago
libgit2 isn't an official library, and even if it did support sha256 dependents would still need to update, so I really don't perceive this as a missing piece. If everyone started using sha256 then all these problems would be addressed practically overnight.

ivoras 4y ago

Is there an explanation of what would go wrong with the naive approach? E.g.: - Change the binary file format in repos to support arbitrary hash algorithms, in a way which unambigously makes old software fail. - Increment the Git major version number to 3.0 - Make the new version support both the old version repos and the new ones. Make it a per-repo config item that allows/disallows old/new hash formats. In theory, there's nothing wrong with having objects hashed with mixed algorithms as long as the software knows how to deal with that. - The old format will probably have to be supported forever because of Linux. Most user-facing utilities don't care what the hash algo actually is, they just use the hash as an opaque string.

deleted 4y ago
[deleted]
runeks 4y ago
Releasing new software is the simple part. The problem is that versioning is lacking in the old software, and therefore it doesn’t know how to talk to the new software. So for the old software there’s no difference between “invalid data” and “I’m too old, please upgrade me”.
- dingleberry420 4y ago
  > So for the old software there’s no difference between “invalid data” and “I’m too old, please upgrade me”. And why is this an issue? Release the new version that can read new repo formats, but doesn't write them yet. Wait a year. Release new version that can write new repo formats and encourage users to upgrade. Anyone who hasn't upgraded in the past year probably doesn't care about security and should be left behind. Besides, once they google the error message they'll figure it out soon enough. It's not like git is known for its great UX anyway.
  - hamilyon2 4y ago
    I might be mistaken, but github could be using their own version of git and accompanying tools. So, unless they implement uprade themselves, no amount of waiting will make git interoperable with them.

wepple 4y ago

I was curious about the “sha1dc” that git uses and reportedly helps protect against collision attacks. Here’s the paper: https://marc-stevens.nl/research/papers/C13-S.pdf https://marc-stevens.nl/research/papers/C13-S.pdf

O__________O 4y ago

Anyone aware of any exploits tied the SHA-1 weakness in the wild? (I have seen proofs of concept [1], but never actually heard of an exploit in the wild using it; for example, on: digital certificate signatures, email PGP/GPG signatures, software vendor signatures, software updates, ISO checksums, backup systems, deduplication systems, Git, etc.) [1] https://shattered.io/ https://shattered.io/

password4321 4y ago
Applications of that collision: https://twitter.com/rauchg/status/834770508633694208 https://twitter.com/rauchg/status/834770508633694208 > a SHA-1 "Pinata" [...] claimed https://news.ycombinator.com/item?id=13723892 https://news.ycombinator.com/item?id=13723892 > Make your own colliding PDFs https://news.ycombinator.com/item?id=13917990 https://news.ycombinator.com/item?id=13917990 > Collision Detection
bawolff 4y ago
Most security critical systems have switched to sha256 at this point, and making a fresh collision still costs tens of thousands, so people arent really doing it for kicks (that said, once you have one collision you can reuse it for free as long as you keep the same prefix, so the proof of concept can be repurposed with certain constraints). The most in the wild one i have ever heard of was when webkit accidentally broke their svn repo by checking in a collision. However you can look at the history of md5 which had a similar flaw which was exploited by the flame malware.
- O__________O 4y ago
  Thanks, agree the Flame’s use of a collision attack was both comparable and notable: https://en.m.wikipedia.org/wiki/Flame_(malware) https://en.m.wikipedia.org/wiki/Flame_(malware)

encryptluks2 4y ago

It seems like something more modern, like b3sum would be better... no? What about b2sum?

kortex 4y ago
I love the performance of blake3 but my understanding is it's still a bit of the new kid on the block. Blake2 is a SHA-3 finalist so should be perfectly sufficient, plus it has variable digest sizes, reasonably fast, and other nice features. Either way, anything relying on hashes for data integrity should at least be flexible to the option of multiple hash algos. But with git, it's going to be hard enough as is to change to SHA-256, and I don't know how parametric it'll be.
- notfed 4y ago
  Actually, BLAKE2 was not a SHA-3 finalist, BLAKE was. Also, BLAKE3 is so significantly faster than it's probably worth waiting for (waiting for it to get vetted cryptanalysis-wise).
MrStonedOne 4y ago
rockdoe 4y ago
I mean, SHA-3?
- encryptluks2 4y ago
  Blake3 is faster and more secure. I also really don't trust NIST to be honest when they have a history of promoting solutions with backdoors.
  - oconnor663 4y ago
    I'm not aware of any reason SHA-3 would be less secure than BLAKE3. Quite the opposite: SHA-3 has a very large security margin, so large that the authors cut the number of rounds in half for KangarooTwelve.

chmaynard 4y ago

Previous articles on this topic: A new hash algorithm for Git https://lwn.net/Articles/811068/ https://lwn.net/Articles/811068/ Updating the Git protocol for SHA-256 https://lwn.net/Articles/823352/ https://lwn.net/Articles/823352/

kazinator 4y ago

> Given the threat that the SHA-1 hash poses I give -3 flying ducks about this, and don't want the Git storage format to be diddled with in any way. Git in 2122 should read and write a git repo made in 2010. Git is not a public crypto system. If you think a commit is important and needs to be signed, you need to sign the files and add the signature to the commit.

kzrdude 4y ago
Migrating to a non broken hash is the better chance at being a century long strong repository format.

pornel 4y ago

The worst thing about the SHA-1 collision is the tedium of explaining the difference between a collision attack and a preimage attack.

heynowheynow 4y ago

It might be wiser to keep SHA1 and use SHA2, SHA3, etc. and GPG as overlays for compatibility and simplicity reasons.

harryvederci 4y ago

Relevant quote from the Fossil website[0]: "Fossil started out using 160-bit SHA-1 hashes to identify check-ins, just as in Git. That changed in early 2017 when news of the SHAttered attack broke, demonstrating that SHA-1 collisions were now practical to create. Two weeks later, the creator of Fossil delivered a new release allowing a clean migration to 256-bit SHA-3 with full backwards compatibility to old SHA-1 based repositories. [...] Meanwhile, the Git community took until August 2018 to publish their first plan for solving the same problem by moving to SHA-256, a variant of the older SHA-2 algorithm. As of this writing in February 2020, that plan hasn't been implemented, as far as this author is aware, but there is now a competing SHA-256 based plan which requires complete repository conversion from SHA-1 to SHA-256, breaking all public hashes in the repo." [0]: https://fossil-scm.org/home/doc/trunk/www/fossil-v-git.wiki#hash https://fossil-scm.org/home/doc/trunk/www/fossil-v-git.wiki#...

ludwigvan 4y ago
Migrations are easier when you are the only one using your software. :p Joking aside, expected from a developer whose work is the recommended storage format for Library of Congress.
tremon 4y ago
This raises an interesting point: given that git has been dragging its feet for so long on the transition to SHA-256, it's better if they were to drag their feet a bit longer and move directly to SHA3-256 too, and never let the current SHA-256 implementation get widely deployed.

YesThatTom2 4y ago

GitHub won’t feel any heat about this until Microsoft salespeople start demanding it. I’ve added to my todo list a reminder to raise this issue with mine. In fact, I’m going to give them a deadline for when we will start evaluating competitors that do support SHA256. I suspect that most people on HN do not interact with their MS account team. That relationship is probably managed by your CIO or IT department. They probably have monthly or quarterly “business review” meetings. You should get this issue on the agenda of that meeting.

sdfhdhjdw3 4y ago
Thank you.
codazoda 4y ago
Is there something special about GitHub on this? This seems like a Git issue and not a GitHub issue to me; unless I'm missing something.
- lucb1e 4y ago
  They don't accept pushes of repositories in that format. The article says "none of the Git hosting providers appear to be supporting SHA-256", and while GH is not mentioned by name (and I applaud them for indeed not strengthening this "git == github-the-brand" trap), I can't imagine GH was left out of scope when checking the major hosting providers.
- evil-olive 4y ago
  as the article says, you can create a local git repository with SHA-256 hashes today, and it should work fine...but the moment you try to push your repo up to Github, you'll hit a brick wall. Gitlab also appears to be lacking support [0], and the same with Gitea [1]. so it's a grey area where Git itself supports SHA-256-based repos, but without the major Git hosting services also supporting them, the support in core Git is somewhat useless. 0: https://gitlab.com/groups/gitlab-org/-/epics/794 https://gitlab.com/groups/gitlab-org/-/epics/794 1: https://github.com/go-gitea/gitea/issues/13794 https://github.com/go-gitea/gitea/issues/13794

slim 4y ago

Why didn't Linux migrate their repo to SHA-256 ? is it that difficult to migrate a repo ?

sdfhdhjdw3 4y ago

> Even if creating a collision were feasible for an attacker, Bjarmason pointed out, that is only the first step in the development of a successful attack. Finding a collision of any type is hard; finding one that is still working code, that has the functionality the attacker is after, and that looks reasonable to both humans and compilers is quite a bit harder — if it is possible at all. Sounds like there's money in this.

jiggawatts 4y ago

If you’re going to “fix” the hash algorithm, do it properly! Sha256 can only be computed in a single sequential stream (thread) by definition. For large files this is increasingly becoming a performance limitation. A Merkle tree based on SHA512 would have significant benefits. SHA512 is faster than SHA256 on modern CPUs because processes 64 bits per internal register instead of 32 bits. A tree-structured hash can be parallelised across all cores. For repositories with files over 100MB in them on an SSD this would make a noticeable difference…

dchest 4y ago
Most git objects are tiny files, so internal tree-based parallelization won't bring much compared to file parallelization (git is a hash tree itself, with variable-length leaves). SHA256 is actually a lot faster on modern CPUs due to https://en.wikipedia.org/wiki/Intel_SHA_extensions https://en.wikipedia.org/wiki/Intel_SHA_extensions (and similar on Arm), which are implemented for SHA-256 but not for SHA-512, e.g. openssl speed sha256 sha512 on M1: type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes sha256 89474.97k 283341.15k 901724.41k 1730980.24k 2339109.86k sha512 66160.19k 262139.03k 365675.96k 487572.26k 545142.91k
- jiggawatts 4y ago
  A fair point about the instruction sets, and it is also true that “most” files are small. But again, due precisely to their size, large files take a disproportionate amount of time to process. Don’t confuse the typical use-case with the fundamental concept: versioning. Git could be a general purpose versioning system with many more use-cases, but limitations like this hold it back unnecessarily…
  - dchest 4y ago
    Hashing is not the only thing that stops git from being useful for large file versioning. For this purpose, splitting files into chunks using a rolling hash (similar to how git packs, rsync, tarsnap or IPFS) would work better. This again doesn't require "internal" tree hashing, since each chunk would be hashed separately.

avar 4y ago

I'm the person and Git developer (Ævar) quoted in the article. I didn't expect this to end up on LWN. I'm happy to answer any questions here that people might have. I don't think the LWN article can be said to take anything out of context. But I think it's worth empathizing that this is a thread on the Git ML in response to a user who's asking if Git/SHA-256 is something "that users should start changing over to[?]". I stand by the comments that I think the current state of Git is that we shouldn't be recommending to users that they use SHA-256 repositories without explaining some major caveats, mainly to do with third party software support, particularly the lack of support from the big online "forges". But I don't think there's any disagreement in the Git development community (and certainly not from me) that Git should be moving towards migrating away from SHA-1.

lalaland1125 4y ago
Have you considered moving over to a combined SHA-1, SHA-256 model where both hashes are calculated, with SHA-1 shown to the user and SHA-256 only used in the background to prevent collisions? There is a compute cost for that, but it should be minimal relative to the security benefits?
- pmarreck 4y ago
  This is what I would have recommended... A transition period where both are used
- avar 4y ago
  Someone probably brought it up at some point, I can't remember. But I'm not aware of any known scenario where the SHA1DC library Git uses doesn't give you the benefits of that and more. "And more" because to detect a collision with a background SHA-256 you'll need both objects, whereas SHA1DC detects attempts to spoof SHA1 in a way that leads to collisions. So it won't pass along an object that collides with another one, even though it only has 1/2 objects. That distinction is something that's generally considered important, e.g. there's been past exploits in Git where you could trick a client into doing something bad by e.g. a crafted .gitmodules file. The fix has not only been to patch clients, but also to patch "git fsck" to detect and reject such bad contents, so that e.g. the forges can't be used to relay a repository exploit to users running older versions. A viable hash collision exploit in the wild might likewise want to make use of such an attack scenarios, so having servers capable of detecting collisions without having both sides is preferable to doing so by re-hashing with SHA-256.

hrgiger 4y ago

Probably performance impact is little but still noticeable for a large project ,maybe even with modern hardware but wouldnt a portable approach would be better? Not the security but performance perspective maybe one would prefer cheap calculation, i.e. ext4 pre-calculate already crc32c , just found out google added alternative highwayhash recommends and claims it is faster. https://github.com/google/highwayhash https://github.com/google/highwayhash

janwas 4y ago
Author here. HighwayHash is indeed reasonably fast for a PRF/MAC/fingerprint, but our security claims are not strong enough for it to serve as a cryptographic hash. No collision nor preimage attacks are known to us, but it would not be appropriate to use HighwayHash in this context.
- hrgiger 4y ago
  Thanks for the response, yes I can understand the preference in security context and dont want to drop 'why cryptographic hash' bomb into discussion, I was just thinking maybe keeping sha256/512 as default but having switchable options through the configuration for alternatives like your implementation or proven implementations or using filesystem features. No matter what that is laziness from my side, just throwing question rather than spending time on source.

pmarreck 4y ago

Couldn't there be a transition period where both are supported?

hmsimha 4y ago

Regarding the difficulty of generating a collision with working code, wouldn't it be as "trivial" as generating a SHA-1 collision in the first place? Just have the malicious code in a file and append a multiline comment block, then have your collision generator insert random junk into that comment block

ed25519FUUU 4y ago
Generating the “junk” is the extraordinarily challenging part, though it’s been proved possible.
- hmsimha 4y ago
  Sure, I had the word "trivial" in quotes for that reason. I meant that if you're able to generate a collision, the rest of this doesn't follow (from the article) > Even if creating a collision were feasible for an attacker, Bjarmason pointed out, that is only the first step in the development of a successful attack. Finding a collision of any type is hard; finding one that is still working code, that has the functionality the attacker is after, and that looks reasonable to both humans and compilers is quite a bit harder — if it is possible at all.

phlogisticfugu 4y ago

at the risk of sounding like a heretic: if backwards compatibility is so hard, why not break it? replace git with git2 in all places (command line, URLs, protocol, etc) and make a git1to2 utility, and you are golden

londons_explore 4y ago

Normally migrations of this type are done in two phases: * Add support for new hash * Migrate all data to new hash, dropping support for clients that don't support it. It appears we're in the waiting phase between the two bullet points. I imagine that could be many years, because many people don't update their git clients often.

userbinator 4y ago

MD5 is still strong to preimage attacks although you can generate collisions in seconds now. Thought experiment: What if Git had used MD5? The news has been proclaimed loudly and often: the SHA-1 hash algorithm is terminally broken and should not be used in any situation where security matters. There are organizations where SHA-1 is blanket banned across the board - regardless of its use I think it's about time people understood the different types of attacks and what they mean for various uses of hash functions instead of blindly cargo-culting as the security (aka paranoia) industry seems to want to do. Then again, the people in those same organizations who promote this sort of anti-thinking would probably suddenly not care if you just renamed all occurrences of "SHA1" in everything they see to something else, because they are such incompetent idiots anyway --- true stories from experience...

iraqmtpizza 4y ago

Curious why SHA-256 and not SHA-512 or SHA-512/256 which have higher throughput. Interoperability? Hardware acceleration?

rnhmjoj 4y ago

I hope this doesn't turn into an IPv4/IPv6 situation, but it looks like it's directly headed that way. Without a proper transition mechanism companies won't implement SHA-256 because there's no demand for it and users can't switch until companies support SHA-256.

red_admiral 4y ago

Is SHA-1 being used in a security-critical way in git, though? I tend to agree with the points made towards the end of the article. If someone malicious can make commits to your repo, you have a much worse problem than a possible hash collision. Most languages these days have a built-in hashtable type, and they use some kind of non-security-critical hash function, albeit usually with a random seed, that's a lot weaker than SHA-1 - often you only need something like 32-64 bits of output in the first place as no-one allocates more than 2^64 buckets as far as I know. From a security perspective that's even worse than MD5! I'd argue (and so does the article, in part) that the use of SHA-1 in git is closer to the use of a hash in a hashtable than to the use of a cryptographic checksum - using SHA-1 for subresource integrity on the web is an obvious security risk (luckily, it's not allowed there) but I can't see an obvious attack on git's use that doesn't also assume you can do much worse things.

haileys 4y ago
> If someone malicious can make commits to your repo, you have a much worse problem than a possible hash collision. This is exactly how GitHub works though: all repositories in a fork network share a single underlying repository, only the ref namespaces are separated.
- wildfire 4y ago
  I am skeptical that is actually the case though, there are so many use-cases where you have to think through the consequences too carefully: - what happens when the source repository rewrites all the commits (to remove one); should the forks also be re-written? - what happens when the source repository disappears? _All_ of the problems go away when you actually do a "git clone" of the source repository and have your own objects. More storage, yes, but way fewer corner cases to think through.
deleted 4y ago
[deleted]

billpg 4y ago

I'm about to start a new repo. Should I use the SHA256 option?