14 ms·
Linus on Git and SHA-1
- claar 10y agoAlso see discussion of Linus's earlier comments at https://news.ycombinator.com/item?id=13719368 https://news.ycombinator.com/item?id=13719368
- bascule 10y agoLinus's transition plan seems to involve truncating SHA-256 to 160-bits. This is bad for several reasons: - Truncating to 160-bits still has a birthday bound at 80-bits. That would still require a lot more brute force than the 2^63 computations involved to find this collision, but it is much weaker than is generally considered secure - Post-quantum, this means there will only be 80-bits of preimage resistance (Also: if he's going to truncate a hash, he use SHA-512, which will be faster on 64-bit platforms) Do either of these weak security levels impact Git? Preimage resistance does matter if we're worried about attackers reversing commit hashes back into their contents. Linus doesn't seem to care about this one, but I think he should. Collision resistance absolutely matters for the commit signing case, and once again Linus is downplaying this. He starts off talking about how they're not doing that, then halfway through adding a "oh wait but some people do that", then trying to downplay it again by talking about how an attacker would need to influence the original commit. Of course, this happens all the time: it's called a pull request. Linus insists that prior proper source code review will prevent an attacker who sends you a malicious pull request from being able to pull off a chosen prefix collision. I have doubts about that, especially in any repos containing binary blobs (and especially if those binary blobs are executables) Linus just doesn't take this stuff seriously. I really wish he would, though.
- tytso 10y agoThat's not the plan. That was an idea that was thrown out if this was an emergency (it's handling different length hashes, and doing so that we don't have to force a flag day conversion which is hard), but once people realized that in fact, the sky was not following, the plan which Linus outlined in his G+ post was devised --- which does not involve truncating a 256-bit hash.
- bascule 10y agoCan you please link me to "the plan" then? I have been trying to follow some of the ML discussion and that was the last plan I saw him put forth, e.g.: https://marc.info/?l=git&m=148787047422954 https://marc.info/?l=git&m=148787047422954
- pedrocr 10y agoBy clicking "next in thread" in that very post you get Linus replying to himself with a non-truncating plan: https://marc.info/?l=git&m=148787163023435&w=2 https://marc.info/?l=git&m=148787163023435&w=2
- bascule 10y agoThanks for the link. Re: snark, I did hit "next in thread" but managed to skim over that in his response. But again, thanks anyway. (Note: this still sounds more like spitballing than "the plan", but at least it's a step in the right direction)
- greglindahl 10y agoDo you have any insight into the sha-256 vs sha-512 choice?
- eslaught 10y agoCan you link the new plan? I didn't see it described in his G+ post.
- yawaramin 10y agoThe G+ post was not meant to describe the plan, it was an overview of the situation for git users who aren't necessarily security experts. The plan is outlined at https://marc.info/?l=git&m=148787163023435&w=2 https://marc.info/?l=git&m=148787163023435&w=2
- deleted 10y ago[deleted]
- ksherlock 10y agoOn the mailing list he suggested displaying 160 bits for backwards compatibility with existing tools. The full 256 bits would be used internally (and displayed externally with the proper --flag)
- diminish 10y agoadditionally cosmetics but instead of hex string could they not use a-zA-Z0-9 in the visual output to the user to make the git command line text output shorter? 0efaa. -> 2AdC..
- pshc 10y agoUnfortunately there's a lot of code out there that assumes commit hashes are case-insensitive and hexadecimal... I've written some myself. It'd be a hugely painful change.
- runeks 10y agoThen you have stuff like 1, I and l, which are difficult to distinguish. Which was why base58 was invented (basically the range you suggested, without visually similar characters).
- kzrdude 10y agoThis really should not be a problem since that's task #1 that a programmer's good font needs to solve.
- cbr 10y agoI often run into git hashes places where a programming font wouldn't apply, like URLs and web pages.
- wongarsu 10y ago>where a programming font wouldn't apply, like URLs Any phisher's wet dream is a URL displayed in a font where l and I look the same. For anything that routinely displays URLs, font matters a lot. And any website displaying git hashes is likely to be programming related and have some code-font (good monospace or similar) in it's repertoir.
- paulddraper 10y ago> especially in any repos containing binary blobs Yeah, that part is the real flaw in the argument.
- hermitdev 10y agoAlthough, I generally defer to someone such as Linus in having far more domain knowledge such as this, but I'm concerned willingness to just drop bits of the hash like this. I mean 20-30 years ago, I could quasi understand for the sake of performance, but are we really so concerned about performance of a few clock cycles vs opening ourselves to a known vulnerable attack?
- hdhzy 10y agoI think the truncation is considered because a lot of software assumes git commit id has 160 bit length. It's not because of performance.
- hermitdev 10y agoThere are only 2 reasonable rationales for the 160 bit restriction: it was arbitrary because that was the length of the chosen hash, or for performance (and no one will ever need more that 640kb of RAM, or 160 bits of hash). I'm guessing it was probably the first, though, and Linus designed for the immediate now, and not for the future.
- derefr 10y agoIt's not even a "restriction", so much as a fragile ecosystem of tools that parse the output of git commands. A lot of them assume particular column widths.
- hueving 10y agoWhat is this known vulnerable attack you are talking about? The birthday paradox applies to every hashing function.
- hermitdev 10y agoWhile there may not be a known, direct, attack against git, WebKit's SVN repository was demonstrably attacked by the SHA-1 vulnerability, oddly by their own developers today (or yesterday?). That it took that little time from a PoC to an actual production issue leads me to believe it wouldn't take long for a dedicated individual or team to extend the vulnerability to git and it's "mitigation" involving the header. I prefer not to take the bury my head in the sand approach to this, at least when it comes to public repositories.
- deleted 10y ago[deleted]
- deleted 10y ago[deleted]
- the_mitsuhiko 10y ago> Linus's transition plan seems to involve truncating SHA-256 to 160-bits. This is not the plan. This is the backwards compatible system for tools that can only deal with 40 characters.
- chainsaw10 10y ago> reversing commit hashes back into their contents Somewhat off topic, but is this actually possible? Given hashing is inherently lossy, I'm inclined to assume it's not possible for anything must longer than a password, but commits are text, which I suppose is low entropy per character, so I don't know.
- bascule 10y agoYes this is possible. It's called a preimage attack: https://en.wikipedia.org/wiki/Preimage_attack https://en.wikipedia.org/wiki/Preimage_attack It's computationally infeasible against most cryptographically secure hash functions. However, Grover's algorithm results in a sqrt(keyspace) reduction in security levels (effectively halving bit-sized security levels): https://en.wikipedia.org/wiki/Grover's_algorithm https://en.wikipedia.org/wiki/Grover's_algorithm
- AgentME 10y agoA successful preimage attack against a cryptographic hash would most likely find you a different message that gets you the same hash. You wouldn't do a preimage attack to find what the original input to create a hash was. You would do a preimage attack to find a new input with the same hash so that you could pass this new input around claiming it was the original (and have any signatures for the old input be valid for your new input, etc).
- oleganza 10y agoWhat you described is called "second preimage attack", that's != "first preimage attack".
- AgentME 10y agoIn a second preimage attack, you start with an input that comes out to a hash, and you're tasked with making a distinct input that comes out to a hash. In a first preimage attack, you start with only the hash, and have to come up with an input from scratch that comes out to that hash. Even in a first preimage attack, you're extremely unlikely to craft the same input that someone else used to create the hash you were given. By pigeonhole principle, the number of inputs that come out to a shorter hash is extreme.
- deleted 10y ago[deleted]
- hueving 10y ago>Preimage resistance does matter Is there a preimage weakness though? I thought this attack only reduced collision resistance.
- OJFord 10y agoGP said that in reference to quantum attacks.
- mappu 10y ago>(Also: if he's going to truncate a hash, he use SHA-512, which will be faster on 64-bit platforms) BLAKE2 is faster still! It's also at least as secure as SHA-3, and produces any choice of output size up to 512bit.
- stouset 10y agoI'm a fan of blake2, but SHA-2 is the conservative choice here given the sheer amount of cryptanalysis it's been subjected to.
- gkya 10y agoIf a repo contains binary blobs, especially executables, well, that's very bad practices right there. Also, how can somebody else modify a binary in a meaningful way and send a patch to it? How can you review a patch to a binary file before applying? I'd say that any sane project, especially if open source, would not include binaries (maybe apart from images), and even if it did, would not accept patches to them (if the members of a project were talking about changing, say, an icon, valid images would be exchanged on a mailing list/issue tracker, nobody would bother making, sending and applying binary diffs; then somebody w/ commit bit just commit it). Let me put it more simply: if you're accepting patches for binary files in your repos you don't care about security at all. Maybe unless if you know how to decode machine code/JPEG manually. Also, the proposition was to use the full hash internally, and truncate the representation. Nobody other than git itself uses full hashes anyways.
- wongarsu 10y agoExecutables in git are bad practise, but not all that uncommon. Images in git are the norm, and if somebody comes in and creates a pull request with an improved version of the existing images (better compression, better adapted for color blind people, fixing whitespace issues etc.) that's pretty unsuspicious and likely to succeed (and I've seen it multiple times).
- usrusr 10y agoBut how would a safer hash improve resistance to that blob update scenario? How would a weaker hash make it more dangerous? If you have a fiercely audited branch next to one that is basically free for all, maybe?
- Dylan16807 10y ago- Truncating to 160-bits still has a birthday bound at 80-bits. That would still require a lot more brute force than the 2^63 computations involved to find this collision, but it is much weaker than is generally considered secure What level is considered secure, then? The numbers for O(time) and O(space) should be many orders of magnitude apart to represent the relative costs.
- simplehuman 10y ago> Linus just doesn't take this stuff seriously. I really wish he would, though. Can't downvote this enough. This is plain FUD. Did you even read the complete thread on the git mailing list? This was just one proposal by him.
- AgentME 10y agoIf he took this stuff seriously, he wouldn't have waited 12 years since SHA-1 was broken to even start considering any changes.
- simplehuman 10y agoSo did anyone on this thread send patched to the mailing list? Obviously you guys are very serious right?
- aseipp 10y agoThis conversation has gone on several times over the course of years on the Git mailing list. In almost every situation it's been completely brushed off as a mostly non-issue to change from SHA-1 inside Git, despite the fact it's been known SHA-1 has basically been on life support. There are lots of opinions on both sides, but ultimately, until now, the Upstream decision seemed to be "WONTFIX". Given this context, of course nobody wrote patches: they would have obviously been rejected and been a total waste of time. Until now, when we actually have to deal with it. This is all aside from your argument being fundamentally weak, however ("you can't criticize anything unless i say so and contributed by meeting this arbitrary standards. i mean, didn't do anything either, i just get to make up the rules you abide by!!")
- simplehuman 10y ago> This is all aside from your argument being fundamentally weak, however ("you can't criticize anything unless i say so and contributed by meeting this arbitrary standards. i mean, didn't do anything either, i just get to make up the rules you abide by!!") Are we both reading the same GP comment? It reads as "If he took this stuff seriously, he wouldn't have waited 12 years since SHA-1 was broken to even start considering any changes.".
- infinity0 10y ago> "A hash that is used for security is basically a statement of trust [..] In contrast, in a project like git, the hash isn't used for "trust". I don't pull on peoples trees because they have a hash of a4d442663580. Our trust is in people, and then we end up having lots of technology measures in place to secure the actual data." This is horseshit, and Linus should not be saying these hugely misleading statements about security principles. The point of a hash is to remove the need for trust between the trusted person who tells you the hash and the infrastructure you get the actual data that was hashed, from (edit: and, between you and the latter). In other words, once you get a good non-colliding hash from a trusted person, then you don't need to worry about malicious infrastructure sending you bad data claiming to be the source of that hash. Linus trusting Tytso to sign the commit object that references the SHA-1 of the tree object, says nothing about whether the infrastructure served him the tree object correctly. Sure, he might also trust the infrastructure providers, but when he says "trusts people" it does not sound like that is what he means. And even if he trusts the infrastructure providers, with a good hash HE DOESN'T HAVE TO. The "trust" wording is serious horseshit. (edit: there is also the case of people downloading "linux" from random git repos in the future. Right now if you GPG-sign a commit or tag, it has SHA-1 references to the tree object underneath it. Once SHA-1 is more broken it basically means you shouldn't trust random git repos across the internet to give you good content, even if it's "signed by Linus".)
- bascule 10y agoYes, exactly. If Linus truly doesn't care about security, then git could use any error correcting code that produces a uniform distribution of tags, such as CRC64. The size of the tag only affects the number of objects we'd expect to be able to commit before we see a collision: over 4 billion in the case of CRC64. Linux mistakenly claims using a cryptographic hash function helps avoid non-malicious collisions, but this is not the case. Where the choice of a cryptographic hash function matters is specifically if we expect an attacker to be trying to collide tags. CRC64 is a linear function of the input data and therefore fails miserably at preventing attackers from colliding tags, but still produces a uniform distribution of tags for non-malicious inputs. git seems to be in the odd place where Linus argues he's using a cryptographic hash function but not for security purposes.
- deleted 10y ago[deleted]
- amluto 10y ago> - Post-quantum, this means there will only be 80-bits of preimage resistance Post-quantum, it's only ~2^53 work to find a collision. IMO that's worse.
- ploxiln 10y agoIf anyone is really interested in more assurance of git commit contents, there's "git-evtag", which does a sha-512 hash over the full contents of a commit, including all trees and blob contents. https://github.com/cgwalters/git-evtag https://github.com/cgwalters/git-evtag
- debatem1 10y agoI'm mystified as to why this is even a discussion. SHA1 is busted. That impacts some git users. The fix is not invasive. Fix the bug. Make the transition. Move on. Super unprofessional.
- deleted 10y ago[deleted]
- AsyncAwait 10y agoIt's not that simple. Git is a widely used software integrated into many places, so keeping some backwards compatibility is important. Just going ahead and start breaking things would really be unprofessional.
- debatem1 10y agoI'm not saying he should go out and break things now. I'm saying that the way forward he outlines could have been followed years ago, as essentially the rest of the industry did. Instead he waited until his user base was totally panicked to address their concerns. In my view, that's not the right-- or even a rational-- way to do things.
- snakeanus 10y agoThey did break backwards compatibility with git v2.0 but sadly they did not bother to change the hash function.
- joatmon-snoo 10y agoWhat backwards compatibility was broken?
- Kubuxu 10y agoLittle bits in CLI interface https://blogs.atlassian.com/2014/06/happened-git-2-0-full-goodies/ https://blogs.atlassian.com/2014/06/happened-git-2-0-full-go...
- dboreham 10y agoUm what? Software written in the past 20 years has a baked-in assumption that the length of some ID can't change?
- theseoafs 10y agoHave there been writings on what exactly git's migration strategy to a new hash function will be? Apparently they have a seamless transition designed that won't require anyone to update their repositories, which seems like a pretty crazy promise in the absence of details.
- keeperofdakeys 10y agoIn git the SHA-1 hash is simply an identifier for an object - it's used in the filename, but not stored in the object. And when a commit or tree object references others, it's just a name that can be looked up in the database. So a commit object hashed with SHA-256 can easily reference a previous commit that was hashed with SHA-1. During the switch, a bit of deduplication may be lost. But the only interesting issue I can see is how git fsck will tell which hash an object was created with when verifying the hash (maybe with length?).
- iamgopal 10y agoGit update repo kind of command may be ??
- runeks 10y agoOne thing SHA-256 has going for it is that millions can be made from finding pre-image weaknesses in it, because it's used in Bitcoin mining. If you could "figure out" SHA-256, and use it to take over Bitcoin mining, you'd make $2M the first 24 hours, at current rates. And if you play it wise, it could take a long time before anyone figure out what's going on. With regards to market price for a successful attack, I don't think any hash function stands close to SHA-256. And for that reason I think it would be the right choice.
- wolf550e 10y agohttps://twitter.com/veorq/status/834872988445065218 https://twitter.com/veorq/status/834872988445065218
- runeks 10y agoI think the claim is interesting, and I certainly wouldn't reject it. But just relaying someone else's claim, without any substantiating argument, I find quite uninteresting.
- wolf550e 10y agoJean Philippe Aumasson[1] is an important currently active cryptographer who studied this particular problem closely. If he says he does not expect an algorithmic break of SHA2, I believe him that I don't need to plan to switch from SHA2 to SHA3 in an hurry. I don't have the math to understand a more detailed explanation for his reasons to make such a statement. If you do have the math, ask him. 1 - https://131002.net/index.html https://131002.net/index.html
- kzrdude 10y agoSHA-512/256 is faster and has the same output length
- colin_fraizer 10y agoThis, btw, is why we have e-cigarette bans. The fact that the generally high-IQ, paid-to-think-about-subtle-categorization community of software developers needs to be inoculated against the "I Heard SHA-1 Was Bad Now" meme, should serve as a reminder for why most things should not be managed by democracy. (Yeah, I know this will be read as a plea for monarchy and downvoted. It simply proves my point: people are WAY too subject to errors in the classes (1) "I hate him because he said something 'bad' about something 'good'." and (2) "I hate him because he said something 'good' about something The Tribe now knows is 'bad.')
- grzm 10y agoSave yourself some downvotes and remove the mention that you expect them.
- colin_fraizer 10y agoSave myself from people proving my point? Why?
- grzm 10y agoThe HN guidelines specifically ask not to express the expectation of downvotes. You may be downvoted purely for ignoring the guidelines, regardless of the rest of your comment. Please don't bait other users by inviting them to downvote you or proclaim that you expect to get downvoted. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- colin_fraizer 10y agoA generally reasonable guideline, but in this case, I am actually criticizing the tribalism that makes people rise to that bait. It similarly leads to the discussion of how "I can't believe Linus is trying to defend SHA-1 when The Tribe already knows it is cryptographically 'bad'."
- butwhynotmore 10y agoIn the specific case of cryptography where it's unknown how bulletproof the algorithm will be why not use multiple hash functions? Perhaps using the top 10 best hash functions of the day. That way you're not putting all your eggs in one basket and if nefarious collisions are able to be created in the future you still have the other hash functions to both "trust" and double check against. It's even more unlikely that nefarious collisions will be able to be constructed that collide all the other hash functions as well. You could just append the hashes to each other or put them in a hash table or something. Maybe my computer science is not up to snuff but it seems like this would provide more resiliency against future and non-public mathematical breakthroughs as well as increased computing power such as quantum computing. Yes, it would take a little longer to compute all the hashes in day to day use, but with the benefit of a more robust system both now and in the future.
- hannob 10y agoOne thing that I think is worth mentioning: This was completely avoidable. Git isn't that old, it wasn't taken by surprise by the SHA1 attacks. The first paper from Wang et al, which should've put SHA1 to rest, was published in 2004, the year before the first ever Git version was released. It could have been easy: Just take a secure hash from the beginning.
- simias 10y agoWhile this post sounds very reasonable to me there's one point that I really don't get: why does he keep saying that git commit hashes have nothing to do with security? If he believes that, why does git allow signing tags and commits and why does Linus himself sign kernel release tags? Isn't that the very definition of "using a hash for security"?
- jmount 10y agoProbably isn'y the sky falling. But if knowing the length fixed all hash function issues then cryptographic hashes would just use a some more bits for length.
- maxander 10y agoI don't really get the threat model here. If an attacker is pushing commits into your repository, you're long since toast on all possible security fronts, right? Is there anything nefarious they could accomplish through hash collisions that couldn't be done simply by editing commit history?
- azernik 10y agoIf they edit the commit history and you're using a secure hash algorithm, then the hash of the current commit will change and no longer match the signed tag your trusted maintainer sent you.
- deleted 10y ago[deleted]
- leoh 10y agoNot really. From Linus — I think the most important point that has not been discussed extensively: > But if you use git for source control like in the kernel, the stuff you really care about is source code, which is very much a transparent medium. If somebody inserts random odd generated crud in the middle of your source code, you will absolutely notice.
- yeukhon 10y agoWell, open source software like Linux kernel has many obvious bugs for years go without notice. But his point is still extremely fair. There may be code no one really actively updating, but to even able to add random odd crud in the source code is really really really really hard. The effort is not worth it for any reasonable attackers. One would rather exploit an existing zero day or just infect network and wait to activate the infection. If someone modifies the local source code you just git cloned (or from a tarball), well you have something to worry about - your network.
- astrodust 10y agoThe kernel, and OpenSSH to name another project, is full of random crud nobody noticed. Everyone does their best to review patches and ensure things are sane when going in, but there's a lot of change in projects at scale, and sometimes the standards one patch wrangler has are different from another. Mistakes happen. If at-patch time is your only code review, you've got problems. If your code base is too huge to look over frequently you've got problems.
- hackuser 10y agoRelated, from Mozilla: * The end of SHA-1 on the Public Web https://blog.mozilla.org/security/2017/02/23/the-end-of-sha-1-on-the-public-web/ https://blog.mozilla.org/security/2017/02/23/the-end-of-sha-... As announced last fall, we’ve been disabling SHA-1 for increasing numbers of Firefox users since the release of Firefox 51 using a gradual phase-in technique. Tomorrow [Feb 24th], this deprecation policy will reach all Firefox users. It is enabled by default in Firefox 52.
- azernik 10y agoTo be fair (although I've been commenting angrily about git's continued use of SHA-1 elsewhere) it's a lot easier for a browser to change hash algorithms than for git.
- joatmon-snoo 10y agoThe actual mailing list discussion thread can be found here, and is infinitely more informative than any of the bull being spouted in this thread: http://public-inbox.org/git/20170226004657.zowlojdzqrrcalsm@sigill.intra.peff.net/T/#t http://public-inbox.org/git/20170226004657.zowlojdzqrrcalsm@...
- azernik 10y agoMy takeaway from all that - the git project is internally taking steps to mitigate vulnerabilities from this particular attack (making it harder to insert the arbitrary binary data necessary into git metadata), but a) is just throwing up their hands at the problem of projects that store binary blobs like image data in their repos, and b) is not taking this as a signal that more serious sha-1 attacks are on the horizon and they should speed up their hash-replacement efforts. This latter leads into the problems with Linus' positions in particular. In that thread, does not take seriously the threats that this poses to the broader git userbase, because he only seems to care about the kernel use-case: trusted hosting infrastructure at kernel.org (itself an iffy assumption, given previous hacks and the use of mirrors), and the exclusive storage of human-readable text in the repo which makes binary garbage harder to sneak in. These do not apply to most users of git. His rather extreme public position (paraphrased, "our security doesn't depend on SHA1") is even more troubling - it absolutely does depend on SHA1, this just isn't (yet) a strong enough attack to absolutely screw over the kernel. A stronger collision attack (eg a chosen-prefix as opposed to identical-prefix, or godforbid a pre-image attack) would absolutely invalidate the whole git security model.
- Chyzwar 10y agoSHA1 is only used for uniqueness. You still need to have write access to repo to perform attack. If you already have write access you do not need to make collision...
- groestl 10y agoExactly. Why waste the time to fabricate a collision when all you have to do is "accidentally" commit exploitable code? At least then you got plausible deniability.
- yuhong 10y agoI do wonder how many outside of crypto circles know about SHA-2 circa 2004.
- deleted 10y ago[deleted]
- markiniotome 10y agoLinus don't want to left his comfort zone and do anything until it's too late and the next heartbleed bug spreads around the world.
- frik 10y agoCan someone correct me. SVN/Subversion and GIT are affected by SHA-1 problem. SVN uses SHA-1 internally, but exposes only a numeric int as revision. GIT uses SHA-1 internally and as revision. So if someone commit a modified PDF that collides he can run havoc on both SVN and GIT at the moment. It seems easier to fix the issue in SVN than GIT.
- hannob 10y agoIt's a somewhat different issue. Git can probably not be havoced by committing two colliding files (and doing so would require doing another chosen prefix attach with a git blob header). But git looses cryptographic integrity promises due to this attack (aka: you can have different source trees with different histories leading to the same top commit hash). svn never had any cryptographic integrity to begin with.
- kzrdude 10y agoSure, here's a correction: There has been no evidence of havoc or repo corruption in git. It appears that to even try to attack git you'd need to spend the same amount of work again ($110K or what it was) to create a new collision with the right git object headers.
- luckydude 10y agoLinus is a little behind the times with this comment: ``Other SCM's have used things like CRC's for error detection, although honestly the most common error handling method in most SCM's tends to be "tough luck, maybe your data is there, maybe it isn't, I don't care".'' BitKeeper has an error detection (CRC per block) and error correction (XOR block at the end) system. Any single block loss is correctable. Block sizes vary with file size so large files have to lose a large amount of data to be non-correctable.
- godzilla82 10y agoNewbie question .. can some one please help me understand the attack scenario. if I, as the attacker, want to inject malicious code/binary into a git repo, then I need to write my malicious code/binary in such a way that the resultant hash collides with one of the commits (? Or the last one?) in the repo. Is this correct?