10 ms·

The fact that this key was apparently not stored in an HSM, and that GH employees had access to this private key (allowing them to accidentally push it) means t

by LeonM 4y ago

The fact that this key was apparently not stored in an HSM, and that GH employees had access to this private key (allowing them to accidentally push it) means that effectively all communication with GH since the founding of the company has to be considered compromised. This basically means that, depending on your level of paranoia, you will have to review all code that has ever interacted with Github repositories, and any code pushed or pulled from private repositories can no longer be considered private. Github's customers trust GH/MS with their code, which, for most businesses, is a high value asset. It wouldn't surprise me if this all results in a massive lawsuit. Not just against GH as a company, but also to those involved (like the CISO). Also, how on earth was it never discovered during an audit that the SSH private key was plain text and accessible? How has GH ever been able to become ISO certified last year [0], when they didn't even place their keys in a HSM? Obviously, as a paying customer I am quite angry with GH right now. So I might be overreacting when I write this, but IMO the responsible persons (not talking about the poor dev that pushed the key, but the managers, CISO, auditors, etc.) should be fined, and/or lose their license. [0] https://github.blog/2022-05-16-github-achieves-iso-iec-270012013-certification/ https://github.blog/2022-05-16-github-achieves-iso-iec-27001...

djbusby 4y ago
What license?
- LeonM 4y ago
  Auditors require a license/accreditation to do certain certifications.
brightball 4y ago
I wonder if they found it by turning on their own secret detection system?
Veliladon 4y ago
> The fact that this key was apparently not stored in an HSM, and that GH employees had access to this private key (allowing them to accidentally push it) means that effectively all communication with GH since the founding of the company has to be considered compromised. For a host key? Like I get that being able to impersonate Github isn't great as far as state level actors having the ability to do this but you do know the actual transport layer keys are ephemeral and aren't derived at all from the host key, right?
- LeonM 4y ago
  > state level actors having the ability to do this Not just nation state actors, but basically anyone in a position to MITM. Also, you don't have to be a nation state actor to extort a GH employee. Any bad guy can do a "give me this key or I'll hurt your kid". People are being extorted for a lot less. There are billions of dollars of assets flowing through GH's infrastructure, for the sake of safety (!= security) of Github's employees, nobody should ever have access to key material.
  - palijer 4y ago
    >There are billions of dollars of assets flowing through GH's infrastructure Do you mean source code here? I have a hard time believing source code holds that much value.
    - nicoburns 4y ago
      > As of January 2023, GitHub reported having over 100 million developers and more than 372 million repositories, including at least 28 million public repositories If there are ~350 million private repos then they'd only need to be worth an average of $30 each to be worth a billion dollars in total. Which doesn't seem farfetched.
      - explaininjs 4y ago
        Considering the looooong tail of these repos are forks with no changes, sample code, toy projects abandoned after a single commit, etc. etc., I'd say it's pretty far fetched. For proof, try searching for a mundane string in GH Code search. The vast majority of repos you see will be basically garbage.
      - palijer 4y ago
        I think that is incredibly farfetched. If you got access to 1,000 random private Github repos, I don't think you could sell them, or otherwise utilize them for anywhere near that value, if anything.
      - 4y ago
j16sdiz 4y ago
I have never knew a single person put ssh host key into HSM. In fact, this is not a supported option in openssh.
- LeonM 4y ago
  > I have never knew a single person put ssh host key into HSM. You probably also never met a single person where the SSH interface sees millions of sessions as day with valuable assets (code) being transported over said sessions. > In fact, this is not a supported option in openssh. This definitely is supported. Though documentation for this is often HSM vendor specific, which if heavily NDA'd. So that's why you probably haven't found much information about it.
  - bob1029 4y ago
    > This definitely is supported. Agreed. I have seen some crazy stuff in the payment card industry. I can't recall what I can and can't talk about so I'll just say "Atalla".
    - lxgr 4y ago
      Yes, but that would either be a fork of OpenSSH, private or open source (both are possible since it's BSD-licensed), or a different SSH server (which Github is of course free to use, since the protocol is standardized and their scale absolutely justifies any efforts in protecting their SSH host key). But GPs comment was about OpenSSH. Edit: Apparently OpenSSH's sshd also supports the SSH agent protocol for host keys, and ssh-agent does support PKCS#11 – so I stand corrected!
  - lxgr 4y ago
    > This definitely is supported. Though documentation for this is often HSM vendor specific [...] How can openssh documentation be vendor-specific? Or are you saying that vendors commonly provide an openssh fork/patchset/plugin allowing for HSM-resident host keys?
    - 4y ago
killerstorm 4y ago
> any code pushed or pulled from private repositories can no longer be considered private. Do you realize that the code just sits on GitHub servers even if it's private? If you have any degree of paranoia, why do you put your code into GitHub?!?! Like, if you work on code which
- BlueTemplar 4y ago
  Ah, I see that the men in black got there just in time ! XD
Shank 4y ago
> How has GH ever been able to become ISO certified last year? ISO/IEC 27001:2013 doesn’t say you have to store private keys in HSMs? It just requires you to have a standards compliant ISMS that implements all Annex A controls and all of the clauses. Annex A and the clauses don’t specifically mandate this. If you can convince an auditor that you have controls in-place that meet the standard for protecting cryptographic media you basically meet the standard. The controls can be a wide variety of options and don’t specifically mandate technical implementation details for many, many things. You shouldn’t rely on ISO/IEC 27001:2013 as attestation for technical implementation details here. Just because your auditor would red flag you doesn’t mean all auditors would. The standard is only as effective as the weakest, cheapest auditor, and there are perverse incentives that make auditors financially incentivized to certify companies due to recurring revenue.
- LeonM 4y ago
  > You shouldn’t rely on ISO/IEC 27001:2013 as attestation for technical implementation details here. Thanks for the insight, good advice. But also from the same GH article [0]: The ISO 27001 certification is the latest addition to GitHub’s compliance portfolio, preceded by SOC and ISAE reports, FedRAMP Tailored LiSaaS ATO, and the Cloud Security Alliance CAIQ. Do you have any knowledge on one of these certifications (for exmaple FedRAMP) that puts any restrictions on handling key material? [0] https://github.blog/2022-05-16-github-achieves-iso-iec-270012013-certification/ https://github.blog/2022-05-16-github-achieves-iso-iec-27001...
  - 1970-01-01 4y ago
    I think you're looking for something like PCI DSS compliance, which requires you to store keys in a HSM, and is much more prescriptive with key management.
    - bob1029 4y ago
      PCI-DSS is incredibly aggressive with key management considerations. They get down to radio frequency concerns in pin pad hardware, etc. It took us ~2 years of back & forth with various parties & auditors to get per-client exclusions for accepting end-customer debit PIN codes in-branch on an iPad screen. These banks do not have fully-compliant solutions and must have exceptions on file.
sgt 4y ago
> How has GH ever been able to become ISO certified last year [0], when they didn't even place their keys in a HSM? ISO 27001 certification does not require you to put keys into an HSM. The standard requires you to have controls in place, be aware of your risks and to maintain a risk register. But in no way does the standard require HSM's. The standard would even be OK with storing this on a floppy drive if the risks surrounding that were identified and mitigated (or accepted).
lxgr 4y ago
> [...] depending on your level of paranoia, you will have to review all code that has ever interacted with Github repositories [...] Not to diminish the problems of having a large entity like Github handle a private key like that, but if that was your level of paranoia, you probably should have used commit signatures all along and not relied on Github to do that job for you.
- dannyincolor 4y ago
  As usual on HN, I find the pragmatic response about 3 pages down in the replies to an extremely hyperbolic top-level comment. I also don't want to diminish the concerns around Github or similar orgs losing control of a private key, but the far more realistic concern for the vast majority of threat models is often put to the wayside in favor of what amounts to a scary story. Rather than the straightforward key removal and replacement that this should be, I (and surely many others) have spent all morning combatting this specific FUD that cropped up on HN with leadership and many engineers. It's actually quite detrimental to quickly remediating the actual concerns introduced by this leak. I understand that security inspires people to be as pedantic as possible - that's where some big exploits come from on occasion - but I really hope the average HN narrative changes toward "what is your actual, real-world threat model" vs. "here is a highly theoretical edge-case scenario, applicable to very few, that I'll state as a general fact so everyone will now wonder if they should spend months auditing their codebase and secrets". Put simply: this is why people just start ignoring security measures in the real world. Surely someone has already coined the term "security fatigue". It's all just a bit unbalanced, and definitely becomes frustrating when those suggesting these "world is burning" scenarios didn't even take the available precautions that apparently would satisfy their threat model (i.e. commit sigs, as you suggested) Ok, end rant :)
  - whydoyoucare 4y ago
    Very cogent explanation, and an important point that you highlight - factual real-world risk/threat model is far more important than hypothetical "the-world-is-burning" scenarios. Having a correct threat model is the first step towards building reasonable security controls. But far too many are willing to pander to the "It rather involved being on the other side of this airtight hatchway" [0] scenarios. [0] https://devblogs.microsoft.com/oldnewthing/20060508-22/?p=31283 https://devblogs.microsoft.com/oldnewthing/20060508-22/?p=31...
grumple 4y ago
There's a few reasons I wouldn't worry too much: 1) Nation state level actors can probably insert or compromise high level staff, or multiple high level staff, at any given company, and perform MITM attacks fairly easily. And some could compel turning over your code or secrets more directly anyway. Not worth worrying about this scenario: nobody working on anything truly sensitive should be using any externally hosted (or even externally networked) repositories. 2) It is much more difficult for other actors to do a MITM attack, and if they did, they'd probably have access to your code more directly. 3) Your code actually isn't worth much to anybody else. Imagine someone launching a complete clone of HN or any other site. Who cares? Nobody. What makes your code valuable is that you have it, and that you have a network and relationship with your customers. If somebody stole my company's codebases, I'd feel sorry for them, that they are going to waste any time wading through these useless piles of shit. The only potential problems are if secrets or major vulnerabilities are exposed and provide a path for exploit (like ability to access servers, services, exposing potential ransomware attacks).
- Art9681 4y ago
  Information has different levels of value depending on what the user needs to do with it. It's kind of like how two individual pieces of "unclassified" info are...well, Unclassified but putting the two together as a cohesive whole that provides further context turns it into "classified" info. All it takes is a little bit of time for actors working with the funding and compute capacity of a major nation to scrape the entirety of Github, dump it in a data processing tool none of us know about, and make the correlations you and I cannot. This leak opened a time window big enough for that to happen. We may or may not know if it did. I doubt this info would be offered to the public because it would sink the business.
j45 4y ago
Makes self-hosting git look more preferable. The cloud is always the convenience of someone else’s computer over some amount of security.
amrb 4y ago
Your 100% right to hold critical infrastructure to higher standards. Putting Solarwinds aside, how many companies could to grind to a halt via this 3rd party.
peterkelly 4y ago
Git provides the ability for authors to sign their commits with their own private key. To ensure the integrity of code in a repository, this method should be relied on rather than whatever hosting provider(s) have a copy of the repository. Requiring all commits to be signed by trusted keys avoids the risks associated with someone tampering with a repository hosted on GitHub if they are able to get access to it, although it doesn't protect against code being leaked. See here for details: https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work
- travisd 4y ago
  Parent comment is concerned with privacy, not authenticity. They're not worried that someone modified their code, they're worried that someone saw it.
  - DistractionRect 4y ago
    They specifically called out the need to review all code that ever interacted with github. The implication is that you can't trust it hasn't been tampered with.
  - Spooky23 4y ago
    The parent was assuming full compromise. The risk of disclosure is pretty obvious with GitHub, and I’d assume anyone with low risk tolerance here is using something else, including the on-prem GitHub. I can think of a dozen higher risks.
- jlokier 4y ago
  About signed commits, but off topic for the article: Many signed commits ("Verified" in green) on GitHub are signed with GitHub's own private key¹, not the committer's key. There's a good technical reason, but many committers don't realise their own signing key isn't the one used on their signed commits to the main branch. That GitHub private key would be a fun one to have leaked! It would invalidate most of the "Verified" flags on commits on most repo main branch histories. (¹ GitHub uses GitHub's own commit-signing key when you use the GitHub GUI to merge or squash-merge commits if you already signed them, and the resulting commits show as "Verified" the same as conmits signed by your own key. So you do have to sign with your own key,, but it's not the key used on the main branch commits everyone sees whrn they visit and doenload. Many controlled workflows require approved PRs to be merged to main/master through GitHub's tools, and many users default to using the GUI for PR merges by choice.)
megous 4y ago
A bit of an overreaction, right? Number of people just not blindly using TOFU with github over ssh must be quite low. Who here went here https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/githubs-ssh-key-fingerprints https://docs.github.com/en/authentication/keeping-your-accou... and added the keys manually to known_hosts before using github for the first time?
- nebulous1 4y ago
  I would have guessed that the significant majority would be using TOFU, but in any case the actual key was leaked so it wouldn't matter which method was used.
  - megous 4y ago
    The point is about how much people care, not about whether verification works or not when the key is leaked.
    - nebulous1 4y ago
      > Number of people just not blindly using TOFU with github over ssh must be quite low. Oh, I missed the "not" the first time I read it, which changed your overall meaning entirely. My bad.
      - jessaustin 4y ago
        In this case "not just" would have been more legible than "just not".
nebulous1 4y ago
> The fact that this key was apparently not stored in an HSM, and that GH employees had access to this private key (allowing them to accidentally push it) means that effectively all communication with GH since the founding of the company has to be considered compromised. I think this suggests we need more information from github. For instance GH employees may not always have had live access to this key, this could have happened as part of an operation that gave temporary access to an employee only recently. Or it could have been stored plaintext on multiple employees' home computers since creation. When was the leaked key created anyway?
tashian 4y ago
It's easy to say "should have used an HSM" (or, in truth, many HSMs), but I can appreciate the technical challenges of acutually doing that at their scale. It would not be a trivial project. There's a ton of operational concerns here, including figuring out how you would go about rotating the key on all those HSMs in an emergency.
- lxgr 4y ago
  These would also need to be very distributed and high-throughput HSMs: You'd need to talk to one for every single SSH login! This is in contrast to e.g. having a CA signing key in a HSM, but distributing keys signed with it more widely. I suppose (Open?)SSH's PKI mode could support a model like that, but as others have noted here, this requires much more manual work on the user's side than comparing a TOFU key hash. Maybe that model could be extended to allow TOFU for CAs, though? But I think PKI/CA mode is an OpenSSH extension to the SSH protocol as it is, and that would be a further extension to that extension...
  - tashian 4y ago
    SSH CAs would make the challenge a lot easier. It sounds like they are using RSA keys here for the widest possible compatibility, and while OpenSSH's certificate support is not at all new, it still may be too new for this application.
    - lxgr 4y ago
      Using SSH certificates would tie every Github user to OpenSSH extensions though. I'm not sure if many git clients use something else, but it's at least worth a consideration.
- JeremyNT 4y ago
  There's a lot of daylight between "use a HSM" specifically and "use a system that prevents junior developers from accessing the key and checking it into public repos." Storing the key in some kind of credential vault that can only be accessed from the hosts that need it at startup would usually be enough to prevent this particular kind of error (unless you're giving root on those boxes to people without enough sense to avoid checking private keys into git, in which case you've probably got worse problems).
nwallin 4y ago
SSH uses ephemeral keys. It's not enough to have the private key and listen to the bytes on the wire, you have to actively MITM the connection. A github employee who has access to the private key and enough network admin privileges to MITM your connection already has access to the disk platters your data is saved on. Regarding the secrecy of the data you host at github, you should operate under the assumption that a sizeable number of github employees will have access to it. You should assume that it's all sitting unencrypted on several different disk platters replicated at several different geographically separated locations. Because it is. One of the things that you give up when you host your private data on the cloud is controlling who and under what circumstances people can view or modify your private data. If the content of the data is enough to sink you/your company without remedy you should not store it on the cloud.
- ajross 4y ago
  Exactly. Host keys are about authentication, not connection security. Presumably the upthread comment is trying to say that "ssh communication with github could have been subject to an undetectable MitM attack by an attacker with access to this key"[1], which isn't remotely the same thing as "all communication with GH since the founding of the company has to be considered compromised". [1] Which is sort of tautological and silly, because that's true of all sites and all host keys. What the comment was trying to imply was that the choice of storage of this key invalidates any trust we might have in GitHub/Microsoft regarding key management, and that therefore we shouldn't trust them. Which is also tautological and silly. Either you trust them or you don't, that's not a technological argument.
- snowwrestler 4y ago
  Agreed; GitHub documentation refers to repo “visibility,” not “security,” and that is an intentional distinction. When we signed on with GH as a paying customer over a decade ago, they were quite clear that private repos should not be considered secure storage for secrets. It’s not encrypted at rest, and GitHub staff have access to it. It takes only a few clicks to go from private to public.
  - shdjhdfh 4y ago
peanut-walrus 4y ago
It is so incredibly rare for public-facing service keys to be stored on an HSM that I don't think anyone could reasonably have expected this to be the case?
nonethewiser 4y ago
> the responsible persons (not talking about the poor dev that pushed the key, but the managers, CISO, auditors, etc.) should be fined, and/or lose their license. By no means do I want to see the dev get fined or blackballed from the industry. But if there is any 1 person responsible, it’s the person who did it. The reason why the dev shouldn’t be fined/blackballed is because it’s not just 1 person’s fault. I mean, fining or booting his manager out of the industry? Really?
belter 4y ago
Looking from the outside, for many of these companies: GitHub, OpenAI, Cloudflare, Facebook, and so on, it seems that they torture their hires with ridiculously code challenges. Spend a lot of time on elegant engineering blogs. Write about how many Phds work at their locations, can't stop talking about how selective of the 0.01% Developers they are... But then, internally, everything seems more or less tied up with Shoestrings and Rube Goldberg machines.
- JasserInicide 4y ago
  "Temporary" solutions and countless "TODO: Fix this" simply are endemic to development, even at the top companies. They just like to pretend they're better than everyone else.
steponlego 4y ago
I don't think there will be any lawsuits. The user agreement precludes that. I don't even know how anybody could be angry about this - your code's on somebody else's computer and if you didn't know that that's a huge risk, you do now.

by LeonM 4y ago

djbusby 4y ago

What license?

LeonM 4y ago
Auditors require a license/accreditation to do certain certifications.

brightball 4y ago

I wonder if they found it by turning on their own secret detection system?

Veliladon 4y ago

> The fact that this key was apparently not stored in an HSM, and that GH employees had access to this private key (allowing them to accidentally push it) means that effectively all communication with GH since the founding of the company has to be considered compromised. For a host key? Like I get that being able to impersonate Github isn't great as far as state level actors having the ability to do this but you do know the actual transport layer keys are ephemeral and aren't derived at all from the host key, right?

LeonM 4y ago
> state level actors having the ability to do this Not just nation state actors, but basically anyone in a position to MITM. Also, you don't have to be a nation state actor to extort a GH employee. Any bad guy can do a "give me this key or I'll hurt your kid". People are being extorted for a lot less. There are billions of dollars of assets flowing through GH's infrastructure, for the sake of safety (!= security) of Github's employees, nobody should ever have access to key material.
- palijer 4y ago
  >There are billions of dollars of assets flowing through GH's infrastructure Do you mean source code here? I have a hard time believing source code holds that much value.
  - nicoburns 4y ago
    > As of January 2023, GitHub reported having over 100 million developers and more than 372 million repositories, including at least 28 million public repositories If there are ~350 million private repos then they'd only need to be worth an average of $30 each to be worth a billion dollars in total. Which doesn't seem farfetched.
    - explaininjs 4y ago
      Considering the looooong tail of these repos are forks with no changes, sample code, toy projects abandoned after a single commit, etc. etc., I'd say it's pretty far fetched. For proof, try searching for a mundane string in GH Code search. The vast majority of repos you see will be basically garbage.
    - palijer 4y ago
      I think that is incredibly farfetched. If you got access to 1,000 random private Github repos, I don't think you could sell them, or otherwise utilize them for anywhere near that value, if anything.
    - 4y ago

j16sdiz 4y ago

I have never knew a single person put ssh host key into HSM. In fact, this is not a supported option in openssh.

LeonM 4y ago
> I have never knew a single person put ssh host key into HSM. You probably also never met a single person where the SSH interface sees millions of sessions as day with valuable assets (code) being transported over said sessions. > In fact, this is not a supported option in openssh. This definitely is supported. Though documentation for this is often HSM vendor specific, which if heavily NDA'd. So that's why you probably haven't found much information about it.
- bob1029 4y ago
  > This definitely is supported. Agreed. I have seen some crazy stuff in the payment card industry. I can't recall what I can and can't talk about so I'll just say "Atalla".
  - lxgr 4y ago
    Yes, but that would either be a fork of OpenSSH, private or open source (both are possible since it's BSD-licensed), or a different SSH server (which Github is of course free to use, since the protocol is standardized and their scale absolutely justifies any efforts in protecting their SSH host key). But GPs comment was about OpenSSH. Edit: Apparently OpenSSH's sshd also supports the SSH agent protocol for host keys, and ssh-agent does support PKCS#11 – so I stand corrected!
- lxgr 4y ago
  > This definitely is supported. Though documentation for this is often HSM vendor specific [...] How can openssh documentation be vendor-specific? Or are you saying that vendors commonly provide an openssh fork/patchset/plugin allowing for HSM-resident host keys?
  - 4y ago

killerstorm 4y ago

> any code pushed or pulled from private repositories can no longer be considered private. Do you realize that the code just sits on GitHub servers even if it's private? If you have any degree of paranoia, why do you put your code into GitHub?!?! Like, if you work on code which

BlueTemplar 4y ago
Ah, I see that the men in black got there just in time ! XD

Shank 4y ago

> How has GH ever been able to become ISO certified last year? ISO/IEC 27001:2013 doesn’t say you have to store private keys in HSMs? It just requires you to have a standards compliant ISMS that implements all Annex A controls and all of the clauses. Annex A and the clauses don’t specifically mandate this. If you can convince an auditor that you have controls in-place that meet the standard for protecting cryptographic media you basically meet the standard. The controls can be a wide variety of options and don’t specifically mandate technical implementation details for many, many things. You shouldn’t rely on ISO/IEC 27001:2013 as attestation for technical implementation details here. Just because your auditor would red flag you doesn’t mean all auditors would. The standard is only as effective as the weakest, cheapest auditor, and there are perverse incentives that make auditors financially incentivized to certify companies due to recurring revenue.

LeonM 4y ago
> You shouldn’t rely on ISO/IEC 27001:2013 as attestation for technical implementation details here. Thanks for the insight, good advice. But also from the same GH article [0]: The ISO 27001 certification is the latest addition to GitHub’s compliance portfolio, preceded by SOC and ISAE reports, FedRAMP Tailored LiSaaS ATO, and the Cloud Security Alliance CAIQ. Do you have any knowledge on one of these certifications (for exmaple FedRAMP) that puts any restrictions on handling key material? [0] https://github.blog/2022-05-16-github-achieves-iso-iec-270012013-certification/ https://github.blog/2022-05-16-github-achieves-iso-iec-27001...
- 1970-01-01 4y ago
  I think you're looking for something like PCI DSS compliance, which requires you to store keys in a HSM, and is much more prescriptive with key management.
  - bob1029 4y ago
    PCI-DSS is incredibly aggressive with key management considerations. They get down to radio frequency concerns in pin pad hardware, etc. It took us ~2 years of back & forth with various parties & auditors to get per-client exclusions for accepting end-customer debit PIN codes in-branch on an iPad screen. These banks do not have fully-compliant solutions and must have exceptions on file.

sgt 4y ago

> How has GH ever been able to become ISO certified last year [0], when they didn't even place their keys in a HSM? ISO 27001 certification does not require you to put keys into an HSM. The standard requires you to have controls in place, be aware of your risks and to maintain a risk register. But in no way does the standard require HSM's. The standard would even be OK with storing this on a floppy drive if the risks surrounding that were identified and mitigated (or accepted).

lxgr 4y ago

> [...] depending on your level of paranoia, you will have to review all code that has ever interacted with Github repositories [...] Not to diminish the problems of having a large entity like Github handle a private key like that, but if that was your level of paranoia, you probably should have used commit signatures all along and not relied on Github to do that job for you.

dannyincolor 4y ago
As usual on HN, I find the pragmatic response about 3 pages down in the replies to an extremely hyperbolic top-level comment. I also don't want to diminish the concerns around Github or similar orgs losing control of a private key, but the far more realistic concern for the vast majority of threat models is often put to the wayside in favor of what amounts to a scary story. Rather than the straightforward key removal and replacement that this should be, I (and surely many others) have spent all morning combatting this specific FUD that cropped up on HN with leadership and many engineers. It's actually quite detrimental to quickly remediating the actual concerns introduced by this leak. I understand that security inspires people to be as pedantic as possible - that's where some big exploits come from on occasion - but I really hope the average HN narrative changes toward "what is your actual, real-world threat model" vs. "here is a highly theoretical edge-case scenario, applicable to very few, that I'll state as a general fact so everyone will now wonder if they should spend months auditing their codebase and secrets". Put simply: this is why people just start ignoring security measures in the real world. Surely someone has already coined the term "security fatigue". It's all just a bit unbalanced, and definitely becomes frustrating when those suggesting these "world is burning" scenarios didn't even take the available precautions that apparently would satisfy their threat model (i.e. commit sigs, as you suggested) Ok, end rant :)
- whydoyoucare 4y ago
  Very cogent explanation, and an important point that you highlight - factual real-world risk/threat model is far more important than hypothetical "the-world-is-burning" scenarios. Having a correct threat model is the first step towards building reasonable security controls. But far too many are willing to pander to the "It rather involved being on the other side of this airtight hatchway" [0] scenarios. [0] https://devblogs.microsoft.com/oldnewthing/20060508-22/?p=31283 https://devblogs.microsoft.com/oldnewthing/20060508-22/?p=31...

grumple 4y ago

There's a few reasons I wouldn't worry too much: 1) Nation state level actors can probably insert or compromise high level staff, or multiple high level staff, at any given company, and perform MITM attacks fairly easily. And some could compel turning over your code or secrets more directly anyway. Not worth worrying about this scenario: nobody working on anything truly sensitive should be using any externally hosted (or even externally networked) repositories. 2) It is much more difficult for other actors to do a MITM attack, and if they did, they'd probably have access to your code more directly. 3) Your code actually isn't worth much to anybody else. Imagine someone launching a complete clone of HN or any other site. Who cares? Nobody. What makes your code valuable is that you have it, and that you have a network and relationship with your customers. If somebody stole my company's codebases, I'd feel sorry for them, that they are going to waste any time wading through these useless piles of shit. The only potential problems are if secrets or major vulnerabilities are exposed and provide a path for exploit (like ability to access servers, services, exposing potential ransomware attacks).

Art9681 4y ago
Information has different levels of value depending on what the user needs to do with it. It's kind of like how two individual pieces of "unclassified" info are...well, Unclassified but putting the two together as a cohesive whole that provides further context turns it into "classified" info. All it takes is a little bit of time for actors working with the funding and compute capacity of a major nation to scrape the entirety of Github, dump it in a data processing tool none of us know about, and make the correlations you and I cannot. This leak opened a time window big enough for that to happen. We may or may not know if it did. I doubt this info would be offered to the public because it would sink the business.

j45 4y ago

Makes self-hosting git look more preferable. The cloud is always the convenience of someone else’s computer over some amount of security.

amrb 4y ago

Your 100% right to hold critical infrastructure to higher standards. Putting Solarwinds aside, how many companies could to grind to a halt via this 3rd party.

peterkelly 4y ago

Git provides the ability for authors to sign their commits with their own private key. To ensure the integrity of code in a repository, this method should be relied on rather than whatever hosting provider(s) have a copy of the repository. Requiring all commits to be signed by trusted keys avoids the risks associated with someone tampering with a repository hosted on GitHub if they are able to get access to it, although it doesn't protect against code being leaked. See here for details: https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work

travisd 4y ago
Parent comment is concerned with privacy, not authenticity. They're not worried that someone modified their code, they're worried that someone saw it.
- DistractionRect 4y ago
  They specifically called out the need to review all code that ever interacted with github. The implication is that you can't trust it hasn't been tampered with.
- Spooky23 4y ago
  The parent was assuming full compromise. The risk of disclosure is pretty obvious with GitHub, and I’d assume anyone with low risk tolerance here is using something else, including the on-prem GitHub. I can think of a dozen higher risks.
jlokier 4y ago
About signed commits, but off topic for the article: Many signed commits ("Verified" in green) on GitHub are signed with GitHub's own private key¹, not the committer's key. There's a good technical reason, but many committers don't realise their own signing key isn't the one used on their signed commits to the main branch. That GitHub private key would be a fun one to have leaked! It would invalidate most of the "Verified" flags on commits on most repo main branch histories. (¹ GitHub uses GitHub's own commit-signing key when you use the GitHub GUI to merge or squash-merge commits if you already signed them, and the resulting commits show as "Verified" the same as conmits signed by your own key. So you do have to sign with your own key,, but it's not the key used on the main branch commits everyone sees whrn they visit and doenload. Many controlled workflows require approved PRs to be merged to main/master through GitHub's tools, and many users default to using the GUI for PR merges by choice.)

megous 4y ago

A bit of an overreaction, right? Number of people just not blindly using TOFU with github over ssh must be quite low. Who here went here https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/githubs-ssh-key-fingerprints https://docs.github.com/en/authentication/keeping-your-accou... and added the keys manually to known_hosts before using github for the first time?

nebulous1 4y ago
I would have guessed that the significant majority would be using TOFU, but in any case the actual key was leaked so it wouldn't matter which method was used.
- megous 4y ago
  The point is about how much people care, not about whether verification works or not when the key is leaked.
  - nebulous1 4y ago
    > Number of people just not blindly using TOFU with github over ssh must be quite low. Oh, I missed the "not" the first time I read it, which changed your overall meaning entirely. My bad.
    - jessaustin 4y ago
      In this case "not just" would have been more legible than "just not".

nebulous1 4y ago

> The fact that this key was apparently not stored in an HSM, and that GH employees had access to this private key (allowing them to accidentally push it) means that effectively all communication with GH since the founding of the company has to be considered compromised. I think this suggests we need more information from github. For instance GH employees may not always have had live access to this key, this could have happened as part of an operation that gave temporary access to an employee only recently. Or it could have been stored plaintext on multiple employees' home computers since creation. When was the leaked key created anyway?

tashian 4y ago

It's easy to say "should have used an HSM" (or, in truth, many HSMs), but I can appreciate the technical challenges of acutually doing that at their scale. It would not be a trivial project. There's a ton of operational concerns here, including figuring out how you would go about rotating the key on all those HSMs in an emergency.

lxgr 4y ago
These would also need to be very distributed and high-throughput HSMs: You'd need to talk to one for every single SSH login! This is in contrast to e.g. having a CA signing key in a HSM, but distributing keys signed with it more widely. I suppose (Open?)SSH's PKI mode could support a model like that, but as others have noted here, this requires much more manual work on the user's side than comparing a TOFU key hash. Maybe that model could be extended to allow TOFU for CAs, though? But I think PKI/CA mode is an OpenSSH extension to the SSH protocol as it is, and that would be a further extension to that extension...
- tashian 4y ago
  SSH CAs would make the challenge a lot easier. It sounds like they are using RSA keys here for the widest possible compatibility, and while OpenSSH's certificate support is not at all new, it still may be too new for this application.
  - lxgr 4y ago
    Using SSH certificates would tie every Github user to OpenSSH extensions though. I'm not sure if many git clients use something else, but it's at least worth a consideration.
JeremyNT 4y ago
There's a lot of daylight between "use a HSM" specifically and "use a system that prevents junior developers from accessing the key and checking it into public repos." Storing the key in some kind of credential vault that can only be accessed from the hosts that need it at startup would usually be enough to prevent this particular kind of error (unless you're giving root on those boxes to people without enough sense to avoid checking private keys into git, in which case you've probably got worse problems).

nwallin 4y ago

SSH uses ephemeral keys. It's not enough to have the private key and listen to the bytes on the wire, you have to actively MITM the connection. A github employee who has access to the private key and enough network admin privileges to MITM your connection already has access to the disk platters your data is saved on. Regarding the secrecy of the data you host at github, you should operate under the assumption that a sizeable number of github employees will have access to it. You should assume that it's all sitting unencrypted on several different disk platters replicated at several different geographically separated locations. Because it is. One of the things that you give up when you host your private data on the cloud is controlling who and under what circumstances people can view or modify your private data. If the content of the data is enough to sink you/your company without remedy you should not store it on the cloud.

ajross 4y ago
Exactly. Host keys are about authentication, not connection security. Presumably the upthread comment is trying to say that "ssh communication with github could have been subject to an undetectable MitM attack by an attacker with access to this key"[1], which isn't remotely the same thing as "all communication with GH since the founding of the company has to be considered compromised". [1] Which is sort of tautological and silly, because that's true of all sites and all host keys. What the comment was trying to imply was that the choice of storage of this key invalidates any trust we might have in GitHub/Microsoft regarding key management, and that therefore we shouldn't trust them. Which is also tautological and silly. Either you trust them or you don't, that's not a technological argument.
snowwrestler 4y ago
Agreed; GitHub documentation refers to repo “visibility,” not “security,” and that is an intentional distinction. When we signed on with GH as a paying customer over a decade ago, they were quite clear that private repos should not be considered secure storage for secrets. It’s not encrypted at rest, and GitHub staff have access to it. It takes only a few clicks to go from private to public.
- shdjhdfh 4y ago

peanut-walrus 4y ago

It is so incredibly rare for public-facing service keys to be stored on an HSM that I don't think anyone could reasonably have expected this to be the case?

nonethewiser 4y ago

> the responsible persons (not talking about the poor dev that pushed the key, but the managers, CISO, auditors, etc.) should be fined, and/or lose their license. By no means do I want to see the dev get fined or blackballed from the industry. But if there is any 1 person responsible, it’s the person who did it. The reason why the dev shouldn’t be fined/blackballed is because it’s not just 1 person’s fault. I mean, fining or booting his manager out of the industry? Really?

belter 4y ago

Looking from the outside, for many of these companies: GitHub, OpenAI, Cloudflare, Facebook, and so on, it seems that they torture their hires with ridiculously code challenges. Spend a lot of time on elegant engineering blogs. Write about how many Phds work at their locations, can't stop talking about how selective of the 0.01% Developers they are... But then, internally, everything seems more or less tied up with Shoestrings and Rube Goldberg machines.

JasserInicide 4y ago
"Temporary" solutions and countless "TODO: Fix this" simply are endemic to development, even at the top companies. They just like to pretend they're better than everyone else.

steponlego 4y ago

I don't think there will be any lawsuits. The user agreement precludes that. I don't even know how anybody could be angry about this - your code's on somebody else's computer and if you didn't know that that's a huge risk, you do now.