6 ms·

Github had one RSA ssh host key, the most widely supported key format. It was trusted to clone code into the infrastructure of hundreds of thousands of organiz

by lrvick 4y ago

Github had one RSA ssh host key, the most widely supported key format. It was trusted to clone code into the infrastructure of hundreds of thousands of organizations. People pin it everywhere. With this key someone could infect the infrastructure of fintech companies and steal billions of dollars. I know this well because I run a security consulting company focusing mostly on that industry. Mainly this is possible because almost no companies check for git commit signing, and Github does not enforce it anyway, and I digress. This key held enough power over value that some might have used violence to acquire it. With that context of course they chose to place this key in a hardware security module controlled with an m-of-n quorum of engineers to ensure no single human can steal it, expose it, or leak it. Right? ... right? Nope. They admit they just stuck it in a git repo in plain text where any engineer could have copied it to a compromised workstation, or been bribed for it, for who knows how many years. Worse, it was not even some separate employee only intranet git repo, but their own regular public production infra and someone had the power to accidentally make it public. I have no words for this level of gross negligence. Maybe, just maybe, centralizing all trust and security for most of the worlds software development to a proprietary for-profit company with an abysmal security reputation was not the best plan. I will just leave this here: https://sfconservancy.org/blog/2022/jun/30/give-up-github-launch/ https://sfconservancy.org/blog/2022/jun/30/give-up-github-la...

CGamesPlay 4y ago
> With that context of course they chose to place this key in a hardware security module controlled with an m-of-n quorum of engineers to ensure no single human can steal it, expose it, or leak it. Right? ... right? This is unfortunately not how SSH works. It needs to be unlocked for every incoming connection. You raise valid hypotheticals about the security of the service... but fixing it involves removing SSH host key verification from Github; better OpsSec would not fully resolve this issue.
- p-e-w 4y ago
  Hardware security modules can perform key operations without allowing anyone to access the key data. Key material being (accidentally or deliberately) leaked has been a solved problem for a long, long time.
  - CGamesPlay 4y ago
    Used in such a way, the entirety of Github's SSH connections would be bottlenecked by this HSM. It wouldn't scale and it would be a single point of failure. As lrvick points out, you'd have to use a certificate-based host key scheme like PKCS#11 to make this scalable. That's fine, but it is a different scheme than RSA host key identification.
    - lrvick 4y ago
      I gave two options but they are separate. PKCS#11 is a protocol for talking to hardware security modules for generic cryptographic operations to keep a key out of system memory. You can totally take a single host key scheme and use HSMs or TEEs at scale. Openssh supports using openssl as a backend which in turn can use the PKCS#11 protocol to delegate private key operations to remote enclaves or TEEs. Nothing stops you from loading your key into a fleet of them and scaling horizontally just like you scale anything else.
      - CGamesPlay 4y ago
        > Nothing stops you from loading your key into a fleet of them and scaling horizontally just like you scale anything else. Isn't having one key the root of the problem? If you've got one key, it has to be transferred to each HSM module, which means it's going to be in their deploy code. My understanding is that the safe way to scale PKI is to have an HSM generate a key internally, and have that key be signed by a CA in a different HSM, so private key material never leaves any HSM. I guess you're saying that SSH RSA host keys support this, but I'm only familiar with doing it using looks up correct terminology X.509 certificates like for HTTPS.
        arianvanp 4y ago
        HSMs can key-wrap a key. E.g. Foo can encrypt the key X with Bars public key and then Bar can import key X.
        fireflash38 4y ago
- lrvick 4y ago
  I am well aware how ssh works. I have written ssh servers and design secure enclave key management solutions for a living. Even if they wanted the most quick and dirty lazy option with no policy management, they could stick the key in a PKCS#11 supporting enclave every cloud provider supports these days. OpenSSH natively supports them today. At a minimum they could have placed their whole ssh connection termination engine in a immutable read only and remotely attestable system like a Nitro enclave or other TEE. You do not need to invent anything new for this. There are just no excuses here for a company with their size and resources, because I do this stuff all the time as just one guy.
  - kelnos 4y ago
    Would these secure storage methods for the key be able to scale up to the ssh connection volume an outfit like GH sees? Genuinely asking; I don't know the answer. I just feel like having to hit a HSM or something similar every time a new ssh connection comes in wouldn't work. Or at the very least I would not see the sub-100ms connection time I can get right now.
    - lrvick 4y ago
      Easily. You could have only the CA key in the enclave and have it sign throw-away session keys on stateless autoscaling machines, and/or you can have your actual ssh terminations happen in cloud enclaves like nitro enclaves which have all the same specs and cost of a regular cloud server.
- marcosdumay 4y ago
  > It needs to be unlocked for every incoming connection. Yep. Well, certificates exist exactly to bridge the GP's requirement with your reality.
kelnos 4y ago
Wow, your assessment of the impact here (or even possible impact) is way way way overblown. In reality, the vast majority of users don't pay attention to SSH host keys at all. Even if an attacker got hold of this private host key, they'd have to be able to MitM the connections of their target. Next, they have to decide what they want to do. If they want to send malicious code to someone doing a 'git pull', they'd have to craft the payload specifically for the repo being pulled from. Not impossible, but difficult. If they want to "steal" source code from someone doing 'git push' (perhaps to a private repo on GitHub), that's a bit easier, as they can just tell the client "I have no objects yet", and then the client will send the entire repo. And, again, they'd have to have the ability to MitM some git user's connections. Regardless, there is no way that they could change any code on github.com; this key would not give them any access to GH's services that they don't already have. So I think your anger here is a bit over the top and unnecessary. I agree that it's pretty bad that some GH employee even had the ability to get hold of this private key (sure, probably should be in an HSM, but I'm not particularly surprised it's not) in order to accidentally leak it, but... shit happens. They admitted the mistake and rotated the key, even though it's likely that there was zero impact from all this.
- lrvick 4y ago
  End users do not pay attention but their clients pin the key after first use. Also everyone is using gitops these days and almost no one is using dns-over-tls. Imagine you control the right router on a company wifi, or any home wifi a production engineer works from and suddenly you can cause them to clone the wrong git submodule, the wrong go package, or the wrong terraform config. If you knew a CI/CD system blindly clones and deploys git repos to prod without signature checks, and that prod is a top 10 crypto exchange with 1b of liquidity in hot wallets, then suddenly a BGP attack to redirect DNS is a good investment. Myetherwallet got taken over for 15 minutes with a BGP so this is not hypothetical. Should that be the case? Of course not. But the reality is I find this in almost all of the security audits I do for fintech companies. Blind trust in Github host keys is industry standard all the way to prod.
  - kelnos 4y ago
mjg59 4y ago
Which HSM are you looking at that would be able to handle the required number of transactions per second?
- ZiiS 4y ago
  The HSM only needs to sign new host keys, transactions per decade at thier current rate.
- lrvick 4y ago
  The same ones that terminate TLS for millions. Most are garbage but at least they keep the key offline. Also you can scale horizontally or only use the HSM as a CA to sign short lived host keys. You could also use things like Nitro enclaves which have all the same specs as a regular EC2 instance. Tons of options. They clearly chose none of them.
  - mjg59 4y ago
    > The same ones that terminate TLS for millions. Which ones? > Also you can scale horizontally For SSH? Only by having the same private key in all of them, which means it's still around somewhere. > or only use the HSM as a CA to sign short lived host keys That would be ideal, except that the user experience around SSH host certificates is currently woeful.
    - Perseids 4y ago
      (Not your parent commenter.) > Which ones? Take Utimaco Security Server and you'll get 10k+ RSA signatures per seconds. Yes, you'll definitely need dozens of them, but you'll probably want several for high availability and low latency anyway. > For SSH? Only by having the same private key in all of them, which means it's still around somewhere. End-to-End encrypted key transfer between HSMs is well established. Ops for such a setup is definitely going to be a pain and lots of manual (and thus expensive) work but it is doable. The banking industry has been operating like that since forever – with symmetric cryptography only. Imagine two people being sent letters with XOR halves of a transport key and physically meeting at an HSM and entering the halves on a PIN pad (not a hexadecimal but a decimal PIN pad, mind you, where you need to press shift for A-F). From a modern perspective it's totally bonkers, but it works. If I was tasked with building something for large scale companies like GitHub, I would probably pass up on HSMs and use commodity measured boot on a minimal Linux for my trusted key servers. Outside of these key servers the SSH key would only be stored split up with Shamir-Secret-Sharing with few trusted employees which will only restore the key on ephemeral offline systems. Is that overkill? Very much depends on your threat model. Investing in the security of their build chain runners and data-at-rest integrity might have higher pay off. But then again, GitHub has become such a big player that we should also hold them to very high standards. And the setup can be re-used for all their secret management, e.g. TLS certificate private keys.
ses1984 4y ago
The key has to be in memory on all of their front end servers. Do you think a quorum of engineers should get together every time a front end server boots or reboots? Genuinely asking because I’ve struggled with this question.
- bob1029 4y ago
  I don't understand how initializing cryptographic keys from an HSM at boot time is an untenable proposition. The quorum would be for accessing the key by human means. You can have a separate, approved path for pre-authorized machines to access cryptographic primitives across an isolated network.
- lrvick 4y ago
  Lots of cloud instances support remote attestation these days which gives you a reasonable path to autoscaling secure enclaves. 1. You compile a deterministic unikernel appliance-style linux kernel with a bare bones init system 2. You deploy it to a system that supports remote attestation like a nitro enclave. 3. It boots and generates a random ephemeral key 4. m-of-n engineers compile the image themselves, get the same hash, and verify the remote attestation proof confirming the system is running the bit-for-bit trusted image 5. m-of-n engineers encrypt and submit shamirs secret shares of the really important private key that needs protecting 6. key is reconstituted in memory of enclave and can start taking requests 7. Traffic goes up and autoscaling is triggered 8. New system boots with an identical account, role, and boot image to the first manually provisioned enclave 9. First enclave (with hot key) remotely attests the new enclave and obtains its ephemeral key (with help of an internet connected coordinator) 10. First enclave encrypts hot key to new autoscaled enclave 11. rinse/repeat
- kccqzy 4y ago
  The key doesn't have to in memory on all of their front end servers. Any respectable company that cares about security wouldn't put their TLS private key on all of their front end servers anyways. You expose a special crypto oracle that your front end servers talk to; the oracle can be a specially hardened process on a dedicated server or better yet a HSM; the point is, the private key is never in memory on any server that handles untrusted data.
ed25519FUUU 4y ago
Thoughts with the sysadmins and devops people out there on this wonderful Friday afternoon. These kinds of changes suuuuuuck. Messing with known_hosts file is not always something easy to do. Might require a image rebuild, if you have access at all.
uejfiweun 4y ago
Is it negligence or just incompetence? I get the sense that security is such a tough problem that all of us, even CISOs and red teamers, are incompetent.
- lrvick 4y ago
  If hospital workers spread disease because they could not be bothered to do the obvious things we -know- prevent this like basic sanitation... then yeah, I would call it negligence. Do not put long lived cryptographic key material in the memory of an internet connected system. Ever. It is a really easy to understand rule.

by lrvick 4y ago

CGamesPlay 4y ago

> With that context of course they chose to place this key in a hardware security module controlled with an m-of-n quorum of engineers to ensure no single human can steal it, expose it, or leak it. Right? ... right? This is unfortunately not how SSH works. It needs to be unlocked for every incoming connection. You raise valid hypotheticals about the security of the service... but fixing it involves removing SSH host key verification from Github; better OpsSec would not fully resolve this issue.

p-e-w 4y ago
Hardware security modules can perform key operations without allowing anyone to access the key data. Key material being (accidentally or deliberately) leaked has been a solved problem for a long, long time.
- CGamesPlay 4y ago
  Used in such a way, the entirety of Github's SSH connections would be bottlenecked by this HSM. It wouldn't scale and it would be a single point of failure. As lrvick points out, you'd have to use a certificate-based host key scheme like PKCS#11 to make this scalable. That's fine, but it is a different scheme than RSA host key identification.
  - lrvick 4y ago
    I gave two options but they are separate. PKCS#11 is a protocol for talking to hardware security modules for generic cryptographic operations to keep a key out of system memory. You can totally take a single host key scheme and use HSMs or TEEs at scale. Openssh supports using openssl as a backend which in turn can use the PKCS#11 protocol to delegate private key operations to remote enclaves or TEEs. Nothing stops you from loading your key into a fleet of them and scaling horizontally just like you scale anything else.
    - CGamesPlay 4y ago
      > Nothing stops you from loading your key into a fleet of them and scaling horizontally just like you scale anything else. Isn't having one key the root of the problem? If you've got one key, it has to be transferred to each HSM module, which means it's going to be in their deploy code. My understanding is that the safe way to scale PKI is to have an HSM generate a key internally, and have that key be signed by a CA in a different HSM, so private key material never leaves any HSM. I guess you're saying that SSH RSA host keys support this, but I'm only familiar with doing it using looks up correct terminology X.509 certificates like for HTTPS.
      - arianvanp 4y ago
        HSMs can key-wrap a key. E.g. Foo can encrypt the key X with Bars public key and then Bar can import key X.
      - fireflash38 4y ago
lrvick 4y ago
I am well aware how ssh works. I have written ssh servers and design secure enclave key management solutions for a living. Even if they wanted the most quick and dirty lazy option with no policy management, they could stick the key in a PKCS#11 supporting enclave every cloud provider supports these days. OpenSSH natively supports them today. At a minimum they could have placed their whole ssh connection termination engine in a immutable read only and remotely attestable system like a Nitro enclave or other TEE. You do not need to invent anything new for this. There are just no excuses here for a company with their size and resources, because I do this stuff all the time as just one guy.
- kelnos 4y ago
  Would these secure storage methods for the key be able to scale up to the ssh connection volume an outfit like GH sees? Genuinely asking; I don't know the answer. I just feel like having to hit a HSM or something similar every time a new ssh connection comes in wouldn't work. Or at the very least I would not see the sub-100ms connection time I can get right now.
  - lrvick 4y ago
    Easily. You could have only the CA key in the enclave and have it sign throw-away session keys on stateless autoscaling machines, and/or you can have your actual ssh terminations happen in cloud enclaves like nitro enclaves which have all the same specs and cost of a regular cloud server.
marcosdumay 4y ago
> It needs to be unlocked for every incoming connection. Yep. Well, certificates exist exactly to bridge the GP's requirement with your reality.

kelnos 4y ago

Wow, your assessment of the impact here (or even possible impact) is way way way overblown. In reality, the vast majority of users don't pay attention to SSH host keys at all. Even if an attacker got hold of this private host key, they'd have to be able to MitM the connections of their target. Next, they have to decide what they want to do. If they want to send malicious code to someone doing a 'git pull', they'd have to craft the payload specifically for the repo being pulled from. Not impossible, but difficult. If they want to "steal" source code from someone doing 'git push' (perhaps to a private repo on GitHub), that's a bit easier, as they can just tell the client "I have no objects yet", and then the client will send the entire repo. And, again, they'd have to have the ability to MitM some git user's connections. Regardless, there is no way that they could change any code on github.com; this key would not give them any access to GH's services that they don't already have. So I think your anger here is a bit over the top and unnecessary. I agree that it's pretty bad that some GH employee even had the ability to get hold of this private key (sure, probably should be in an HSM, but I'm not particularly surprised it's not) in order to accidentally leak it, but... shit happens. They admitted the mistake and rotated the key, even though it's likely that there was zero impact from all this.

lrvick 4y ago
End users do not pay attention but their clients pin the key after first use. Also everyone is using gitops these days and almost no one is using dns-over-tls. Imagine you control the right router on a company wifi, or any home wifi a production engineer works from and suddenly you can cause them to clone the wrong git submodule, the wrong go package, or the wrong terraform config. If you knew a CI/CD system blindly clones and deploys git repos to prod without signature checks, and that prod is a top 10 crypto exchange with 1b of liquidity in hot wallets, then suddenly a BGP attack to redirect DNS is a good investment. Myetherwallet got taken over for 15 minutes with a BGP so this is not hypothetical. Should that be the case? Of course not. But the reality is I find this in almost all of the security audits I do for fintech companies. Blind trust in Github host keys is industry standard all the way to prod.
- kelnos 4y ago

mjg59 4y ago

Which HSM are you looking at that would be able to handle the required number of transactions per second?

ZiiS 4y ago
The HSM only needs to sign new host keys, transactions per decade at thier current rate.
lrvick 4y ago
The same ones that terminate TLS for millions. Most are garbage but at least they keep the key offline. Also you can scale horizontally or only use the HSM as a CA to sign short lived host keys. You could also use things like Nitro enclaves which have all the same specs as a regular EC2 instance. Tons of options. They clearly chose none of them.
- mjg59 4y ago
  > The same ones that terminate TLS for millions. Which ones? > Also you can scale horizontally For SSH? Only by having the same private key in all of them, which means it's still around somewhere. > or only use the HSM as a CA to sign short lived host keys That would be ideal, except that the user experience around SSH host certificates is currently woeful.
  - Perseids 4y ago
    (Not your parent commenter.) > Which ones? Take Utimaco Security Server and you'll get 10k+ RSA signatures per seconds. Yes, you'll definitely need dozens of them, but you'll probably want several for high availability and low latency anyway. > For SSH? Only by having the same private key in all of them, which means it's still around somewhere. End-to-End encrypted key transfer between HSMs is well established. Ops for such a setup is definitely going to be a pain and lots of manual (and thus expensive) work but it is doable. The banking industry has been operating like that since forever – with symmetric cryptography only. Imagine two people being sent letters with XOR halves of a transport key and physically meeting at an HSM and entering the halves on a PIN pad (not a hexadecimal but a decimal PIN pad, mind you, where you need to press shift for A-F). From a modern perspective it's totally bonkers, but it works. If I was tasked with building something for large scale companies like GitHub, I would probably pass up on HSMs and use commodity measured boot on a minimal Linux for my trusted key servers. Outside of these key servers the SSH key would only be stored split up with Shamir-Secret-Sharing with few trusted employees which will only restore the key on ephemeral offline systems. Is that overkill? Very much depends on your threat model. Investing in the security of their build chain runners and data-at-rest integrity might have higher pay off. But then again, GitHub has become such a big player that we should also hold them to very high standards. And the setup can be re-used for all their secret management, e.g. TLS certificate private keys.

ses1984 4y ago

The key has to be in memory on all of their front end servers. Do you think a quorum of engineers should get together every time a front end server boots or reboots? Genuinely asking because I’ve struggled with this question.

bob1029 4y ago
I don't understand how initializing cryptographic keys from an HSM at boot time is an untenable proposition. The quorum would be for accessing the key by human means. You can have a separate, approved path for pre-authorized machines to access cryptographic primitives across an isolated network.
lrvick 4y ago
Lots of cloud instances support remote attestation these days which gives you a reasonable path to autoscaling secure enclaves. 1. You compile a deterministic unikernel appliance-style linux kernel with a bare bones init system 2. You deploy it to a system that supports remote attestation like a nitro enclave. 3. It boots and generates a random ephemeral key 4. m-of-n engineers compile the image themselves, get the same hash, and verify the remote attestation proof confirming the system is running the bit-for-bit trusted image 5. m-of-n engineers encrypt and submit shamirs secret shares of the really important private key that needs protecting 6. key is reconstituted in memory of enclave and can start taking requests 7. Traffic goes up and autoscaling is triggered 8. New system boots with an identical account, role, and boot image to the first manually provisioned enclave 9. First enclave (with hot key) remotely attests the new enclave and obtains its ephemeral key (with help of an internet connected coordinator) 10. First enclave encrypts hot key to new autoscaled enclave 11. rinse/repeat
kccqzy 4y ago
The key doesn't have to in memory on all of their front end servers. Any respectable company that cares about security wouldn't put their TLS private key on all of their front end servers anyways. You expose a special crypto oracle that your front end servers talk to; the oracle can be a specially hardened process on a dedicated server or better yet a HSM; the point is, the private key is never in memory on any server that handles untrusted data.

ed25519FUUU 4y ago

Thoughts with the sysadmins and devops people out there on this wonderful Friday afternoon. These kinds of changes suuuuuuck. Messing with known_hosts file is not always something easy to do. Might require a image rebuild, if you have access at all.

uejfiweun 4y ago

Is it negligence or just incompetence? I get the sense that security is such a tough problem that all of us, even CISOs and red teamers, are incompetent.

lrvick 4y ago
If hospital workers spread disease because they could not be bothered to do the obvious things we -know- prevent this like basic sanitation... then yeah, I would call it negligence. Do not put long lived cryptographic key material in the memory of an internet connected system. Ever. It is a really easy to understand rule.