8 ms·

How is there an alternative here?

ansible 4y ago
There isn't an alternative, really. The private key has been exposed, and presumably it is unknown if or how far it has spread. The SSH keys must be changed, and the sooner the better. All that can be done is to notify people after the change has occurred.
- p-e-w 4y ago
  > and presumably it is unknown if or how far it has spread Why would that be unknown? GitHub has HTTP and SSH access logs, right?
  - tgsovlerkhgsel 4y ago
    Since the post doesn't mention anything like "we reviewed logs and confirm the key was not accessed", it is very likely that they either don't have logs that are reliable enough to rule it out (e.g. they may be sampled or otherwise incomplete), or that the key was accessed.
    - renonce 4y ago
      Keeping a complete log of all GET requests to random files in a public repository in a reliable way would be insane.
      - otoolep 4y ago
        No, it wouldn’t be - assuming by “insane” you mean “silly to do”. I build systems at Google that do exactly that. Whether it’s worth the cost is a decision each company makes. Also, you don’t need to keep the log forever. Max of a few weeks retention would be common.
        pbhjpbhj 4y ago
        Presumably, keeping 'last remotely accessed' and 'last remotely modified' for every file (or other stats that are a digest of the logs) is sane for pretty much any system too. Having a handle on how much space one is dedicating to files that are never viewed and or never updated seems like something web companies that have public file access would all want?
        tgsovlerkhgsel 4y ago
        What guarantees do these systems provide? Are 100% of requests where data was served guaranteed to either end up in the log or at least create a detectable "logs may have been lost here" event? Or does it log all the requests all the time as long as everything goes well, but if the wrong server crashes at the wrong time, a couple hundred requests may get lost because in the end who cares?
      - est31 4y ago
        It's not just GET requests. Someone could have cloned/refreshed the repo using ssh. The repo might have been indexed by github's internal search daemon which might not use the public HTTP API but uses internal access ways however those might look like. You might have purged the database of that daemon but what about backups of it? What about people who have subscribed to public events happening in the github.com/github org via the API? You'd have to have logging set up for all of these services and it would have to work over your entire CDN... and what if a CDN node crashed before it was able to submit the log entry to the log collector? You'll never know.
  - blitzar 4y ago
    One H4X0R gave it to four friends, who in turn gave it to between 9 and 14 friends, who in turn gave it to between one and 6 friends. If train A leaves New York going 60 miles per hour with 167 people on board and train B leaves Chicago one hour later with 361 people on board going 85 miles per hour, how many people now have the key? The answer is 31337.
p-e-w 4y ago
Same as with any other decision: Do a cost/benefit analysis of whether the security risk created by rotating the key is actually outweighed by the security risk of doing nothing, taking into account logs that should tell you whether the exposed key was indeed accessed by unauthorized parties. To be 100% clear: Both courses of action come with associated security risks. The problem is not choosing one course of action over the other, the problem is thinking you can just skip the cost/benefit analysis because the answer is somehow 'obvious'. It's not obvious at all.
- johngalt_ 4y ago
  No, you cannot keep using an exposed key. You must replace it. There is no cost/benefit analysis needed in this situation.
  - p-e-w 4y ago
    Wrong. A CBA is always needed. If the potential damage from MITM attacks made possible by rotating the key is greater than the potential damage from a rogue key multiplied by the likelihood that someone actually accessed the key, then it is wrong to rotate the key. It's that simple. The only way a CBA would be unnecessary is if rotating the key didn't have any security risks. But it does.
    - eyelidlessness 4y ago
      Here I’ll do the CBA: - if they have evidence that the key was exposed to one person, even with zero usage of the key, failing to rotate the key is tantamount to knowingly accepting widespread compromise at a potential attacker’s whim. At GitHub’s scale, that’s untenable. - rotating the key is the only correct reaction to that - they should have better communications in place to help users mitigate MITM - there really isn’t an option, because they’re critical infrastructure; I’m glad they know that and acted accordingly - on principle this speculation makes sense, but understanding the threat makes it moot - you hopefully know that, and it’s good to insist on thoughtful security practices but it’s important to also understand the actual risk
    - 4y ago
rakoo 4y ago
That's why you need certificates and not just a key pair. Certificates make key rotation easier, and you want key rotation to be easy. I guess the proper way forward is a small utility that gets the latest signature through http+tls, and replaces the line in your known_hosts file, all in the background. Looking long term, maybe we need to get rid of all the security stuff in ssh and just pipe the rest of its functionalities inside a TLS pipe. Let the os do its certificate management, reuse security bricks that are way more studied, ...
- oleganza 4y ago
  Certificates just add more keys to worry about. The beauty of SSH is that it does not add hugely trusted parties in the name of convenience, while the UX of TOFU (trust on first use) is pretty decent. The real solution to break out of these UX/security tradeoffs is to put domain names on a blockchain: then you can simply rotate the key in your DNS record, while the blockchain model is such that you need to compromise many parties, instead of "one out of many parties", as with CAs. Tracking Bitcoin chain for DNS updates is lightweight enough that it can be built into OS alongside other modern components such as secure enclave, TCP/IP stack and WiFi/BT/5G radios.
  - hnfong 4y ago
    > The beauty of SSH is that it does not add hugely trusted parties in the name of convenience Even with a certificate authority model, you don't have to trust any CAs if you don't want to. Not having the option to do so is more of a problem.
  - datadeft 4y ago
    We should use a separate system that could reliably verify which certs belong to which entity. Blockchain is a perfect solution to this. I wonder why it is not considered yet.
    - jtsiskin 4y ago
      https://certificate.transparency.dev https://certificate.transparency.dev it is
ithkuil 4y ago
The alternative would be to use certificate authorities (ssh has CA support) which allow to effectively have private keys at different levels and allow you to keep the root private key in a physical vault and use it very rarely to issue other private keys
- terom 4y ago
  And then don't forget to setup key revocation as well, and make sure that an attacker in a position to MITM the connection cannot cause the revocation checks to fail-open. I hope you don't need that SSH connection to fix your broken CRL endpoint!
- datadeft 4y ago
  This would just offload the problem to a separate entity. CAs can be (and have been) compromised.
  - ithkuil 4y ago
    Sure, but isn't it more likely that a key that has to be shared by who knows how many ssh load balancer machines at GitHub and can't be easily rotated because it's pinned by millions of users, isn't it more likely that that private key gets eventually compromised or thought to be at risk at being compromised? We need to compare the relative risks within the same context, namely within a company like GitHub So it's not relevant to bring up failures of other CAs
michaelt 4y ago
Jump into a time machine, go back to the creation of SSH, and adopt SSL-style trusted third-party certificate authorities. Somehow get it adopted anyway, even though loads of people use SSH on internal networks where host-based authentication is difficult; SSH is how many headless machines are bootstrapped; and that you've got to do it 19 years before Lets Encrypt. Jump into a lesser time machine, go back to when Github were creating their SSH key, and put it into a hardware security module. Somehow share that hardware-backed security key to loads of servers over a network, without letting hackers do the same thing. Somehow get an HSM that isn't a closed-source black box from a huge defence contractor riddled with foreign spies. Somehow avoid vendor lock-in or the HSM becoming obsolete. Somehow do this when you're a scrappy startup with barely any free time.
- nibbleshifter 4y ago
  Ssh certificate authorities are a thing that exists. We also have a way to put SSH host key fingerprints in DNS records already.
  - Arch-TK 4y ago
    Yes but the option to do verify host keys using ("VerifyHostKeyDNS") is not enabled by default.
  - roblabla 4y ago
    DNS can trivially be mitm'd. DNS-stored fingerprints are strictly less secure than TOFU.
    - tialaramex 4y ago
      If you use DNSSEC (cue inevitable rant from Thomas) this just works. If you have DoH (and why wouldn't you?) and your trusted resolver uses DNSSEC (which popular ones do), you get the same benefits. https://en.wikipedia.org/wiki/SSHFP_record https://en.wikipedia.org/wiki/SSHFP_record
  - datadeft 4y ago
pid-1 4y ago
Clone repos using oauth2 with two factor enabled - both GitHub and GitLab support that though their CLIs.
marcosdumay 4y ago
Well, at least SSH could allow for signing a new key with the old one. So they could say it's signed, and people would know to accept only a different prompt. There is DNS verification, but people have been trained all their lives to accept insecure DNS information (and set their systems accordingly), and I really doubt the SSH client checks the DNSSEC data.

There isn't an alternative, really. The private key has been exposed, and presumably it is unknown if or how far it has spread. The SSH keys must be changed, and the sooner the better. All that can be done is to notify people after the change has occurred.

p-e-w 4y ago
> and presumably it is unknown if or how far it has spread Why would that be unknown? GitHub has HTTP and SSH access logs, right?
- tgsovlerkhgsel 4y ago
  Since the post doesn't mention anything like "we reviewed logs and confirm the key was not accessed", it is very likely that they either don't have logs that are reliable enough to rule it out (e.g. they may be sampled or otherwise incomplete), or that the key was accessed.
  - renonce 4y ago
    Keeping a complete log of all GET requests to random files in a public repository in a reliable way would be insane.
    - otoolep 4y ago
      No, it wouldn’t be - assuming by “insane” you mean “silly to do”. I build systems at Google that do exactly that. Whether it’s worth the cost is a decision each company makes. Also, you don’t need to keep the log forever. Max of a few weeks retention would be common.
      - pbhjpbhj 4y ago
        Presumably, keeping 'last remotely accessed' and 'last remotely modified' for every file (or other stats that are a digest of the logs) is sane for pretty much any system too. Having a handle on how much space one is dedicating to files that are never viewed and or never updated seems like something web companies that have public file access would all want?
      - tgsovlerkhgsel 4y ago
        What guarantees do these systems provide? Are 100% of requests where data was served guaranteed to either end up in the log or at least create a detectable "logs may have been lost here" event? Or does it log all the requests all the time as long as everything goes well, but if the wrong server crashes at the wrong time, a couple hundred requests may get lost because in the end who cares?
    - est31 4y ago
      It's not just GET requests. Someone could have cloned/refreshed the repo using ssh. The repo might have been indexed by github's internal search daemon which might not use the public HTTP API but uses internal access ways however those might look like. You might have purged the database of that daemon but what about backups of it? What about people who have subscribed to public events happening in the github.com/github org via the API? You'd have to have logging set up for all of these services and it would have to work over your entire CDN... and what if a CDN node crashed before it was able to submit the log entry to the log collector? You'll never know.
- blitzar 4y ago
  One H4X0R gave it to four friends, who in turn gave it to between 9 and 14 friends, who in turn gave it to between one and 6 friends. If train A leaves New York going 60 miles per hour with 167 people on board and train B leaves Chicago one hour later with 361 people on board going 85 miles per hour, how many people now have the key? The answer is 31337.

p-e-w 4y ago

Same as with any other decision: Do a cost/benefit analysis of whether the security risk created by rotating the key is actually outweighed by the security risk of doing nothing, taking into account logs that should tell you whether the exposed key was indeed accessed by unauthorized parties. To be 100% clear: Both courses of action come with associated security risks. The problem is not choosing one course of action over the other, the problem is thinking you can just skip the cost/benefit analysis because the answer is somehow 'obvious'. It's not obvious at all.

johngalt_ 4y ago
No, you cannot keep using an exposed key. You must replace it. There is no cost/benefit analysis needed in this situation.
- p-e-w 4y ago
  Wrong. A CBA is always needed. If the potential damage from MITM attacks made possible by rotating the key is greater than the potential damage from a rogue key multiplied by the likelihood that someone actually accessed the key, then it is wrong to rotate the key. It's that simple. The only way a CBA would be unnecessary is if rotating the key didn't have any security risks. But it does.
  - eyelidlessness 4y ago
    Here I’ll do the CBA: - if they have evidence that the key was exposed to one person, even with zero usage of the key, failing to rotate the key is tantamount to knowingly accepting widespread compromise at a potential attacker’s whim. At GitHub’s scale, that’s untenable. - rotating the key is the only correct reaction to that - they should have better communications in place to help users mitigate MITM - there really isn’t an option, because they’re critical infrastructure; I’m glad they know that and acted accordingly - on principle this speculation makes sense, but understanding the threat makes it moot - you hopefully know that, and it’s good to insist on thoughtful security practices but it’s important to also understand the actual risk
  - 4y ago

rakoo 4y ago

That's why you need certificates and not just a key pair. Certificates make key rotation easier, and you want key rotation to be easy. I guess the proper way forward is a small utility that gets the latest signature through http+tls, and replaces the line in your known_hosts file, all in the background. Looking long term, maybe we need to get rid of all the security stuff in ssh and just pipe the rest of its functionalities inside a TLS pipe. Let the os do its certificate management, reuse security bricks that are way more studied, ...

oleganza 4y ago
Certificates just add more keys to worry about. The beauty of SSH is that it does not add hugely trusted parties in the name of convenience, while the UX of TOFU (trust on first use) is pretty decent. The real solution to break out of these UX/security tradeoffs is to put domain names on a blockchain: then you can simply rotate the key in your DNS record, while the blockchain model is such that you need to compromise many parties, instead of "one out of many parties", as with CAs. Tracking Bitcoin chain for DNS updates is lightweight enough that it can be built into OS alongside other modern components such as secure enclave, TCP/IP stack and WiFi/BT/5G radios.
- hnfong 4y ago
  > The beauty of SSH is that it does not add hugely trusted parties in the name of convenience Even with a certificate authority model, you don't have to trust any CAs if you don't want to. Not having the option to do so is more of a problem.
- datadeft 4y ago
  We should use a separate system that could reliably verify which certs belong to which entity. Blockchain is a perfect solution to this. I wonder why it is not considered yet.
  - jtsiskin 4y ago
    https://certificate.transparency.dev https://certificate.transparency.dev it is

ithkuil 4y ago

The alternative would be to use certificate authorities (ssh has CA support) which allow to effectively have private keys at different levels and allow you to keep the root private key in a physical vault and use it very rarely to issue other private keys

terom 4y ago
And then don't forget to setup key revocation as well, and make sure that an attacker in a position to MITM the connection cannot cause the revocation checks to fail-open. I hope you don't need that SSH connection to fix your broken CRL endpoint!
datadeft 4y ago
This would just offload the problem to a separate entity. CAs can be (and have been) compromised.
- ithkuil 4y ago
  Sure, but isn't it more likely that a key that has to be shared by who knows how many ssh load balancer machines at GitHub and can't be easily rotated because it's pinned by millions of users, isn't it more likely that that private key gets eventually compromised or thought to be at risk at being compromised? We need to compare the relative risks within the same context, namely within a company like GitHub So it's not relevant to bring up failures of other CAs

michaelt 4y ago

Jump into a time machine, go back to the creation of SSH, and adopt SSL-style trusted third-party certificate authorities. Somehow get it adopted anyway, even though loads of people use SSH on internal networks where host-based authentication is difficult; SSH is how many headless machines are bootstrapped; and that you've got to do it 19 years before Lets Encrypt. Jump into a lesser time machine, go back to when Github were creating their SSH key, and put it into a hardware security module. Somehow share that hardware-backed security key to loads of servers over a network, without letting hackers do the same thing. Somehow get an HSM that isn't a closed-source black box from a huge defence contractor riddled with foreign spies. Somehow avoid vendor lock-in or the HSM becoming obsolete. Somehow do this when you're a scrappy startup with barely any free time.

nibbleshifter 4y ago
Ssh certificate authorities are a thing that exists. We also have a way to put SSH host key fingerprints in DNS records already.
- Arch-TK 4y ago
  Yes but the option to do verify host keys using ("VerifyHostKeyDNS") is not enabled by default.
- roblabla 4y ago
  DNS can trivially be mitm'd. DNS-stored fingerprints are strictly less secure than TOFU.
  - tialaramex 4y ago
    If you use DNSSEC (cue inevitable rant from Thomas) this just works. If you have DoH (and why wouldn't you?) and your trusted resolver uses DNSSEC (which popular ones do), you get the same benefits. https://en.wikipedia.org/wiki/SSHFP_record https://en.wikipedia.org/wiki/SSHFP_record
- datadeft 4y ago

pid-1 4y ago

Clone repos using oauth2 with two factor enabled - both GitHub and GitLab support that though their CLIs.

marcosdumay 4y ago

Well, at least SSH could allow for signing a new key with the old one. So they could say it's signed, and people would know to accept only a different prompt. There is DNS verification, but people have been trained all their lives to accept insecure DNS information (and set their systems accordingly), and I really doubt the SSH client checks the DNSSEC data.