4 ms·

Imagine the amount of ssh connections that github must handle every second, I don't think it's so easy to hold the private key in a HSM while maintaining the ne

by 0x0 4y ago

Imagine the amount of ssh connections that github must handle every second, I don't think it's so easy to hold the private key in a HSM while maintaining the necessary performance and availability.

StopHammoTime 4y ago
Yeah but surely there’s a happy medium ground between HSM and storing it on some old mates desktop (which would have to be what’s going on here). It still shouldn’t have been easily accessible to anyone except the instances running the SSH service.
parhamn 4y ago
Loading it from HSM to memory/keychain is probably fine too. It's certainly odd it found its way to a repo and makes you wonder how that could have even possibly happened. And what that indicates about their security practices in general. Github is host to a large percent of US tech IP. Pretty concerning if you extrapolate.
- 0x0 4y ago
  The whole point of a HSM is to never reveal the private key, but instead to have the HSM perform the necessary crypto operation for each request. This wouldn't scale to the amount of ssh traffic I would imagine github sees. I think it's completely reasonable that they have the ssh private key in some sort of configuration management repository, because they need to be able to deploy that key to all their public facing ssh servers. You would hope they would have more than 1 ssh server instance world wide for availability and resiliency.
  - Dylan16807 4y ago
    > The whole point of a HSM is to never reveal the private key That's not "the whole point". You can have methods to copy a key between HSMs, methods that can scale just fine while being much more secure than a file in a repository.
  - retrocryptid 4y ago
    Or you could just buy HSMs that can do the number of private key operations per second that you require. I've never seen the inside of GH's network, but I would be surprised to discover they're not distributing the load of SSH termination across a fleet of machines. Just put a HSM on each of the machines that terminates a SSH connection.
vbezhenar 4y ago
HSM is just a computer. May be with crypto accelerators. You can set up OpenBSD boxes, put them into safes and disable any incoming connections other than your custom HSM protocol and that would work better than storing private keys on web frontends. Yes, it makes things hard and unconventional to set up. But GitHub is not some small website.
- retrocryptid 4y ago
  That's not all of what an HSM is. Or should be. The beefier ones come with rf shielding to prevent bad guys from trying to reduce the key search space by listening to EM energy coming out of the box. And active key zeroation if it thinks you're trying to drill through the epoxy surrounding the crypto boundary.
  - joezydeco 4y ago
    Is it technically possible (and/or wise) to duplicate the key across multiple HSMs running in parallel? I'm guessing if you have a super massive CDN like Apple or Meta this is a necessary thing.
    - retrocryptid 4y ago
      Yes.
  - seri4l 4y ago
    >And active key zeroation if it thinks you're trying to drill through the epoxy surrounding the crypto boundary. Or passive! Probably it wasn't a real product but I recall reading about one that derived its key from the field generated by randomly arranged magnetic particles in the resin, or something like that. The point was to make it impossible to disturb the resin without altering the key.
red_admiral 4y ago
That sounds like an engineering problem. I have my political opinions on Meta as much as the next person, but every bit of evidence I've seen so far is they take this kind of security thing seriously and they definitely operate on a larger scale than github, although most of their traffic is HTTPS not SSH.
- deathanatos 4y ago
  > I have my political opinions on Meta as much as the next person, but every bit of evidence I've seen so far is they take this kind of security thing seriously Meta doesn't own Github, MS does.
retrocryptid 4y ago
HSMs don't handle the entire load of the the connection, only the private key operations. At the 10k per sec sites I worked at, we attached some beefy HSMs to the few (less than 40) machines that terminated TLS.

by 0x0 4y ago

Imagine the amount of ssh connections that github must handle every second, I don't think it's so easy to hold the private key in a HSM while maintaining the necessary performance and availability.

StopHammoTime 4y ago

Yeah but surely there’s a happy medium ground between HSM and storing it on some old mates desktop (which would have to be what’s going on here). It still shouldn’t have been easily accessible to anyone except the instances running the SSH service.

parhamn 4y ago

Loading it from HSM to memory/keychain is probably fine too. It's certainly odd it found its way to a repo and makes you wonder how that could have even possibly happened. And what that indicates about their security practices in general. Github is host to a large percent of US tech IP. Pretty concerning if you extrapolate.

0x0 4y ago
The whole point of a HSM is to never reveal the private key, but instead to have the HSM perform the necessary crypto operation for each request. This wouldn't scale to the amount of ssh traffic I would imagine github sees. I think it's completely reasonable that they have the ssh private key in some sort of configuration management repository, because they need to be able to deploy that key to all their public facing ssh servers. You would hope they would have more than 1 ssh server instance world wide for availability and resiliency.
- Dylan16807 4y ago
  > The whole point of a HSM is to never reveal the private key That's not "the whole point". You can have methods to copy a key between HSMs, methods that can scale just fine while being much more secure than a file in a repository.
- retrocryptid 4y ago
  Or you could just buy HSMs that can do the number of private key operations per second that you require. I've never seen the inside of GH's network, but I would be surprised to discover they're not distributing the load of SSH termination across a fleet of machines. Just put a HSM on each of the machines that terminates a SSH connection.

vbezhenar 4y ago

HSM is just a computer. May be with crypto accelerators. You can set up OpenBSD boxes, put them into safes and disable any incoming connections other than your custom HSM protocol and that would work better than storing private keys on web frontends. Yes, it makes things hard and unconventional to set up. But GitHub is not some small website.

retrocryptid 4y ago
That's not all of what an HSM is. Or should be. The beefier ones come with rf shielding to prevent bad guys from trying to reduce the key search space by listening to EM energy coming out of the box. And active key zeroation if it thinks you're trying to drill through the epoxy surrounding the crypto boundary.
- joezydeco 4y ago
  Is it technically possible (and/or wise) to duplicate the key across multiple HSMs running in parallel? I'm guessing if you have a super massive CDN like Apple or Meta this is a necessary thing.
  - retrocryptid 4y ago
    Yes.
- seri4l 4y ago
  >And active key zeroation if it thinks you're trying to drill through the epoxy surrounding the crypto boundary. Or passive! Probably it wasn't a real product but I recall reading about one that derived its key from the field generated by randomly arranged magnetic particles in the resin, or something like that. The point was to make it impossible to disturb the resin without altering the key.

red_admiral 4y ago

That sounds like an engineering problem. I have my political opinions on Meta as much as the next person, but every bit of evidence I've seen so far is they take this kind of security thing seriously and they definitely operate on a larger scale than github, although most of their traffic is HTTPS not SSH.

deathanatos 4y ago
> I have my political opinions on Meta as much as the next person, but every bit of evidence I've seen so far is they take this kind of security thing seriously Meta doesn't own Github, MS does.

retrocryptid 4y ago

HSMs don't handle the entire load of the the connection, only the private key operations. At the 10k per sec sites I worked at, we attached some beefy HSMs to the few (less than 40) machines that terminated TLS.