5 ms·

We looked into them once at work for our "conventional" SSH infrastructure, but immediately rammed into the wall that a certificate can sign a key, but you can'

by jerf 4y ago

We looked into them once at work for our "conventional" SSH infrastructure, but immediately rammed into the wall that a certificate can sign a key, but you can't have a full PKI infrastructure like SSL where certs can sign certs that can sign keys. Without the ability to delegate, the logistics of needing to create an absolutely centralized authority to do the useful things we wanted to do within the organization raised the cost of "first value" too high for us to fuss over. So for large-ish organizations (~1000 affected employees) we had a hard time justifying any effort put into it versus the established infrastructures for key management that already exist. On the other side, as a personal user of SSH with basically one person to worry about, the effort of setting up a certificate seems like just a waste versus the existing key-based infrastructure; I don't understand at all what attack it would prevent or what convenience it would provide for the cost of learning it. A general principle is you get good at what you do a lot, and you aren't good at what you do rarely. Key-based management has its issues, but people have been doing it for a while and they're good at it. Not necessarily "great" and not necessarily "unable to be improved", but they're good at it. People understand it and know how to use it. Nobody uses certificates, so nobody understands how they work or what to do with them, and they have the weakness I cited above. So, even though in a perfect world certificates may be superior, in the world we live in now, the competition between the existing polished infrastructure for pure key-based management is pretty good, even if you think it's a polished turd, and the infrastructure for SSH certs is pretty bad, even if you think it's an uncut diamond in the rough. It's difficult to look at that situation and honestly choose to proceed with the certificates. As others point out, the cloud services are proceeding in other directions as well which will probably continue to make using SSH certificates directly a very difficult choice to practically justify.

gunapologist99 4y ago
I agree. We use a distributed Userify infrastructure to take some of the sting out of key management for our teams; this gives us the ability to tie user authentication into our enterprise directory but still delegate an entire Userify infrastructure on a BU basis, which is pretty awesome. To your point about clouds making it more difficult in the future, it's becoming more challenging to scale the CA across auto scaling groups with dynamic IPs and where you can't control TXT records. We've found that SSH keys are very flexible, and we simply needed a centralized management point (which we have now), but with decentralized operation in case anything is offline along the way. Also, it's very important to us to be able to kill active sessions globally when a user is removed, and there doesn't seem to be any other tools that do that except Userify.
cubesnooper 4y ago
> On the other side, as a personal user of SSH with basically one person to worry about, the effort of setting up a certificate seems like just a waste versus the existing key-based infrastructure; I don't understand at all what attack it would prevent or what convenience it would provide for the cost of learning it. The main benefit I get from using SSH certs at home is expiration. Before, I always had a niggling feeling at the back of my mind that the public key I’d been using for four years could have been surreptitiously exfiltrated by some shell script three years ago, and I’d never notice. Now my certificates expire regularly, so I no longer have to worry about whether my machine remained secure throughout the entire continuous past; I only have to worry if my machine is secure in the present.
- xyzzyz 4y ago
  How do you issue client certificates? Do you not worry that the private key needed to issue these can also leak just the same as your personal private key?
  - cubesnooper 4y ago
    I have a Raspberry Pi dedicated to generating certificates. It serves the files to my LAN statically via a webserver, and is otherwise heavily firewalled. I don't run any other software on the Pi, so barring an exploit in the webserver, I’m not worried about the signing key getting compromised. Compared to my desktop, where over the years I ran all kinds of stuff from the package manager, downloaded Python scripts and configure scripts from GitHub and Sourceforge… I tried to be careful but you can’t audit everything.
    - xyzzyz 4y ago
      So anyone on your LAN can visit the URL and download the CA private key? Isn’t it only marginally more secure than just keeping the private key on your workstation in the first place, and foregoing the entire rigamarole with certificates? I mean, if you are worried about your something compromising your workstation and stealing your individual private keys, nothing is stopping whoever compromised your workstation from asking your RaspberryPi nicely for CA signing keys and stealing those too.
xyzzyz 4y ago
> Without the ability to delegate, the logistics of needing to create an absolutely centralized authority to do the useful things we wanted to do within the organization raised the cost of "first value" too high for us to fuss over. Have you considered integrating this with the SSO provider you are almost certainly already using as 1k strong org? Basically, create groups in Okta or whatever you’re using, delegate management of these groups to whoever you want, and then configure Teleport to use those groups for authorization in various contexts.
- jerf 4y ago
  Well, that is either 1. exactly the sort of development that I was talking about on top of the tech, rather than the tech being directly useful and 2. the sort of thing the clouds are naturally doing on their own.
tptacek 4y ago
Essentially the whole point of modern SSH CAs is to centralize authentication and authorization for servers, and to keep long-lived keys off your developer machine fleet. If "no long-lived keys on laptops" and "central authority for who gets access to what with which credentials" aren't principles for your security practice, then you're right: there's not much of a win to switching to a CA model.
- gunapologist99 4y ago
  Sarcastic remarks like "if [mostly good things] aren't principles for your security practice" an strawman arguments seem a bit provocative for this type of debate.
  - tptacek 4y ago
    There was absolutely nothing sarcastic about that remark. I wouldn't snark at 'jerf. Most security practices don't have those principles for SSH. Thanks for calling it out though; if you read it that way, they might have too, which would be a little mortifying.
    - gunapologist99 4y ago
      That's fair. thanks for the clarification!
- jerf 4y ago
  We use different central authorities than certificates, which manage keys directly; the end result is much the same even if the tools are theoretically inferior. Long-lived keys on laptops might be a weakness at this point, honestly. I expect that to be resolved by a slow but steady move to "whatever the cloud supports" as AWS and such rolls out their various solutions. I know we've got teams that use them, I imagine it'll be obligatory at some point here. I know it's not perfect, which is why I contrasted "polished turd" to the "uncut diamond in the rough". We're at a size where it's really hard to justify a whole lot of work on the uncut diamond, in contrast with the various cloud efforts to fix their problems. Given that such things now exist it's more a matter of moving over to them than developing anything ourselves now, which is probably an improvement over where we were two years ago. We probably couldn't have beaten the clouds even if we did put the effort in; this is in that class of problem where the effort is nominally small but the price of not breaking things is so large than it ends up a multi-year project anyhow. (One of my least favorite classes of project.) I'd also add that I report this not in the spirit of "we've solved this problem", but just an experience report.
d0gsg0w00f 4y ago
We set up an ssh cert signing server using Hashicorp Vault [1] so we didn't have to keep rotating keys across a fleet of boxes we couldn't always reach. It was cool but it made me nervous to centralize something like that. Cert sigantures expired after 30 min so leaks were not a problem. It was backed by DynamoDB and the keys were in memory so everytime the containers rebooted someone would have to manually unseal before SSH would work again. [1] - https://www.hashicorp.com/blog/managing-ssh-access-at-scale-with-hashicorp-vault https://www.hashicorp.com/blog/managing-ssh-access-at-scale-...
- seti0Cha 4y ago
  We did this as well. To mitigate the danger of a vault outage, we had a very tightly restricted endpoint which generated 24 hour certs that were fetched by automation on a single box. In the event of a vault outage, we'd have a day to either fix it or deploy public key access.

by jerf 4y ago

gunapologist99 4y ago

I agree. We use a distributed Userify infrastructure to take some of the sting out of key management for our teams; this gives us the ability to tie user authentication into our enterprise directory but still delegate an entire Userify infrastructure on a BU basis, which is pretty awesome. To your point about clouds making it more difficult in the future, it's becoming more challenging to scale the CA across auto scaling groups with dynamic IPs and where you can't control TXT records. We've found that SSH keys are very flexible, and we simply needed a centralized management point (which we have now), but with decentralized operation in case anything is offline along the way. Also, it's very important to us to be able to kill active sessions globally when a user is removed, and there doesn't seem to be any other tools that do that except Userify.

cubesnooper 4y ago

> On the other side, as a personal user of SSH with basically one person to worry about, the effort of setting up a certificate seems like just a waste versus the existing key-based infrastructure; I don't understand at all what attack it would prevent or what convenience it would provide for the cost of learning it. The main benefit I get from using SSH certs at home is expiration. Before, I always had a niggling feeling at the back of my mind that the public key I’d been using for four years could have been surreptitiously exfiltrated by some shell script three years ago, and I’d never notice. Now my certificates expire regularly, so I no longer have to worry about whether my machine remained secure throughout the entire continuous past; I only have to worry if my machine is secure in the present.

xyzzyz 4y ago
How do you issue client certificates? Do you not worry that the private key needed to issue these can also leak just the same as your personal private key?
- cubesnooper 4y ago
  I have a Raspberry Pi dedicated to generating certificates. It serves the files to my LAN statically via a webserver, and is otherwise heavily firewalled. I don't run any other software on the Pi, so barring an exploit in the webserver, I’m not worried about the signing key getting compromised. Compared to my desktop, where over the years I ran all kinds of stuff from the package manager, downloaded Python scripts and configure scripts from GitHub and Sourceforge… I tried to be careful but you can’t audit everything.
  - xyzzyz 4y ago
    So anyone on your LAN can visit the URL and download the CA private key? Isn’t it only marginally more secure than just keeping the private key on your workstation in the first place, and foregoing the entire rigamarole with certificates? I mean, if you are worried about your something compromising your workstation and stealing your individual private keys, nothing is stopping whoever compromised your workstation from asking your RaspberryPi nicely for CA signing keys and stealing those too.

xyzzyz 4y ago

> Without the ability to delegate, the logistics of needing to create an absolutely centralized authority to do the useful things we wanted to do within the organization raised the cost of "first value" too high for us to fuss over. Have you considered integrating this with the SSO provider you are almost certainly already using as 1k strong org? Basically, create groups in Okta or whatever you’re using, delegate management of these groups to whoever you want, and then configure Teleport to use those groups for authorization in various contexts.

jerf 4y ago
Well, that is either 1. exactly the sort of development that I was talking about on top of the tech, rather than the tech being directly useful and 2. the sort of thing the clouds are naturally doing on their own.

tptacek 4y ago

Essentially the whole point of modern SSH CAs is to centralize authentication and authorization for servers, and to keep long-lived keys off your developer machine fleet. If "no long-lived keys on laptops" and "central authority for who gets access to what with which credentials" aren't principles for your security practice, then you're right: there's not much of a win to switching to a CA model.

gunapologist99 4y ago
Sarcastic remarks like "if [mostly good things] aren't principles for your security practice" an strawman arguments seem a bit provocative for this type of debate.
- tptacek 4y ago
  There was absolutely nothing sarcastic about that remark. I wouldn't snark at 'jerf. Most security practices don't have those principles for SSH. Thanks for calling it out though; if you read it that way, they might have too, which would be a little mortifying.
  - gunapologist99 4y ago
    That's fair. thanks for the clarification!
jerf 4y ago
We use different central authorities than certificates, which manage keys directly; the end result is much the same even if the tools are theoretically inferior. Long-lived keys on laptops might be a weakness at this point, honestly. I expect that to be resolved by a slow but steady move to "whatever the cloud supports" as AWS and such rolls out their various solutions. I know we've got teams that use them, I imagine it'll be obligatory at some point here. I know it's not perfect, which is why I contrasted "polished turd" to the "uncut diamond in the rough". We're at a size where it's really hard to justify a whole lot of work on the uncut diamond, in contrast with the various cloud efforts to fix their problems. Given that such things now exist it's more a matter of moving over to them than developing anything ourselves now, which is probably an improvement over where we were two years ago. We probably couldn't have beaten the clouds even if we did put the effort in; this is in that class of problem where the effort is nominally small but the price of not breaking things is so large than it ends up a multi-year project anyhow. (One of my least favorite classes of project.) I'd also add that I report this not in the spirit of "we've solved this problem", but just an experience report.

d0gsg0w00f 4y ago

We set up an ssh cert signing server using Hashicorp Vault [1] so we didn't have to keep rotating keys across a fleet of boxes we couldn't always reach. It was cool but it made me nervous to centralize something like that. Cert sigantures expired after 30 min so leaks were not a problem. It was backed by DynamoDB and the keys were in memory so everytime the containers rebooted someone would have to manually unseal before SSH would work again. [1] - https://www.hashicorp.com/blog/managing-ssh-access-at-scale-with-hashicorp-vault https://www.hashicorp.com/blog/managing-ssh-access-at-scale-...

seti0Cha 4y ago
We did this as well. To mitigate the danger of a vault outage, we had a very tightly restricted endpoint which generated 24 hour certs that were fetched by automation on a single box. In the event of a vault outage, we'd have a day to either fix it or deploy public key access.