3 ms·

So naive, so wrong. It's hard to know where to start. > Operating SSH at scale is a disaster. Key approval & distribution is a silly waste of time. Host names

So naive, so wrong. It's hard to know where to start. > Operating SSH at scale is a disaster. Key approval & distribution is a silly waste of time. Host names can’t be reused. Homegrown tools scatter key material across your fleet that must be cleaned up later to off-board users. Outright lie or incompetence at worst, or a "my salary depends on not understanding this" case at best, being demonstrated here. Even if you don't have chef/puppet/etc managing at least some level of fleet configuration (if you don't, your org is hopeless anyway and ssh-at-scale is not a first problem to solve), it's oh-so-easy to put just your pubkeys somewhere (S3, eg) and manage them via, say, git. Next problem is that openssh-style certs do not support cert chains. So you cannot revoke an intermediate. A huge aspect of PKI is that you must be able to recover from a compromise. The rest of this point is left as an exercise for the reader. You have to secure the CA! You have to properly authorize signing requests! These things fall into the realm of hard.

ryan_lane 7y ago
You should checkout BLESS (or better, Lyft's changes to BLESS), which drastically reduce the work traditionally needed to do this. Basically, you piggy-back on your cloud provider's security tooling.
mschuster91 7y ago
> Outright lie or incompetence at worst, or a "my salary depends on not understanding this" case at best, being demonstrated here. Even if you don't have chef/puppet/etc managing at least some level of fleet configuration (if you don't, your org is hopeless anyway and ssh-at-scale is not a first problem to solve), it's oh-so-easy to put just your pubkeys somewhere (S3, eg) and manage them via, say, git. Puppet's ssh key management only adds and removes public keys that you specify, but leaves other keys intact. Which means that a hacker can place a backdoor key in authorized_keys that is not going to be spotted by Puppet. Cert-only auth prevents this class of attack from the beginning.
- jiveturkey 7y ago
  Why would you use the puppet-esque style of management then? Just generate the complete list of authorized_keys yourself, don't let puppet modify it for you. But do let puppet distribute the file. It's beyond easy. But this is a distraction. Anyone that can influence the insertion of a permanent backdoor in authorized_keys is almost certainly in deep enough to not care about any specific mechanism. Being able to populate arbitrary, unaudited data on your hosts is an intolerable situation to be in. And anyway, you're wrong. The 'purge_ssh_keys' directive can be set to 'true', disabling merge. > Cert-only auth prevents this class of attack from the beginning. Cert auth is a subset of pubkey auth, which includes both. You could set AuthorizedKeysFile to 'none' but you'd have to explain how an attacker that can influence authorized_keys cannot also influence sshd_config. Of course these are different files likely populated via different mechanism, but the threat model you are implying here seems overly specific. It might work for some people but, given the tone of the article (unspecified 'you' are doing in wrong), I don't buy it.
mmalone 7y ago
Am author. > chef/puppet/etc ... put just your pubkeys somewhere (S3, eg) ... Yea, cool, and now you have a second authentication system that you have to manually administer to onboard and offboard people. And you have permanent long-lived single-factor credentials on a bunch of (possibly unmanaged) endpoints, that are easy to exfiltrate, and are a common target for attackers if a laptop is lost/stolen/compromised. You haven't addressed host name reuse or host rekeying which, again, are real operational problems. And somehow running and administering chef/puppet/etc or this S3 bucket is easier than running a simple certificate signing service connected to SSO that requires basically zero maintenance once it's setup? Argumentative, at best. > Next problem is that openssh-style certs do not support cert chains ... recover from compromise. Since only hopeless organizations don't have config management, we can solve this there. SSH PKI is simple and doesn't have intermediates. But you can configure multiple roots. To rotate or revoke a compromised private key you'd simply deploy a new root using cert management and issue new certificates. In a non-emergency situation you can slowly roll certificates by having two trusted roots for a transitional period while everyone gets new certs. > You have to secure the CA! You have to secure your configuration management that's deploying public keys everywhere and your authorized keys files on every host! Pubkey distribution has a bigger attack surface area and the same exact compromise consequences as a CA. This is not even a contest. > You have to properly authorize signing requests! You have to authorize requests to deploy public keys! There are tools for this. What's the hard part here? Verifying an OIDC identity token? Mapping a token subject to an SSH principal? Worst case, you could keep whatever crappy manual process you have for authorizing public key distribution and issue a certificate instead. > These things fall into the realm of hard. Ok, I guess? This is all relative. It's easier than pubkey authentication at any non-trivial scale. There's a difference between something being hard vs. being unfamiliar.
- jiveturkey 7y ago
  > and now you have a second authentication system ... [and a host of other problems] You have that anyway. I wasn't suggesting to dump authorized_keys into S3 if you aren't already using S3. I was just using a throwaway example. Any large enough environment to care, is large enough to need to manage this type of resource as a sunk cost. > somehow running and administering chef/puppet/etc or this S3 bucket is easier Yes, because it's a sunk cost, and mandatory anyway, outside of any ssh consideration. Any org should be striving to do less things, not more things. > simple certificate signing service there's no such thing > configure multiple roots You got me there. That is a reasonable enough solution. Of course your 'backup' root has to be offline, but that's the same as with an intermediate. I could probably manufacture a case where having a cert chain is easier but since it's not obvious to me at a moment, I'm sure would be pretty contrived. But, aren't you getting farther and farther away from simple? > You have to secure your configuration management that's deploying public keys everywhere and your authorized keys files on every host! True. However, that's a sunk cost. You have to do that anyway. > Worst case, you could keep whatever crappy manual process you have for authorizing public key distribution and issue a certificate instead. Then why bother with the cert at all? Also, you are insisting that an org's manual process is crappy. If your process for pubkey authz and distribution is crappy, you have crappy practices and it's a sure thing your shiny new CA is a huge liability as another crappily deployed and crappily maintained system. > It's easier than pubkey authentication at any non-trivial scale. Well, I dispute that. At large scale you have even better config mgmt and if you're doing it right, can handle either way. CA is still probably better because the benefit to you at large scale is not relief from managing pubkey resources, it's from being able to do short-lived certs as well as an easier route to tighter authz. At moderate scale, get real, these things aren't a concern. Your main problem isn't that you're wrong, it's that your absolutist view of it is a distortion. (Which makes it wrong.)

ryan_lane 7y ago

You should checkout BLESS (or better, Lyft's changes to BLESS), which drastically reduce the work traditionally needed to do this. Basically, you piggy-back on your cloud provider's security tooling.

mschuster91 7y ago

> Outright lie or incompetence at worst, or a "my salary depends on not understanding this" case at best, being demonstrated here. Even if you don't have chef/puppet/etc managing at least some level of fleet configuration (if you don't, your org is hopeless anyway and ssh-at-scale is not a first problem to solve), it's oh-so-easy to put just your pubkeys somewhere (S3, eg) and manage them via, say, git. Puppet's ssh key management only adds and removes public keys that you specify, but leaves other keys intact. Which means that a hacker can place a backdoor key in authorized_keys that is not going to be spotted by Puppet. Cert-only auth prevents this class of attack from the beginning.

jiveturkey 7y ago
Why would you use the puppet-esque style of management then? Just generate the complete list of authorized_keys yourself, don't let puppet modify it for you. But do let puppet distribute the file. It's beyond easy. But this is a distraction. Anyone that can influence the insertion of a permanent backdoor in authorized_keys is almost certainly in deep enough to not care about any specific mechanism. Being able to populate arbitrary, unaudited data on your hosts is an intolerable situation to be in. And anyway, you're wrong. The 'purge_ssh_keys' directive can be set to 'true', disabling merge. > Cert-only auth prevents this class of attack from the beginning. Cert auth is a subset of pubkey auth, which includes both. You could set AuthorizedKeysFile to 'none' but you'd have to explain how an attacker that can influence authorized_keys cannot also influence sshd_config. Of course these are different files likely populated via different mechanism, but the threat model you are implying here seems overly specific. It might work for some people but, given the tone of the article (unspecified 'you' are doing in wrong), I don't buy it.

mmalone 7y ago

Am author. > chef/puppet/etc ... put just your pubkeys somewhere (S3, eg) ... Yea, cool, and now you have a second authentication system that you have to manually administer to onboard and offboard people. And you have permanent long-lived single-factor credentials on a bunch of (possibly unmanaged) endpoints, that are easy to exfiltrate, and are a common target for attackers if a laptop is lost/stolen/compromised. You haven't addressed host name reuse or host rekeying which, again, are real operational problems. And somehow running and administering chef/puppet/etc or this S3 bucket is easier than running a simple certificate signing service connected to SSO that requires basically zero maintenance once it's setup? Argumentative, at best. > Next problem is that openssh-style certs do not support cert chains ... recover from compromise. Since only hopeless organizations don't have config management, we can solve this there. SSH PKI is simple and doesn't have intermediates. But you can configure multiple roots. To rotate or revoke a compromised private key you'd simply deploy a new root using cert management and issue new certificates. In a non-emergency situation you can slowly roll certificates by having two trusted roots for a transitional period while everyone gets new certs. > You have to secure the CA! You have to secure your configuration management that's deploying public keys everywhere and your authorized keys files on every host! Pubkey distribution has a bigger attack surface area and the same exact compromise consequences as a CA. This is not even a contest. > You have to properly authorize signing requests! You have to authorize requests to deploy public keys! There are tools for this. What's the hard part here? Verifying an OIDC identity token? Mapping a token subject to an SSH principal? Worst case, you could keep whatever crappy manual process you have for authorizing public key distribution and issue a certificate instead. > These things fall into the realm of hard. Ok, I guess? This is all relative. It's easier than pubkey authentication at any non-trivial scale. There's a difference between something being hard vs. being unfamiliar.

jiveturkey 7y ago
> and now you have a second authentication system ... [and a host of other problems] You have that anyway. I wasn't suggesting to dump authorized_keys into S3 if you aren't already using S3. I was just using a throwaway example. Any large enough environment to care, is large enough to need to manage this type of resource as a sunk cost. > somehow running and administering chef/puppet/etc or this S3 bucket is easier Yes, because it's a sunk cost, and mandatory anyway, outside of any ssh consideration. Any org should be striving to do less things, not more things. > simple certificate signing service there's no such thing > configure multiple roots You got me there. That is a reasonable enough solution. Of course your 'backup' root has to be offline, but that's the same as with an intermediate. I could probably manufacture a case where having a cert chain is easier but since it's not obvious to me at a moment, I'm sure would be pretty contrived. But, aren't you getting farther and farther away from simple? > You have to secure your configuration management that's deploying public keys everywhere and your authorized keys files on every host! True. However, that's a sunk cost. You have to do that anyway. > Worst case, you could keep whatever crappy manual process you have for authorizing public key distribution and issue a certificate instead. Then why bother with the cert at all? Also, you are insisting that an org's manual process is crappy. If your process for pubkey authz and distribution is crappy, you have crappy practices and it's a sure thing your shiny new CA is a huge liability as another crappily deployed and crappily maintained system. > It's easier than pubkey authentication at any non-trivial scale. Well, I dispute that. At large scale you have even better config mgmt and if you're doing it right, can handle either way. CA is still probably better because the benefit to you at large scale is not relief from managing pubkey resources, it's from being able to do short-lived certs as well as an easier route to tighter authz. At moderate scale, get real, these things aren't a concern. Your main problem isn't that you're wrong, it's that your absolutist view of it is a distortion. (Which makes it wrong.)