5 ms·

SSH Keys Don't Scale. SSH Certificates Do

20 points by dangtony98 2y ago

karmicthreat 2y ago
This seems kind of like teleport without the logging and remote infrastructure.
- dangtony98 2y ago
  Definitely some overlap there for sure but with different architecture — this is also an extension of the broader Infisical platform that a lot of companies use for secrets management.
sepositus 2y ago
I was curious if Infisical would eventually turn into Vault (i.e., move beyond pure secret management), and this appears to be the first step in that direction. Granted, the onboarding/usage steps in the article are dramatically simplified compared to Vault. We used to use a method that was identical to this using Vault. Even wrote our own mini-CLI with a similar usage pattern. However, nowadays, we rely on Tailscale SSH (with a break-glass key) and have never really felt the need for an alternative.
- dangtony98 2y ago
  Co-founder of Infisical here! You're right in noticing that we're moving beyond secrets management into adjacent related verticals as well, specifically branching out more into the larger identity and access management space; secrets management continues to be the bread and butter of Infisical but the way we see it, it would be super nice in the long run to have one unified control plane over all related aspects spanning secrets management to certificate management, SSH access, and more. One of the core product philosophies of Infisical has always been to abstract away complexity as much as possible while still giving users the ability to customize the tool. Whereas Vault might, for example, have you explicitly create SSH certificate authorities and require 12+ steps to configure a working SSH certificate-based authentication model, Infisical SSH makes it so you only have to care about users and hosts that is who has access to what with something like 4 steps and if you want more power out of Infisical then that is always possible to build atop. Good to know regarding Tailscale SSH!
fisian 2y ago
It's just an ad for their SSH cert service... I feel like for SSH certs to expand beyond large companies, there's the need for an open-source service which does the issuing of short-lived certs after a user authenticates. I know smallstep, but their offer feels open-core/freemium.
- dangtony98 2y ago
  Hey totally agree with the open source aspect here in order for SSH certificates to reach broader adoption (coupled with seamless admin and user experience). Infisical SSH is actually an extension of the Infisical platform which is open source and used by a ton of companies for secrets management.
szszrk 2y ago
I would love to ask you lovely HN people, assuming you actually use SSH for work: - does you organization use ssh certs? - how big is the org? ("I know most IT", or "it's impossible to know the whole IT"?) - were you the ones that proposed and implemented that change? :) SSH certs make so much sense and I know about them for a long time. Yet never implemented that approach - we had at most 2 people that were actually interested. And now for me everything is http API's and oauth, so I don't need it any more. I wonder what are reasons orgs don't use ssh certs by default.
- 77pt77 2y ago
  I have never met anyone who used certs with ssh, despite being the obvious choice. But since apparently ssh is now obsolete (as I was told here a week ago or so) maybe it makes no difference.
  - szszrk 2y ago
    I will throw it in the bin with other obsolete tech from at least 40+ years ago. Like NFS. Oh, wait.
    - 77pt77 2y ago
      I'd say ssh is here to stay far more than NFS.
- hylaride 2y ago
  We implemented it at a previous company, but it was a managed services provider that had access to customer VM's or other infrastructure. It was part of the soc2 process to "approve, audit, and then grant timed access" to various infra. Everywhere else I worked just used ssh keys, either deployed via ansible or via LDAP because it's so much more simple unless there's a high need for an audit trail with proof of approval. SSH keys scale perfectly fine otherwise. Unless I'm desperate, I'll never work for a managed services company again, so hopefully I'll never need to bother.
- dangtony98 2y ago
thinkingkong 2y ago
If you need easy distributed key management, modern SSH makes this fairly straightforward with some config values. It supports executing a program to get the SSH key at login time, dynamically. This way you can still maintain local certificates for fallback, and you can plug into anything. For example in the past I wrote a simple golang based app that loaded all of the SSH pubkeys from my organizations github, for users in a specific team.
ltbarcly3 2y ago
github uses ssh keys. I would wager github's auth volume is orders of magnitude bigger than anything infiscale has ever done or will ever do. therefore ssh keys do in fact scale.
- szszrk 2y ago
  To be fair, they do just part of that process. They don't generate keys at all (unless I'm mistaken?) and are vulnerable to that very issue discussed here. You can't easily create rules how to expire that access, or manage compromised keys.
  - deleted 2y ago
    [deleted]
  - ltbarcly3 2y ago
    I see what you mean, but this just trades one thing for another. On the one hand I could write a script that goes over every host and rsync's everyone's public key to it. You could have a cron job that does this. If you want to set up access control you would have sets of keys and different sets of keys would go to different servers. If you revoke a key you just need to remove it from the set and run whatever mechanism syncs the keys (a script that runs rsync under xargs would let you manage a few thousand servers pretty easily). Ok compare that to this solution: you still have to define sets of users or certificates. You still have to go edit the configuration on every server to put the correct list of certificates on there, so you are managing the lists just the same. If you find out something needs to be revoked you have to go do that. What's good: You don't have to go touch the servers in a loop except when changing the ACL setup for the servers. You can avoid having anything that can access the key setup for those servers (not really, you are just pretending this is true, but kindof). What's bad: You have to buy into their model. You have to go change how you do auth on all the servers. Users can't generate their own keys anymore, which is annoying. Some things that magically work with keys will be hard or impossible to configure with certificates, especially home grown stuff. Whenever inifiwhatever gets hacked you are in serious trouble.
    - 2y ago
abricq 2y ago
I'm actually really bad at all this networking certificates, and have a question for the experts. If a user have to sign a short-lived CA and then present that certificate to the host he wishes to connect with, isn't that basically allowing CA emitters to track the user's activity for this host ? This feels like replacing ssh for shh-with-tracking. Am i missing something ?
- szszrk 2y ago
  It's probably a typo, but a user rather uses short lived Cert, not a CA. CA (certificate authority) issues a Cert. CA doesn't have to be (and likely isn't) in users full control. Hosts answering to connections using that cert don't have to send anything back to CA. They just need rules "I trust this CA. If a user has a cert from it, and this precise combination of fields match, I consider this user's response trustworthy".
  - dangtony98 2y ago
    That's correct — The user obtains a short-lived SSH certificate (signed/issued by a CA).
Bleibeidl 2y ago
... or use soemthing like FreeIPA
bananapub 2y ago
Why do people upvote this sort of lazy and shitty content marketing?
chasil 2y ago
The content of the article is OpenSSH-centric. Especially on Windows, this needs to include the PuTTY pageant.exe. I am on a corporate desktop where the agent service is not running: C:\>ssh-agent unable to start ssh-agent service, error :1058 Because of this, I use pageant.exe for the Microsoft OpenSSH binaries: https://the.earth.li/~sgtatham/putty/0.80/htmldoc/Chapter9.html#pageant-cmdline-openssh https://the.earth.li/~sgtatham/putty/0.80/htmldoc/Chapter9.h... I do see that pageant.exe supports certificates, but the discussion is quite brief.

20 points by dangtony98 2y ago

karmicthreat 2y ago

This seems kind of like teleport without the logging and remote infrastructure.

dangtony98 2y ago
Definitely some overlap there for sure but with different architecture — this is also an extension of the broader Infisical platform that a lot of companies use for secrets management.

sepositus 2y ago

I was curious if Infisical would eventually turn into Vault (i.e., move beyond pure secret management), and this appears to be the first step in that direction. Granted, the onboarding/usage steps in the article are dramatically simplified compared to Vault. We used to use a method that was identical to this using Vault. Even wrote our own mini-CLI with a similar usage pattern. However, nowadays, we rely on Tailscale SSH (with a break-glass key) and have never really felt the need for an alternative.

dangtony98 2y ago
Co-founder of Infisical here! You're right in noticing that we're moving beyond secrets management into adjacent related verticals as well, specifically branching out more into the larger identity and access management space; secrets management continues to be the bread and butter of Infisical but the way we see it, it would be super nice in the long run to have one unified control plane over all related aspects spanning secrets management to certificate management, SSH access, and more. One of the core product philosophies of Infisical has always been to abstract away complexity as much as possible while still giving users the ability to customize the tool. Whereas Vault might, for example, have you explicitly create SSH certificate authorities and require 12+ steps to configure a working SSH certificate-based authentication model, Infisical SSH makes it so you only have to care about users and hosts that is who has access to what with something like 4 steps and if you want more power out of Infisical then that is always possible to build atop. Good to know regarding Tailscale SSH!

fisian 2y ago

It's just an ad for their SSH cert service... I feel like for SSH certs to expand beyond large companies, there's the need for an open-source service which does the issuing of short-lived certs after a user authenticates. I know smallstep, but their offer feels open-core/freemium.

dangtony98 2y ago
Hey totally agree with the open source aspect here in order for SSH certificates to reach broader adoption (coupled with seamless admin and user experience). Infisical SSH is actually an extension of the Infisical platform which is open source and used by a ton of companies for secrets management.

szszrk 2y ago

I would love to ask you lovely HN people, assuming you actually use SSH for work: - does you organization use ssh certs? - how big is the org? ("I know most IT", or "it's impossible to know the whole IT"?) - were you the ones that proposed and implemented that change? :) SSH certs make so much sense and I know about them for a long time. Yet never implemented that approach - we had at most 2 people that were actually interested. And now for me everything is http API's and oauth, so I don't need it any more. I wonder what are reasons orgs don't use ssh certs by default.

77pt77 2y ago
I have never met anyone who used certs with ssh, despite being the obvious choice. But since apparently ssh is now obsolete (as I was told here a week ago or so) maybe it makes no difference.
- szszrk 2y ago
  I will throw it in the bin with other obsolete tech from at least 40+ years ago. Like NFS. Oh, wait.
  - 77pt77 2y ago
    I'd say ssh is here to stay far more than NFS.
hylaride 2y ago
We implemented it at a previous company, but it was a managed services provider that had access to customer VM's or other infrastructure. It was part of the soc2 process to "approve, audit, and then grant timed access" to various infra. Everywhere else I worked just used ssh keys, either deployed via ansible or via LDAP because it's so much more simple unless there's a high need for an audit trail with proof of approval. SSH keys scale perfectly fine otherwise. Unless I'm desperate, I'll never work for a managed services company again, so hopefully I'll never need to bother.
dangtony98 2y ago

thinkingkong 2y ago

If you need easy distributed key management, modern SSH makes this fairly straightforward with some config values. It supports executing a program to get the SSH key at login time, dynamically. This way you can still maintain local certificates for fallback, and you can plug into anything. For example in the past I wrote a simple golang based app that loaded all of the SSH pubkeys from my organizations github, for users in a specific team.

ltbarcly3 2y ago

github uses ssh keys. I would wager github's auth volume is orders of magnitude bigger than anything infiscale has ever done or will ever do. therefore ssh keys do in fact scale.

szszrk 2y ago
To be fair, they do just part of that process. They don't generate keys at all (unless I'm mistaken?) and are vulnerable to that very issue discussed here. You can't easily create rules how to expire that access, or manage compromised keys.
- deleted 2y ago
  [deleted]
- ltbarcly3 2y ago
  I see what you mean, but this just trades one thing for another. On the one hand I could write a script that goes over every host and rsync's everyone's public key to it. You could have a cron job that does this. If you want to set up access control you would have sets of keys and different sets of keys would go to different servers. If you revoke a key you just need to remove it from the set and run whatever mechanism syncs the keys (a script that runs rsync under xargs would let you manage a few thousand servers pretty easily). Ok compare that to this solution: you still have to define sets of users or certificates. You still have to go edit the configuration on every server to put the correct list of certificates on there, so you are managing the lists just the same. If you find out something needs to be revoked you have to go do that. What's good: You don't have to go touch the servers in a loop except when changing the ACL setup for the servers. You can avoid having anything that can access the key setup for those servers (not really, you are just pretending this is true, but kindof). What's bad: You have to buy into their model. You have to go change how you do auth on all the servers. Users can't generate their own keys anymore, which is annoying. Some things that magically work with keys will be hard or impossible to configure with certificates, especially home grown stuff. Whenever inifiwhatever gets hacked you are in serious trouble.
  - 2y ago

abricq 2y ago

I'm actually really bad at all this networking certificates, and have a question for the experts. If a user have to sign a short-lived CA and then present that certificate to the host he wishes to connect with, isn't that basically allowing CA emitters to track the user's activity for this host ? This feels like replacing ssh for shh-with-tracking. Am i missing something ?

szszrk 2y ago
It's probably a typo, but a user rather uses short lived Cert, not a CA. CA (certificate authority) issues a Cert. CA doesn't have to be (and likely isn't) in users full control. Hosts answering to connections using that cert don't have to send anything back to CA. They just need rules "I trust this CA. If a user has a cert from it, and this precise combination of fields match, I consider this user's response trustworthy".
- dangtony98 2y ago
  That's correct — The user obtains a short-lived SSH certificate (signed/issued by a CA).

Bleibeidl 2y ago

... or use soemthing like FreeIPA

bananapub 2y ago

Why do people upvote this sort of lazy and shitty content marketing?

chasil 2y ago

The content of the article is OpenSSH-centric. Especially on Windows, this needs to include the PuTTY pageant.exe. I am on a corporate desktop where the agent service is not running: C:\>ssh-agent unable to start ssh-agent service, error :1058 Because of this, I use pageant.exe for the Microsoft OpenSSH binaries: https://the.earth.li/~sgtatham/putty/0.80/htmldoc/Chapter9.html#pageant-cmdline-openssh https://the.earth.li/~sgtatham/putty/0.80/htmldoc/Chapter9.h... I do see that pageant.exe supports certificates, but the discussion is quite brief.