17 ms·

How to properly manage SSH keys for server access

373 points by mpaepper 6y ago

mpaepper 6y ago
A while ago, I asked here: "Ask HN: What do you use for SSH key management of teams?" (https://news.ycombinator.com/item?id=24157180 https://news.ycombinator.com/item?id=24157180) And in this blog entry I am summarizing what I learned and what I think is a very good approach now.
- romanoderoma 6y ago
  Did you forget to add the link to the blog entry?
  - mpaepper 6y ago
    It's the one I submitted: https://www.paepper.com/blog/posts/how-to-properly-manage-ssh-keys-for-server-access/ https://www.paepper.com/blog/posts/how-to-properly-manage-ss...
    - romanoderoma 6y ago
      Oh sorry, I didn't realise you submitted it. Thanks!
jgilias 6y ago
Very nice! Thank you for breaking it down in such a succinct and straight to the point way, this is definitely something to bookmark.
cjsawyer 6y ago
Article content aside, I’m a big fan of the mobile site design.
znpy 6y ago
Once it reaches this level of complexity, one might consider setting up some kind of centralised authentication system. You can hook openssh to pull public keys off openldap, and possibly add some additional checks on it (like group membership). This way when a developer leaves the company, you just remove their entry in the LDAP DIT and all and every access is removed.
sgt 6y ago
In terms of the problem of removing SSH public keys from authorized_keys, etc we solve that problem by means of Ansible. If a developer leaves the company, he/she is removed and that Ansible task is applied to all the servers. Works fine too.
- LogicX 6y ago
  This. The author presents a different solution which is equally not scalable without automation and suggests you automate his new layer instead of just using ansible to automate handling of ssh keys.
  - leetrout 6y ago
    And certificate revocation lists (CRLs) are potentially more brittle in my experience, too. It’s pretty easy to audit a server for a key in a file or a key file on disk.
- b0afc375b5 6y ago
  Hi, I'm new at using ansible. This is the module you're referring to right? https://docs.ansible.com/ansible/latest/collections/ansible/posix/authorized_key_module.html https://docs.ansible.com/ansible/latest/collections/ansible/...
  - sgt 6y ago
    Actually we wrote our own, just leveraging built in modules within Ansible, but that one looks good. Give it a shot, and loop through a datasource e.g a variable.
mrintegrity 6y ago
Why not store the public key in a users ldap profile, then when they login ssh can pull that with the AuthorizedKeyCommand option to sshd config. As it's in AD / ldap you can allow them to manage their own public keys via a simple portal. When a user quits or is fired, their account is disabled and this will stop the key from being used.
- SahAssar 6y ago
  I'm not sure I'd call AD or it's portal simple...
  - marvion 6y ago
    If we're talking about internal servers, isn't an existing ldap/AD infrastructure so uncommon? There would be almost no additional work to implement this(depending on the size..) The discussed sining flow probably works better with cloud infrastructure. Afaik it's one of the ways hashicorps vault can be used for SSH.
    - SahAssar 6y ago
      I think that if you already have all users in a authz/authn system then anything will feel easy compared to the alternative, but I'd definitely not call them simple.
- marvion 6y ago
  Yep, pretty neat solution. This occasionally pops up at reddit and the "key in ldap" way feels surprisingly unknown/uncommon. Many use ansible.. but it requires guaranteed cleanup of revoked keys....
  - ViViDboarder 6y ago
    I’m using Ansible for this for my personal servers on a small scale, but revocation is pretty easy for me. I have all keys I want to distribute in my Playbook and I remove all authorized keys from the server and write only the ones in the playbook.
juangacovas 6y ago
I have team members that run on Windows, using private-public keys with Putty and Pageant for passphrases. When you read "put the third certificate" under ".ssh" directory, how would you do it on windows?
- DarthGhandi 6y ago
  %USERPROFILE%\.ssh\
  - juangacovas 6y ago
    Thanks, in my case that directory didn't exist until I lookup how to properly start ssh-agent under windows
- throwaway2048 6y ago
  Windows 10 has native openssh built in now, it has a lot more features than putty, and wont ever have long lag time for features like certificates.
misnome 6y ago
This is good. There's not much information around on this topic, and often roles seem only vaguely hinted at. The lab I work at recently had a "worry" during recent attacks on UK science infrastructure that someone might have compromised an unknown private key someone might have stored on a third part server. The institutional response was to switch to password-based ssh only for a month (so lots of people used sshpass...) and then revoke - at the bastion - every single key listed in every single authorized_keys for every one of thousands of users in the system, before turning key access on again. I've wondered why they didn't switch to an ssh certificate system. My best guess is the complication, maintenance, and risk profile of setting up a secure key-signing system to do this signing automatically - presumably this acts as a single point-of-failure for compromising the entire network.
- gnufx 6y ago
  I've not seen any rationale why sites haven't done something sensible. Sensible, like allowing automation without storing your password to use sshpass, for instance; then if you insist on MFA, that can be done when issuing an ephemeral certificate. "Certificate" might bring back bad memories of Globus et al, but that was different, and I haven't seen it articulated. The general response to those compromises has been a disaster area. Especially as (in the absence of any post mortem, as far as I know) the attack presumably involved the general password-spraying that was happening around then. Users were made to look responsible and suffer because systems had local privilege escalations due to poor management -- which, to be fair, might be due to relying on cluster vendors who "take security very seriously", as they say. That has been around the top of the threat list for decades too, it's just that it's now much larger scale. Then you got private keys purged that might be used to access systems on other sites with less risk than typing passwords on insecure systems -- arbitrarily deleting users' data.
- AA1B7B76 6y ago
  I have given up on OpenSSH's built-in certificate handling because it does not properly support revocation. It's not possible to have 100% of your keys on a short duration, so you have to cook up a revocation system. That's not my idea of a fun time. Auditing keys on disk is much less trouble than properly managing CRLs.
yakshaving_jgt 6y ago
Luckily for my team at least, we're using NixOS for everything so the `authorized_keys` are part of the system's configuration, which is all defined in code. This means we don't have the problem of any machine's configuration going out of sync with the rest.
- ryukafalz 6y ago
  Is the updated system configuration applied on all machines automatically? I haven’t used Nix personally but I’ve used Guix - I want to go further with it and I’m curious how you’re managing this.
  - yakshaving_jgt 6y ago
    Yes. We're using NixOps which does this by default, though of course it can be configured otherwise.
- neolog 6y ago
  What happens if a user manually places an ssh key in ~/.ssh/ ? Do you have a way to automatically remove those?
  - GekkePrutser 6y ago
    You can just modify the sshd_config not to look at the user authorized keys file
- mpaepper 6y ago
  What about having different access roles for certain people to certain servers? Is that covered in the NixOS approach as well? Sounds interesting!
larelli 6y ago
I think I understand how signing keys removes the need to update every server when adding a user to the system, but it seems like that comes at the price of having to update them all when someone leaves to revoke their certificates. What is the benefit of pulling revoked certificates to all servers periodically vs pulling authorized_keys files? Is it possible to work around this at all, e.g. conceive a system that eliminates all needs to push to servers? Is an online lookup like LDAP the way to go there?
- brianpan 6y ago
  If you use certs, then you also get Certificate revocation lists. You don't just trust the CA, you also trust the CA's CRL. https://en.wikipedia.org/wiki/Certificate_revocation_list https://en.wikipedia.org/wiki/Certificate_revocation_list
- cicloid 6y ago
  Certificates should have an expiration date, any system implemented with this pattern should expire after a couple of hours. If manual, I would even consider doing it every week with the caveat that it would be a large attack vector.
- mpaepper 6y ago
  The certificates have an expiration date, so if something goes wrong on that end at least it will expire after some time. Also, this takes care of role-based access - this you might not have with an authorized_keys file solution?
60Vhipx7b4JL 6y ago
With a ssh jump box you could also centrally manage access, revoke instantly and don't have to sign users keys repetitively..
- rantwasp 6y ago
  this. if you have access to all your machines through public internet you're doing it wrong (ie stop solving a problem that you should not have in the first place). The proper way to do this is to have a bastion host (or a jump host) where you strictly control access. Someone leaves you revoke their access. (extra: have the access be protected via 2FA that depends on the person having an active [LDAP/corp] account) There is another layer to this where the access to the bastion is allowed only from the corporate network (ie you need to VPN into the corpnet to be able to access the jump host). You leave, you no longer can access the bastion. the ultimate level to this is that you should not ssh willy nilly into your production hosts (to the degree that this should not even be possible). you should have a solution for pushing the logs + instrumentation (ie metrics) that makes it so that you don't need to do this in production.
altdatathrow 6y ago
We use authorized_keys and that file is managed by ansible across our servers. Adding/removing users consists of updating a yaml file (list of tuples {username, public key, status}) and running the playbook.
k__ 6y ago
When do you need to access a server via SSH?
Extigy 6y ago
For our users we use Kerberos authentication with AD, rather than SSH keys. Once a user has a TGT on their desktop, ticket forwarding takes care of SSH SSO from there. The same TGT can also be used for other neat things like secure and transparently automounted homes and other directories via NFS4 or CIFS.
- gnufx 6y ago
  Right, but people seem remarkably reluctant to use the facility which is just there, and it might even be proscribed for systems that aren't "joined to AD", for unexplained reasons. If ephemeral certificates are also used, your ticket can presumably cover getting them too. It's probably not an option for systems with off-site users, though, since sites won't expose their AD systems or put something in front of them.
- tha0x5 6y ago
  Kerberos (sssd-ad) backed authentication for SSH is really the best. You no longer have to deal with SSH keys whatsoever and all the management that goes with them: When users get their access revoked on AD, they get their SSH access revoked as well. You can have group based authorization (only those in the SRE group can access this class of QA endpoints), so when dozens of people a month are being added and removed from the various groups, you don't have to worry about giving them keys/access. They can SSO from their laptops, so all they have to do is open PuTTY and they can connect away without even typing their usernames and passwords. etc. Lots of these new generation "devops" and "full-stack developers" haven't had the experience of AD and Kerberos, so they spend all this time, blog posts, money, etc. to reinvent the wheel. Sad really.
  - solatic 6y ago
    That's great until you work for a company that bought Macs for everyone for their design and upper-management likes to keep it that way.
    - GekkePrutser 6y ago
      You can do it on Mac. I wouldn't recommend binding Macs anymore since Apple broke filevault for AD accounts in high Sierra (AD accounts don't get the secure token by default which is needed to unlock the drive) But since Catalina there's now a great Kerberos SSO plugin that you can push through MDM. Previously this was known as enterprise connect but only available from Apple professional services.
danbmil99 6y ago
Can't some Hacker News reader start a company that can do this right so I don't have to think about it? It would be really nice to have a solution to this problem that doesn't require the user to go to first principles involving Alice, Bob, and Charlie.
- tptacek 6y ago
  You mean, like: https://gravitational.com/teleport/ https://gravitational.com/teleport/
- ViViDboarder 6y ago
  Smallstep has done a few blog posts of doing this simply using their tools.
- different_sort 6y ago
  Hashicorp vault? You probably have more secrets than SSH to protect. If you’re on AWS also consider high value add built ins like EC2 instance connect or ssm session manager so you can manage host access via IAM.
- avanai 6y ago
  Something like https://www.okta.com/products/advanced-server-access/ https://www.okta.com/products/advanced-server-access/ ?
- paolomrg 6y ago
  Solved all my headaches and saved a lot of time managing my (and teammates) SSH keys with this service: https://authkeys.io/ https://authkeys.io/
  - mpaepper 6y ago
    Nice, looks like they pretty much automated what I described in my article as a service.
    - gizero76 6y ago
AbortedLaunch 6y ago
Suppose I grant a user access to system X for 2 weeks via a cert. When this user then requires 8 hours of access to system Y, can I just provide an additional cert with this claim and have the users ssh client figure it all out? Or does this scenario either require the user juggling certs, or me generating certs containing all concurrent claims?
- mpaepper 6y ago
  Great question, I haven't tested this, yet. One thing I'm sure would work if the user has generated two separate public-private key pairs and you sign two different certs. Not so sure about having several certs for the same key. If I was in that situation, I would probably generate a new cert which contains the concurrent claims and is short-lived, but we also don't have extremely many different roles.
readingnews 6y ago
Am I mistaken? The author notes in alternative solutions that the alternative solution is a single point of failure and that is a problem. Is the CA not a single point of failure also? I agree with one users comment, ansible can solve all of this too...
- amerine 6y ago
  How does ansible solve this?
  - readingnews 6y ago
    Playbooks to update all keys on all machines. Run one script and they are all updated.
- sigjuice 6y ago
  The CA is not a single point of failure in the traditional sense where a single machine going down has a large impact.
- mpaepper 6y ago
  Of course, if your CA is taken over, you are in big trouble. However, the attack vector here is much lower given you can have this on a local machine which is not even connected to a network if you like. The single point of failure for central management solutions is in having them as a running service - if that goes down, you are in trouble. And that is far more likely than your CA becoming compromised somehow. Also in practice, you can also run two or more CAs (we do that), so you can deprecate a full CA if the need arises.
daneel_w 6y ago
So the solution to the problem brought up by the author - having to regularly deploy a new authorized_keys file to servers - is to instead regularly deploy a new revocation document to servers. I don't see how this is any different or a solution to the primary problem discussed. It just adds the additional effort of having to sign and revoke certificates. We use a cronjob to regularly fetch, verify and deploy updated authorized_keys on servers.
- Xylakant 6y ago
  The primary difference is that you can have short-lived certificates, but ssh-keys are eternal. If you fail to remove one, it sticks around and may come back to bite you if compromised at a later point. Certificates expire, after which they’re useless to an attacker.
  - daper 6y ago
    One of nice options in recent versions of OpenSSH is the "expiry-time" key option you can put in authorized_keys. There are also other options handy to restrict the usage of the key, for example you can limit the key to be used only as a jump host. One can combine them so the users on the jump host will not be able to execute any command and even not able to edit authorizd_keys file and remove the "expiry-time" option.
- amerine 6y ago
  I think that you’re overreacting a tiny bit. One area that the OPs solution allows is rapid onboarding, quick/automatable generation for machine to machine roles and the ability to actually do some RBAC(if that matters to you). To make their system fail safe, they do the exact same thing you are except only making sure to synchronize the block-list. I don’t know about you, but at least at my org, waiting for tens of thousands of machines to sync an authorized keys file when doing a ton of onboarding or a very very selective off-boarding... I’d rather minimize the amount of machines I touch and doing that via an authorized_keys push via ansible is untenable at scale or volume.
  - 6y ago
lomereiter 6y ago
Related (IMO a somewhat better explanation of why): https://smallstep.com/blog/use-ssh-certificates/ https://smallstep.com/blog/use-ssh-certificates/ (note: you can also use Hashicorp Vault for the same)
different_sort 6y ago
I saw ssh keys and was ready to get on my moral high horse about ssh certs, only to read the article to be talked down. Nice guide!
regen 6y ago
Love the idea but revokation became the problem now
gabereiser 6y ago
>Every developer needs access to some servers for example to check the application logs. I fundamentally disagree with this. I’ve been writing software a long time and I used to demand I have server access so I can tail logs, creating the problem this article talks about resolving (good read btw). But I can’t help but wonder why we keep teaching this mindset. We have log analysis tools available on pretty much every cloud now. Docker has an aws log driver, a gcp log driver... etc. Backend developers should make a conscious decision on how to ship log events out of the box and into something searchable, indexable, and can derive metrics from. I ran a cloud infrastructure group that vehemently said “No” to any dev requesting ssh access. Not because they couldn’t, but because we don’t have access either. We created our platform without the ability to modify it. Infrastructure as code. Only way is to redeploy the stack. You’d be surprised at how much less stress there is when you can have alerts on log events from your application when things break instead of a support call or a support ticket. Proactive > Reactive debugging. I also understand not all shops are at that level of maturity. I’d love it if the community as a whole stopped teaching people to treat their app and server as a second home, as a pet, that must be nurtured. Obviously these are my opinions and the article itself addresses how to handle ssh keys for server access in a logical way, my only issue is why create that mess in the first place?
- mpaepper 6y ago
  Thank you, I agree that this should be avoided as much as possible in general. Actually, in my company, we are pushing the logs to an external service, so I guess that line I wrote there was not totally thought out. ;) However, it still might be useful for certain people to check servers for health or other metrics and you might not always have everything as perfect as you wish. I guess you already stated that yourself. So in general I'd agree. Good comment!
  - gabereiser 6y ago
    Health is a function of the application, not it’s host. A health check endpoint is paramount to ensuring the app is healthy. I still disagree with having _any_ access to a “box”. Local dev, console log, deployed debug? Better make sure you are logging events and not non-sense. Actionable events with request tracing (preferably). But yeah, it was a good article. Bitwarden is something I’ve used to share privileged keys before but the whole signing stuff was the right way to go. Also, if you aren’t on “cloud”, odds are you are still using something like Kubernetes or DC/OS or Swarm or the like. If you aren’t then well, wordpress sites aren’t really in the same ballpark technically. (Joke, Wordpress sites get traffic, some lots of traffic, I don’t discriminate against the PHP tribe).
teeray 6y ago
Anyone have a good solution for doing this with host keys? I’m not big on TOFU. It’d be nice if cloud-init would generate some keys and pass those off to a CA to sign (maybe vault?). That’s my rough idea, but I’m curious what others have come up with.
how_gauche 6y ago
It's even better if you force a hardware two-factor authentication before you grant the ssh cert.
ws66 6y ago
Openssh also supports OCSP, which can be used to more efficiently manage revoked certificates.
effnorwood 6y ago
Don’t
tbrock 6y ago
Why not just use something like Foxpass? It’s well worth the money!
atmosx 6y ago
I prefer AWS SSM or something like teleport which offloads user management to AWS or an Idp. The CA still needs management and special handling and policies if you want to be compliant.
- zokier 6y ago
  Annoyingly SSM still requires managing ssh keys and users somehow, it really only takes care of the network layer. I have been thinking of combining SSM with EC2 Instance Connect to deal with that issue, but haven't gotten around to actually implement that. Also SSM has some annoying glitches like e.g. https://github.com/aws/amazon-ssm-agent/issues/274 https://github.com/aws/amazon-ssm-agent/issues/274 Edit: found this blog post that shows how the two can work together https://skorfmann.com/blog/aws-ecs-instance-connect-meets-aws-session-manager/ https://skorfmann.com/blog/aws-ecs-instance-connect-meets-aw...
  - atmosx 6y ago
    > Annoyingly SSM still requires managing ssh keys and users somehow No, doesn't require managing ssh keys. SSM can work with SSH keys but it's not required. The downside I found is that all users connect with the same uid/gid which does not work great for bastion hosts where user profiles are required (e.g. connect to EKS API). You could work your way around this with documents permissions per user, but it's not great. The user management is handled by IAM.
    - zokier 6y ago
      To be more specific, ssh keys (and their management) are required for ssh access, something which I took as a given considering the thread. Sure, the rest of SSM is not reliant on ssh or ssh keys in any other way
forty 6y ago
Hashicorp Vault is able to issue short lived ssh certificate. It's pretty cool and you don't have to deal with revocations.
mh-cx 6y ago
Maybe I've missed it but in the first setup (without roles) where on the server is defined as which user someone can ssh into the machine now? There's only this global setting: TrustedUserCAKeys /etc/ssh/ca.pub Where can I define that a user can log in e.g. as "john" on a server? Is this the '-I USER_ID' part when creating the cert? If so that would mean that a certificate is bound to one username only and that every user needs his own account with exactly that name on every server he has access to, right?
binarysneaker 6y ago
"Every developer needs access to some servers for example to check the application logs" What year is this? 2010? No, application logs shouldn't reside on servers, and Devs shouldn't need access to servers either.
byteofbits 6y ago
I’ve seen a variety of solutions to this working on different products but by far the best I have worked with so far is OS Login from Google with their Identity Aware Proxy product. It’s allows developers to manage their own certificates (adding a new machine or rotating a key) whilst allowing us to use the GCP IAM tooling to grant access to certain machines - all without hosting bastion servers ourselves or exposing the servers themselves to the public internet. As it’s based on PAM its also been relatively painless to integrate with other functions like audit logging. If you’re using GCP already - I’d highly recommend it!
tomohawk 6y ago
Depending on your scale, this can be made much easier and simpler - just manage the keys using something like ansible. Got a new developer? Push out the key to the systems they need access to. Developer leaving? Remove the key from the systems. Configure the ssh server config to not use keys in home dirs.
deleted 6y ago
[deleted]
chomezski 6y ago
how does one manage new user creation dynamically? is the expectation here that there is a shared user that is preconfigured on the target host?

373 points by mpaepper 6y ago

A while ago, I asked here: "Ask HN: What do you use for SSH key management of teams?" (https://news.ycombinator.com/item?id=24157180 https://news.ycombinator.com/item?id=24157180) And in this blog entry I am summarizing what I learned and what I think is a very good approach now.

romanoderoma 6y ago
Did you forget to add the link to the blog entry?
- mpaepper 6y ago
  It's the one I submitted: https://www.paepper.com/blog/posts/how-to-properly-manage-ssh-keys-for-server-access/ https://www.paepper.com/blog/posts/how-to-properly-manage-ss...
  - romanoderoma 6y ago
    Oh sorry, I didn't realise you submitted it. Thanks!

jgilias 6y ago

Very nice! Thank you for breaking it down in such a succinct and straight to the point way, this is definitely something to bookmark.

cjsawyer 6y ago

Article content aside, I’m a big fan of the mobile site design.

znpy 6y ago

Once it reaches this level of complexity, one might consider setting up some kind of centralised authentication system. You can hook openssh to pull public keys off openldap, and possibly add some additional checks on it (like group membership). This way when a developer leaves the company, you just remove their entry in the LDAP DIT and all and every access is removed.

sgt 6y ago

In terms of the problem of removing SSH public keys from authorized_keys, etc we solve that problem by means of Ansible. If a developer leaves the company, he/she is removed and that Ansible task is applied to all the servers. Works fine too.

LogicX 6y ago
This. The author presents a different solution which is equally not scalable without automation and suggests you automate his new layer instead of just using ansible to automate handling of ssh keys.
- leetrout 6y ago
  And certificate revocation lists (CRLs) are potentially more brittle in my experience, too. It’s pretty easy to audit a server for a key in a file or a key file on disk.
b0afc375b5 6y ago
Hi, I'm new at using ansible. This is the module you're referring to right? https://docs.ansible.com/ansible/latest/collections/ansible/posix/authorized_key_module.html https://docs.ansible.com/ansible/latest/collections/ansible/...
- sgt 6y ago
  Actually we wrote our own, just leveraging built in modules within Ansible, but that one looks good. Give it a shot, and loop through a datasource e.g a variable.

mrintegrity 6y ago

Why not store the public key in a users ldap profile, then when they login ssh can pull that with the AuthorizedKeyCommand option to sshd config. As it's in AD / ldap you can allow them to manage their own public keys via a simple portal. When a user quits or is fired, their account is disabled and this will stop the key from being used.

SahAssar 6y ago
I'm not sure I'd call AD or it's portal simple...
- marvion 6y ago
  If we're talking about internal servers, isn't an existing ldap/AD infrastructure so uncommon? There would be almost no additional work to implement this(depending on the size..) The discussed sining flow probably works better with cloud infrastructure. Afaik it's one of the ways hashicorps vault can be used for SSH.
  - SahAssar 6y ago
    I think that if you already have all users in a authz/authn system then anything will feel easy compared to the alternative, but I'd definitely not call them simple.
marvion 6y ago
Yep, pretty neat solution. This occasionally pops up at reddit and the "key in ldap" way feels surprisingly unknown/uncommon. Many use ansible.. but it requires guaranteed cleanup of revoked keys....
- ViViDboarder 6y ago
  I’m using Ansible for this for my personal servers on a small scale, but revocation is pretty easy for me. I have all keys I want to distribute in my Playbook and I remove all authorized keys from the server and write only the ones in the playbook.

juangacovas 6y ago

I have team members that run on Windows, using private-public keys with Putty and Pageant for passphrases. When you read "put the third certificate" under ".ssh" directory, how would you do it on windows?

DarthGhandi 6y ago
%USERPROFILE%\.ssh\
- juangacovas 6y ago
  Thanks, in my case that directory didn't exist until I lookup how to properly start ssh-agent under windows
throwaway2048 6y ago
Windows 10 has native openssh built in now, it has a lot more features than putty, and wont ever have long lag time for features like certificates.

misnome 6y ago

This is good. There's not much information around on this topic, and often roles seem only vaguely hinted at. The lab I work at recently had a "worry" during recent attacks on UK science infrastructure that someone might have compromised an unknown private key someone might have stored on a third part server. The institutional response was to switch to password-based ssh only for a month (so lots of people used sshpass...) and then revoke - at the bastion - every single key listed in every single authorized_keys for every one of thousands of users in the system, before turning key access on again. I've wondered why they didn't switch to an ssh certificate system. My best guess is the complication, maintenance, and risk profile of setting up a secure key-signing system to do this signing automatically - presumably this acts as a single point-of-failure for compromising the entire network.

gnufx 6y ago
I've not seen any rationale why sites haven't done something sensible. Sensible, like allowing automation without storing your password to use sshpass, for instance; then if you insist on MFA, that can be done when issuing an ephemeral certificate. "Certificate" might bring back bad memories of Globus et al, but that was different, and I haven't seen it articulated. The general response to those compromises has been a disaster area. Especially as (in the absence of any post mortem, as far as I know) the attack presumably involved the general password-spraying that was happening around then. Users were made to look responsible and suffer because systems had local privilege escalations due to poor management -- which, to be fair, might be due to relying on cluster vendors who "take security very seriously", as they say. That has been around the top of the threat list for decades too, it's just that it's now much larger scale. Then you got private keys purged that might be used to access systems on other sites with less risk than typing passwords on insecure systems -- arbitrarily deleting users' data.
AA1B7B76 6y ago
I have given up on OpenSSH's built-in certificate handling because it does not properly support revocation. It's not possible to have 100% of your keys on a short duration, so you have to cook up a revocation system. That's not my idea of a fun time. Auditing keys on disk is much less trouble than properly managing CRLs.

yakshaving_jgt 6y ago

Luckily for my team at least, we're using NixOS for everything so the `authorized_keys` are part of the system's configuration, which is all defined in code. This means we don't have the problem of any machine's configuration going out of sync with the rest.

ryukafalz 6y ago
Is the updated system configuration applied on all machines automatically? I haven’t used Nix personally but I’ve used Guix - I want to go further with it and I’m curious how you’re managing this.
- yakshaving_jgt 6y ago
  Yes. We're using NixOps which does this by default, though of course it can be configured otherwise.
neolog 6y ago
What happens if a user manually places an ssh key in ~/.ssh/ ? Do you have a way to automatically remove those?
- GekkePrutser 6y ago
  You can just modify the sshd_config not to look at the user authorized keys file
mpaepper 6y ago
What about having different access roles for certain people to certain servers? Is that covered in the NixOS approach as well? Sounds interesting!

larelli 6y ago

I think I understand how signing keys removes the need to update every server when adding a user to the system, but it seems like that comes at the price of having to update them all when someone leaves to revoke their certificates. What is the benefit of pulling revoked certificates to all servers periodically vs pulling authorized_keys files? Is it possible to work around this at all, e.g. conceive a system that eliminates all needs to push to servers? Is an online lookup like LDAP the way to go there?

brianpan 6y ago
If you use certs, then you also get Certificate revocation lists. You don't just trust the CA, you also trust the CA's CRL. https://en.wikipedia.org/wiki/Certificate_revocation_list https://en.wikipedia.org/wiki/Certificate_revocation_list
cicloid 6y ago
Certificates should have an expiration date, any system implemented with this pattern should expire after a couple of hours. If manual, I would even consider doing it every week with the caveat that it would be a large attack vector.
mpaepper 6y ago
The certificates have an expiration date, so if something goes wrong on that end at least it will expire after some time. Also, this takes care of role-based access - this you might not have with an authorized_keys file solution?

60Vhipx7b4JL 6y ago

With a ssh jump box you could also centrally manage access, revoke instantly and don't have to sign users keys repetitively..

rantwasp 6y ago
this. if you have access to all your machines through public internet you're doing it wrong (ie stop solving a problem that you should not have in the first place). The proper way to do this is to have a bastion host (or a jump host) where you strictly control access. Someone leaves you revoke their access. (extra: have the access be protected via 2FA that depends on the person having an active [LDAP/corp] account) There is another layer to this where the access to the bastion is allowed only from the corporate network (ie you need to VPN into the corpnet to be able to access the jump host). You leave, you no longer can access the bastion. the ultimate level to this is that you should not ssh willy nilly into your production hosts (to the degree that this should not even be possible). you should have a solution for pushing the logs + instrumentation (ie metrics) that makes it so that you don't need to do this in production.

altdatathrow 6y ago

We use authorized_keys and that file is managed by ansible across our servers. Adding/removing users consists of updating a yaml file (list of tuples {username, public key, status}) and running the playbook.

k__ 6y ago

When do you need to access a server via SSH?

Extigy 6y ago

For our users we use Kerberos authentication with AD, rather than SSH keys. Once a user has a TGT on their desktop, ticket forwarding takes care of SSH SSO from there. The same TGT can also be used for other neat things like secure and transparently automounted homes and other directories via NFS4 or CIFS.

gnufx 6y ago
Right, but people seem remarkably reluctant to use the facility which is just there, and it might even be proscribed for systems that aren't "joined to AD", for unexplained reasons. If ephemeral certificates are also used, your ticket can presumably cover getting them too. It's probably not an option for systems with off-site users, though, since sites won't expose their AD systems or put something in front of them.
tha0x5 6y ago
Kerberos (sssd-ad) backed authentication for SSH is really the best. You no longer have to deal with SSH keys whatsoever and all the management that goes with them: When users get their access revoked on AD, they get their SSH access revoked as well. You can have group based authorization (only those in the SRE group can access this class of QA endpoints), so when dozens of people a month are being added and removed from the various groups, you don't have to worry about giving them keys/access. They can SSO from their laptops, so all they have to do is open PuTTY and they can connect away without even typing their usernames and passwords. etc. Lots of these new generation "devops" and "full-stack developers" haven't had the experience of AD and Kerberos, so they spend all this time, blog posts, money, etc. to reinvent the wheel. Sad really.
- solatic 6y ago
  That's great until you work for a company that bought Macs for everyone for their design and upper-management likes to keep it that way.
  - GekkePrutser 6y ago
    You can do it on Mac. I wouldn't recommend binding Macs anymore since Apple broke filevault for AD accounts in high Sierra (AD accounts don't get the secure token by default which is needed to unlock the drive) But since Catalina there's now a great Kerberos SSO plugin that you can push through MDM. Previously this was known as enterprise connect but only available from Apple professional services.

danbmil99 6y ago

Can't some Hacker News reader start a company that can do this right so I don't have to think about it? It would be really nice to have a solution to this problem that doesn't require the user to go to first principles involving Alice, Bob, and Charlie.

tptacek 6y ago
You mean, like: https://gravitational.com/teleport/ https://gravitational.com/teleport/
ViViDboarder 6y ago
Smallstep has done a few blog posts of doing this simply using their tools.
different_sort 6y ago
Hashicorp vault? You probably have more secrets than SSH to protect. If you’re on AWS also consider high value add built ins like EC2 instance connect or ssm session manager so you can manage host access via IAM.
avanai 6y ago
Something like https://www.okta.com/products/advanced-server-access/ https://www.okta.com/products/advanced-server-access/ ?
paolomrg 6y ago
Solved all my headaches and saved a lot of time managing my (and teammates) SSH keys with this service: https://authkeys.io/ https://authkeys.io/
- mpaepper 6y ago
  Nice, looks like they pretty much automated what I described in my article as a service.
  - gizero76 6y ago

AbortedLaunch 6y ago

Suppose I grant a user access to system X for 2 weeks via a cert. When this user then requires 8 hours of access to system Y, can I just provide an additional cert with this claim and have the users ssh client figure it all out? Or does this scenario either require the user juggling certs, or me generating certs containing all concurrent claims?

mpaepper 6y ago
Great question, I haven't tested this, yet. One thing I'm sure would work if the user has generated two separate public-private key pairs and you sign two different certs. Not so sure about having several certs for the same key. If I was in that situation, I would probably generate a new cert which contains the concurrent claims and is short-lived, but we also don't have extremely many different roles.

readingnews 6y ago

Am I mistaken? The author notes in alternative solutions that the alternative solution is a single point of failure and that is a problem. Is the CA not a single point of failure also? I agree with one users comment, ansible can solve all of this too...

amerine 6y ago
How does ansible solve this?
- readingnews 6y ago
  Playbooks to update all keys on all machines. Run one script and they are all updated.
sigjuice 6y ago
The CA is not a single point of failure in the traditional sense where a single machine going down has a large impact.
mpaepper 6y ago
Of course, if your CA is taken over, you are in big trouble. However, the attack vector here is much lower given you can have this on a local machine which is not even connected to a network if you like. The single point of failure for central management solutions is in having them as a running service - if that goes down, you are in trouble. And that is far more likely than your CA becoming compromised somehow. Also in practice, you can also run two or more CAs (we do that), so you can deprecate a full CA if the need arises.

daneel_w 6y ago

So the solution to the problem brought up by the author - having to regularly deploy a new authorized_keys file to servers - is to instead regularly deploy a new revocation document to servers. I don't see how this is any different or a solution to the primary problem discussed. It just adds the additional effort of having to sign and revoke certificates. We use a cronjob to regularly fetch, verify and deploy updated authorized_keys on servers.

Xylakant 6y ago
The primary difference is that you can have short-lived certificates, but ssh-keys are eternal. If you fail to remove one, it sticks around and may come back to bite you if compromised at a later point. Certificates expire, after which they’re useless to an attacker.
- daper 6y ago
  One of nice options in recent versions of OpenSSH is the "expiry-time" key option you can put in authorized_keys. There are also other options handy to restrict the usage of the key, for example you can limit the key to be used only as a jump host. One can combine them so the users on the jump host will not be able to execute any command and even not able to edit authorizd_keys file and remove the "expiry-time" option.
amerine 6y ago
I think that you’re overreacting a tiny bit. One area that the OPs solution allows is rapid onboarding, quick/automatable generation for machine to machine roles and the ability to actually do some RBAC(if that matters to you). To make their system fail safe, they do the exact same thing you are except only making sure to synchronize the block-list. I don’t know about you, but at least at my org, waiting for tens of thousands of machines to sync an authorized keys file when doing a ton of onboarding or a very very selective off-boarding... I’d rather minimize the amount of machines I touch and doing that via an authorized_keys push via ansible is untenable at scale or volume.
- 6y ago

lomereiter 6y ago

Related (IMO a somewhat better explanation of why): https://smallstep.com/blog/use-ssh-certificates/ https://smallstep.com/blog/use-ssh-certificates/ (note: you can also use Hashicorp Vault for the same)

different_sort 6y ago

I saw ssh keys and was ready to get on my moral high horse about ssh certs, only to read the article to be talked down. Nice guide!

regen 6y ago

Love the idea but revokation became the problem now

gabereiser 6y ago

>Every developer needs access to some servers for example to check the application logs. I fundamentally disagree with this. I’ve been writing software a long time and I used to demand I have server access so I can tail logs, creating the problem this article talks about resolving (good read btw). But I can’t help but wonder why we keep teaching this mindset. We have log analysis tools available on pretty much every cloud now. Docker has an aws log driver, a gcp log driver... etc. Backend developers should make a conscious decision on how to ship log events out of the box and into something searchable, indexable, and can derive metrics from. I ran a cloud infrastructure group that vehemently said “No” to any dev requesting ssh access. Not because they couldn’t, but because we don’t have access either. We created our platform without the ability to modify it. Infrastructure as code. Only way is to redeploy the stack. You’d be surprised at how much less stress there is when you can have alerts on log events from your application when things break instead of a support call or a support ticket. Proactive > Reactive debugging. I also understand not all shops are at that level of maturity. I’d love it if the community as a whole stopped teaching people to treat their app and server as a second home, as a pet, that must be nurtured. Obviously these are my opinions and the article itself addresses how to handle ssh keys for server access in a logical way, my only issue is why create that mess in the first place?

mpaepper 6y ago
Thank you, I agree that this should be avoided as much as possible in general. Actually, in my company, we are pushing the logs to an external service, so I guess that line I wrote there was not totally thought out. ;) However, it still might be useful for certain people to check servers for health or other metrics and you might not always have everything as perfect as you wish. I guess you already stated that yourself. So in general I'd agree. Good comment!
- gabereiser 6y ago
  Health is a function of the application, not it’s host. A health check endpoint is paramount to ensuring the app is healthy. I still disagree with having _any_ access to a “box”. Local dev, console log, deployed debug? Better make sure you are logging events and not non-sense. Actionable events with request tracing (preferably). But yeah, it was a good article. Bitwarden is something I’ve used to share privileged keys before but the whole signing stuff was the right way to go. Also, if you aren’t on “cloud”, odds are you are still using something like Kubernetes or DC/OS or Swarm or the like. If you aren’t then well, wordpress sites aren’t really in the same ballpark technically. (Joke, Wordpress sites get traffic, some lots of traffic, I don’t discriminate against the PHP tribe).

teeray 6y ago

Anyone have a good solution for doing this with host keys? I’m not big on TOFU. It’d be nice if cloud-init would generate some keys and pass those off to a CA to sign (maybe vault?). That’s my rough idea, but I’m curious what others have come up with.

how_gauche 6y ago

It's even better if you force a hardware two-factor authentication before you grant the ssh cert.

ws66 6y ago

Openssh also supports OCSP, which can be used to more efficiently manage revoked certificates.

effnorwood 6y ago

Don’t

tbrock 6y ago

Why not just use something like Foxpass? It’s well worth the money!

atmosx 6y ago

I prefer AWS SSM or something like teleport which offloads user management to AWS or an Idp. The CA still needs management and special handling and policies if you want to be compliant.

zokier 6y ago
Annoyingly SSM still requires managing ssh keys and users somehow, it really only takes care of the network layer. I have been thinking of combining SSM with EC2 Instance Connect to deal with that issue, but haven't gotten around to actually implement that. Also SSM has some annoying glitches like e.g. https://github.com/aws/amazon-ssm-agent/issues/274 https://github.com/aws/amazon-ssm-agent/issues/274 Edit: found this blog post that shows how the two can work together https://skorfmann.com/blog/aws-ecs-instance-connect-meets-aws-session-manager/ https://skorfmann.com/blog/aws-ecs-instance-connect-meets-aw...
- atmosx 6y ago
  > Annoyingly SSM still requires managing ssh keys and users somehow No, doesn't require managing ssh keys. SSM can work with SSH keys but it's not required. The downside I found is that all users connect with the same uid/gid which does not work great for bastion hosts where user profiles are required (e.g. connect to EKS API). You could work your way around this with documents permissions per user, but it's not great. The user management is handled by IAM.
  - zokier 6y ago
    To be more specific, ssh keys (and their management) are required for ssh access, something which I took as a given considering the thread. Sure, the rest of SSM is not reliant on ssh or ssh keys in any other way

forty 6y ago

Hashicorp Vault is able to issue short lived ssh certificate. It's pretty cool and you don't have to deal with revocations.

mh-cx 6y ago

Maybe I've missed it but in the first setup (without roles) where on the server is defined as which user someone can ssh into the machine now? There's only this global setting: TrustedUserCAKeys /etc/ssh/ca.pub Where can I define that a user can log in e.g. as "john" on a server? Is this the '-I USER_ID' part when creating the cert? If so that would mean that a certificate is bound to one username only and that every user needs his own account with exactly that name on every server he has access to, right?

binarysneaker 6y ago

"Every developer needs access to some servers for example to check the application logs" What year is this? 2010? No, application logs shouldn't reside on servers, and Devs shouldn't need access to servers either.

byteofbits 6y ago

I’ve seen a variety of solutions to this working on different products but by far the best I have worked with so far is OS Login from Google with their Identity Aware Proxy product. It’s allows developers to manage their own certificates (adding a new machine or rotating a key) whilst allowing us to use the GCP IAM tooling to grant access to certain machines - all without hosting bastion servers ourselves or exposing the servers themselves to the public internet. As it’s based on PAM its also been relatively painless to integrate with other functions like audit logging. If you’re using GCP already - I’d highly recommend it!

tomohawk 6y ago

Depending on your scale, this can be made much easier and simpler - just manage the keys using something like ansible. Got a new developer? Push out the key to the systems they need access to. Developer leaving? Remove the key from the systems. Configure the ssh server config to not use keys in home dirs.

deleted 6y ago

[deleted]

chomezski 6y ago

how does one manage new user creation dynamically? is the expectation here that there is a shared user that is preconfigured on the target host?