13 ms·

Poll HN: Do you use SSH certificates (not mere public-key authentication)?

142 points by tonyg 4y ago

Do you use SSH certificates for work and/or personal SSH logins? I have just switched, for my personal machines, but dealing with the CA keys is a royal pain. What on earth do people to do keep their CA keys accessible but secure? (You can choose multiple options below)

lkuty 4y ago
Infos about how to use them would be great.
- jeroenhd 4y ago
  I find this quick introduction to be quite helpful if you want to learn the basics: https://jameshfisher.com/2018/03/16/how-to-create-an-ssh-certificate-authority/ https://jameshfisher.com/2018/03/16/how-to-create-an-ssh-cer... If you don't know about them, you can probably answer "no" in this poll :)
- traceroute66 4y ago
  See the link I just posted in my reply, i.e. https://smallstep.com/blog/diy-single-sign-on-for-ssh/ https://smallstep.com/blog/diy-single-sign-on-for-ssh/
  - jeroenhd 4y ago
    Edit: parent comment got corrected. Old comment left below for reference. Note that this is a blog about X509 certificates, not SSH certificates! That blog page does link to https://smallstep.com/blog/diy-single-sign-on-for-ssh/ https://smallstep.com/blog/diy-single-sign-on-for-ssh/ which is about SSH certificates.
    - traceroute66 4y ago
      @jeroenhd Mea-culpa, had a choice of two smallstep bookmarks and I copy/pasted the wrong one ! Now corrected.
traceroute66 4y ago
I've used public-key basically forever. I have wanted to go that step further and implement certificates for a good few years now, I absolutely see the value, no question about it (i.e. no more out of date authorized_keys files, no need to even install authorized_keys files in the first place, the list goes on). I wish YubiKeys were more flexible in allowing SSH certificates to be hosted on them (currently can only host X509 type). Because if I could use a Yubi for SSH certificate signing I would have already been using certificates ! (Yes I know, theoretically you can do it with GPG keys, but frankly that's a PITA with many issues). My current plan is to take inspiration from this : https://smallstep.com/blog/diy-single-sign-on-for-ssh/ https://smallstep.com/blog/diy-single-sign-on-for-ssh/
- jeroenhd 4y ago
  You can use this great-but-also-terrible system to do X509 certificate validation with SSH: https://github.com/mjg59/ssh_pki https://github.com/mjg59/ssh_pki Maybe after some deeper security analysis this idea could help you work around the limitations of Yubikey's hardware. However, if the author of a piece of security software tells me something is a bad idea, I tend to accept that, especially if that author is mjg59.
  - traceroute66 4y ago
    @jeroenhd Mea-culpa, had a choice of two smallstep bookmarks and I copy/pasted the wrong one ! Now corrected.
- XiS 4y ago
  As an alternative, you might consider using an ed25519-sk ssh key so you can use your YubiKey as a second factor during authentication. I've got my keys loaded up in the SSH agent by KeepassXC when I open my database and can log in to a box just by pressing my YubiKey. No keys on disk, no passwords to remember (except your Keepass master key ofcourse).
  - tomaslaureano 4y ago
    Would love to know more about this workflow!
ultra_nick 4y ago
What's the difference between a certificate and a public/private keys?
- tfigment 4y ago
  You add the intermediate key to config for trust. You do not need authorized_keys files for each user. Good for team and central management and faster key rotation but then needs trusted intermediate authority which is secure. No password or key rotation on machine itself which is good if you have a lot of machines.
- sofixa 4y ago
  Certificate is short lived and signed on the fly for when you need it. The server trusts your SSH CA, and you can ask it to sign any private key you want, with a ttl. So you get extra security (credentials are dynamic and short -lived, and also per-user if needed) and lower maintenance because you only have the SSH CA public key to setup on all machines and that one needs less rotation/updates than public keys.
  - hawski 4y ago
    But then don't you need to maintain this CA certificate? I guess it makes sense for wider centralized access.
  - thayne 4y ago
    A certificate doesn't necessarily have to be short lived, although in practice they usually are. The main difference is that with an ssh key, you install the public key directly on the target host. With a certificate, you Just tell the host to trust a CA, and use that CA to sign client certificates.
  - goodpoint 4y ago
    > lower maintenance because you only have the SSH CA public key to setup on all machines This is hardly "lower maintenance" than pushing updates to the authorized_keys across servers. Or having servers poll for such file.
jeroenhd 4y ago
I've set them up for both identification and authorization with most of my virtual machines. I've set up two CAs (one for server identification, one for client identification) and provision virtual machines to trust the client authorization certificate by default. I still need to work something out for setting up server identification automatically but I haven't had the time (or interest) to find out how to set proper random hostnames through cloudinit scripts. I think the certificate system is especially great for certificate login from multiple devices. At some point I was just constantly copying/updating authorized_keys files and with SSH certificates I can just provision my clients once and the problem is solved, assuming I use the same username everywhere. On the server side it's kind of annoying to set up right because there's no easy process like ACME for SSH certificates. Depending on how much effort I want to put into (temporary) virtual machines I'll either TOFU the host certificate or go through the effort to properly sign it if I have easy access to the CA key from the device I'm working on.
lstoll 4y ago
We use an AWS KMS asymmetric key for the CA keys, they're cheap and avoids exposing the private key material in an any way. For signing SSH certificates, we run a small service (prototype code dump at https://github.com/pardot/sshsigner https://github.com/pardot/sshsigner) that uses this key to sign short lived certificates. Auth to the service is via OIDC issued ID tokens. On the client side we have a custom SSH agent that uses an ephemeral in-memory private key. The agent manages the OIDC web flow and calling out to the service for signing on demand. This lets us keep the cert duration small and scoped, and allows us to force re-auth for sudo etc. via the web flow. We also do a similar thing for host keys, IAM auth the instances and sign certificates. Altogether works well, provides a nice user experience, and keeps long-lived/leakable creds out of out environment.
yrro 4y ago
I use Kerberos (KDC hands out TGT only after MFA)
jerf 4y ago
We looked into them once at work for our "conventional" SSH infrastructure, but immediately rammed into the wall that a certificate can sign a key, but you can't have a full PKI infrastructure like SSL where certs can sign certs that can sign keys. Without the ability to delegate, the logistics of needing to create an absolutely centralized authority to do the useful things we wanted to do within the organization raised the cost of "first value" too high for us to fuss over. So for large-ish organizations (~1000 affected employees) we had a hard time justifying any effort put into it versus the established infrastructures for key management that already exist. On the other side, as a personal user of SSH with basically one person to worry about, the effort of setting up a certificate seems like just a waste versus the existing key-based infrastructure; I don't understand at all what attack it would prevent or what convenience it would provide for the cost of learning it. A general principle is you get good at what you do a lot, and you aren't good at what you do rarely. Key-based management has its issues, but people have been doing it for a while and they're good at it. Not necessarily "great" and not necessarily "unable to be improved", but they're good at it. People understand it and know how to use it. Nobody uses certificates, so nobody understands how they work or what to do with them, and they have the weakness I cited above. So, even though in a perfect world certificates may be superior, in the world we live in now, the competition between the existing polished infrastructure for pure key-based management is pretty good, even if you think it's a polished turd, and the infrastructure for SSH certs is pretty bad, even if you think it's an uncut diamond in the rough. It's difficult to look at that situation and honestly choose to proceed with the certificates. As others point out, the cloud services are proceeding in other directions as well which will probably continue to make using SSH certificates directly a very difficult choice to practically justify.
- gunapologist99 4y ago
  I agree. We use a distributed Userify infrastructure to take some of the sting out of key management for our teams; this gives us the ability to tie user authentication into our enterprise directory but still delegate an entire Userify infrastructure on a BU basis, which is pretty awesome. To your point about clouds making it more difficult in the future, it's becoming more challenging to scale the CA across auto scaling groups with dynamic IPs and where you can't control TXT records. We've found that SSH keys are very flexible, and we simply needed a centralized management point (which we have now), but with decentralized operation in case anything is offline along the way. Also, it's very important to us to be able to kill active sessions globally when a user is removed, and there doesn't seem to be any other tools that do that except Userify.
xaduha 4y ago
I recently got SmartCard HSM (also available as Nitrokey HSM 1 and 2 I think), but didn't get around to setting it up yet. For personal use it does seem like an overkill, when FIDO2 security keys are much more convenient.
derefr 4y ago
If you use Google Cloud’s “OS Login” feature, you’re using (semi-ephemeral) SSH certificates.
dotwaffle 4y ago
I store my authorized_keys in DNS TXT records, that are DNSSEC signed, with a validating resolver on the box. I then just use "/usr/bin/hesinfo %u ssh" as my AuthorizedKeysCommand in OpenSSH. I wrote a little tool that allowed you to "#include" other DNS records etc, but "hesinfo" is generally easily installable/available so it's just easier.
- aliqot 4y ago
  Is there some way I can purchase you a pint?
  - dotwaffle 4y ago
    It would be rude of me to decline such an offer ;)
- bshep 4y ago
  I would love a HOWTO link. Also, can the DNS server be public facing? Any issues with the authorized keys being public (AFAICT there isnt but i am not a security expert)?
  - theblazehen 4y ago
    There are none. In fact yours is already public: https://github.com/bshep.keys https://github.com/bshep.keys
    - bombcar 4y ago
      There is technically one minor one, which really isn't one - but you should be aware of. Someone can take your authorized keys and add them to a box they control, and trick you into logging in. However, this would trigger the "new host" warning SSH gives you, and you can minimize this by minimizing which hosts you allow your private keys to be used on. And if someone is so actively trying to attack you they probably have more direct methods available.
nsheridan 4y ago
I use them for personal machines, and have deployed them in the past in work environments. A few years ago I wrote a CA which can exchange oauth tokens for signed keys: https://github.com/nsheridan/cashier https://github.com/nsheridan/cashier Auth is handled in a browser e.g. by Google, and the CA will sign a key and return a cert with a valid token.
deleted 4y ago
[deleted]
GekkePrutser 4y ago
I use private keys but store them on a Yubikey. In OpenPGP format, not using Fido2 at present. I use OpenPGP for other stuff too like my password manager so I'm not ready to move to fido. But certificates - no. I don't see the need for this in my home environment. In my opinion it only makes sense if you have a PKI, in which case you already have the whole CA scenario covered.
baybal2 4y ago
blueflow 4y ago
I use certificates to deal with the n clients -> m servers i have privately. On some machines i have AuthorizedPrincipalsFile include my username, so i can directly login as root there. I found these SSH features while studying the ssh-keygen(1) man page. I don't use extra tools aside what SSH ships.
dingosity 4y ago
I did for a while, then didn't because neither my chromebook nor my android devices supported them. Then they did and now I do again.
Joel_Mckay 4y ago
Hypothetically, SSH over SSL/TLS or an iodine tunnel along with a few IDS tripwires. Every approach has strengths and weaknesses, the password-less method is convenient... but is the number one target for worms/bots. =)
Ottolay 4y ago
I use public/private with the private key handled by Yubikey PIV functionality.
vngzs 4y ago
Doing this without something like Vault is very difficult. Vault is designed to provide similar guarantees to an HSM (it's a service designed to make keys usable but not exfiltratable), and that's essential for any long-lived certificates.
- tptacek 4y ago
  I don't know that I disagree with the broader point here, but long-lived certificates are what you don't want.
  - archi42 4y ago
    Not GP, but I'd assume they talked about the root cert? You don't want to roll out a new one every other week ;-)
    - vngzs 4y ago
      Yeah, the root is long-lived. The ones you issue to SSH to a machine are short-lived, because they're not stored in something like Vault.
- goodpoint 4y ago
  Vault is not able to provide guarantees similar to an HSM.
  - vngzs 4y ago
    You're right - an adversary with physical access to machines running Vault can steal keys from it. An HSM is designed to survive at least temporary physical access by an adversary. And if you have management infrastructure that lets admins access the servers running Vault, then they can exfiltrate the keys (an HSM can be configured to prevent this). There exist threats that HSMs will stop that Vault simply won't stop. But they are both systems intended to provide secure storage of key material with policy-based access to that key material. As an SSH CA, you can configure Vault to sign SSH pubkeys but never divulge the key material. Depending on your threat model, it might get you what you want for this use-case, but you should definitely be aware of the limitations of Vault's security model.
ninefathom 4y ago
Bearing in mind that, on principle, I never expose ssh to untrusted hosts/networks, for many years my solution for ssh-SSO was GSSAPI. Typically this would be a two part auth, where gssapi-kex had to succeed, _and_ the connecting user had to supply the kerberos password on the server side via PAM/keyboard-interactive to reduce the risk of "sysadmin walked away and left screen unlocked." I think this approach has fallen out of favour these days compared to PKI, but I haven't been an enterprise sysadmin in a few years. Edit: for those curious, ephemeral access in this case is easily enough managed via e.g. the usual AD-on-*nix methods. It doesn't scale beyond a few dozen systems, but the scenario described above was back before scale-out became the fashion.
StewardMcOy 4y ago
I don't see the benefit of using certificates for personal use. In my mind, certificates only really shine when administering a fleet of servers in a large company, as it eliminates the need to update the authorized_keys file on every server someone is hired or leaves. I could be missing some less obvious use cases, but if you're just authenticating to your personal machines, it doesn't seem worth it. It's not any more secure than SSH keys, and you have to also manage and secure your CA key.
- stormbrew 4y ago
  I think this is true if you only really have one or two computers you use ssh from regularly but if you have a few or you get new ones frequently you can get back to that experience of not having to frequently generate and distribute public keys by using a yubikey to sign local keys and then your authorized_keys only needs the cert authorization in it.
  - NjU24E48YqdBGnG 4y ago
    I just store my SSH key in my Yubikey and then I can SSH from any computer that has SSH and GPG installed. Gets rid of the "generate and sign a key that then sits on that computer that might not be fully trusted" step...
    - zhfliz 4y ago
      not as flexible, as it has more dependencies, but you can make it work without gpg as well: https://developers.yubico.com/yubico-piv-tool/YKCS11/ https://developers.yubico.com/yubico-piv-tool/YKCS11/
      - NjU24E48YqdBGnG 4y ago
        Yes, I could. In fact I actually also use the PIV functionality, just not for SSH. The GPG answer seems simpler, at least for my use-case, since it really does just require SSH and GPG (okay, and GPG configured to `enable-ssh-support`). Also, I was actually using a GPG-as-SSH-key before I started hosting it on my Yubikey. Sadly, the Windows GPG build and Win32-OpenSSH don't talk to the agent the same way, so it doesn't work there. (AFAIR Windows GPG - and Putty - uses "Assuan" while Win32-OpenSSH uses named pipes just like *nix SSH/GPG do.) I don't think PIV would help that though :(
dmuth 4y ago
For those curious about how SSH CAs and key signing works, I wrote a blog post about that awhile ago: https://dmuth.medium.com/ssh-at-scale-cas-and-principals-b27edca3a5d https://dmuth.medium.com/ssh-at-scale-cas-and-principals-b27... I also created a "playground" which can be used to stand up such an environment running in Docker containers to better understand the process: https://github.com/dmuth/ssh-principal-and-ca-playground https://github.com/dmuth/ssh-principal-and-ca-playground Hope folks here find them helpful.
- LionTamer 4y ago
  That that looks rly helpful. I’ll check that out tonight, thanks for sharing
rkangel 4y ago
My hope is to make this problem mostly redundant among my machines through use of tailscale. Tailscale SSH means that I don't have to worry about SSH keys at all - it just uses the same auth as the rest of tailscale. Only downside is that at the moment it doesn't play with SELinux which is a bit unhelpful for me given I mostly use Fedora.
- mdeeks 4y ago
  For personal use cases this the the best option by far IMHO. At work we deployed Tailscale company wide but we almost entirely use subnet routers and don't have Tailscale on each node. So it only gets you as far as an SSH bastion. We instead rely on standard SSH keys to reach individual nodes and another third party service that dynamically provisions accounts and authorized_keys on machines when you try to login. I would kill for something as simple and clean as Tailscale SSH for all of our nodes at work. Especially since it has ACLs and optional "check mode" which forces re-auth/2FA for high risk nodes.
rkeene2 4y ago
I use X.509 certificates for SSH (via PKIXSSH [0])... but I'm not sure where that goes here [0] https://roumenpetrov.info/secsh/ https://roumenpetrov.info/secsh/
zie 4y ago
No, because I needed to interoperate with Windows and at the time it didn't handle SSH CA certs. Maybe that's fixed now? Anyways, we use Hashicorp Vault's SSH OTP system. Works fine.
vaporup 4y ago
Yes, I use them for work. Checkout ssh-certinfo also: https://github.com/vaporup/ssh-tools/blob/master/ssh-certinfo https://github.com/vaporup/ssh-tools/blob/master/ssh-certinf...
robohoe 4y ago
IAP wherever I can
lapser 4y ago
For folks on AWS I highly recommend investing in SSH over SSM in combination with EC2 Instance Connect. You can generate temporary ssh keys, send it to EC2 Instance Connect, it'll be on the server for 60 seconds. Then you can use ssm start-session to connect to the server over SSM. This also allows you to disable SSH ingress, and doesn't need a bastion or VPN.
- darkr 4y ago
  If you’re all in on AWS, this is the easiest and possibly best solution, solves a bunch of security, operational and usability problems if implemented in tandem with SSO, CloudTrail and well defined IAM roles.
GNOMES 4y ago
Google Auth on SSH is awesome too https://wiki.archlinux.org/title/Google_Authenticator https://wiki.archlinux.org/title/Google_Authenticator
hedora 4y ago
Note that using a CA cert with SSH (or putting a host cert in DNS, etc) expands the attack surface for your server from just that server to the server + the CA's servers + the CA's employees + potentially other CA's + whatever governments are able to compel the CA['s] to issue a MITM cert for your server. For big organizations security is by-definition lax (you have to trust lots of non-trustworthy employees), but there's no reason to accept this compromise for smaller companies or personal use.
- pphysch 4y ago
  That's just the nature of trusting someone else to be your CA. You don't have to do that to use SSH certs.
  - hedora 4y ago
    At the minimum, you need to trust yourself to be a CA. That's strictly more attack surface than not having a CA.
    - pphysch 4y ago
      It's not "strictly" more because you reduce the attack surface of your leaf certs through expiration, etc.
      - hedora 4y ago
        What attack does that prevent? If the leaf certs are compromised, then so is the server, and it is already game over.
        pphysch 4y ago
        It prevents the attack where thousands of users have thousands of long-forgotten keys lying around, and eventually one of them gets compromised months or years after it was last legitimately used. AKA a huge attack surface that can't reliably be culled, because you are effectively relying on users to be their individual CAs. This particular attack took down my former organization once.
pritambaral 4y ago
I used to put CA keys, encrypted, in a repo only ops admins have access to. The passphrase was shared manually with each admin, once. If/when an admin left, the plan was to rotate the CAs. All SSH and TLS servers I used this for had the ability to use multiple CAs, so the rotation itself worked.
yankaf 4y ago
people here are quite honest.
onetimeusename 4y ago
It's possible to store the private CA keys on something like YubiKey hardware offline. That seems like a good trade off if you don't have access to a cloud based HSM. Ideally, the private CA keys would be stored in an HSM and used to sign any keys you needed with them. Note: this is not to be confused with using a FIDO/U2F hardware authenticators which is also a good idea. This can double not only has a PKI for hosts and users using SSH connections but you can also use the PKI for signature verification of arbitrary data.
NjU24E48YqdBGnG 4y ago
We use SSH certificates for host keys at work. That way we can avoid the constant "is this the host key you were expecting?" that everyone just pounds 'y' through. I've used user certificates personally in the past but no more. These days my SSH key is a GPG key on my Yubikey. For (centralized) user auth at work we use Kerberos/GSSAPIAuthentication (Windows 10's ssh client finally has support for it! You don't have to go download a newer version off their Github!). Users are free to use that to drop an authorized_keys on systems themselves as desired.
zikduruqe 4y ago
I just use my GPG key for SSH authentication. One GPG ID is personal, one GPG ID is work... and the twain shall not cross.
stavros 4y ago
I have a FIDO2 key I use for SSH, can't beat the convenience nor the security: https://www.stavros.io/posts/u2f-fido2-with-ssh/ https://www.stavros.io/posts/u2f-fido2-with-ssh/
71a54xd 4y ago
Anyone here done cool things with the YubiHSM? I've been dying to buy one for a year or so!
exabrial 4y ago
We keep user keys in LDAP. LDAP is mainly read-only having many many replicas is pretty easy. This is by far the most elegant solution we've determined and allows for instant revoking of user accounts.
_jal 4y ago
We use old-style public keys, but store them in LDAP. sshd has AuthorizedKeysCommand pointing to a script to retrieve for login. And we scan for things that look like keys.
rav 4y ago
To SSH into servers at $work, I need to run a command to 2FA-authenticate against our SSO and obtain a short-lived SSH cert for my local SSH key. We have around 10 tech employees and around 40 servers and it's really nice when onboarding new servers or new employees that we don't have to juggle authorized_keys around, and when offboarding employees we no longer have stale authorized_keys files to worry about, since access will expire automatically after 20 hours once the employee is removed in SSO. There's no hardware backing the CA, and it's all homebrew (NIH style) 2FA-SSO with openssh CA invocations, but it works and is quite convenient. We also use the same CA for host keys (again, certs issued by our NIH-style homebrew fleet management), which allows us to avoid TOFU prompts when onboarding new servers.
jleahy 4y ago
An option that's not in the list: check you CA key into source control (but with a very strong password). If you set the number of rounds large enough and have an excellent password this is sufficient (not as good as HSM, but probably safer than Vault). The real thing that gets me though, is how to deal with different users needing access to different subsets of machines. That's where I miss Kerberos.
romx 4y ago
I save CA root into private github repo for the pass app. Then i can sign certs whenever i need them.
tehno 4y ago
Has anyone here implemented Google's HIBA (https://github.com/google/hiba https://github.com/google/hiba) in addition to SSH CA/certs for the extra identity and grant checks? I'm thinking about implementing HIBA with Vault managed SSH certs, and would appreciate anyone's comments about this approach.
fjni 4y ago
At work there's a command I run that then requires 2fa auth through our identity provider. As a result, my ssh key gets signed with both a time restriction and an ip restriction. Same mechanism for git interactions and server logins. I believe on the backend it's an HSM which does the signing.
mutant 4y ago
All in on teleport here

cubesnooper 4y ago

> On the other side, as a personal user of SSH with basically one person to worry about, the effort of setting up a certificate seems like just a waste versus the existing key-based infrastructure; I don't understand at all what attack it would prevent or what convenience it would provide for the cost of learning it. The main benefit I get from using SSH certs at home is expiration. Before, I always had a niggling feeling at the back of my mind that the public key I’d been using for four years could have been surreptitiously exfiltrated by some shell script three years ago, and I’d never notice. Now my certificates expire regularly, so I no longer have to worry about whether my machine remained secure throughout the entire continuous past; I only have to worry if my machine is secure in the present.

xyzzyz 4y ago
How do you issue client certificates? Do you not worry that the private key needed to issue these can also leak just the same as your personal private key?
- cubesnooper 4y ago
  I have a Raspberry Pi dedicated to generating certificates. It serves the files to my LAN statically via a webserver, and is otherwise heavily firewalled. I don't run any other software on the Pi, so barring an exploit in the webserver, I’m not worried about the signing key getting compromised. Compared to my desktop, where over the years I ran all kinds of stuff from the package manager, downloaded Python scripts and configure scripts from GitHub and Sourceforge… I tried to be careful but you can’t audit everything.
  - xyzzyz 4y ago
    So anyone on your LAN can visit the URL and download the CA private key? Isn’t it only marginally more secure than just keeping the private key on your workstation in the first place, and foregoing the entire rigamarole with certificates? I mean, if you are worried about your something compromising your workstation and stealing your individual private keys, nothing is stopping whoever compromised your workstation from asking your RaspberryPi nicely for CA signing keys and stealing those too.
    - cubesnooper 4y ago
      > So anyone on your LAN can visit the URL and download the CA private key? No, the only files served to the LAN are the certificates, which contain the signatures by the CA of the public keys of other machines. Those are safe to distribute openly because they’re useless without the private key of the public key that was signed.
      - xyzzyz 4y ago
        Okay, so let me make sure I understand it. You have your own personal key, and CA key. Your personal (private) key lives on your workstation, and CA key lives on RPi. There are some other machines that you want to SSH into. They trust CA public key. To log into them, you download client certificates for your personal key, signed by CA key, from the RPi, and present them to remote SSH servers, while proving to them ownership of private key corresponding to the certificate. The remote servers trust the certificate issuer, verify that you own the certificate’s private key, and let you in. Do I get it correct? If so, how do you issue the certificates that live on RPi? Those certificates have some limited validity period, so that you can worry less about leaking them, which was your concern in the first place. Therefore, you must reissue them on a regular basis. How do you do that? What’s your process for that?
        cubesnooper 4y ago
        > Your personal (private) key lives on your workstation, and CA key lives on RPi. Yep. > There are some other machines that you want to SSH into. They trust CA public key. Yep, with TrustedUserCAKeys. > The remote servers trust the certificate issuer, verify that you own the certificate’s private key, and let you in. Yep. Additionally, the servers themselves present CA‐signed certificates alongside their host keys. My GlobalKnownHostsFile contains the CA public key, so when I connect to a host for the first time, there’s no “unknown host” warning and my user’s known_hosts file is not updated. > If so, how do you issue the certificates that live on RPi? The CA has a directory containing the public keys of every user and host that I set up. A cronjob periodically runs ssh-keygen -s against these files, and copies them to htdocs. Each host and user has a cronjob that periodically fetches its certificate and copies it to /etc/ssh or ~/.ssh, respectively.
- blablabla123 4y ago
  By doing a Certificate Signing Request (CSR) from the client. Not sure when it's actually worth doing this extra effort but with the right automation (e.g. with Ansible) it's doable. Never tried this for ssh though. Additional security if you use HSMs.
  - xyzzyz 4y ago
    That actually doesn’t answer my question (though the actual addressee answered it already quite satisfactorily). First, there is no such thing as CSR in context of ssh certificates: ssh certificates are not x.509 certificates (known for their use in TLS). Second, even assuming that they were, a client creates a CSR with their key, and then what? Where is the root of trust? Who processes these CSRs? How is it deciding which CSRs to accept, and which to reject? That’s what I’m actually asking about.
    - blablabla123 4y ago
      I mean ssh certificates can be x509, Azure VMs used to be provisioned with PEM actually, also PGP can be used. Not sure, I guess a use case could be to create client certificates remotely that are part of a certificate hierarchy. Obviously this doesn't solve any root of trust issues if the creation is initiated by the client.
  - cubesnooper 4y ago
    SSH provides native support for certificates; they’re a custom (non‐X.509) format. The signatures are generated with the ssh-keygen command; I set my infrastructure up purely by reading the manpage, not referring to any blog posts or anything, so I think the documentation is a good way to get started. https://man.openbsd.org/ssh-keygen.1#CERTIFICATES https://man.openbsd.org/ssh-keygen.1#CERTIFICATES
gunapologist99 4y ago
The certificates expire, but not the keys. This turns the CA and all points in between into a pretty major target, while SSH private keys reside only on the user's workstation, are personal to the user (reducing the scope), and are easily rotated by the user (well, more easily than CA!). Tools like userify also empowers the user to revoke their own keys globally, even from their phone, just by blanking out the authorized_keys box. Userify also lets you force key rotation for your team and automatically removes their accounts from servers if they don't rotate quickly enough.
- cubesnooper 4y ago
  Rotation is actually a lot easier with certificates. I just generate a new key on the client, copy the public key to the CA, and I’m done, with no need to repopulate authorized_keys on all my other machines. In another comment I went into more detail about how I keep the CA secure.
  - gunapologist99 4y ago
    > Rotation is actually a lot easier with certificates. I just generate a new key on the client, copy the public key to the CA, and I’m done, with no need to repopulate authorized_keys on all my other machines. Or, just paste your public key into your Userify profile and the same thing happens in seconds for every server that you have authorization for. Even better, there's no dependency on having a CA up and running in order to be able to log in; since your account is a regular local Linux account (just managed centrally), you log directly into the server with no need for that server to confirm your login elsewhere. But I've been burned before when a central auth server was down and I couldn't log into my servers (through no fault of my own), so I wouldn't want to go back to the old "please wait until we check your login against a central server" model again.
    - cubesnooper 4y ago
      I’m glad Userify works well for you. For my purposes, whipping up a couple of cronjobs involving curl and OpenSSH is more appropriate than relying on an external cloud service. > But I've been burned before when a central auth server was down and I couldn't log into my servers There is no central auth server involved here. My servers check login credentials against the CA’s public key, which is installed alongside my sshd config file. I make my certificates valid for three weeks, but regenerate them every two, so if some failure happens with creating or fetching certificates I have a week to notice and fix the problem.
goodpoint 4y ago
How is this more secure than simply creating new certificates and replacing the old ones is the authorized_keys files? > I only have to worry if my machine is secure in the present No. If a host has been accessed by an attacker due to an exfiltrated key in the past it's tainted forever.
- cubesnooper 4y ago
  > How is this more secure than simply creating new certificates and replacing the old ones is the authorized_keys files? It’s more convenient for me than updating authorized_keys. When I build a new machine, for example, I first generate a new SSH keypair on the machine. Then I copy the server and user public keys to the CA. Once I drop them in the right folder, certificates get generated automatically and served over HTTP. Then on the new machine I set up cronjobs to refresh the certificates every two weeks. I didn’t have to update authorized_keys on a dozen other machines, and I didn’t have to inspect any host fingerprints. > If a host has been accessed by an attacker due to an exfiltrated key in the past it's tainted forever. Yes, that’s obvious (although I did mention shell script in my previous comment—been a while since I thought this through). The scenario I imagined at the time was more like, did I ever copy my private key to an unencrypted flash drive, and then lose the flash drive? I never let private keys leave a machine anymore, but maybe I wasn’t so strict about that five years ago, and when did I generate my main SSH keypair? Was it four, five, six years ago? I no longer have to worry about such things. I could have just rotated my keys, but with short‐lived certificates I now get the benefits of key rotation without having to do any of the work. Now I tie all my SSH keys to a WebAuthn key, which provides even more protection, because within the three‐week window that my certificates are valid, an attacker would have to also physically possess my Yubikey.
  - goodpoint 4y ago
    How do you implement emergency certificate revocation?
    - cubesnooper 4y ago
      I have another Raspberry Pi sitting next to my desk, with a keyboard and a tiny screen, dedicated to systems administration. My user on this machine has an SSH key that on every machine logs into an account with sudo access. To revoke a user key, I run a script from this machine that logs into each host and updates sshd’s RevokedKeys. I have no mechanism at the moment for revoking host keys, which is a harder problem to solve as it would involve updating a number of laptops, phones, etc. that may not be powered on at a given time, but that’s less of a problem since if I knew a host key had been compromised I wouldn’t be logging into it anyway.

tptacek 4y ago

Essentially the whole point of modern SSH CAs is to centralize authentication and authorization for servers, and to keep long-lived keys off your developer machine fleet. If "no long-lived keys on laptops" and "central authority for who gets access to what with which credentials" aren't principles for your security practice, then you're right: there's not much of a win to switching to a CA model.

gunapologist99 4y ago
Sarcastic remarks like "if [mostly good things] aren't principles for your security practice" an strawman arguments seem a bit provocative for this type of debate.
- tptacek 4y ago
  There was absolutely nothing sarcastic about that remark. I wouldn't snark at 'jerf. Most security practices don't have those principles for SSH. Thanks for calling it out though; if you read it that way, they might have too, which would be a little mortifying.
  - gunapologist99 4y ago
    That's fair. thanks for the clarification!
jerf 4y ago
We use different central authorities than certificates, which manage keys directly; the end result is much the same even if the tools are theoretically inferior. Long-lived keys on laptops might be a weakness at this point, honestly. I expect that to be resolved by a slow but steady move to "whatever the cloud supports" as AWS and such rolls out their various solutions. I know we've got teams that use them, I imagine it'll be obligatory at some point here. I know it's not perfect, which is why I contrasted "polished turd" to the "uncut diamond in the rough". We're at a size where it's really hard to justify a whole lot of work on the uncut diamond, in contrast with the various cloud efforts to fix their problems. Given that such things now exist it's more a matter of moving over to them than developing anything ourselves now, which is probably an improvement over where we were two years ago. We probably couldn't have beaten the clouds even if we did put the effort in; this is in that class of problem where the effort is nominally small but the price of not breaking things is so large than it ends up a multi-year project anyhow. (One of my least favorite classes of project.) I'd also add that I report this not in the spirit of "we've solved this problem", but just an experience report.

tptacek 4y ago

I know people do this, but I can't get my head around it. SSH is end-to-end secure even if the entire DNS hierarchy is corrupted. The DNSSEC PKI is controlled at its roots by governments, and one level of branches down by a set of companies not known for integrity and especially strong security practices. Why would you give any of these entities any influence over your authorized keys?

Fnoord 4y ago
DNSSEC is top-down securing chain, DNSCrypt bottom-up. Each has their pros and cons. Relying on your government to keep you secure can be a valuable factor, depending on your threat model.
- tptacek 4y ago
  Ok, these are words, but again I'm not talking about DNS security here, I'm talking about SSH key distribution. Why would you elect to have your key distribution controlled by the DNS PKI? What's the upside? The downside is, an actor with control over the DNS PKI (there are many of those; see, for instance, every DOJ seizure of a domain) gets a degree of control over your SSH authorized keys. Seems... bad?
  - deleted 4y ago
    [deleted]
  - gunapologist99 4y ago
    Agreed completely. The threat model for DNSSEC is vast; why would you diminish the security of a perfectly good end-to-end model in SSH keys or certificates, as long as you control 100% of that infrastructure. Introducing any outside actors at all objectively diminishes the security of the whole model by becoming another link in the chain, even if that link isn't necessarily the weakest (and I certainly believe it would be, because both DNSSEC and the public Certificate Authority industry are object lessons for the abject failure of highly centralized global, government-wide, or even just company-wide security), but simply increasing any of the surface area is enough to decrease the security of the system.
    - dotwaffle 4y ago
      You'd have to compromise not only the root keys just to get into my infrastructure, but sign all the zones from there downwards, and intercept all DNS traffic leaving that server in the first place. That's secure enough for me.