8 ms·

> On the other side, as a personal user of SSH with basically one person to worry about, the effort of setting up a certificate seems like just a waste versus t

> On the other side, as a personal user of SSH with basically one person to worry about, the effort of setting up a certificate seems like just a waste versus the existing key-based infrastructure; I don't understand at all what attack it would prevent or what convenience it would provide for the cost of learning it. The main benefit I get from using SSH certs at home is expiration. Before, I always had a niggling feeling at the back of my mind that the public key I’d been using for four years could have been surreptitiously exfiltrated by some shell script three years ago, and I’d never notice. Now my certificates expire regularly, so I no longer have to worry about whether my machine remained secure throughout the entire continuous past; I only have to worry if my machine is secure in the present.

xyzzyz 4y ago
How do you issue client certificates? Do you not worry that the private key needed to issue these can also leak just the same as your personal private key?
- cubesnooper 4y ago
  I have a Raspberry Pi dedicated to generating certificates. It serves the files to my LAN statically via a webserver, and is otherwise heavily firewalled. I don't run any other software on the Pi, so barring an exploit in the webserver, I’m not worried about the signing key getting compromised. Compared to my desktop, where over the years I ran all kinds of stuff from the package manager, downloaded Python scripts and configure scripts from GitHub and Sourceforge… I tried to be careful but you can’t audit everything.
  - xyzzyz 4y ago
    So anyone on your LAN can visit the URL and download the CA private key? Isn’t it only marginally more secure than just keeping the private key on your workstation in the first place, and foregoing the entire rigamarole with certificates? I mean, if you are worried about your something compromising your workstation and stealing your individual private keys, nothing is stopping whoever compromised your workstation from asking your RaspberryPi nicely for CA signing keys and stealing those too.
    - cubesnooper 4y ago
      > So anyone on your LAN can visit the URL and download the CA private key? No, the only files served to the LAN are the certificates, which contain the signatures by the CA of the public keys of other machines. Those are safe to distribute openly because they’re useless without the private key of the public key that was signed.
      - xyzzyz 4y ago
        Okay, so let me make sure I understand it. You have your own personal key, and CA key. Your personal (private) key lives on your workstation, and CA key lives on RPi. There are some other machines that you want to SSH into. They trust CA public key. To log into them, you download client certificates for your personal key, signed by CA key, from the RPi, and present them to remote SSH servers, while proving to them ownership of private key corresponding to the certificate. The remote servers trust the certificate issuer, verify that you own the certificate’s private key, and let you in. Do I get it correct? If so, how do you issue the certificates that live on RPi? Those certificates have some limited validity period, so that you can worry less about leaking them, which was your concern in the first place. Therefore, you must reissue them on a regular basis. How do you do that? What’s your process for that?
        cubesnooper 4y ago
        > Your personal (private) key lives on your workstation, and CA key lives on RPi. Yep. > There are some other machines that you want to SSH into. They trust CA public key. Yep, with TrustedUserCAKeys. > The remote servers trust the certificate issuer, verify that you own the certificate’s private key, and let you in. Yep. Additionally, the servers themselves present CA‐signed certificates alongside their host keys. My GlobalKnownHostsFile contains the CA public key, so when I connect to a host for the first time, there’s no “unknown host” warning and my user’s known_hosts file is not updated. > If so, how do you issue the certificates that live on RPi? The CA has a directory containing the public keys of every user and host that I set up. A cronjob periodically runs ssh-keygen -s against these files, and copies them to htdocs. Each host and user has a cronjob that periodically fetches its certificate and copies it to /etc/ssh or ~/.ssh, respectively.
- blablabla123 4y ago
  By doing a Certificate Signing Request (CSR) from the client. Not sure when it's actually worth doing this extra effort but with the right automation (e.g. with Ansible) it's doable. Never tried this for ssh though. Additional security if you use HSMs.
  - xyzzyz 4y ago
    That actually doesn’t answer my question (though the actual addressee answered it already quite satisfactorily). First, there is no such thing as CSR in context of ssh certificates: ssh certificates are not x.509 certificates (known for their use in TLS). Second, even assuming that they were, a client creates a CSR with their key, and then what? Where is the root of trust? Who processes these CSRs? How is it deciding which CSRs to accept, and which to reject? That’s what I’m actually asking about.
    - blablabla123 4y ago
      I mean ssh certificates can be x509, Azure VMs used to be provisioned with PEM actually, also PGP can be used. Not sure, I guess a use case could be to create client certificates remotely that are part of a certificate hierarchy. Obviously this doesn't solve any root of trust issues if the creation is initiated by the client.
  - cubesnooper 4y ago
    SSH provides native support for certificates; they’re a custom (non‐X.509) format. The signatures are generated with the ssh-keygen command; I set my infrastructure up purely by reading the manpage, not referring to any blog posts or anything, so I think the documentation is a good way to get started. https://man.openbsd.org/ssh-keygen.1#CERTIFICATES https://man.openbsd.org/ssh-keygen.1#CERTIFICATES
gunapologist99 4y ago
The certificates expire, but not the keys. This turns the CA and all points in between into a pretty major target, while SSH private keys reside only on the user's workstation, are personal to the user (reducing the scope), and are easily rotated by the user (well, more easily than CA!). Tools like userify also empowers the user to revoke their own keys globally, even from their phone, just by blanking out the authorized_keys box. Userify also lets you force key rotation for your team and automatically removes their accounts from servers if they don't rotate quickly enough.
- cubesnooper 4y ago
  Rotation is actually a lot easier with certificates. I just generate a new key on the client, copy the public key to the CA, and I’m done, with no need to repopulate authorized_keys on all my other machines. In another comment I went into more detail about how I keep the CA secure.
  - gunapologist99 4y ago
    > Rotation is actually a lot easier with certificates. I just generate a new key on the client, copy the public key to the CA, and I’m done, with no need to repopulate authorized_keys on all my other machines. Or, just paste your public key into your Userify profile and the same thing happens in seconds for every server that you have authorization for. Even better, there's no dependency on having a CA up and running in order to be able to log in; since your account is a regular local Linux account (just managed centrally), you log directly into the server with no need for that server to confirm your login elsewhere. But I've been burned before when a central auth server was down and I couldn't log into my servers (through no fault of my own), so I wouldn't want to go back to the old "please wait until we check your login against a central server" model again.
    - cubesnooper 4y ago
      I’m glad Userify works well for you. For my purposes, whipping up a couple of cronjobs involving curl and OpenSSH is more appropriate than relying on an external cloud service. > But I've been burned before when a central auth server was down and I couldn't log into my servers There is no central auth server involved here. My servers check login credentials against the CA’s public key, which is installed alongside my sshd config file. I make my certificates valid for three weeks, but regenerate them every two, so if some failure happens with creating or fetching certificates I have a week to notice and fix the problem.
goodpoint 4y ago
How is this more secure than simply creating new certificates and replacing the old ones is the authorized_keys files? > I only have to worry if my machine is secure in the present No. If a host has been accessed by an attacker due to an exfiltrated key in the past it's tainted forever.
- cubesnooper 4y ago
  > How is this more secure than simply creating new certificates and replacing the old ones is the authorized_keys files? It’s more convenient for me than updating authorized_keys. When I build a new machine, for example, I first generate a new SSH keypair on the machine. Then I copy the server and user public keys to the CA. Once I drop them in the right folder, certificates get generated automatically and served over HTTP. Then on the new machine I set up cronjobs to refresh the certificates every two weeks. I didn’t have to update authorized_keys on a dozen other machines, and I didn’t have to inspect any host fingerprints. > If a host has been accessed by an attacker due to an exfiltrated key in the past it's tainted forever. Yes, that’s obvious (although I did mention shell script in my previous comment—been a while since I thought this through). The scenario I imagined at the time was more like, did I ever copy my private key to an unencrypted flash drive, and then lose the flash drive? I never let private keys leave a machine anymore, but maybe I wasn’t so strict about that five years ago, and when did I generate my main SSH keypair? Was it four, five, six years ago? I no longer have to worry about such things. I could have just rotated my keys, but with short‐lived certificates I now get the benefits of key rotation without having to do any of the work. Now I tie all my SSH keys to a WebAuthn key, which provides even more protection, because within the three‐week window that my certificates are valid, an attacker would have to also physically possess my Yubikey.
  - goodpoint 4y ago
    How do you implement emergency certificate revocation?
    - cubesnooper 4y ago
      I have another Raspberry Pi sitting next to my desk, with a keyboard and a tiny screen, dedicated to systems administration. My user on this machine has an SSH key that on every machine logs into an account with sudo access. To revoke a user key, I run a script from this machine that logs into each host and updates sshd’s RevokedKeys. I have no mechanism at the moment for revoking host keys, which is a harder problem to solve as it would involve updating a number of laptops, phones, etc. that may not be powered on at a given time, but that’s less of a problem since if I knew a host key had been compromised I wouldn’t be logging into it anyway.

xyzzyz 4y ago

How do you issue client certificates? Do you not worry that the private key needed to issue these can also leak just the same as your personal private key?

cubesnooper 4y ago
I have a Raspberry Pi dedicated to generating certificates. It serves the files to my LAN statically via a webserver, and is otherwise heavily firewalled. I don't run any other software on the Pi, so barring an exploit in the webserver, I’m not worried about the signing key getting compromised. Compared to my desktop, where over the years I ran all kinds of stuff from the package manager, downloaded Python scripts and configure scripts from GitHub and Sourceforge… I tried to be careful but you can’t audit everything.
- xyzzyz 4y ago
  So anyone on your LAN can visit the URL and download the CA private key? Isn’t it only marginally more secure than just keeping the private key on your workstation in the first place, and foregoing the entire rigamarole with certificates? I mean, if you are worried about your something compromising your workstation and stealing your individual private keys, nothing is stopping whoever compromised your workstation from asking your RaspberryPi nicely for CA signing keys and stealing those too.
  - cubesnooper 4y ago
    > So anyone on your LAN can visit the URL and download the CA private key? No, the only files served to the LAN are the certificates, which contain the signatures by the CA of the public keys of other machines. Those are safe to distribute openly because they’re useless without the private key of the public key that was signed.
    - xyzzyz 4y ago
      Okay, so let me make sure I understand it. You have your own personal key, and CA key. Your personal (private) key lives on your workstation, and CA key lives on RPi. There are some other machines that you want to SSH into. They trust CA public key. To log into them, you download client certificates for your personal key, signed by CA key, from the RPi, and present them to remote SSH servers, while proving to them ownership of private key corresponding to the certificate. The remote servers trust the certificate issuer, verify that you own the certificate’s private key, and let you in. Do I get it correct? If so, how do you issue the certificates that live on RPi? Those certificates have some limited validity period, so that you can worry less about leaking them, which was your concern in the first place. Therefore, you must reissue them on a regular basis. How do you do that? What’s your process for that?
      - cubesnooper 4y ago
        > Your personal (private) key lives on your workstation, and CA key lives on RPi. Yep. > There are some other machines that you want to SSH into. They trust CA public key. Yep, with TrustedUserCAKeys. > The remote servers trust the certificate issuer, verify that you own the certificate’s private key, and let you in. Yep. Additionally, the servers themselves present CA‐signed certificates alongside their host keys. My GlobalKnownHostsFile contains the CA public key, so when I connect to a host for the first time, there’s no “unknown host” warning and my user’s known_hosts file is not updated. > If so, how do you issue the certificates that live on RPi? The CA has a directory containing the public keys of every user and host that I set up. A cronjob periodically runs ssh-keygen -s against these files, and copies them to htdocs. Each host and user has a cronjob that periodically fetches its certificate and copies it to /etc/ssh or ~/.ssh, respectively.
blablabla123 4y ago
By doing a Certificate Signing Request (CSR) from the client. Not sure when it's actually worth doing this extra effort but with the right automation (e.g. with Ansible) it's doable. Never tried this for ssh though. Additional security if you use HSMs.
- xyzzyz 4y ago
  That actually doesn’t answer my question (though the actual addressee answered it already quite satisfactorily). First, there is no such thing as CSR in context of ssh certificates: ssh certificates are not x.509 certificates (known for their use in TLS). Second, even assuming that they were, a client creates a CSR with their key, and then what? Where is the root of trust? Who processes these CSRs? How is it deciding which CSRs to accept, and which to reject? That’s what I’m actually asking about.
  - blablabla123 4y ago
    I mean ssh certificates can be x509, Azure VMs used to be provisioned with PEM actually, also PGP can be used. Not sure, I guess a use case could be to create client certificates remotely that are part of a certificate hierarchy. Obviously this doesn't solve any root of trust issues if the creation is initiated by the client.
- cubesnooper 4y ago
  SSH provides native support for certificates; they’re a custom (non‐X.509) format. The signatures are generated with the ssh-keygen command; I set my infrastructure up purely by reading the manpage, not referring to any blog posts or anything, so I think the documentation is a good way to get started. https://man.openbsd.org/ssh-keygen.1#CERTIFICATES https://man.openbsd.org/ssh-keygen.1#CERTIFICATES

gunapologist99 4y ago

The certificates expire, but not the keys. This turns the CA and all points in between into a pretty major target, while SSH private keys reside only on the user's workstation, are personal to the user (reducing the scope), and are easily rotated by the user (well, more easily than CA!). Tools like userify also empowers the user to revoke their own keys globally, even from their phone, just by blanking out the authorized_keys box. Userify also lets you force key rotation for your team and automatically removes their accounts from servers if they don't rotate quickly enough.

cubesnooper 4y ago
Rotation is actually a lot easier with certificates. I just generate a new key on the client, copy the public key to the CA, and I’m done, with no need to repopulate authorized_keys on all my other machines. In another comment I went into more detail about how I keep the CA secure.
- gunapologist99 4y ago
  > Rotation is actually a lot easier with certificates. I just generate a new key on the client, copy the public key to the CA, and I’m done, with no need to repopulate authorized_keys on all my other machines. Or, just paste your public key into your Userify profile and the same thing happens in seconds for every server that you have authorization for. Even better, there's no dependency on having a CA up and running in order to be able to log in; since your account is a regular local Linux account (just managed centrally), you log directly into the server with no need for that server to confirm your login elsewhere. But I've been burned before when a central auth server was down and I couldn't log into my servers (through no fault of my own), so I wouldn't want to go back to the old "please wait until we check your login against a central server" model again.
  - cubesnooper 4y ago
    I’m glad Userify works well for you. For my purposes, whipping up a couple of cronjobs involving curl and OpenSSH is more appropriate than relying on an external cloud service. > But I've been burned before when a central auth server was down and I couldn't log into my servers There is no central auth server involved here. My servers check login credentials against the CA’s public key, which is installed alongside my sshd config file. I make my certificates valid for three weeks, but regenerate them every two, so if some failure happens with creating or fetching certificates I have a week to notice and fix the problem.

goodpoint 4y ago

How is this more secure than simply creating new certificates and replacing the old ones is the authorized_keys files? > I only have to worry if my machine is secure in the present No. If a host has been accessed by an attacker due to an exfiltrated key in the past it's tainted forever.

cubesnooper 4y ago
> How is this more secure than simply creating new certificates and replacing the old ones is the authorized_keys files? It’s more convenient for me than updating authorized_keys. When I build a new machine, for example, I first generate a new SSH keypair on the machine. Then I copy the server and user public keys to the CA. Once I drop them in the right folder, certificates get generated automatically and served over HTTP. Then on the new machine I set up cronjobs to refresh the certificates every two weeks. I didn’t have to update authorized_keys on a dozen other machines, and I didn’t have to inspect any host fingerprints. > If a host has been accessed by an attacker due to an exfiltrated key in the past it's tainted forever. Yes, that’s obvious (although I did mention shell script in my previous comment—been a while since I thought this through). The scenario I imagined at the time was more like, did I ever copy my private key to an unencrypted flash drive, and then lose the flash drive? I never let private keys leave a machine anymore, but maybe I wasn’t so strict about that five years ago, and when did I generate my main SSH keypair? Was it four, five, six years ago? I no longer have to worry about such things. I could have just rotated my keys, but with short‐lived certificates I now get the benefits of key rotation without having to do any of the work. Now I tie all my SSH keys to a WebAuthn key, which provides even more protection, because within the three‐week window that my certificates are valid, an attacker would have to also physically possess my Yubikey.
- goodpoint 4y ago
  How do you implement emergency certificate revocation?
  - cubesnooper 4y ago
    I have another Raspberry Pi sitting next to my desk, with a keyboard and a tiny screen, dedicated to systems administration. My user on this machine has an SSH key that on every machine logs into an account with sudo access. To revoke a user key, I run a script from this machine that logs into each host and updates sshd’s RevokedKeys. I have no mechanism at the moment for revoking host keys, which is a harder problem to solve as it would involve updating a number of laptops, phones, etc. that may not be powered on at a given time, but that’s less of a problem since if I knew a host key had been compromised I wouldn’t be logging into it anyway.