7 ms·

SSH certificates: the better SSH experience

268 points by jandeboevrie 7mo ago

thomashabets2 7mo ago
Every couple of months someone re-discovers SSH certificates, and blogs about them. I'm guilty of it too. My blog post from 15 years ago is nowhere near as good as OP's post, but if I though me of 15 years ago lived up to my standards of today, I'd be really disappointed: https://blog.habets.se/2011/07/OpenSSH-certificates.html https://blog.habets.se/2011/07/OpenSSH-certificates.html
- kaoD 7mo ago
  I've known SSH certs for a while but never went through the effort of migrating away from keys. I'm very frustrated about manually managing my SSH keys across my different servers and devices though. I assume you gathered a lot of thoughts over these 15 years. Should I invest in making the switch?
  - ibotty 7mo ago
    Yes. Caveat: It might not really be worth it if all your infrastructure is managed by these newfangled infrastructure-as-code-things that are quick to roll out (OpenShift/OKD, Talos, etc.) and you have only one repo to change SSH keys (single cluster or single repo for all clusters). There are some serious security benefits for larger organizations but it does not sound as if you are part of one.
  - deleted 7mo ago
    [deleted]
  - thomashabets2 7mo ago
    If your use case is such that you are frustrated about managing keys, host or user keys, then yes it does sound like SSH certs would help you. E.g. when you have many users, servers, or high enough cartesian product of the two. In environment where they don't cause frustration they're not worth it. Not really more to it than that, from my point of view.
  - otabdeveloper4 7mo ago
    You will have to manage your SSH CA certificates instead of your keys. The workflows SSH CA's are extremely janky and insecure. With some creative use of `AuthorizedKeysCommand` you can make SSH key rotation painless and secure. With SSH certificates you have to go back to the "keys to the kingdom" antipattern and just hope for the best.
    - jamiesonbecker 7mo ago
      Exactly. We'd had discussions about building https://Userify.com https://Userify.com (plug!) around SSH certificates, but elected to go with keys instead, because Userify delivers most of the good things around certificates without the jank and insecurity. It's not that certificates themselves are insecure themselves, it's that the workflows (as the parent points out) are awful. We might still add some automation around that (and I think I saw some competitor tooling out there if you're committed to that path) but I personally feel like it's an answer to the wrong question.
    - cyberax 7mo ago
      > With SSH certificates you have to go back to the "keys to the kingdom" antipattern and just hope for the best. Whut? This is literally the opposite. With CA certs you can create short-lived certificates, so you can easily grant access to a system for a short time.
      - namibj 7mo ago
        And what about the CA?
        cyberax 7mo ago
        It's no different compared to regular SSH private keys. You need to protect it from compromise. However, it provides you an additional layer of protection, because it does not need to be on the critical path for every SSH connection. My CA is a Nitrokey HSM, for example. I issue myself temporary certs that are valid only for 6 hours for ephemeral private keys.
  - anyfoo 7mo ago
    A big problem I have with ssh carts is that they are not universally supported. For me, there is always some device or daemon (for example tinyssh in the initramfs of my gaming pc so that I can unlock it remotely) that only works with “plain old ssh keys”. And if I have to distribute and sync my keys onto a few hosts anyway, it takes away the benefits.
    - TZubiri 7mo ago
      Might actually be a positive instead of a negative. Gaming use-cases should have not any effect on security policies, these should be as separate as possible, different auth mechanisms for your gaming stuff and your professional stuff ensures nothing gets mixed.
      - nagaiaida 6mo ago
        remote unlock is also useful when you're not gaming so that feels like the wrong aspect to focus on
      - anyfoo 6mo ago
        Hah? It being my gaming machine has nothing to do with the problem. It’s also my FPGA development machine, though it gets used less for that. It only happens to be the only Linux workstation in my home (the others are Macs or OpenBSD).
        TZubiri 6mo ago
        If you care about security, I recommend investing into a separate computer for developing hardware and software and another for downloading games on. You can setup your security any way you like, but nothing beats an air gap in terms of security and simplicity.
    - namibj 7mo ago
      Upgrade to a better one in initramfs?
    - AceJohnny2 6mo ago
  - dizhn 7mo ago
    I am keeping an eye on the new (and alpha) Authentik agent which will allow idp based ssh logins. There's also SSSD already supported but it requires glibc (due to needing NSS) meaning it's not available on Alpine.
    - gnufx 7mo ago
      If you mean using OIDC, in that space there's at least https://github.com/EOSC-synergy/ssh-oidc https://github.com/EOSC-synergy/ssh-oidc, https://dianagudu.github.io/mccli/ https://dianagudu.github.io/mccli/ and OpenPubkey-ssh discussed in https://news.ycombinator.com/item?id=43470906 https://news.ycombinator.com/item?id=43470906 (which might mention more). How does SSSD support help with SSH authN? I know you can now get Kerberos tickets from FreeIPA using OIDC(?), but I forget if SSSD is involved.
      - floating-io 6mo ago
        Can't really speak to the point of the guy you're replying to, but the FreeIPA implementation via SSSD does more than just Kerberos tickets. Actually, I think the Kerberos based stuff as it relates to SSH is GSSAPI as part of sshd itself and has little to do with sssd, though I could be wrong. That said... If I'm remembering things correctly (and it's been a looong while since I've played with this), FreeIPA's client configures sshd with an AuthorizedKeysCommand that executes a program that queries sssd for the list of authorized keys for a given user. Sssd then uses a plugin to query the LDAP server @ FreeIPA to get the list of keys. There's also SSHFP (I think) records in DNS if you're using FreeIPA's DNS servers. These provide the host keys for servers for your ssh client to check against. Not sure if that's integrated into ssh itself or something else -- I can't remember how it's implemented offhand -- but it's fairly nifty since you never see the TOFU prompt (or it would be if DNS was actually secure, anyway).
        gnufx 6mo ago
        Yes, FreeIPA is Kerberos+LDAP+X.509 CA, and GSSAPI is in OpenSSH (normally with the key exchange patch). SSSD is a local mechanism, not network authentication. I mentioned authorized keys distribution mechanisms elsewhere, but I was thinking authentication (c.f. OIDC), not authorization.
  - cyberax 7mo ago
    It depends on what you want to do. CA certs are easy to manage, you just put the CA key instead of the SSH public key in authorized_keys. They also provide a way to get hardware-backed security without messing with SSH agent forwarding and crappy USB security devices. You can use an HSM to issue a temporary certificate for your (possibly temporary) public key and use it as normal. The certificate can be valid for just 1 hour, enough to not worry about it leaking.
- papyDoctor 7mo ago
  Another useful feature of SSH certificates is that you can sign a user’s public key to grant them access to a remote machine for a limited time and as a specific remote user.
  - TZubiri 7mo ago
    The capacity to grant access as a specific remote user is present without certs as well right? The typical authorized_keys file lives under a user directory and grants access only to that user.
    - blueflow 7mo ago
      The main advantage of certificates is that you are able to do that from the CA without touching the target machine.
      - lokar 7mo ago
        Exactly. This is really useful in larger organizations where you may want more complex rules on access. For example, you can easily build "break glass" or 2nd party approved access on demand. You can put whatever logic you need in a CA front-end. You can also make all the certs short-lived (and only store them in ram).
        TZubiri 6mo ago
        The way I've been doing that is with Shamir Secret Sharing and encrypting keys until glass-breaking is necessary.
        lokar 6mo ago
        generating tons of keys? or just broad keys? What I've done is generate a cert for the host(s) the user needs, for the time-span they need (subject to authorization logic).
      - pphysch 6mo ago
- Stefan-H 7mo ago
  I think the scary reality is most people conflate "keys" and "certificates". I have worked with security engineers that I need to remind that we do not use SSH certs, but rather key auth, and they have to think it through to make it click.
  - tracker1 7mo ago
    I'm consistently amazed how many developers and security professionals don't have a clear understanding how PPK even works conceptually. Things like deploying dev keys to various production environments, instead of generating/registering them within said environment. One of the worst recent security examples... You can't get this data over HTTPS from $OtherAgency, it's "not secure" ... then their suggestion is a "secure" read-only account to the other agency's SQL server (which uses the same TLS 1.3 as HTTPS). This is from person in charge of digital security for a government org.
    - tolciho 7mo ago
      Or when the security team at some other company emails you their private key.
      - tracker1 6mo ago
        LOL, yeah.. had that happen quite a few times... Also, re-using the ssh server key for the client connecting to the sftp server.
    - laserlight 6mo ago
      > Things like deploying dev keys to various production environments, instead of generating/registering them within said environment. I can see this happening when a developer is authorized to generate, but not to register. So, they just reuse an already-registered one.
      - tracker1 6mo ago
        In the example, it wasn't even that complex... I have used patterns to register allowed signer keys based on environment variables that an application runs under, initializing at startup... so "register" just meant assigning the correct values for 2-4 environment variables per public signer allowed... and removing the dev signer. (JWT based auth)
- V-eHGsd_ 7mo ago
  oh man, I referred back to your blog post when I wrote the ssh certificate authority for $job ... ~10 years ago. Thank for writing it!
- functional_dev 6mo ago
  maybe it keeps getting rediscovered, becaues a good tooling around it is still missing?
Thom2000 7mo ago
Sadly services such as Github don't support these so it's mostly good for internal infrastructure.
- lights0123 7mo ago
  They do, for Enterprise customers only: https://docs.github.com/en/enterprise-cloud@latest/organizations/managing-git-access-to-your-organizations-repositories/about-ssh-certificate-authorities https://docs.github.com/en/enterprise-cloud@latest/organizat... They've rolled their host key one time, so there's little reason for them to use it on the host side.
  - rpicard 6mo ago
    I’d love if they supported an SSH certificate CA for commit signing verification though.
linsomniac 7mo ago
In our dev/stg environment we reinstall half our machines every morning (largely to test our machine setup automation), and SSH host certificates make that so much nicer than having to persist host keys or remove/replace them in known_hosts. Highly recommended.
- grave88 7mo ago
  [flagged]
Tepix 7mo ago
The author lists all the advantes of CA certificates, yet doesn't list the disadvantages. OTOH, all the many steps required to set it up make the disadvantages rather obvious. Also, I've never had a security issue due to TOFU, have you?
- GandalfHN 7mo ago
  [flagged]
- adrian_b 7mo ago
  TOFU is convenient, but not necessary. Choosing to use TOFU is a distinct choice from the choice of using the keys generated by SSH, instead of using certificates. If you do not want to use TOFU, for extra security, you just have to pair the computers by copying between them the corresponding public keys through a secure channel, e.g. by using a USB memory. Using certificates does not add any simplification or any extra security. For real security, you still must pair the communicating computers by copying between them the corresponding certificates, through a secure channel, e.g. a USB memory. When you use for HTTPS the certificates that have come with your Internet browser, you trust that the installer package for the browser has come to that computer through a secure channel from the authority that has created the certificates. This is usually an assumption much more far fetched than the assumption that you can trust TOFU between computers under your control. Certificates may be useful in big organizations, if other functionality is needed beyond just establishing secure communication channels, e.g. if you want to use certificate revocation. In the list of "advantages" enumerated in the parent article, more than half of them are false, because if certificates are implemented correctly, completely equivalent actions must be executed when SSH keys without TOFU are used and when certificates are used. Perhaps the author meant by writing some of the "advantages" that the actions that supposedly are no longer needed with certificates are done by an administrator, not by the user. However that is also applicable with SSH. An administrator could install the certificates, so that no action is required from the user, but an administrator can also install the SSH public keys, so that no TOFU is ever needed from the user. Using certificates requires exactly the same steps like using keys generated by SSH (i.e. generating certificates and copying them between computers through secure channels, to pair the servers and the authorized users), but it may need additional steps, caused by the fact that certificates provide additional functionality.
jcalvinowens 7mo ago
You can also address TOFU to some extent using SSHFP DNS records. Openssh supports checking the DNSSEC signature in the client, in theory, but it's a configure option and I'm not sure if distros build with it.
- jsiepkes 7mo ago
  On top of that you would need something to secure DNS. Like DNSSEC or at the very least use DNS with TLS or DNS over HTTP. None of these are typically enabled by default.
  - jcalvinowens 7mo ago
    Anything that uses system-resolved is probably doing DNSSEC validation by default. It's becoming much more common. Additionally, as I mentioned, openssh itself has support for validating the DNSSEC signature even if your local resolver doesn't. I actually don't think it can use the standard resolver for SSHFP records at all, but I'm not sure.
- fc417fc802 7mo ago
  Any idea if there's a standardized location, something like /.well-known/ssh?
  - NicolaiS 6mo ago
    This exists, but not very well supported: https://github.com/C2SP/C2SP/blob/main/well-known-ssh-hosts.md https://github.com/C2SP/C2SP/blob/main/well-known-ssh-hosts.... But DNSSEC + SSHFP is a better supported solution at the moment
bobo56539 7mo ago
With the recent wave of npm hacks stealing private keys, I wanted to limit key's lifetimes. I've set up a couple of yubikeys as SSH CAs on hosts I manage. I use them to create short lived certs (say 24h) at the start of the day. This way i only have to enter the yubikey pin once a day. I could not find an easy way to limit maximum certificate lifetime in openssh, except for using the AuthorizedPrincipalCommand, which feels very fragile. Does anyone else have any experience with a similar setup? How do you limit cert max lifetime?
- agwa 7mo ago
  Instead of using a CA, why not set the key's PIN policy to "once" and use an agent (e.g. https://github.com/FiloSottile/yubikey-agent/ https://github.com/FiloSottile/yubikey-agent/) that holds an active session to the yubikey? You start the agent at the beginning of the day, enter the PIN once, and then stop the agent at the end of the day.
  - bobo56539 7mo ago
    Wanted to avoid having the key in USB slot all the time. I have the version that sticks out and i carry it on my key chain. So it's easy to break in a laptop.
- dplgk 6mo ago
  In addition to ssh key, I have my servers also ask for a one-time password
moviuro 7mo ago
All those articles about SSH certificates fall short of explaining how the revocation list can/should be published. Is that yet another problem that I need to solve with syncthing? https://man.openbsd.org/ssh-keygen.1#KEY_REVOCATION_LISTS https://man.openbsd.org/ssh-keygen.1#KEY_REVOCATION_LISTS
- blipvert 7mo ago
  If you generate short lived certificates via an automated process/service then you don’t really need to manage a revocation list as they will have expired in short order.
  - jamiesonbecker 7mo ago
    But then you can't log in if your box goes offline for any reason.
    - blipvert 7mo ago
      Hmm. For user certs you can have the service sign them for, say an hour, so long as you can ssh to your server in that time then there’s no need for any other interaction. Sure you need your signing service to be reasonably available, but that’s easily accomplished. Maybe I misunderstand?
      - jamiesonbecker 7mo ago
        That works for authn in the happy path: short-lived cert, grab it, connect, done. Except for everything around that: * user lifecycle (create/remove/rename accounts) * authz (who gets sudo, what groups, per-host differences) * cleanup (what happens when someone leaves) * visibility (what state is this box actually in right now?) SSH certs don’t really touch any of that. They answer can this key log in right now, not what should exist on this machine. So in practice, something else ends up managing users, groups, sudoers, home dirs, etc. Now there are two systems that both have to be correct. On the availability point: "reasonably available" is doing a lot of work ;) Even with 1-hour certs: * new sessions depend on the signer * fleet-wide issues hit everything at once * incident response gets awkward if the signer is part of the blast radius The failure mode shifts from a few boxes don't work to nobody can get in anywhere The pull model just leans the other way: * nodes converge to desired state * access continues even if control plane hiccups * authn and authz live together on the box Both models can work - it’s more about which failure mode is tolerable to you.
gunapologist99 7mo ago
Anyone tried out Userify? It creates/removes ssh pubkeys locally so (like a CA) no authn server needs to be online. But unlike certs, active sessions and processes are terminated when the user access is revoked.
- jamiesonbecker 7mo ago
  We're in the process of updating the experience to this century! ;) We've always taken the stance that crusty is better than vulnerable, but it turns out that not having a modern experience after 15 years is starting to feel like maybe we need to step up the features and shininess :)
sqbic 7mo ago
I've had very good experiences with SSH Communication Security company's (the guys who invented SSH) PrivX product to manage secure remote access, including SSH certificates and also cert based Windows authentication. It supports other kinds of remote targets too, via webui or with native clients. Great product.
jamiesonbecker 7mo ago
SSH certs quietly hurt in prod. Short-lived creds + centralized CA just moves complexity upward without solving the core problem: user management. The system shifts from many small local states to one highly coupled control point. That control point has to be correct and reachable all the time. When it isn’t, failures go wide instead of narrow. Example: a few boxes get popped and start hammering the CA. Now what? Access is broken everywhere at once. Common friction points: 1. your signer that has to be up and correct all the time 2. trust roots everywhere (and drifting) 3. TTL tuning nonsense (too short = random lockouts, too long = what was the point) 4. limited on-box state makes debugging harder than it should be 5. failures tend to fan out instead of staying contained Revocation is also kind of a lie. Just waiting for expiry and hoping that’s good enough. What actually happens is people reintroduce state anyway: sidecars, caches, agents… because you need it. We went the opposite direction: 1. nodes pull over outbound HTTPS 2. local authorized_keys is the source of truth locally 3. users/roles are visible on the box 4. drift fixes itself quickly 5. no inbound ports, no CA signatures (WELL, not strictly true*!) You still get central control, but operation and failure modes are local instead of "everyone is locked out right now." That’s basically what we do at Userify (https://userify.com https://userify.com). Less elegant than certs, more survivable at 2am. Also actually handles authz, not just part of authn. And the part that usually gets hand-waved with SSH CAs: 1. creating the user account 2. managing sudo roles 3. deciding what happens to home directories on removal 4. cleanup vs retention for compliance/forensics Those don’t go away - they're just not part of the certificate solution. * (TLS still exists here, just at the transport layer using the system trust store. That channel delivers users, keys, and roles. The rest is handled explicitly instead of implied.)
- ngrilly 7mo ago
  How do you solve TOFU?
  - jamiesonbecker 7mo ago
    Well, TOFU is really just the model for how the chain of trust is established. In practice there isn’t really trust on first use: there’s verify the key matches what’s expected, or distribute keys out-of-band (including certs). If that verification step isn’t happening, then it’s not TOFU, it’s just blind trust. From an automation/autoscaling angle, the same thing shows up again: 1. either keys are pre-baked / distributed 2. or, something signs them at boot Signing an instance key is just another way of distributing trust. It doesn’t remove the need for a root of trust, it moves it. Certificates just add extra steps around the same underlying task.
longislandguido 7mo ago
This discussion is full of schizo solutions to "secure" SSH, most of which make no practical sense or have no technical basis. There really needs to be a definitive best practices guide published by a trusted authority.
- kackerlacker 7mo ago
  In my view it is more important to stop using software keys so probably use sk (fido) for both host and user.. From there CAs would be a next step.. The level of documentation and example setups is astoundingly poor if you even look at step 2 for any feature. I.e. SK keys are reasonably understood for user keys but the setup as host keys is vague and needs testing to see if it really works.
- elric 6mo ago
  Can you give some examples of which suggestions you think are schizo?
Serhii-Set 7mo ago
[dead]
TZubiri 7mo ago
>then I don’t need to type the target user’s password; instead I enter the key’s passphrase, a hopefully much more complicated combination of words, to unlock the private key. This sentence is a bit of a red flag, it looks like the author is making a (subtle) mistake in the category of too much security, or at least misjudging the amount of security (objectively measurable entropy) needed. This is of course a less consequential error than too little entropy/security measures, but still if one wants to be a cybersecurity professional, especially one with influence, they must know exactly the right amount needed, because our resources are limited, and each additional bit of entropy and security step not only costs time of the admin that implements it, but of the users that have to follow it, and this can even impact security itself by fatiguing the user and causing them to circumvent measures or ignore alerts. On to specifically what's wrong: Either a key file or a password can be used to log in to a server or authenticate to any service in general. Besides the technical implementation, the main difference is whether the secret is stored on the device, or in the user's brain. One is not more correct than the other, there's a lot of tradeoffs, one can ensure more bits and is more ergonomic, the other is not stored on device so it cannot be compromised that way. That said a 2FA approach, in whatever format, is (generally speaking) safer than any individual method, in that the two secrets are necessary to be granted access. In this scenario one needs both the file and the password to authenticate, even if the password is 4 digits long, that increases the security of the system when compared to no password. An attacker would have to setup a brute force attempt along with a way to verify the decryption was successful. If local decryption confirmation is not possible, then such a brute force attack would require to submit erroneous logins to the server potentially activating tripwires or alerting a monitoring admin. There's nothing special about the second factor authorization being equal or equivalent in entropy to the first, and there's especially no requirement that a password have more entropy when it's a second authorization, in fact it's the other way around. tl;dr You can consider each security mechanism in the wider context rather than in isolation and you will see security fatigue go down without compromising security.
aquafox 7mo ago
I work in a corporate setting and the money and time we wasted because of Zscaler and its SSL inspection [1] is beyond your wildest imagination. Whenever I see a "SSL certificate problem: self-signed certificate in certificate chain" error, I know I'm in trouble. [1] https://www.zscaler.com/resources/security-terms-glossary/what-is-ssl-inspection https://www.zscaler.com/resources/security-terms-glossary/wh...
- nayuki 7mo ago
  Zscaler was deployed on a Windows 11 machine at a place that a friend worked at. When I assessed the software, its behavior was downright evil like malware. As we all know, it injects its own root certificate into the operating system in order to conduct man-in-the-middle attacks on TLS/HTTPS connections to monitor the user's web activity. Furthermore, it locks down the web browser's settings so that you cannot use a proxy server to bypass Zscaler's MITM. I saw this behavior in Mozilla Firefox, where the proxy option is set to "No proxy" and all other options are disabled and grayed out; I imagine that it does the same to Google Chrome. If you try to modify the browser's .ini(?) file for proxy settings, Zscaler immediately overwrites it against your will. Zscaler worked very hard to enforce its settings in order to spy on the computer user. And as you'd expect, if you open up the Zscaler GUI in the system tray, you are presented with the option to disable the software if you have the IT password. Which of course, you don't have. Then again, that might be an epsilon better than the Cybereason antivirus software, which just has a system tray icon with no exit option, and cannot be killed in Task Manager even if you are a local administrator, and imposes a heavy slowdown if you're open hundreds of small text files per second.
- pxc 7mo ago
  I feel the same way about Cisco Umbrella where I work. The worst breakage by far is protocol breakage; basically anything that uses HTTP as a basis for building some other protocol gets broken all the time. None of the people implementing it seem aware. They buy the vendor's claim that it's "transparent", when in fact even "inspect/trace-only" modes often break all kinds of shit. I've seen Umbrella break: - Git - RubyGems - `go mod` - OrbStack - Matrix - Cargo - all JDKs - Nix - Pkgsrc - all VMs and probably some other things I'm forgetting. When this breakage is reported, the first round of replies is typically "I visited that domain in my enterprise-managed browser and it's not blocked". That is, of course, a useless and irrelevant test. Often it takes hours to even fully diagnose the breakage with enough confidence to point the finger at that tool and not some other endpoint security tool. I'm not sure if the people buying and deploying tools in this category don't know how much stuff it breaks or just don't care. But the breakage is everywhere and nobody seems prepared for it.
yason 7mo ago
Despite the drawbacks of its grassroot nature TOFU goes a looooong way. With my own machines I can just physically check that the server host key matches what the ssh client sees. Once TOFU looks good I'm all set with that host because I don't change any of the keys ever. In a no-frills corporate unix environment it's enough to have a list of the internal servers' public keys listed on an internal website, accessible via SSL so it's effectively signed by a known corporate identity. You only need to check this list once to validate carrying out TOFU after which you can trust future connections. In settings with huge fleet of machines or in a very dynamic environment where new machines are rolled out all the time it probably makes things easier to use certificates. Of course, certificates come with some extra work and some extra features so the amount of benefit depend on the case. But at this scale TOFU is breaking down bad on multiple levels so you can't afford a strong opinion against certificates, really. I wish web browsers could remember server TLS host keys easily too and at least notify me whenever they change even if they'd still accept the new keys via ~trusted CAs.
- icedchai 6mo ago
  I've been using SSH since 1996 and I've never once seen anyone actually verify a public key manually like that. If they can MITM your SSH server, why can't they MITM your internal web site? In common practice, TOFU is type "yes" and YOLO.
gnufx 7mo ago
Life is easier if you can use Kerberos SSO, i.e. GSSAPIAuthentication in OpenSSH. (If we're talking certificates, presumably it is OpenSSH, or does anything else implement them?)
- tptacek 7mo ago
  Why would you do that rather than just hooking SSH up to a real IdP with certificates?
  - gnufx 6mo ago
    I don't want to have to get a special purpose credential when I have a TGT which can work generally, and is at least required for secure remote filesystem access. You have to manage extra infrastructure for certificates and, as a user, have the friction of firing up a JavaScript-enabled web browser via an additional tool, assuming "real IdP" means using OIDC. Unfortunately that flow is actually needed for remote systems and something like Edugain federation, since Moonshot/IETF ABFAB failed, but at least Shibboleth can use the TGT, and it's not the Globus horror.
    - tptacek 6mo ago
      Every serious shop of any real size is already managing an OIDC IdP (you need one for whatever SAAS apps your team is using along with any internal web applications you're using). Why not just link it to something that can issue short-lived SSH certificates? That's also the cleanest way to get strong multifactor auth for SSH (certificates issued only through an OIDC progress minted with MFA requirements). Setting up Kerberos in 2026 feels somewhat close to malpractice to me.
      - gnufx 6mo ago
        I'm happy for anyone who doesn't have MS Windows/Active Directory -- so Kerberos -- in their organization, but I'd need (Free)IPA or similar for user/access management anyway. Certificates are an extra layer of SSH-specific complexity, which concerns me for security even if it doesn't involve some third party. MFA is needed once a day, say, for SSO to all Kerberized services. [As I understand it, "managing an OIDC IdP" includes shipping the contents of Active Directory to Entra, heaven help us.] > Setting up Kerberos in 2026 feels somewhat close to malpractice to me. Microsoft (if that means anything, but they've done good work) and Red Hat obviously disagree, along with decades' experience. It is malpractice not to secure NFS mounts (and other network filesystems with sensitive data), and that means Kerberos.
tacostakohashi 7mo ago
One constant source of amazement for me is people not using ssh keys / using passwords with ssh. Especially at a BigCo, where there are different environments, with different passwords, and password expiry/rotation/complexity rules. Like, when asking for help, or working together... you say to them "ok, lets ssh to devfoo1234", and they do it, and then type in their password, and maybe get it wrong, then need to reset it, or whatever... and it takes half a minute or more for them to just ssh to some host. Maybe there are several hosts involved, and it all multiplies out. I mention to them "you know... i never use ssh passwords, i don't actually know my devfoo1234 password... maybe you should google for ssh-keygen, set it up, let me know if you have any problems?" and they're like "oh yeah, thats cool. i should do that sometime later!".... and then they never do, and they are forever messing with passwords. I just don't get it.
- pphysch 6mo ago
  Unregulated/decentralized SSH key usage (i.e. allowing ssh-copy-id) is a dream for hackers to move laterally through a network. That's why many orgs disable it, and otherwise haven't invested resources in getting a proper centralized CA/authz server set up.
- bombcar 6mo ago
  Keys is great for individual use, or for company use if you have centralized key control (and issue one or more keys per user). Often you either end up with one "dev ssh key" for all machines (which is bad) or you end up with people sharing around keys and unidentified keys on machines. Passwords at least are "simple" for people to work with.
  - elric 6mo ago
    > Often you either end up with one "dev ssh key" for all machines (which is bad) or you end up with people sharing around keys and unidentified keys on machines That hasn't been my experience at all. I've never encountered ssh key sharing in any environment, that would be insane. We enforced different ssh keys per environment at my previous company: fingerprint of your key would get logged, and if the SIEM detected a reuse of keys across environments (dev, test, prod, etc) you'd get a stern talking to.
znpy 6mo ago
It's very interesting that you can force the execution of a specific command. I wonder if that can be somehow employed to somehow get kubectl to invoke ssh in order to get an authentication tokens. I would be cool to authenticate to kubernetes via ssh.
jamiesonbecker 6mo ago
The experience might be better right up until you're running it in prod and someone happens to ask about: Cert revocation (or even expiration) Sudo roles User removal and process termination Is the cert server HA and locked down How you log in when the cert server is down or under attack (rich target!) How to easily add Alice to server group A, Bob to B, and Carlos to both A and B, and then to remove them.. (disclaimer we're celebrating our 15th anniversary at https://Userify.com https://Userify.com, but those are actually legit concerns and not only a sales pitch. You certainly can build a solid and secure ssh cert infra, but doing it in production is just not an easy set-it-and-forget-it sort of thing.)
- deleted 6mo ago
  [deleted]
- waynesonfire 6mo ago
  Sorry to pop your bubble, but a SaaS is the worst possible option.
  - jamiesonbecker 6mo ago
    Then install your own: curl i.userify.com | sudo - sE
antoniomika 6mo ago
We added SSH certificates support to pico.sh [1] and it's been great. Utilizing principals gave us the ability to implement a RBAC like system for specific parts of the pico.sh ecosystem. Users get the flexibility they want with limited complexity. [1] https://pico.sh/access-control#ssh-certificates https://pico.sh/access-control#ssh-certificates
azophy_2 6mo ago
shameless plug: I've been tinkering with a tool to make SSH certificate-based login a bit easier. it's called Sshifu. basically, you set up a sshifu-server that acts as a certificate authority + SSO server. then on your SSH servers, you configure them to trust this CA (there are helper npx commands / bash scripts to make this easy). after that, for each user who wants access, they just run: npx sshifu <sshifu-server-url> <ssh-server-url> this starts the SSO login flow, sets up the CA public key if needed, and immediately opens an SSH session. npx is just the easiest way to get started, there are other install options too. I built this as a smaller alternative to Smallstep / Teleport. it's still very early and mostly vibe-coded, but it's already scratching my own itch. would love to hear what you guys think repo: github.com/azophy/sshifu
linsomniac 6mo ago
This post inspired me to add SSH host key signing to WeEncrypt (formerly LessEncrypt): https://github.com/linsomniac/weencrypt https://github.com/linsomniac/weencrypt

268 points by jandeboevrie 7mo ago

Every couple of months someone re-discovers SSH certificates, and blogs about them. I'm guilty of it too. My blog post from 15 years ago is nowhere near as good as OP's post, but if I though me of 15 years ago lived up to my standards of today, I'd be really disappointed: https://blog.habets.se/2011/07/OpenSSH-certificates.html https://blog.habets.se/2011/07/OpenSSH-certificates.html

kaoD 7mo ago
I've known SSH certs for a while but never went through the effort of migrating away from keys. I'm very frustrated about manually managing my SSH keys across my different servers and devices though. I assume you gathered a lot of thoughts over these 15 years. Should I invest in making the switch?
- ibotty 7mo ago
  Yes. Caveat: It might not really be worth it if all your infrastructure is managed by these newfangled infrastructure-as-code-things that are quick to roll out (OpenShift/OKD, Talos, etc.) and you have only one repo to change SSH keys (single cluster or single repo for all clusters). There are some serious security benefits for larger organizations but it does not sound as if you are part of one.
- deleted 7mo ago
  [deleted]
- thomashabets2 7mo ago
  If your use case is such that you are frustrated about managing keys, host or user keys, then yes it does sound like SSH certs would help you. E.g. when you have many users, servers, or high enough cartesian product of the two. In environment where they don't cause frustration they're not worth it. Not really more to it than that, from my point of view.
- otabdeveloper4 7mo ago
  You will have to manage your SSH CA certificates instead of your keys. The workflows SSH CA's are extremely janky and insecure. With some creative use of `AuthorizedKeysCommand` you can make SSH key rotation painless and secure. With SSH certificates you have to go back to the "keys to the kingdom" antipattern and just hope for the best.
  - jamiesonbecker 7mo ago
    Exactly. We'd had discussions about building https://Userify.com https://Userify.com (plug!) around SSH certificates, but elected to go with keys instead, because Userify delivers most of the good things around certificates without the jank and insecurity. It's not that certificates themselves are insecure themselves, it's that the workflows (as the parent points out) are awful. We might still add some automation around that (and I think I saw some competitor tooling out there if you're committed to that path) but I personally feel like it's an answer to the wrong question.
  - cyberax 7mo ago
    > With SSH certificates you have to go back to the "keys to the kingdom" antipattern and just hope for the best. Whut? This is literally the opposite. With CA certs you can create short-lived certificates, so you can easily grant access to a system for a short time.
    - namibj 7mo ago
      And what about the CA?
      - cyberax 7mo ago
        It's no different compared to regular SSH private keys. You need to protect it from compromise. However, it provides you an additional layer of protection, because it does not need to be on the critical path for every SSH connection. My CA is a Nitrokey HSM, for example. I issue myself temporary certs that are valid only for 6 hours for ephemeral private keys.
- anyfoo 7mo ago
  A big problem I have with ssh carts is that they are not universally supported. For me, there is always some device or daemon (for example tinyssh in the initramfs of my gaming pc so that I can unlock it remotely) that only works with “plain old ssh keys”. And if I have to distribute and sync my keys onto a few hosts anyway, it takes away the benefits.
  - TZubiri 7mo ago
    Might actually be a positive instead of a negative. Gaming use-cases should have not any effect on security policies, these should be as separate as possible, different auth mechanisms for your gaming stuff and your professional stuff ensures nothing gets mixed.
    - nagaiaida 6mo ago
      remote unlock is also useful when you're not gaming so that feels like the wrong aspect to focus on
    - anyfoo 6mo ago
      Hah? It being my gaming machine has nothing to do with the problem. It’s also my FPGA development machine, though it gets used less for that. It only happens to be the only Linux workstation in my home (the others are Macs or OpenBSD).
      - TZubiri 6mo ago
        If you care about security, I recommend investing into a separate computer for developing hardware and software and another for downloading games on. You can setup your security any way you like, but nothing beats an air gap in terms of security and simplicity.
  - namibj 7mo ago
    Upgrade to a better one in initramfs?
  - AceJohnny2 6mo ago
- dizhn 7mo ago
  I am keeping an eye on the new (and alpha) Authentik agent which will allow idp based ssh logins. There's also SSSD already supported but it requires glibc (due to needing NSS) meaning it's not available on Alpine.
  - gnufx 7mo ago
    If you mean using OIDC, in that space there's at least https://github.com/EOSC-synergy/ssh-oidc https://github.com/EOSC-synergy/ssh-oidc, https://dianagudu.github.io/mccli/ https://dianagudu.github.io/mccli/ and OpenPubkey-ssh discussed in https://news.ycombinator.com/item?id=43470906 https://news.ycombinator.com/item?id=43470906 (which might mention more). How does SSSD support help with SSH authN? I know you can now get Kerberos tickets from FreeIPA using OIDC(?), but I forget if SSSD is involved.
    - floating-io 6mo ago
      Can't really speak to the point of the guy you're replying to, but the FreeIPA implementation via SSSD does more than just Kerberos tickets. Actually, I think the Kerberos based stuff as it relates to SSH is GSSAPI as part of sshd itself and has little to do with sssd, though I could be wrong. That said... If I'm remembering things correctly (and it's been a looong while since I've played with this), FreeIPA's client configures sshd with an AuthorizedKeysCommand that executes a program that queries sssd for the list of authorized keys for a given user. Sssd then uses a plugin to query the LDAP server @ FreeIPA to get the list of keys. There's also SSHFP (I think) records in DNS if you're using FreeIPA's DNS servers. These provide the host keys for servers for your ssh client to check against. Not sure if that's integrated into ssh itself or something else -- I can't remember how it's implemented offhand -- but it's fairly nifty since you never see the TOFU prompt (or it would be if DNS was actually secure, anyway).
      - gnufx 6mo ago
        Yes, FreeIPA is Kerberos+LDAP+X.509 CA, and GSSAPI is in OpenSSH (normally with the key exchange patch). SSSD is a local mechanism, not network authentication. I mentioned authorized keys distribution mechanisms elsewhere, but I was thinking authentication (c.f. OIDC), not authorization.
- cyberax 7mo ago
  It depends on what you want to do. CA certs are easy to manage, you just put the CA key instead of the SSH public key in authorized_keys. They also provide a way to get hardware-backed security without messing with SSH agent forwarding and crappy USB security devices. You can use an HSM to issue a temporary certificate for your (possibly temporary) public key and use it as normal. The certificate can be valid for just 1 hour, enough to not worry about it leaking.
papyDoctor 7mo ago
Another useful feature of SSH certificates is that you can sign a user’s public key to grant them access to a remote machine for a limited time and as a specific remote user.
- TZubiri 7mo ago
  The capacity to grant access as a specific remote user is present without certs as well right? The typical authorized_keys file lives under a user directory and grants access only to that user.
  - blueflow 7mo ago
    The main advantage of certificates is that you are able to do that from the CA without touching the target machine.
    - lokar 7mo ago
      Exactly. This is really useful in larger organizations where you may want more complex rules on access. For example, you can easily build "break glass" or 2nd party approved access on demand. You can put whatever logic you need in a CA front-end. You can also make all the certs short-lived (and only store them in ram).
      - TZubiri 6mo ago
        The way I've been doing that is with Shamir Secret Sharing and encrypting keys until glass-breaking is necessary.
        lokar 6mo ago
        generating tons of keys? or just broad keys? What I've done is generate a cert for the host(s) the user needs, for the time-span they need (subject to authorization logic).
    - pphysch 6mo ago
Stefan-H 7mo ago
I think the scary reality is most people conflate "keys" and "certificates". I have worked with security engineers that I need to remind that we do not use SSH certs, but rather key auth, and they have to think it through to make it click.
- tracker1 7mo ago
  I'm consistently amazed how many developers and security professionals don't have a clear understanding how PPK even works conceptually. Things like deploying dev keys to various production environments, instead of generating/registering them within said environment. One of the worst recent security examples... You can't get this data over HTTPS from $OtherAgency, it's "not secure" ... then their suggestion is a "secure" read-only account to the other agency's SQL server (which uses the same TLS 1.3 as HTTPS). This is from person in charge of digital security for a government org.
  - tolciho 7mo ago
    Or when the security team at some other company emails you their private key.
    - tracker1 6mo ago
      LOL, yeah.. had that happen quite a few times... Also, re-using the ssh server key for the client connecting to the sftp server.
  - laserlight 6mo ago
    > Things like deploying dev keys to various production environments, instead of generating/registering them within said environment. I can see this happening when a developer is authorized to generate, but not to register. So, they just reuse an already-registered one.
    - tracker1 6mo ago
      In the example, it wasn't even that complex... I have used patterns to register allowed signer keys based on environment variables that an application runs under, initializing at startup... so "register" just meant assigning the correct values for 2-4 environment variables per public signer allowed... and removing the dev signer. (JWT based auth)
V-eHGsd_ 7mo ago
oh man, I referred back to your blog post when I wrote the ssh certificate authority for $job ... ~10 years ago. Thank for writing it!
functional_dev 6mo ago
maybe it keeps getting rediscovered, becaues a good tooling around it is still missing?

Thom2000 7mo ago

Sadly services such as Github don't support these so it's mostly good for internal infrastructure.

lights0123 7mo ago
They do, for Enterprise customers only: https://docs.github.com/en/enterprise-cloud@latest/organizations/managing-git-access-to-your-organizations-repositories/about-ssh-certificate-authorities https://docs.github.com/en/enterprise-cloud@latest/organizat... They've rolled their host key one time, so there's little reason for them to use it on the host side.
- rpicard 6mo ago
  I’d love if they supported an SSH certificate CA for commit signing verification though.

linsomniac 7mo ago

In our dev/stg environment we reinstall half our machines every morning (largely to test our machine setup automation), and SSH host certificates make that so much nicer than having to persist host keys or remove/replace them in known_hosts. Highly recommended.

grave88 7mo ago
[flagged]

Tepix 7mo ago

The author lists all the advantes of CA certificates, yet doesn't list the disadvantages. OTOH, all the many steps required to set it up make the disadvantages rather obvious. Also, I've never had a security issue due to TOFU, have you?

GandalfHN 7mo ago
[flagged]
adrian_b 7mo ago
TOFU is convenient, but not necessary. Choosing to use TOFU is a distinct choice from the choice of using the keys generated by SSH, instead of using certificates. If you do not want to use TOFU, for extra security, you just have to pair the computers by copying between them the corresponding public keys through a secure channel, e.g. by using a USB memory. Using certificates does not add any simplification or any extra security. For real security, you still must pair the communicating computers by copying between them the corresponding certificates, through a secure channel, e.g. a USB memory. When you use for HTTPS the certificates that have come with your Internet browser, you trust that the installer package for the browser has come to that computer through a secure channel from the authority that has created the certificates. This is usually an assumption much more far fetched than the assumption that you can trust TOFU between computers under your control. Certificates may be useful in big organizations, if other functionality is needed beyond just establishing secure communication channels, e.g. if you want to use certificate revocation. In the list of "advantages" enumerated in the parent article, more than half of them are false, because if certificates are implemented correctly, completely equivalent actions must be executed when SSH keys without TOFU are used and when certificates are used. Perhaps the author meant by writing some of the "advantages" that the actions that supposedly are no longer needed with certificates are done by an administrator, not by the user. However that is also applicable with SSH. An administrator could install the certificates, so that no action is required from the user, but an administrator can also install the SSH public keys, so that no TOFU is ever needed from the user. Using certificates requires exactly the same steps like using keys generated by SSH (i.e. generating certificates and copying them between computers through secure channels, to pair the servers and the authorized users), but it may need additional steps, caused by the fact that certificates provide additional functionality.

jcalvinowens 7mo ago

You can also address TOFU to some extent using SSHFP DNS records. Openssh supports checking the DNSSEC signature in the client, in theory, but it's a configure option and I'm not sure if distros build with it.

jsiepkes 7mo ago
On top of that you would need something to secure DNS. Like DNSSEC or at the very least use DNS with TLS or DNS over HTTP. None of these are typically enabled by default.
- jcalvinowens 7mo ago
  Anything that uses system-resolved is probably doing DNSSEC validation by default. It's becoming much more common. Additionally, as I mentioned, openssh itself has support for validating the DNSSEC signature even if your local resolver doesn't. I actually don't think it can use the standard resolver for SSHFP records at all, but I'm not sure.
fc417fc802 7mo ago
Any idea if there's a standardized location, something like /.well-known/ssh?
- NicolaiS 6mo ago
  This exists, but not very well supported: https://github.com/C2SP/C2SP/blob/main/well-known-ssh-hosts.md https://github.com/C2SP/C2SP/blob/main/well-known-ssh-hosts.... But DNSSEC + SSHFP is a better supported solution at the moment

bobo56539 7mo ago

With the recent wave of npm hacks stealing private keys, I wanted to limit key's lifetimes. I've set up a couple of yubikeys as SSH CAs on hosts I manage. I use them to create short lived certs (say 24h) at the start of the day. This way i only have to enter the yubikey pin once a day. I could not find an easy way to limit maximum certificate lifetime in openssh, except for using the AuthorizedPrincipalCommand, which feels very fragile. Does anyone else have any experience with a similar setup? How do you limit cert max lifetime?

agwa 7mo ago
Instead of using a CA, why not set the key's PIN policy to "once" and use an agent (e.g. https://github.com/FiloSottile/yubikey-agent/ https://github.com/FiloSottile/yubikey-agent/) that holds an active session to the yubikey? You start the agent at the beginning of the day, enter the PIN once, and then stop the agent at the end of the day.
- bobo56539 7mo ago
  Wanted to avoid having the key in USB slot all the time. I have the version that sticks out and i carry it on my key chain. So it's easy to break in a laptop.
dplgk 6mo ago
In addition to ssh key, I have my servers also ask for a one-time password

moviuro 7mo ago

All those articles about SSH certificates fall short of explaining how the revocation list can/should be published. Is that yet another problem that I need to solve with syncthing? https://man.openbsd.org/ssh-keygen.1#KEY_REVOCATION_LISTS https://man.openbsd.org/ssh-keygen.1#KEY_REVOCATION_LISTS

blipvert 7mo ago
If you generate short lived certificates via an automated process/service then you don’t really need to manage a revocation list as they will have expired in short order.
- jamiesonbecker 7mo ago
  But then you can't log in if your box goes offline for any reason.
  - blipvert 7mo ago
    Hmm. For user certs you can have the service sign them for, say an hour, so long as you can ssh to your server in that time then there’s no need for any other interaction. Sure you need your signing service to be reasonably available, but that’s easily accomplished. Maybe I misunderstand?
    - jamiesonbecker 7mo ago
      That works for authn in the happy path: short-lived cert, grab it, connect, done. Except for everything around that: * user lifecycle (create/remove/rename accounts) * authz (who gets sudo, what groups, per-host differences) * cleanup (what happens when someone leaves) * visibility (what state is this box actually in right now?) SSH certs don’t really touch any of that. They answer can this key log in right now, not what should exist on this machine. So in practice, something else ends up managing users, groups, sudoers, home dirs, etc. Now there are two systems that both have to be correct. On the availability point: "reasonably available" is doing a lot of work ;) Even with 1-hour certs: * new sessions depend on the signer * fleet-wide issues hit everything at once * incident response gets awkward if the signer is part of the blast radius The failure mode shifts from a few boxes don't work to nobody can get in anywhere The pull model just leans the other way: * nodes converge to desired state * access continues even if control plane hiccups * authn and authz live together on the box Both models can work - it’s more about which failure mode is tolerable to you.

gunapologist99 7mo ago

Anyone tried out Userify? It creates/removes ssh pubkeys locally so (like a CA) no authn server needs to be online. But unlike certs, active sessions and processes are terminated when the user access is revoked.

jamiesonbecker 7mo ago
We're in the process of updating the experience to this century! ;) We've always taken the stance that crusty is better than vulnerable, but it turns out that not having a modern experience after 15 years is starting to feel like maybe we need to step up the features and shininess :)

sqbic 7mo ago

I've had very good experiences with SSH Communication Security company's (the guys who invented SSH) PrivX product to manage secure remote access, including SSH certificates and also cert based Windows authentication. It supports other kinds of remote targets too, via webui or with native clients. Great product.

jamiesonbecker 7mo ago

SSH certs quietly hurt in prod. Short-lived creds + centralized CA just moves complexity upward without solving the core problem: user management. The system shifts from many small local states to one highly coupled control point. That control point has to be correct and reachable all the time. When it isn’t, failures go wide instead of narrow. Example: a few boxes get popped and start hammering the CA. Now what? Access is broken everywhere at once. Common friction points: 1. your signer that has to be up and correct all the time 2. trust roots everywhere (and drifting) 3. TTL tuning nonsense (too short = random lockouts, too long = what was the point) 4. limited on-box state makes debugging harder than it should be 5. failures tend to fan out instead of staying contained Revocation is also kind of a lie. Just waiting for expiry and hoping that’s good enough. What actually happens is people reintroduce state anyway: sidecars, caches, agents… because you need it. We went the opposite direction: 1. nodes pull over outbound HTTPS 2. local authorized_keys is the source of truth locally 3. users/roles are visible on the box 4. drift fixes itself quickly 5. no inbound ports, no CA signatures (WELL, not strictly true*!) You still get central control, but operation and failure modes are local instead of "everyone is locked out right now." That’s basically what we do at Userify (https://userify.com https://userify.com). Less elegant than certs, more survivable at 2am. Also actually handles authz, not just part of authn. And the part that usually gets hand-waved with SSH CAs: 1. creating the user account 2. managing sudo roles 3. deciding what happens to home directories on removal 4. cleanup vs retention for compliance/forensics Those don’t go away - they're just not part of the certificate solution. * (TLS still exists here, just at the transport layer using the system trust store. That channel delivers users, keys, and roles. The rest is handled explicitly instead of implied.)

ngrilly 7mo ago
How do you solve TOFU?
- jamiesonbecker 7mo ago
  Well, TOFU is really just the model for how the chain of trust is established. In practice there isn’t really trust on first use: there’s verify the key matches what’s expected, or distribute keys out-of-band (including certs). If that verification step isn’t happening, then it’s not TOFU, it’s just blind trust. From an automation/autoscaling angle, the same thing shows up again: 1. either keys are pre-baked / distributed 2. or, something signs them at boot Signing an instance key is just another way of distributing trust. It doesn’t remove the need for a root of trust, it moves it. Certificates just add extra steps around the same underlying task.

longislandguido 7mo ago

This discussion is full of schizo solutions to "secure" SSH, most of which make no practical sense or have no technical basis. There really needs to be a definitive best practices guide published by a trusted authority.

kackerlacker 7mo ago
In my view it is more important to stop using software keys so probably use sk (fido) for both host and user.. From there CAs would be a next step.. The level of documentation and example setups is astoundingly poor if you even look at step 2 for any feature. I.e. SK keys are reasonably understood for user keys but the setup as host keys is vague and needs testing to see if it really works.
elric 6mo ago
Can you give some examples of which suggestions you think are schizo?

Serhii-Set 7mo ago

[dead]

TZubiri 7mo ago

>then I don’t need to type the target user’s password; instead I enter the key’s passphrase, a hopefully much more complicated combination of words, to unlock the private key. This sentence is a bit of a red flag, it looks like the author is making a (subtle) mistake in the category of too much security, or at least misjudging the amount of security (objectively measurable entropy) needed. This is of course a less consequential error than too little entropy/security measures, but still if one wants to be a cybersecurity professional, especially one with influence, they must know exactly the right amount needed, because our resources are limited, and each additional bit of entropy and security step not only costs time of the admin that implements it, but of the users that have to follow it, and this can even impact security itself by fatiguing the user and causing them to circumvent measures or ignore alerts. On to specifically what's wrong: Either a key file or a password can be used to log in to a server or authenticate to any service in general. Besides the technical implementation, the main difference is whether the secret is stored on the device, or in the user's brain. One is not more correct than the other, there's a lot of tradeoffs, one can ensure more bits and is more ergonomic, the other is not stored on device so it cannot be compromised that way. That said a 2FA approach, in whatever format, is (generally speaking) safer than any individual method, in that the two secrets are necessary to be granted access. In this scenario one needs both the file and the password to authenticate, even if the password is 4 digits long, that increases the security of the system when compared to no password. An attacker would have to setup a brute force attempt along with a way to verify the decryption was successful. If local decryption confirmation is not possible, then such a brute force attack would require to submit erroneous logins to the server potentially activating tripwires or alerting a monitoring admin. There's nothing special about the second factor authorization being equal or equivalent in entropy to the first, and there's especially no requirement that a password have more entropy when it's a second authorization, in fact it's the other way around. tl;dr You can consider each security mechanism in the wider context rather than in isolation and you will see security fatigue go down without compromising security.

aquafox 7mo ago

I work in a corporate setting and the money and time we wasted because of Zscaler and its SSL inspection [1] is beyond your wildest imagination. Whenever I see a "SSL certificate problem: self-signed certificate in certificate chain" error, I know I'm in trouble. [1] https://www.zscaler.com/resources/security-terms-glossary/what-is-ssl-inspection https://www.zscaler.com/resources/security-terms-glossary/wh...

nayuki 7mo ago
Zscaler was deployed on a Windows 11 machine at a place that a friend worked at. When I assessed the software, its behavior was downright evil like malware. As we all know, it injects its own root certificate into the operating system in order to conduct man-in-the-middle attacks on TLS/HTTPS connections to monitor the user's web activity. Furthermore, it locks down the web browser's settings so that you cannot use a proxy server to bypass Zscaler's MITM. I saw this behavior in Mozilla Firefox, where the proxy option is set to "No proxy" and all other options are disabled and grayed out; I imagine that it does the same to Google Chrome. If you try to modify the browser's .ini(?) file for proxy settings, Zscaler immediately overwrites it against your will. Zscaler worked very hard to enforce its settings in order to spy on the computer user. And as you'd expect, if you open up the Zscaler GUI in the system tray, you are presented with the option to disable the software if you have the IT password. Which of course, you don't have. Then again, that might be an epsilon better than the Cybereason antivirus software, which just has a system tray icon with no exit option, and cannot be killed in Task Manager even if you are a local administrator, and imposes a heavy slowdown if you're open hundreds of small text files per second.
pxc 7mo ago
I feel the same way about Cisco Umbrella where I work. The worst breakage by far is protocol breakage; basically anything that uses HTTP as a basis for building some other protocol gets broken all the time. None of the people implementing it seem aware. They buy the vendor's claim that it's "transparent", when in fact even "inspect/trace-only" modes often break all kinds of shit. I've seen Umbrella break: - Git - RubyGems - `go mod` - OrbStack - Matrix - Cargo - all JDKs - Nix - Pkgsrc - all VMs and probably some other things I'm forgetting. When this breakage is reported, the first round of replies is typically "I visited that domain in my enterprise-managed browser and it's not blocked". That is, of course, a useless and irrelevant test. Often it takes hours to even fully diagnose the breakage with enough confidence to point the finger at that tool and not some other endpoint security tool. I'm not sure if the people buying and deploying tools in this category don't know how much stuff it breaks or just don't care. But the breakage is everywhere and nobody seems prepared for it.

yason 7mo ago

Despite the drawbacks of its grassroot nature TOFU goes a looooong way. With my own machines I can just physically check that the server host key matches what the ssh client sees. Once TOFU looks good I'm all set with that host because I don't change any of the keys ever. In a no-frills corporate unix environment it's enough to have a list of the internal servers' public keys listed on an internal website, accessible via SSL so it's effectively signed by a known corporate identity. You only need to check this list once to validate carrying out TOFU after which you can trust future connections. In settings with huge fleet of machines or in a very dynamic environment where new machines are rolled out all the time it probably makes things easier to use certificates. Of course, certificates come with some extra work and some extra features so the amount of benefit depend on the case. But at this scale TOFU is breaking down bad on multiple levels so you can't afford a strong opinion against certificates, really. I wish web browsers could remember server TLS host keys easily too and at least notify me whenever they change even if they'd still accept the new keys via ~trusted CAs.

icedchai 6mo ago
I've been using SSH since 1996 and I've never once seen anyone actually verify a public key manually like that. If they can MITM your SSH server, why can't they MITM your internal web site? In common practice, TOFU is type "yes" and YOLO.

gnufx 7mo ago

Life is easier if you can use Kerberos SSO, i.e. GSSAPIAuthentication in OpenSSH. (If we're talking certificates, presumably it is OpenSSH, or does anything else implement them?)

tptacek 7mo ago
Why would you do that rather than just hooking SSH up to a real IdP with certificates?
- gnufx 6mo ago
  I don't want to have to get a special purpose credential when I have a TGT which can work generally, and is at least required for secure remote filesystem access. You have to manage extra infrastructure for certificates and, as a user, have the friction of firing up a JavaScript-enabled web browser via an additional tool, assuming "real IdP" means using OIDC. Unfortunately that flow is actually needed for remote systems and something like Edugain federation, since Moonshot/IETF ABFAB failed, but at least Shibboleth can use the TGT, and it's not the Globus horror.
  - tptacek 6mo ago
    Every serious shop of any real size is already managing an OIDC IdP (you need one for whatever SAAS apps your team is using along with any internal web applications you're using). Why not just link it to something that can issue short-lived SSH certificates? That's also the cleanest way to get strong multifactor auth for SSH (certificates issued only through an OIDC progress minted with MFA requirements). Setting up Kerberos in 2026 feels somewhat close to malpractice to me.
    - gnufx 6mo ago
      I'm happy for anyone who doesn't have MS Windows/Active Directory -- so Kerberos -- in their organization, but I'd need (Free)IPA or similar for user/access management anyway. Certificates are an extra layer of SSH-specific complexity, which concerns me for security even if it doesn't involve some third party. MFA is needed once a day, say, for SSO to all Kerberized services. [As I understand it, "managing an OIDC IdP" includes shipping the contents of Active Directory to Entra, heaven help us.] > Setting up Kerberos in 2026 feels somewhat close to malpractice to me. Microsoft (if that means anything, but they've done good work) and Red Hat obviously disagree, along with decades' experience. It is malpractice not to secure NFS mounts (and other network filesystems with sensitive data), and that means Kerberos.

tacostakohashi 7mo ago

One constant source of amazement for me is people not using ssh keys / using passwords with ssh. Especially at a BigCo, where there are different environments, with different passwords, and password expiry/rotation/complexity rules. Like, when asking for help, or working together... you say to them "ok, lets ssh to devfoo1234", and they do it, and then type in their password, and maybe get it wrong, then need to reset it, or whatever... and it takes half a minute or more for them to just ssh to some host. Maybe there are several hosts involved, and it all multiplies out. I mention to them "you know... i never use ssh passwords, i don't actually know my devfoo1234 password... maybe you should google for ssh-keygen, set it up, let me know if you have any problems?" and they're like "oh yeah, thats cool. i should do that sometime later!".... and then they never do, and they are forever messing with passwords. I just don't get it.

pphysch 6mo ago
Unregulated/decentralized SSH key usage (i.e. allowing ssh-copy-id) is a dream for hackers to move laterally through a network. That's why many orgs disable it, and otherwise haven't invested resources in getting a proper centralized CA/authz server set up.
bombcar 6mo ago
Keys is great for individual use, or for company use if you have centralized key control (and issue one or more keys per user). Often you either end up with one "dev ssh key" for all machines (which is bad) or you end up with people sharing around keys and unidentified keys on machines. Passwords at least are "simple" for people to work with.
- elric 6mo ago
  > Often you either end up with one "dev ssh key" for all machines (which is bad) or you end up with people sharing around keys and unidentified keys on machines That hasn't been my experience at all. I've never encountered ssh key sharing in any environment, that would be insane. We enforced different ssh keys per environment at my previous company: fingerprint of your key would get logged, and if the SIEM detected a reuse of keys across environments (dev, test, prod, etc) you'd get a stern talking to.

znpy 6mo ago

It's very interesting that you can force the execution of a specific command. I wonder if that can be somehow employed to somehow get kubectl to invoke ssh in order to get an authentication tokens. I would be cool to authenticate to kubernetes via ssh.

jamiesonbecker 6mo ago

The experience might be better right up until you're running it in prod and someone happens to ask about: Cert revocation (or even expiration) Sudo roles User removal and process termination Is the cert server HA and locked down How you log in when the cert server is down or under attack (rich target!) How to easily add Alice to server group A, Bob to B, and Carlos to both A and B, and then to remove them.. (disclaimer we're celebrating our 15th anniversary at https://Userify.com https://Userify.com, but those are actually legit concerns and not only a sales pitch. You certainly can build a solid and secure ssh cert infra, but doing it in production is just not an easy set-it-and-forget-it sort of thing.)

deleted 6mo ago
[deleted]
waynesonfire 6mo ago
Sorry to pop your bubble, but a SaaS is the worst possible option.
- jamiesonbecker 6mo ago
  Then install your own: curl i.userify.com | sudo - sE

antoniomika 6mo ago

We added SSH certificates support to pico.sh [1] and it's been great. Utilizing principals gave us the ability to implement a RBAC like system for specific parts of the pico.sh ecosystem. Users get the flexibility they want with limited complexity. [1] https://pico.sh/access-control#ssh-certificates https://pico.sh/access-control#ssh-certificates

azophy_2 6mo ago

shameless plug: I've been tinkering with a tool to make SSH certificate-based login a bit easier. it's called Sshifu. basically, you set up a sshifu-server that acts as a certificate authority + SSO server. then on your SSH servers, you configure them to trust this CA (there are helper npx commands / bash scripts to make this easy). after that, for each user who wants access, they just run: npx sshifu <sshifu-server-url> <ssh-server-url> this starts the SSO login flow, sets up the CA public key if needed, and immediately opens an SSH session. npx is just the easiest way to get started, there are other install options too. I built this as a smaller alternative to Smallstep / Teleport. it's still very early and mostly vibe-coded, but it's already scratching my own itch. would love to hear what you guys think repo: github.com/azophy/sshifu

linsomniac 6mo ago

This post inspired me to add SSH host key signing to WeEncrypt (formerly LessEncrypt): https://github.com/linsomniac/weencrypt https://github.com/linsomniac/weencrypt