15 ms·
We need better support for SSH host certificates
- ptman 4y agoHow about delivering the public keys via another trusted path: https://paul.totterman.name/posts/ssh-pki-web-pki/ https://paul.totterman.name/posts/ssh-pki-web-pki/
- hartator 4y agoI like the `StrictHostKeyChecking accept-new` setting. Accept new host key without prompting the usual `Are you sure you want to continue connecting (yes/no/[fingerprint])?` (great when scripting on 100+ servers). Still reject known hosts mismatch (so shity Wifi can't inject their ads). Everything can be setting up again by just removing `~/.ssh/known_hosts` (instead of that GitHub messy blog post about curling and seding output to `~/.ssh/known_hosts`).
- zamnos 4y agoEw, no. The point that mjg59 points out, just not super explicitly and using many more words, is that the confirmation step wouldn't be necessary, if only all of our tooling were just a little bit better. Instead of ssh $ip_address and getting that prompt, instead, you'd do ssh $hostname, where hostname can be generated and contain numbers, and then the same mechanism that checks SSL certificates then also checks if the host's key actually matches what the rest of the system (the configured CAs) is saying it should be. If it does, great! No need to ask the user "Are you sure you want to continue connecting (yes/no/[fingerprint])", with fingerprint being the one thing to check out of band, but few people are really that fastidious. Competent Security/IT departments are able to get this to work going from corp managed endpoints (laptops) going to corp managed servers. It's just that the rest of the world hasn't caught up yet.
- Operyl 4y agoSSH certificates (host and client) were the best thing I could have ever implemented at my job, it has significantly improved a _ton_ of internal pain points.
- est31 4y ago> and then the same mechanism that checks SSL certificates then also checks if the host's key actually matches what the rest of the system (the configured CAs) is saying it should be. That's not what mjg59 is suggesting, to establish a trust chain to system shipped CAs. In fact that wouldn't be so easy anways because SSH certificates aren't X.509 compatible at all but use their completely homegrown specification. They explicitly say "please do not do this" about this idea. mjg59 suggests to make the TOFU approach on the client trust certificates if a certificate is provided. That's a way narrower suggestion: > OpenSSH has no way to do TOFU for CAs, just the keys themselves. This means there's no way to do a git clone ssh://git@github.com/whatever and get a prompt asking you to trust Github's CA. Instead, you need to add a @cert-authority github.com (key) line to your known_hosts file by hand, and since approximately nobody's going to do that there's only marginal benefit in going to the effort to implement this infrastructure. The most important thing we can do to improve the security of the SSH ecosystem is to make it easier to use certificates, and that means improving the behaviour of the clients.
- mjg59 4y agoTo be clear, I'm not suggesting trusting the certificate itself - I'm suggesting trusting the signing CA from the certificate, and so also trusting any future certificates for the same host signed by the same CA
- est31 4y agoI stand corrected. Thanks for pointing out the difference, I was using the terms "CA" and "certificate" interchangeably because in X.509 world CAs also have certificates of their own but it seems that OpenSSH only deals with CA keys, that is, certificates are always leaf certificates, and CAs consist of their keys only. > "Chained" certificates, where the signature key type is a certificate type itself are NOT supported. https://cvsweb.openbsd.org/src/usr.bin/ssh/PROTOCOL.certkeys?annotate=HEAD https://cvsweb.openbsd.org/src/usr.bin/ssh/PROTOCOL.certkeys...
- pyrolistical 4y agoSeems like a pretty way to solve this by having ssh clients check the TLS of the host and then fetch the public key in a well know sub path. But the hard part is getting everybody to support this. We need to start somewhere. Maybe GitHub can use this bad publicity and turn this into one good
- throwaway892238 4y ago> Seems like a pretty way to solve this by having ssh clients check the TLS of the host If you're doing TLS then you might as well do HTTPS. GitHub already supports HTTPS and way more features for it than SSH, and HTTPS works over more networks than SSH does. Continuing to use SSH is literally just being obsessed with a backwards old protocol for nostalgia reasons.
- Datagenerator 4y agoCorporate networks will happily do DPI on TLS, no so much with SSH. So this 'error' of GitHub might be intentional in order to mold everything into something which they can monitor.
- runeks 4y ago> Continuing to use SSH is literally just being obsessed with a backwards old protocol for nostalgia reasons. Honest question: Client authorization (for private repos) is an important feature of SSH. How does this work with HTTPS? EDIT: I just remembered the existence of TLS client certificate authentication. I wonder if it's possible to use an SSH client key for this (making the server accept a self-signed client certificate from the list of keys in authorized_keys).
- mjg59 4y agoThe usual way HTTPS client auth works is to issue some sort of persistent access token that then ends up sitting on the filesystem without any sort of additional protection, or alternatively to just fall back to username and password without any kind of MFA. It's somehow even worse than the SSH situation. The use of x509 client certs would be a huge improvement here, but I don't know of any git hosting services that support that.
- rhco 4y agoYes! SSH certificates are awesome, both for host- and client-verification. Avoiding Trust on First Use is potentially a big benefit, but the workflow improvements for developers, and especially non-technical people, is a huge win too. At work, we switched to Step CA [1] about 2 years ago. The workflow for our developers looks like: 1. `ssh client-hosts-01` 2. Browser window opens prompting for AzureAD login 3. SSH connection is accepted It really is that simple, and is extremely secure. During those 3 steps, we've verified the host key (and not just TOFU'd it!), verified the user identity, and verified that the user should have access to this server. In the background, we're using `@cert-authority` for host cert verification. A list of "allowed principals" is embedded in the users' cert, which are checked against the hosts' authorized_principals [2] file, so we have total control over who can access which hosts (we're doing this through Azure security groups, so it's all managed at our Azure portal). The generated user cert lasts for 24 hours, so we have some protection against stolen laptops. And finally, the keys are stored in `ssh-agent`, so they work seamlessly with any app that supports `ssh-agent` (either the new Windows named pipe style, or "pageant" style via winssh-pageant [3]) - for us, that means VSCode, DBeaver, and GitLab all work nicely. My personal wishlist addition for GitHub: Support for `@cert-authority` as an alternative to SSH/GPG keys. That would effectively allow us to delegate access control to our own CA, independent of GitHub. [1] https://smallstep.com/docs/step-ca https://smallstep.com/docs/step-ca [2] https://man.openbsd.org/sshd_config#AuthorizedPrincipalsFile https://man.openbsd.org/sshd_config#AuthorizedPrincipalsFile [3] https://github.com/ndbeals/winssh-pageant https://github.com/ndbeals/winssh-pageant
- OrvalWintermute 4y agoI've always thought of SSH host certificates as somewhat of an abomination and maybe I am in the minority because the better solution was always about making SSH speak x509 properly https://roumenpetrov.info/secsh/index.html https://roumenpetrov.info/secsh/index.html Roumen has been regularly updating this for a pretty long time now.
- vaporary 4y agoSSH host certificates as somewhat of an abomination Many would say x509 is the real abomination.
- OrvalWintermute 4y agoI might not disagree with you there. However, for all its warts, x509 due to hardware implementations, seems a great deal more secure than sitting on the FS SSH host certificates.
- fmajid 4y agoOpenSSH supports FIDO keys since 8.2p1 and has supported smart cards via GPG longer.
- Reitet00 4y agoYeah. Actually ssh agent speaks PKCS#11 (both client and server) so it's possible to interface with the hardware token quite easily. I'm using that to store my client key in TPM for example.
- aorth 4y agoCan't view this site on mobile. Constantly refreshing and redirecting to itself on Firefox / Android...
- throwaway892238 4y agoNo, you need to abolish SSH. It's an incredibly limiting protocol that does not support modern computing needs. Hacks and add-ons solve some but never all of its many problems. Wire-level network protocols like Wireguard are somewhat useful, but largely a large step away from the modern best practices. We need more Zero Trust, Federated Identity, Fine-grained Access Control, and Least Privilege. Those solutions exist, but they are almost always for-pay, because SSH is always used as the default option, and so no more effort is put into better security practices. And even without putting any real effort into a better protocol, you can just implement on top of HTTPS. Look at HTTPS+Git, compared to SSH+Git. First and foremost, this RSA key leak bullshit just wouldn't happen. Even if the TLS key on the server got leaked (and why the hell would it?! it gets generated automatically by Let's Encrypt), revoking and issuing a new one happens automatically for the client with no fraught extra options, and nobody's client configuration by default disables validating the certs! Then there's the fact that you can use a variety of AuthN+Z options, it goes over standard ports, and most providers give more fine-grained access control for it. Nerds love SSH. But it is literally worse than the alternatives, and is in practice often not used in a secure manner. Kill your darlings and use something demonstrably better.
- est 4y ago> you need to abolish SSH. It's an incredibly limiting protocol that does not support modern computing needs Yes! Please invent something else, and leave SSH alone.
- throwaway892238 4y agoI meant the protocol, not the program. A secure shell is still useful, but the old protocol is like a unix neckbeard that doesn't wanna learn containers. (And while we're on the subject, SSHD should support an HTTPS port and either serve a javascript client or accept websocket connections, because it is 2023 and that's what everyone wants anyway)
- jmspring 4y agoYou may need to do some more research. SSHD can run on an HTTP port, but why bundle a web server and additional protocols into something? There are web based ssh clients - https://github.com/billchurch/webssh2 https://github.com/billchurch/webssh2 - simple google search. Plenty of "neckbeards" understand containers and probably understand the underlying technologies (cgroups, etc) as well as comparable (or historical) approaches (jails, zones, etc) better than most. But, because something doesn't cater to your whims, must complicate things that actually work.
- purpleidea 4y agoSince the blog author didn't mention it, or doesn't know about it, fun fact, you can store host keys in DNS (make sure to use DNSSEC too of course!) https://en.wikipedia.org/wiki/SSHFP_record https://en.wikipedia.org/wiki/SSHFP_record Of course this has its own challenges, but if you automate your DNS, this can be neat! Cheers!
- xurukefi 4y agoThis is such a nice and pragmatic solution. And yet for some reason people would rather prefer the CA madness we know in the HTTPS ecosystem to be established in the SSH world. I understand that DANE was ahead of its time which is why we settled with CAs for SSL certificates but can we please take a different route this time for SSH?
- ShroudedNight 4y agoAs far as I can tell, DNS-based authentication puts all the trust eggs in the DNS root basket, and then again, in the TLD basket. This seems incredibly brittle and fraught with peril. Am I missing something?
- bawolff 4y agoThe usual counter-argument for https, is that the eggs are already in that basket since anyone with control of dns can just get a new certificate. For ssh that doesn't really apply as clearly though.
- xurukefi 4y ago> For ssh that doesn't really apply as clearly though. Do you think that DNS-based proof of ownership is not something that CAs would use for SSH certification?
- mjg59 4y agoGetting a cert requires someone to either compromise the primary DNS server for the domain, or to compromise DNS in multiple independent locations to serve consistent false answers to the probes. It's true that much of the TLS ecosystem is somewhat bound to DNS being trustworthy, but not to the same extent that SSHFP is.
- forty 4y agoI have had a pretty good experience with SSH certificates generated by Hashicorp Vault, both on the client and host sides (it was not for git but for general purpose SSH).
- AtNightWeCode 4y agoAnother problem with SSH is that people remove the key check completly in automations since some services just rotates the keys without telling. Or for being lazy.
- megous 4y agoThat's also solved with host CA. You can rotate the host keys, add new machines/keys however you want, and all that matters is whether the host keys are signed with a trusted CA, when you setup automation to trust that CA.
- AtNightWeCode 4y agoYes, that would work.
- deleted 4y ago[deleted]
- mschuster91 4y agoThere is SSHFP records in DNS that could be used... but unfortunately, Amazon Route53 doesn't support that record type [1] - or a bunch of other potentially useful DNS records. [1] https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/ResourceRecordTypes.html https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/Re...
- AtNightWeCode 4y agoAnother thing with SSH. Is there something like a host header in SSH to be used for blocking direct IP access?
- SuperSandro2000 4y agoNot that I know of
- SuperSandro2000 4y agoPart of the problem is also that ssh has host key rotation support but openssh client has it turned off by default
- lxgr 4y agoAt least on macOS and Debian it is on by default: > UpdateHostKeys is enabled by default if the user has not overridden the default UserKnownHostsFile setting and has not enabled VerifyHostKeyDNS, otherwise UpdateHostKeys will be set to no. It's not that useful for actual key rotations, though; the use case seems to be more geared towards switching to newer key types (e.g. adding an Ed25529 key seamlessly) than replacing a potentially compromised key. At least I couldn't find a straightforward way for the OpenSSH sshd to actually provide two keys of the same type to my client. All in all, I‘m not a fan of the feature – it seems to complicate key security for pretty marginal benefits. That said, it did auto-upgrade the Github key for me as far as I can tell!
- xurukefi 4y agoHow about something very pragmatic: Use the same key you use for your webserver, which already has a valid SSL cert. Clients first connect to port 443 to get the public key and can verify the certificate. During the SSH handshake, the client then compares the host key with the previously obtained key of the certificate.
- stpe 4y agoI wonder how many thousands (millions?) of man-hours of lost productivity over the world the Github SSH debacle resulted in. Not only for teams to fix the issue in a build pipeline no one touched for several years, but also the lost productivity for teams build/deploy pipelines to break, etc.
- nwellnhof 4y agoIt's just a tiny, tiny fraction of the productivity gained by using platforms like Github.
- fmajid 4y agoSSH doesn’t have a good revocation procedure. Perhaps integrating SSH certificates with OCSP is the way forward.
- yrro 4y agoAnyone else relying on GSSAPI key exchange & authentication instead?
- gnufx 4y agoI assume that means Kerberos, as opposed to GSSAPI mechanisms like (the moribund?) ABFAB, (now rightly deprecated?) Globus GSI, or even NTLM. I've used Kerberos GSSAPI with ssh long ago, but no longer can at work. Incidentally, people arguing for X.509 might consider why GSI is such a dirty word in the research computing circles I know! I think it's still widely used in HEP, and even the US Access system, though.
- yrro 4y agoYeah, Kerberos. Don't think I've ever seen the GSSAPI used for anything else!
- lbriner 4y agoI think the comments just make another proof that the OP assertion was correct and also that we live in a Wild West industry. One day, I hope there will be a much more standardised way of doing things properly, perhaps caveated with "more secure + more work" vs "Simple and OK secure". At the moment, there are too many strong opinions about 100 different "best" ways to do things leaving most of the rest of us just using whatever is easiest to find on Google rather than what the industry has discussed and approved.
- jerf 4y agoWhen we looked into it at work, one major problem we had with SSH certificates is that a cert can sign a key, but that's it. You can't have a cert that signs a cert that signs a key. So, the one and only cert that can sign keys still ends up having to be out there in a relatively live position, signing new keys as new machines come up and such. We really wanted to be able to create a cert, put it into super-cold storage, then sign a cert or possibly a number of other certs that could be used to sign keys. Then if one of those was compromised we could handle that particular cert, but the root cert could still be trusted by everything without breaking everything. We ended up assessing the current host certificate situation as bringing some benefit, but it was difficult to assert that the benefits were commensurate with the costs, especially considering the many other quirks, and we judged it as distinctly possible that if we did try to roll them out we'd find some other "quirk" was actually a significant stopper. It's not the same scale, but it's a very similar problem to what would happen if TLS could only sign at one degree of remove like that. If you occasionally take the time to poke through the padlock icon in your browser, you'll find the bare minimum trust chain you'll ever find is three; a root cert in your browser trust store, some signing cert, then the cert for the domain. I've seen some with more layers than that, but 3 is fairly common. Those 3 would be 2 in the SSH context since the last one would be the key rather than a cert. Can you imagine how much rougher it would be on the web if you could only have two levels, if a root cert could sign your site but that was it?
- DarkCrusader2 4y ago> (It's actually possible to glue the general PKI infrastructure into SSH certificates. Please do not do this) Can someone elaborate why? We are already depending on lists of known good CAs for everything (even banking). Why not leverage the same for SSH as well?
- mattpallissard 4y ago<sigh>, I really wish that the kerberos/gssapi folks had packaged everything up nicely for the web people to use. Between keytabs, derived credentials, and cross realm trust it really did solve nearly every authentication situation that people have. I pine for a world in which we had made it more approachable for the people who just wanted to quickly build an application without learning about the underlying infrastructure. I've been dealing with authn/authz for a long time and kerberos is still one of the best protocols in existence.
- mbreese 4y agoI've been thinking the same thing... all of this has been done before, and will be done again. With LDAP and Kerberos, many of these workflows were possible decades ago. But having servers connected to a centralized auth infrastructure wasn't popular (probably due to automated setups). And if you wanted TLS, you might even be working with an in-house CA with LDAPS (that's how I did it). Now we're swinging back to recognizing the benefits to some level of centralization in authentication. From a historical point of view, this all seems very familiar.
- gnufx 4y ago"I've long ago made up a corollary to Greenspun's tenth rule; any sufficiently complex or mature access regime will re-implement half of kerberos, poorly." -- cduzz, https://news.ycombinator.com/item?id=30798057 https://news.ycombinator.com/item?id=30798057
- gnufx 4y agoDefinitely <sigh>, particularly as Kerberos provides general single sign-on, rather than the uniform sign-on (just the same credentials for different services) that's typically labelled "SSO". SSH GSSAPI seems pretty simple, assuming you can populate the keytab. Is Kerberos setup really worse than alternatives for Apache (all I know)? However, the SPNEGO mechanism itself is cocked up in some way I've forgotten. One problem is people not wanting to expose the KDC to the WAN -- specifically Active Directory, as I recall Microsoft says. MIT have been doing that for ever and, for instance, Fedora uses FreeIPA over https.
- Valien 4y agoCan also use tools like Teleport that handles all of this programatically. Open-Source core as well (github.com/gravitational/teleport). Source: I work for Teleport so a little biased, but love the fact that more and more orgs are going away from static creds/keys and using short-lived certs and/or passwordless solutions. Spent many years in a production role and wish I had these tools (or was aware of these tools) back then.