15 ms·
Scalable and secure access with SSH
- thenewwazoo 10y agoThis looks very similar to Netflix' BLESS package[1] [1] HN discussion at https://news.ycombinator.com/item?id=11746425 https://news.ycombinator.com/item?id=11746425
- spydum 10y agoYes, I was coming to post the exact same thing.
- nsheridan 10y agoFor (possibly) smaller scale I wrote a self-service SSH CA: https://github.com/nsheridan/cashier https://github.com/nsheridan/cashier
- hug 10y agoI thought this was mostly a solved problem? SSSD or FreeIPA and a bunch of LDAP servers. Cached credentials will keep you online in the case of a transient authentication issue. HA LDAP is pretty much a solved problem as well. Say what you want about various Microsoft server products, Active Directory is second to none in this regard, and is a highly flexible and extremely robust service. Am I missing something specific about Facebook that makes this not an option?
- skywhopper 10y agoI haven't read this article closely yet, but based on other similar protocols, I _think_ the key element is, can you still log in to your servers when all your LDAP is catastrophically down for some reason. Sure, it's unlikely, but for someone like Facebook that's an unacceptable risk. This also means you don't have to actually _worry_ about maintaining an HA LDAP infrastructure that can support all the SSH auth you need.
- hug 10y agoThat exact scenario is covered by the concept of "cached credentials" in the case of SSSD. If your authentication services or down (or the box you're attempting to log into is offline) there's a period in which you can log in using the last used password for your account. Granted you only have the configurable cache time out in which to solve your problem, but that's usually on the matter of days. If your authentication infrastructure is down for longer than that you have bigger fish to fry.
- subway 10y agoCached credentials only save the day if the cache has been populated. Engineers should be logging in infrequently enough that cache is rarely populated.
- deathanatos 10y agoIs it not possible to force the cache to have a particular set of users (say, critical devops folks) always populated?
- windowsworkstoo 10y agoI guess they are worried about total network isolation for some hosts...even if the LDAP infra is up, if there is a segment isolated and you haven't logged in for a while, cached creds won't help. That said, if it's that isolated how are you connecting to it in the first place :) I agree with you though, we just use AD and SSSD, with the additional step of having extended our AD schema to store pubkeys and sudo roles for the Linux part of our infra.
- tptacek 10y agoSigned certificates work even when your AAA system is entirely offline. An individual server can make a valid authentication/authorization decision just by validating the signature. There are some significant advantages to working that way.
- mfdutra 10y agoWe have hundreds of thousands of servers and we also run sshd inside containers everywhere, so we probably have over a million endpoints we can SSH into. Cache is only good for things you login frequently, like your workstation or a handful of servers. We don't accept the risk of not being able to login in a system. Our only dependency is a signed certificate. If things go south really really bad, we can just get the private key and sign certificates by hand.
- hug 10y agoThat's an extremely good answer to the question I posed. Thank you. Essentially the answer is that there's less immediate reliance on moving parts, which makes sense when you get to a certain level of scale and risk aversion -- a scale much greater than the one I work at, for sure.
- otterley 10y agoWhy do you run sshd inside containers? I'm sort of surprised that Facebook does anything but run servers inside them. Running another init seems like an antipattern to me.
- markild 10y agoWould be interested in hearing this as well. Could be that you would want to have a more generic interface for your containers, and not have to integrate too tightly with your container framework?
- otterley 10y ago> We don't accept the risk of not being able to login in a system. There are plenty of ways to mitigate the risk of LDAP server unavailability. Besides, the document said that the bastion server uses it, so why isn't what's good for the bastion good enough for the rest of the fleet?
- timv 10y ago> why isn't what's good for the bastion good enough for the rest of the fleet? mfdutra's comment that you replied to said: If things go south really really bad, we can just get the private key and sign certificates by hand. That's obviously an incomplete answer - it doesn't explain how you connect to the server if you can't get onto the bastion due to a directory failure - but the basic premise is that the core system is isolated & contains everything it needs to handle the authentication process.
- skywhopper 10y agoIf I'm reading this right, it looks like this improves over Netflix BLESS IIRC. My concern with something like BLESS was that it allows you in to root or some deploy/app user based on a cert trust chain, but doesn't seem to trace back to the actual user who's making the changes and/or differentiate between two users logged in at once. This embeds the user information in the cert. Looks nice.
- AceJohnny2 10y agoTangent: "When designing a security system, regardless of purpose or protocol, you need to think of authentication and authorization separately. Authentication securely verifies that the client is who it claims to be, but does not grant any permissions. After the successful authentication, authorization decides whether or not the client can perform a specific action." These two operations are often abbreviated "authn" (for autheNtication) and "authz" (for authoriZation) in security frameworks.
- mfdutra 10y agoIn fact, we use a lot the terms authn, authz, authnz and AAA at Facebook. :-)
- AceJohnny2 10y agoBut not referred to in the article, which I found surprising since they're such common terms for those in the know. :)
- beagle3 10y agoWhen I was doing security work, AAA stood for "authorization, authentication and audit"
- otterley 10y agoThe last one was also known as "accounting."
- vacri 10y agoHow confusing for those of us in British English countries! (or should I say 'confuzing'? :p )
- lox 10y agoHow do they handle revocation? The problem I had with this approach in the past was no CRL support.
- mfdutra 10y agoWe can use RevokedKeys for that, but in fact we normally issue short-lived certificates. If we ever have a mass certificate leak, we'll just rotate the entire CA.
- mfdutra 10y agoAlso, we bind certificates to the hosts that requested them. If a certificate and its private key move somewhere else, they will be useless. ssh-keygen -O source-address=1.2.3.4...
- lox 10y agoInteresting, so I'm assuming this wouldn't work for staff that roam with laptops?
- mfdutra 10y agoWe only do that in bastion servers. It would be a bit trickier with laptops, but doable I guess.
- lox 10y agoMakes sense. I missed that section, much clearer now, thanks. Any plans to release more details on how the bastion hosts are configured? Seems like that would be the complex bit to get right.
- mfdutra 10y agoAbsolutely nothing special about them. They trust the same LDAP/Kerberos infra the laptops are part of. We have two-factor authentication, which is pretty simple with PAM. And they're only accessible if you are in our corporate network.
- ianunruh 10y agoAnother promising solution that uses these principles is Teleport (http://gravitational.com/teleport http://gravitational.com/teleport)
- mcpherrinm 10y agoHow do you manage what servers clients trust? I've seen a few SSH CA projects for signing user certs, but not for the server side certificates. When you're running containers, if they're spawned frequently, that can mean a significant amount of known_hosts file churning. It sounds like everyone funnels through a small set of bastions, so you'd only need to update them there, but I'm wondering if anyone else uses an SSH CA to solve that side of the equation? All the ingredients needed are in `ssh-keygen`, but that doesn't feel super awesome to me. Solving server identity is why I built https://github.com/square/sharkey https://github.com/square/sharkey
- knorker 10y agoYou've not seen that? It's HostCertificate in sshd_config. Then in known_hosts: @cert-authority *.example.com ssh-rsa AAAAB… https://www.digitalocean.com/community/tutorials/how-to-create-an-ssh-ca-to-validate-hosts-and-clients-with-ubuntu https://www.digitalocean.com/community/tutorials/how-to-crea... Edit: Oh, you mean for making them more short lived.
- mcpherrinm 10y agoYou can trust a CA easily enough. So do you just glob some shell scripts around ssh-keygen? Short-lived is one solution to limiting their lifetime: the other is to use a CRL format. Either way, you need software to manage the issuance and later revocation. I suppose you could build this into your host imaging profile, or use config management software. I'm just interested in what people do.
- cakoose 10y ago> We do not place SSH certificates on laptops because it is difficult to control everything that individual employees run on them. Even though they are centrally managed, laptops are more prone to vulnerabilities than the bastion servers. Therefore, we do not trust them with SSH private keys. If the laptop doesn't have an SSH private key, how do you SSH to the bastion host? > In your .ssh/ directory, you'll see id_ecdsa and id_ecdsa.pub. Copy the latter to the CA server and get it signed. Because this is public information, the transport isn't important. You can copy and paste, or fax it; just don't copy id_ecdsa anywhere. The transfer of id_ecdsa.pub doesn't need secrecy, but it does need integrity. You don't want to accidentally sign an attacker's public key.
- timv 10y ago> If the laptop doesn't have an SSH private key, how do you SSH to the bastion host? These hosts use centralized LDAP and Kerberos installations to share account information, and they require two-factor authentication to protect against password leakage It appears that you use your password + some unspecified second factor.
- crypt1d 10y agoThe implementation seems quite novel, but I can't shake the feeling that its a bit over-engineered. The additional complexity of this over a standard solution (eg FreeIPA + SSSD) may introduce other issues that need to be mitigated. The approach of using certificates over a cache like sssd is just shifting the issue from cache expiration over to certificate expiration. With certificates, you are likely to set some period where the user certificates expire, and if u have an expired certificate and CA is down u still cannot log in. If you dont expire your certificates u have a security risk. With FreeIPA you can also map public keys to specific users and have sshd pull authorized_keys from SSSD instead of the actual file. So if a user leaves the system, his keys get removed. So no need to worry about old keys lingering on your servers. As somebody already mentioned, LDAP (which FreeIPA uses as backend) has a battle-proven HA capability. With some sane configurations of the underlying servers (separate physical equipment, networking, etc), you don't need to worry about your backend mysteriously going down.
- deleted 10y ago[deleted]
- jorangreef 10y agoI found this really useful. It's lightweight, simple and powerful. However, some pieces of advice in the article could prove catastrophic: "You need to distribute ca.pub to your entire fleet. Remember, this is meant to be public, so complete access lockdown isn't the goal." Complete access lockdown is vital. You need to prevent the CA's public key from being altered, corrupted or deleted across your fleet. When distributing the CA's public key, you definitely need to use a secure transport. The CA's public key is what your system of trust is built on. You need to make sure you are trusting the right thing. "Copy the latter to the CA server and get it signed. Because this is public information, the transport isn't important. You can copy and paste, or fax it" When copying the user's public key to the server, the transport is still important, otherwise a MITM could swap the public key and your CA would happily sign the wrong thing.
- knorker 10y agoHow anticlimactic. Nothing new or internal, just "hey here's what's in the OpenSSH manpage: TrustedUserCAKeys and AuthorizedPrincipalsFile".