10 ms·

I thought this was mostly a solved problem? SSSD or FreeIPA and a bunch of LDAP servers. Cached credentials will keep you online in the case of a transient auth

by hug 10y ago

I thought this was mostly a solved problem? SSSD or FreeIPA and a bunch of LDAP servers. Cached credentials will keep you online in the case of a transient authentication issue. HA LDAP is pretty much a solved problem as well. Say what you want about various Microsoft server products, Active Directory is second to none in this regard, and is a highly flexible and extremely robust service. Am I missing something specific about Facebook that makes this not an option?

skywhopper 10y ago
I haven't read this article closely yet, but based on other similar protocols, I _think_ the key element is, can you still log in to your servers when all your LDAP is catastrophically down for some reason. Sure, it's unlikely, but for someone like Facebook that's an unacceptable risk. This also means you don't have to actually _worry_ about maintaining an HA LDAP infrastructure that can support all the SSH auth you need.
- hug 10y ago
  That exact scenario is covered by the concept of "cached credentials" in the case of SSSD. If your authentication services or down (or the box you're attempting to log into is offline) there's a period in which you can log in using the last used password for your account. Granted you only have the configurable cache time out in which to solve your problem, but that's usually on the matter of days. If your authentication infrastructure is down for longer than that you have bigger fish to fry.
  - subway 10y ago
    Cached credentials only save the day if the cache has been populated. Engineers should be logging in infrequently enough that cache is rarely populated.
    - deathanatos 10y ago
      Is it not possible to force the cache to have a particular set of users (say, critical devops folks) always populated?
windowsworkstoo 10y ago
I guess they are worried about total network isolation for some hosts...even if the LDAP infra is up, if there is a segment isolated and you haven't logged in for a while, cached creds won't help. That said, if it's that isolated how are you connecting to it in the first place :) I agree with you though, we just use AD and SSSD, with the additional step of having extended our AD schema to store pubkeys and sudo roles for the Linux part of our infra.
tptacek 10y ago
Signed certificates work even when your AAA system is entirely offline. An individual server can make a valid authentication/authorization decision just by validating the signature. There are some significant advantages to working that way.
mfdutra 10y ago
We have hundreds of thousands of servers and we also run sshd inside containers everywhere, so we probably have over a million endpoints we can SSH into. Cache is only good for things you login frequently, like your workstation or a handful of servers. We don't accept the risk of not being able to login in a system. Our only dependency is a signed certificate. If things go south really really bad, we can just get the private key and sign certificates by hand.
- hug 10y ago
  That's an extremely good answer to the question I posed. Thank you. Essentially the answer is that there's less immediate reliance on moving parts, which makes sense when you get to a certain level of scale and risk aversion -- a scale much greater than the one I work at, for sure.
- otterley 10y ago
  Why do you run sshd inside containers? I'm sort of surprised that Facebook does anything but run servers inside them. Running another init seems like an antipattern to me.
  - markild 10y ago
    Would be interested in hearing this as well. Could be that you would want to have a more generic interface for your containers, and not have to integrate too tightly with your container framework?
- otterley 10y ago
  > We don't accept the risk of not being able to login in a system. There are plenty of ways to mitigate the risk of LDAP server unavailability. Besides, the document said that the bastion server uses it, so why isn't what's good for the bastion good enough for the rest of the fleet?
  - timv 10y ago
    > why isn't what's good for the bastion good enough for the rest of the fleet? mfdutra's comment that you replied to said: If things go south really really bad, we can just get the private key and sign certificates by hand. That's obviously an incomplete answer - it doesn't explain how you connect to the server if you can't get onto the bastion due to a directory failure - but the basic premise is that the core system is isolated & contains everything it needs to handle the authentication process.

by hug 10y ago

skywhopper 10y ago

I haven't read this article closely yet, but based on other similar protocols, I _think_ the key element is, can you still log in to your servers when all your LDAP is catastrophically down for some reason. Sure, it's unlikely, but for someone like Facebook that's an unacceptable risk. This also means you don't have to actually _worry_ about maintaining an HA LDAP infrastructure that can support all the SSH auth you need.

hug 10y ago
That exact scenario is covered by the concept of "cached credentials" in the case of SSSD. If your authentication services or down (or the box you're attempting to log into is offline) there's a period in which you can log in using the last used password for your account. Granted you only have the configurable cache time out in which to solve your problem, but that's usually on the matter of days. If your authentication infrastructure is down for longer than that you have bigger fish to fry.
- subway 10y ago
  Cached credentials only save the day if the cache has been populated. Engineers should be logging in infrequently enough that cache is rarely populated.
  - deathanatos 10y ago
    Is it not possible to force the cache to have a particular set of users (say, critical devops folks) always populated?

windowsworkstoo 10y ago

I guess they are worried about total network isolation for some hosts...even if the LDAP infra is up, if there is a segment isolated and you haven't logged in for a while, cached creds won't help. That said, if it's that isolated how are you connecting to it in the first place :) I agree with you though, we just use AD and SSSD, with the additional step of having extended our AD schema to store pubkeys and sudo roles for the Linux part of our infra.

tptacek 10y ago

Signed certificates work even when your AAA system is entirely offline. An individual server can make a valid authentication/authorization decision just by validating the signature. There are some significant advantages to working that way.

mfdutra 10y ago

We have hundreds of thousands of servers and we also run sshd inside containers everywhere, so we probably have over a million endpoints we can SSH into. Cache is only good for things you login frequently, like your workstation or a handful of servers. We don't accept the risk of not being able to login in a system. Our only dependency is a signed certificate. If things go south really really bad, we can just get the private key and sign certificates by hand.

hug 10y ago
That's an extremely good answer to the question I posed. Thank you. Essentially the answer is that there's less immediate reliance on moving parts, which makes sense when you get to a certain level of scale and risk aversion -- a scale much greater than the one I work at, for sure.
otterley 10y ago
Why do you run sshd inside containers? I'm sort of surprised that Facebook does anything but run servers inside them. Running another init seems like an antipattern to me.
- markild 10y ago
  Would be interested in hearing this as well. Could be that you would want to have a more generic interface for your containers, and not have to integrate too tightly with your container framework?
otterley 10y ago
> We don't accept the risk of not being able to login in a system. There are plenty of ways to mitigate the risk of LDAP server unavailability. Besides, the document said that the bastion server uses it, so why isn't what's good for the bastion good enough for the rest of the fleet?
- timv 10y ago
  > why isn't what's good for the bastion good enough for the rest of the fleet? mfdutra's comment that you replied to said: If things go south really really bad, we can just get the private key and sign certificates by hand. That's obviously an incomplete answer - it doesn't explain how you connect to the server if you can't get onto the bastion due to a directory failure - but the basic premise is that the core system is isolated & contains everything it needs to handle the authentication process.