5 ms·

We have hundreds of thousands of servers and we also run sshd inside containers everywhere, so we probably have over a million endpoints we can SSH into. Cache

• hug 10y ago
  That's an extremely good answer to the question I posed. Thank you. Essentially the answer is that there's less immediate reliance on moving parts, which makes sense when you get to a certain level of scale and risk aversion -- a scale much greater than the one I work at, for sure.
• otterley 10y ago
  Why do you run sshd inside containers? I'm sort of surprised that Facebook does anything but run servers inside them. Running another init seems like an antipattern to me.
  • markild 10y ago
    Would be interested in hearing this as well. Could be that you would want to have a more generic interface for your containers, and not have to integrate too tightly with your container framework?
• otterley 10y ago
  > We don't accept the risk of not being able to login in a system. There are plenty of ways to mitigate the risk of LDAP server unavailability. Besides, the document said that the bastion server uses it, so why isn't what's good for the bastion good enough for the rest of the fleet?
  • timv 10y ago
    > why isn't what's good for the bastion good enough for the rest of the fleet? mfdutra's comment that you replied to said: If things go south really really bad, we can just get the private key and sign certificates by hand. That's obviously an incomplete answer - it doesn't explain how you connect to the server if you can't get onto the bastion due to a directory failure - but the basic premise is that the core system is isolated & contains everything it needs to handle the authentication process.
    • otterley 10y ago
      That doesn't answer my question at all.
      • timv 10y ago
        I'll assume that you didn't mean that to be as snarky as it sounds. The bastion host is not strictly necessary in the process. Although mfdutra didn't spell it out explicitly, they clearly have a means for accessing their core servers without needing to get on to the bastion - that's where the signing by hand comes in. The benefits of using LDAP in that case justify the cost of having an external dependency - by integrating with the central password store, the bastion provides a path into the system that delivers the desired security features (centralised identity management, auditing back to individual user accounts), but other paths can exist if needed. That trade-off isn't applicable to core servers. If they use LDAP (exclusively) for authentication and the directory is down then you're hosed. There's no other path into the server that you're trying to admin other than to get onto the server. You could use LDAP with a fallback to some other scheme (like signed certs, or a small set of locally defined users managed with a password vault, etc) but then you've got 2 different paths onto each server and both paths needs to be maintained, tested, and regularly audited. That's certainly possible, but why would you want to when you can just have 1 authentication mechanism that has zero dependencies.
        • otterley 10y ago
          I dislike guessing what someone's intent and rationale are, when we can simply ask, as we can here. In any event: First, the CA itself is a dependency: if it fails, no login certificates will issue, and users won't be able to log in. So they traded one kind of service dependency for another. Second, it hasn't been established that a backup login path is available. In many environments this may not even be acceptable (e.g. PCI and billing systems, which I know Facebook has).
• deleted 10y ago
  [deleted]
• andy_ppp 10y ago
  How many engineers can sign into live servers and read information about my Facebook account. Rather scarily it sounds like all of them! How many engineers have access/a copy of you global CA; if i get a copy of your global CA can I create a cert allowing me to log into any of the machines?