3 ms·
1. Get physical access to the systems; employee badges were also offline, so they needed to bring in a different set of people with physical access to the premi
by PeterCorless 5y ago
1. Get physical access to the systems; employee badges were also offline, so they needed to bring in a different set of people with physical access to the premises.
2. Login directly to their systems and update the A-name record.
3. A-name records do not take effect immediately. There's a propagation delay as other DNS services accept the domain name.
4. Any sort of "damage assessment" to check the conditions of the now-offline systems. [Not a gating factor; you can turn things on, but you are likely doing "blast radius" analysis in parallel.]
5. Restart services one-by-one. Web servers, databases, API gateways, etc.
6. Allow external access; turn the systems back on for the world. [This is where they are currently, even though some parts of Facebook do not seem to be working too well. i.e., uploads and new posts seem broken still; at least they were for me.]
- jeffbee 5y agoIt doesn’t sound to me like this was a DNS outage. Their records existed but their name servers were unreachable because they’d withdrawn their BGP route. So I don’t think the rest of your comment is really consistent with the available information.