3 ms·
So, experts, someone explain why it would take half a day to advertise your routes. What did they need to do?
by jeffbee 5y ago
So, experts, someone explain why it would take half a day to advertise your routes. What did they need to do?
- PeterCorless 5y ago1. Get physical access to the systems; employee badges were also offline, so they needed to bring in a different set of people with physical access to the premises. 2. Login directly to their systems and update the A-name record. 3. A-name records do not take effect immediately. There's a propagation delay as other DNS services accept the domain name. 4. Any sort of "damage assessment" to check the conditions of the now-offline systems. [Not a gating factor; you can turn things on, but you are likely doing "blast radius" analysis in parallel.] 5. Restart services one-by-one. Web servers, databases, API gateways, etc. 6. Allow external access; turn the systems back on for the world. [This is where they are currently, even though some parts of Facebook do not seem to be working too well. i.e., uploads and new posts seem broken still; at least they were for me.]
- jeffbee 5y agoIt doesn’t sound to me like this was a DNS outage. Their records existed but their name servers were unreachable because they’d withdrawn their BGP route. So I don’t think the rest of your comment is really consistent with the available information.
- reilly3000 5y agoNot an expert, but I have some theories: 1. Since all internal comms runs through email, messenger, and other systems all run under facebook.com, communication and coordination must have been extremely difficult. 2. eBGP servers/appliances that can serve 3 billion users must take a while to spin up. Assuming there was no cache, it would have to register thousands of internal services. 3. Per a now deleted reddit thread, the people who pushed the change and knew how to fix it weren't able to be onsite at the data center, so they were flying blind and working with lower level techs trying to implement their instructions. 4. DNS is usually heavily cached at multiple layers, but today they were getting DDOS'ed with DNS requests from FB clients. They would probably have to block those requests while the service is coming online so it doesn't just melt down again, and have some method of slowly allowing requests. How? IDK. Maybe host by host. 5. Whatever the case, some of the world's most sophisticated, highest paid engineers were under the gun to solve at $3000/second problem. I have to assume they were working as fast as they could.
- InsomniacL 5y agoThese systems are highly automated. Someone didn’t send an update with an empty table, hundreds of BGP updates were issued to remove blocks of addresses. So either it was sabotage or they had to fix the underlying issue which caused the system to remove the addresses in the first place.