3 ms·
Not an expert, but I have some theories: 1. Since all internal comms runs through email, messenger, and other systems all run under facebook.com, communication
by reilly3000 5y ago
Not an expert, but I have some theories:
1. Since all internal comms runs through email, messenger, and other systems all run under facebook.com, communication and coordination must have been extremely difficult.
2. eBGP servers/appliances that can serve 3 billion users must take a while to spin up. Assuming there was no cache, it would have to register thousands of internal services.
3. Per a now deleted reddit thread, the people who pushed the change and knew how to fix it weren't able to be onsite at the data center, so they were flying blind and working with lower level techs trying to implement their instructions.
4. DNS is usually heavily cached at multiple layers, but today they were getting DDOS'ed with DNS requests from FB clients. They would probably have to block those requests while the service is coming online so it doesn't just melt down again, and have some method of slowly allowing requests. How? IDK. Maybe host by host.
5. Whatever the case, some of the world's most sophisticated, highest paid engineers were under the gun to solve at $3000/second problem. I have to assume they were working as fast as they could.