2 ms·

Networking people will be shocked, SHOCKED, to hear that the root cause appears to have been a broken BGP push. Taking 14 hours to recover from that is all on

Networking people will be shocked, SHOCKED, to hear that the root cause appears to have been a broken BGP push. Taking 14 hours to recover from that is all on Optus though. https://en.wikipedia.org/wiki/2023_Optus_outage https://en.wikipedia.org/wiki/2023_Optus_outage And the damage went well beyond spotty emergency calls: for example, if you run a small business that relies on credit card payments, you were fucked if your terminals were on the Optus network. The situation was so bad that prepaid SIM cards for Telstra (the main competitor) were selling out in much of the country.

robocat 3y ago
From Wikipedia: Causes: A Border Gateway Protocol (BGP) routing problem played a role in the outage. Public data from CloudFlare showed a spike in BGP route announcements from the Optus network around the time the outage occurred — over 940,000 announcements in an hour from a node that normally makes less than 3,000 announcements per hour — indicative of a BGP routing problem. [snip] committee describes the outage as a gradual event triggered by loss of connectivity between neighbouring computer networks. The report suggests that approximately 90 edge provider routers disconnected as an automated protective measure against routing update overload. The failures occurred following a software upgrade at a North American Singtel exchange that caused one of the routers to disconnect. This, in turn, triggered Optus's routers to rapidly update its own routing tables which triggered the shutdown due to the pre-configured default threshold limits set by Cisco Systems being exceeded. The tabled report and Singtel stressed that the software upgrade was not the cause of the fault