5 ms·

BGP filter failure brings Aussie Internet down

85 points by chewxy 15y ago

jacques_chester 15y ago
Ah, that explains that particular mystery.
beedogs 15y ago
go figure, Dodo was involved.
bobbles 15y ago
I think this is the related thread on whirlpool (aussie broadband forums/news) http://forums.whirlpool.net.au/forum-replies.cfm?t=1871831 http://forums.whirlpool.net.au/forum-replies.cfm?t=1871831
DrStalker 15y ago
It's not the entire Australian internet that is down, just one of the major providers; it will also affect a lot of secondary providers that use Telstra for their upstream links.
outside1234 15y ago
telestra is a total disaster of an telco. not a surprise.
- elliotanderson 15y ago
  Maybe so, but in this case it looks like Dodo was the one that hosed them. Since the internet is designed to be as decentralised as possible, it relies on a certain level of trust when it comes to BGP advertisements. In this case Dodo advertised an incorrect routing table upstream to Telstra which then propagated it out as per the protocol.
  - i386 15y ago
    I am no expert, but it appears that Telstra had the ability to block the kind of erroneous BGP broadcast that Dodo made.
asharp 15y ago
For the non network admins amongst us: http://lists.ausnog.net/pipermail/ausnog/2012-February/012225.html http://lists.ausnog.net/pipermail/ausnog/2012-February/01222...
- marquis 15y ago
  that was fascinating. i thought i understood the internet but i followed through with discovering what the terminology meant and feel a few times more clever right now. also, overwhelmed with putting the pieces together in my mind about where decentralisation breaks down.
  - davidu 15y ago
    The lack of security and authenticity in BGP is really one of the great weak points of the Internet. Problems are resolved quickly, but problems and mistakes are easy to make which is the first order problem. There is work being done to better secure the BGP and prefix delegation / announcement infrastructure now: http://www.nanog.org/meetings/nanog49/abstracts.php?pt=MTU3NSZuYW5vZzQ5&nm=nanog49 http://www.nanog.org/meetings/nanog49/abstracts.php?pt=MTU3N...
davidu 15y ago
Since BGP related topics are super rare here... If any of you are programmers interested in writing code that impacts the global routing table, we're hiring at OpenDNS. We are AS36692 and anycast thousands of IPs, route a few dozen prefixes and have about 200+ neighbors across 14 datacenters. You would be hard pressed to find a network this large run by such a small team.
- B0Z 15y ago
  Uhh... Did Monster cancel your guys' account for overuse or something?
shimon_e 15y ago
Dodo! Dodo, internet that dies. Local internet was being routed via the US. I was visiting Australia as it happened. My connection had to route to the USA get to the ISP's own DNS servers!
shimon_e 15y ago
Here is a list of everyone who was affected http://bgp.he.net/AS1221#_peers http://bgp.he.net/AS1221#_peers
- nl 15y ago
  Meh.. Everyone in Australia peers with Telstra. "Affected" in this case in meaningless.
atoonk 15y ago
Telstra's outage visualized: http://www.bgpmon.net/telstra-feb23-2012.png http://www.bgpmon.net/telstra-feb23-2012.png National Australia Bank was completely down for 53 minutes as well. https://twitter.com/#!/bgpmon/status/172608854855647233/photo/1 https://twitter.com/#!/bgpmon/status/172608854855647233/phot...
timClicks 15y ago
Dodo?

85 points by chewxy 15y ago

jacques_chester 15y ago

Ah, that explains that particular mystery.

beedogs 15y ago

go figure, Dodo was involved.

bobbles 15y ago

I think this is the related thread on whirlpool (aussie broadband forums/news) http://forums.whirlpool.net.au/forum-replies.cfm?t=1871831 http://forums.whirlpool.net.au/forum-replies.cfm?t=1871831

DrStalker 15y ago

It's not the entire Australian internet that is down, just one of the major providers; it will also affect a lot of secondary providers that use Telstra for their upstream links.

outside1234 15y ago

telestra is a total disaster of an telco. not a surprise.

elliotanderson 15y ago
Maybe so, but in this case it looks like Dodo was the one that hosed them. Since the internet is designed to be as decentralised as possible, it relies on a certain level of trust when it comes to BGP advertisements. In this case Dodo advertised an incorrect routing table upstream to Telstra which then propagated it out as per the protocol.
- i386 15y ago
  I am no expert, but it appears that Telstra had the ability to block the kind of erroneous BGP broadcast that Dodo made.

asharp 15y ago

For the non network admins amongst us: http://lists.ausnog.net/pipermail/ausnog/2012-February/012225.html http://lists.ausnog.net/pipermail/ausnog/2012-February/01222...

marquis 15y ago
that was fascinating. i thought i understood the internet but i followed through with discovering what the terminology meant and feel a few times more clever right now. also, overwhelmed with putting the pieces together in my mind about where decentralisation breaks down.
- davidu 15y ago
  The lack of security and authenticity in BGP is really one of the great weak points of the Internet. Problems are resolved quickly, but problems and mistakes are easy to make which is the first order problem. There is work being done to better secure the BGP and prefix delegation / announcement infrastructure now: http://www.nanog.org/meetings/nanog49/abstracts.php?pt=MTU3NSZuYW5vZzQ5&nm=nanog49 http://www.nanog.org/meetings/nanog49/abstracts.php?pt=MTU3N...

davidu 15y ago

Since BGP related topics are super rare here... If any of you are programmers interested in writing code that impacts the global routing table, we're hiring at OpenDNS. We are AS36692 and anycast thousands of IPs, route a few dozen prefixes and have about 200+ neighbors across 14 datacenters. You would be hard pressed to find a network this large run by such a small team.

B0Z 15y ago
Uhh... Did Monster cancel your guys' account for overuse or something?

shimon_e 15y ago

Dodo! Dodo, internet that dies. Local internet was being routed via the US. I was visiting Australia as it happened. My connection had to route to the USA get to the ISP's own DNS servers!

shimon_e 15y ago

Here is a list of everyone who was affected http://bgp.he.net/AS1221#_peers http://bgp.he.net/AS1221#_peers

nl 15y ago
Meh.. Everyone in Australia peers with Telstra. "Affected" in this case in meaningless.

atoonk 15y ago

Telstra's outage visualized: http://www.bgpmon.net/telstra-feb23-2012.png http://www.bgpmon.net/telstra-feb23-2012.png National Australia Bank was completely down for 53 minutes as well. https://twitter.com/#!/bgpmon/status/172608854855647233/photo/1 https://twitter.com/#!/bgpmon/status/172608854855647233/phot...

timClicks 15y ago

Dodo?