8 ms·

BGP leaks causing internet outages in Japan and beyond

311 points by zakki 9y ago

imdsm 9y ago
What is BGP?
- deleted 9y ago
  [deleted]
- vitus 9y ago
  Border Gateway Protocol, which is the standard for routing on a super-organizational scale. There are two (main) flavors, but here eBGP (external) is the relevant one. Instead of explaining how it'll route traffic through itself, Google can tell its neighbors that it can reach, say, Vodafone (but with a certain cost). Importantly, this approach allows for autonomous systems to decide which routes to advertise to which neighbors. Content providers generally don't also provide transit, e.g. Google typically doesn't carry traffic from AT&T customers to Comcast customers free of charge, even if it is directly connected to both of them.
  - sillysaurus3 9y ago
    Now talk about how it used to be hackable, and how they fixed it. :) Actually, it's still hackable: https://www.theguardian.com/technology/2014/aug/07/hacker-bitcoin-mining-pool-internet-service-providers-canada-dell https://www.theguardian.com/technology/2014/aug/07/hacker-bi...
    - vitus 9y ago
      Of course it's still hackable. There's nothing stopping you from advertising specific prefixes and hijacking all of Youtube traffic globally. https://dyn.com/blog/pakistan-hijacks-youtube-1/ https://dyn.com/blog/pakistan-hijacks-youtube-1/ Oh wait, that happened almost a decade ago, and there's been no fundamental change to the protocol to protect against that. Except, if you keep something like that in place persistently, nobody will want to peer with you. * My understanding is that the global hijack was accidental. Pakistan Telecom just wanted to block Youtube traffic internally, and they used a bad mechanism for doing that. Oops.
      - notyourday 9y ago
        Oh rubbish. It is not hackable. Proper BGP implementations drop packets with TTL > 1 for non-multihop. This means that the only way to attack BGP between A and B is to have a control of the router on the other side. By doing that one's adversary controls the other side's inbound. It is always the case: Since my outbound is your inbound and since I control my outbound ( by definition ) then you cannot control your inbound. This means that the only defense is filtering.
- geofft 9y ago
  Let's say you have three ISPs, Red, Green, and Blue. Red and Green have a cable connecting them somewhere, and Green and Blue have a cable connecting them somewhere. Each of them send notices using BGP to each other saying, "Hi, this is my subnet, if you want to route to that subnet you should send packets to me." But Red and Blue also want to communicate to each other, so Green will also send a message to Red saying "Hey, I know how to reach Blue's subnet, I can indirectly route packets there too," and to Blue saying the same thing about Red's subnet. Now if Red and Blue get a cable connecting the two of them, they'll start speaking BGP to each other. Green will continue advertising the indirect path, but Red's routers and Blue's routers will see that they have a more direct path to each other, and not go through green. So while network engineers need to tell their routers about their direct connections, BGP will automatically help distant ISPs figure out the best indirect paths. It looks like what happened is that Google, which is not an ISP but peers directly with a bunch of ISPs (so they get better performance than being behind a single ISP, as most smaller companies do), started advertising routes that looked more efficient than the actual routes between various ISPs. Technically, those packets can flow over Google, although Google doesn't have the capacity to route traffic on behalf of the internet at large (it's not an ISP), and probably has routing rules configured to not actually accept packets that aren't destined for Google sites. So two things happened. The first is that internet connectivity was disrupted. The second is that we got to see, a bit, what Google's peering relationships look like, because of the routes that Google advertised.
  - exikyut 9y ago
    Thanks for that explanation. I'm not 100% on BGP yet, although I do want to learn more about it. It sounds like Google's SDN (software-designed networking) stack had a glitch then. I read an overview of that a while ago, I think I got it from HN. https://www.nextplatform.com/2017/07/17/google-wants-rewire-internet/ https://www.nextplatform.com/2017/07/17/google-wants-rewire-... On a side note, I'm aware that BGP and similar "super-large-scale" infrastructure can be examined using publicly-accessible resources, but the sites out there are mostly geared toward people doing lookups for specific info, not people who want to learn. That's completely understandable, but as someone who learns more easily if something is tangible and "tinkerable", it's difficult for me to look at these sites and make the effort to [get oriented and then] piece everything together. How can I go "oooh, I get it" looking at the data from these sites? I'm also vaguely aware you can take the TCP/IP stacks on Linux (and probably Windows) completely to bits and put everything back together so the OS speaks BGP over Ethernet (or something equivalent) instead. It would be kind of cool if I could do that and then actually use the connection, with BGP setup as a replacement for standard IP addresses... if I could do that I'd actually leave everything configured that way, and thus be able to tinker with it. Sure, people don't generally connect their systems together via BGP - but BGP is used to transfer data (in the sense of "link-layer protocol"), right? I'm going to learn something.
- tptacek 9y ago
  BGP associates IP networks (prefixes) to ASNs (numeric addresses representing collections of networks) and establishes the paths between ASNs. You can think of an ASN as representing a (routinely changing) bundle of different IP prefixes. A large ISP will have several ASNs. A very big company that manages its own multihomed connectivity might have one. Most companies on the Internet do not have one; their ISP handles it for them. The fact that virtually no ISP will peer with you† explains why you can't easily subscribe to two ISPs and multihome through both of them. BGP basically accomplishes two things: 1. It establishes which network operators have claims on which specific IP networks. 2. It establishes which network operators (via their ASNs) are connected to which other network operators, so that a participating router can figure out where to send traffic to an arbitrary IP address. † for random message board values of "you"
redm 9y ago
A really interesting read. I agree BGP leaks are a great risk to instability, but that could be said about any glitches that affect major backbones like NTT and Google (not a backbone but it's so well connected...). BGP Routing issues happened numerously, and will likely continue. Last year with Telia too had 4 or 5 "glitches" as they upgraded their network. [1] I talked to them about it and the mitigation is always the same, be more careful, peer review, additional filters, etc. Since each ISP implements BGP/Routing Tables/Topology in their own way, I'm not sure what you would do about this, other than choosing your peers carefully and filter any crazy route changes. [1] http://www.theregister.co.uk/2016/06/20/telia_engineer_blamed_massive_net_outage/ http://www.theregister.co.uk/2016/06/20/telia_engineer_blame...
- scurvy 9y ago
  The NTT in the article isn't the NTT America that you're thinking of. NTT America is the "backbone" company formed from the Verio purchase, then left to do their own thing largely outside the control of NTT Japan.
justinjlynn 9y ago
What ever happened to network ops implementing RFC7454/BCP38! > Google is not a transit provider and traffic for 3rd party networks should never go through the Google network. Why would you ever purposely configure your router to transit traffic via them?
- toast0 9y ago
  Google runs a couple different ASNs, so you might need to allow that as transit. BGP is notoriously easy to configure too open, but prefix limits might help; OTOH, dropping sessions to Google is going to cause a lot of headaches, so extra caution is required.
  - justinjlynn 9y ago
    That's certainly true. Generally speaking though, it's up to one's own network to know which ASs Google are routing for - and filter ranges not owned by any of those ASs. At worst you should route out of your known transit networks and (for the time google is being stupid) simply get non-optimal routing. Ideally your PNI agreement will include cost recovery for bad-acts like advertising non-owned space (but they probably won't) causing session drops and thus making your transit bill go up. Google being stupid (or any PNI, non-transit peer) should never cause your network to drop customer packets on the floor due to completely invalid routing.
    - notyourday 9y ago
      They can't. Google refuses to register its routes!
      - justinjlynn 9y ago
        If true, WTF Google...
        trapperkeeper74 9y ago
        That's the point. Google trying to be special instead of playing nicely with other kids.
blinkingled 9y ago
> necessity to have filters on both sides of an EBGP session. In this case it appears Verizon had little or no filters, and accepted most if not all BGP announcements from Google which lead to widespread service disruptions. At the minimum Verizon should probably have a maximum-prefix limit on their side and perhaps some as-path filters which would have prevented the wide spread impact. Wow that's just stupid on Verizon's part given the magnitude of potential impacts.
- akvadrako 9y ago
  It's not that stupid; it's an intentional tradeoff. It's also quite common, especially with a customer like Google.
  - blinkingled 9y ago
    Everybody (including Google as we find) makes mistakes and not having safeguards doesn't sound like a good trade-off to me. (What are they trading off here again?)
    - rnxrx 9y ago
      Operational complexity. For a carrier the size of Verizon to build an authoritative list of prefixes to advertise they would have to have a registration process for every single customer, peer, etc that was both reliable and up to the minute. That list would have to be compiled and driven on a near constant basis to many tens of thousands of routers and potentially hundreds of thousands of BGP sessions. It's not that this isn't doable, but it touches far more process in far more places than might be obvious and potentially requires additional capacity/capability on a lot of hardware. For a giant network this all means a great deal of time and money. Should they do it? Absolutely. Is it cheap/easy? Absolutely not. More cynically, does the potential cost and disruption of doing the right thing map favorably against their view of the risk of NOT doing it? Unknown.
      - jlgaddis 9y ago
notyourday 9y ago
That's because for all the talk about Google its network engineering drives networks like a buzzed 19 year old drives his fathers Porsche. Google does not believe in BGP filtering. They just don't. When someone brings up a BGP peering with Google that someone announces any prefixes to Google without first registering it. When asked "Huh? How do you ensure that I do not announce someone else's address space to you" Google's response becomes something akin to "We are Google, we have a very complicated system that prevents that from happening. It will detect the issue and address it automatically. We would build your filter lists based on those announcements" At the same time, the same people say that prefixes advertised to Google over PNIs take hours to propagate across the entire Google network. BGP filtering of prefixes to the address space registered to the peer is a basic hygiene, something that Google simply does not believe it has to do.
- fach 9y ago
  Based on the article, you probably should be s/Google/Verizon here. Yes, Google's export policy was misconfigured towards Verizon but Verizon blindly accepted and propagated these prefixes.
  - notyourday 9y ago
    Since Google does not register or filter routes it receives it is impossible for VZ or anyone else to know what are the "correct" routes that Google advertises v. what should be filtered. Look, we have been though this before, in 1994, 1995, 1996, 1997, etc. Sprint (1239) used to filter based on AS_PATH. They stopped after FLIX incident.
    - fach 9y ago
      They don't?: https://pastebin.com/5ngq1cJi https://pastebin.com/5ngq1cJi
      - notyourday 9y ago
        Right now Google announces slightly more than 440 IPv4 routes over PNIs. The list in radb is : whois -h whois.radb.net '!oMAINT-AS15169' | grep ^route | awk '{print $2}' | grep -v "::" | wc -l 6180 I picked four prefixes. 3 were in RADB. 1 was not.
thomas_howland 9y ago
Every time I see basic internet infrastructure experiencing issues, my default is to think about who is testing a new cyberweapon or censorship mechanism. BGP has been known as trust-reliant for a while.
- simondedalus 9y ago
  (: these aren't the droids you're looking for. taking down the internet (or specific pockets of it) via BGP is not a question of method, it's a question of access. BGP hijacking can be easily done by computer science grad students... state actors would not need to test a "cyberweapon" that messes with BGP routes. edit: the point being, how to do it is not an issue. they're not in the position to do it. ...but if you were thinking of censorship or outright disruptive terrorism via BGP, you'd be looking for infiltrating network operator jobs, not developing an attack. the attacks themselves are trivial, well-documented, and often happen accidentally. edit2: "sorry we broke the internet" http://seclists.org/nanog/1997/Apr/444 http://seclists.org/nanog/1997/Apr/444
  - notyourday 9y ago
    Internet Truth #33, as told by Mr. V: "If the interstate highway system ran the way we run internet, then in the middle of the rush hour a giant sinkhole would open in the middle of i495, swallow twenty thousand cars without a trace, a sinkhole would close and we would still call it a good day"
- campuscodi 9y ago
  "A while" being "two decades"
fundabulousrIII 9y ago
When I was doing routing and switching Cisco was still undisputed king and I never had to do anything more than IGP routing. Still, at the time, using two protocol stacks (IPX/SPX and TCP/IP) and redistribution into IS-IS using NLSP and RIP was painful. Still rather do that then deal with BGP.

311 points by zakki 9y ago

imdsm 9y ago

What is BGP?

deleted 9y ago
[deleted]
vitus 9y ago
Border Gateway Protocol, which is the standard for routing on a super-organizational scale. There are two (main) flavors, but here eBGP (external) is the relevant one. Instead of explaining how it'll route traffic through itself, Google can tell its neighbors that it can reach, say, Vodafone (but with a certain cost). Importantly, this approach allows for autonomous systems to decide which routes to advertise to which neighbors. Content providers generally don't also provide transit, e.g. Google typically doesn't carry traffic from AT&T customers to Comcast customers free of charge, even if it is directly connected to both of them.
- sillysaurus3 9y ago
  Now talk about how it used to be hackable, and how they fixed it. :) Actually, it's still hackable: https://www.theguardian.com/technology/2014/aug/07/hacker-bitcoin-mining-pool-internet-service-providers-canada-dell https://www.theguardian.com/technology/2014/aug/07/hacker-bi...
  - vitus 9y ago
    Of course it's still hackable. There's nothing stopping you from advertising specific prefixes and hijacking all of Youtube traffic globally. https://dyn.com/blog/pakistan-hijacks-youtube-1/ https://dyn.com/blog/pakistan-hijacks-youtube-1/ Oh wait, that happened almost a decade ago, and there's been no fundamental change to the protocol to protect against that. Except, if you keep something like that in place persistently, nobody will want to peer with you. * My understanding is that the global hijack was accidental. Pakistan Telecom just wanted to block Youtube traffic internally, and they used a bad mechanism for doing that. Oops.
    - notyourday 9y ago
      Oh rubbish. It is not hackable. Proper BGP implementations drop packets with TTL > 1 for non-multihop. This means that the only way to attack BGP between A and B is to have a control of the router on the other side. By doing that one's adversary controls the other side's inbound. It is always the case: Since my outbound is your inbound and since I control my outbound ( by definition ) then you cannot control your inbound. This means that the only defense is filtering.
geofft 9y ago
Let's say you have three ISPs, Red, Green, and Blue. Red and Green have a cable connecting them somewhere, and Green and Blue have a cable connecting them somewhere. Each of them send notices using BGP to each other saying, "Hi, this is my subnet, if you want to route to that subnet you should send packets to me." But Red and Blue also want to communicate to each other, so Green will also send a message to Red saying "Hey, I know how to reach Blue's subnet, I can indirectly route packets there too," and to Blue saying the same thing about Red's subnet. Now if Red and Blue get a cable connecting the two of them, they'll start speaking BGP to each other. Green will continue advertising the indirect path, but Red's routers and Blue's routers will see that they have a more direct path to each other, and not go through green. So while network engineers need to tell their routers about their direct connections, BGP will automatically help distant ISPs figure out the best indirect paths. It looks like what happened is that Google, which is not an ISP but peers directly with a bunch of ISPs (so they get better performance than being behind a single ISP, as most smaller companies do), started advertising routes that looked more efficient than the actual routes between various ISPs. Technically, those packets can flow over Google, although Google doesn't have the capacity to route traffic on behalf of the internet at large (it's not an ISP), and probably has routing rules configured to not actually accept packets that aren't destined for Google sites. So two things happened. The first is that internet connectivity was disrupted. The second is that we got to see, a bit, what Google's peering relationships look like, because of the routes that Google advertised.
- exikyut 9y ago
  Thanks for that explanation. I'm not 100% on BGP yet, although I do want to learn more about it. It sounds like Google's SDN (software-designed networking) stack had a glitch then. I read an overview of that a while ago, I think I got it from HN. https://www.nextplatform.com/2017/07/17/google-wants-rewire-internet/ https://www.nextplatform.com/2017/07/17/google-wants-rewire-... On a side note, I'm aware that BGP and similar "super-large-scale" infrastructure can be examined using publicly-accessible resources, but the sites out there are mostly geared toward people doing lookups for specific info, not people who want to learn. That's completely understandable, but as someone who learns more easily if something is tangible and "tinkerable", it's difficult for me to look at these sites and make the effort to [get oriented and then] piece everything together. How can I go "oooh, I get it" looking at the data from these sites? I'm also vaguely aware you can take the TCP/IP stacks on Linux (and probably Windows) completely to bits and put everything back together so the OS speaks BGP over Ethernet (or something equivalent) instead. It would be kind of cool if I could do that and then actually use the connection, with BGP setup as a replacement for standard IP addresses... if I could do that I'd actually leave everything configured that way, and thus be able to tinker with it. Sure, people don't generally connect their systems together via BGP - but BGP is used to transfer data (in the sense of "link-layer protocol"), right? I'm going to learn something.
tptacek 9y ago
BGP associates IP networks (prefixes) to ASNs (numeric addresses representing collections of networks) and establishes the paths between ASNs. You can think of an ASN as representing a (routinely changing) bundle of different IP prefixes. A large ISP will have several ASNs. A very big company that manages its own multihomed connectivity might have one. Most companies on the Internet do not have one; their ISP handles it for them. The fact that virtually no ISP will peer with you† explains why you can't easily subscribe to two ISPs and multihome through both of them. BGP basically accomplishes two things: 1. It establishes which network operators have claims on which specific IP networks. 2. It establishes which network operators (via their ASNs) are connected to which other network operators, so that a participating router can figure out where to send traffic to an arbitrary IP address. † for random message board values of "you"

redm 9y ago

A really interesting read. I agree BGP leaks are a great risk to instability, but that could be said about any glitches that affect major backbones like NTT and Google (not a backbone but it's so well connected...). BGP Routing issues happened numerously, and will likely continue. Last year with Telia too had 4 or 5 "glitches" as they upgraded their network. [1] I talked to them about it and the mitigation is always the same, be more careful, peer review, additional filters, etc. Since each ISP implements BGP/Routing Tables/Topology in their own way, I'm not sure what you would do about this, other than choosing your peers carefully and filter any crazy route changes. [1] http://www.theregister.co.uk/2016/06/20/telia_engineer_blamed_massive_net_outage/ http://www.theregister.co.uk/2016/06/20/telia_engineer_blame...

scurvy 9y ago
The NTT in the article isn't the NTT America that you're thinking of. NTT America is the "backbone" company formed from the Verio purchase, then left to do their own thing largely outside the control of NTT Japan.

justinjlynn 9y ago

What ever happened to network ops implementing RFC7454/BCP38! > Google is not a transit provider and traffic for 3rd party networks should never go through the Google network. Why would you ever purposely configure your router to transit traffic via them?

toast0 9y ago
Google runs a couple different ASNs, so you might need to allow that as transit. BGP is notoriously easy to configure too open, but prefix limits might help; OTOH, dropping sessions to Google is going to cause a lot of headaches, so extra caution is required.
- justinjlynn 9y ago
  That's certainly true. Generally speaking though, it's up to one's own network to know which ASs Google are routing for - and filter ranges not owned by any of those ASs. At worst you should route out of your known transit networks and (for the time google is being stupid) simply get non-optimal routing. Ideally your PNI agreement will include cost recovery for bad-acts like advertising non-owned space (but they probably won't) causing session drops and thus making your transit bill go up. Google being stupid (or any PNI, non-transit peer) should never cause your network to drop customer packets on the floor due to completely invalid routing.
  - notyourday 9y ago
    They can't. Google refuses to register its routes!
    - justinjlynn 9y ago
      If true, WTF Google...
      - trapperkeeper74 9y ago
        That's the point. Google trying to be special instead of playing nicely with other kids.

blinkingled 9y ago

> necessity to have filters on both sides of an EBGP session. In this case it appears Verizon had little or no filters, and accepted most if not all BGP announcements from Google which lead to widespread service disruptions. At the minimum Verizon should probably have a maximum-prefix limit on their side and perhaps some as-path filters which would have prevented the wide spread impact. Wow that's just stupid on Verizon's part given the magnitude of potential impacts.

akvadrako 9y ago
It's not that stupid; it's an intentional tradeoff. It's also quite common, especially with a customer like Google.
- blinkingled 9y ago
  Everybody (including Google as we find) makes mistakes and not having safeguards doesn't sound like a good trade-off to me. (What are they trading off here again?)
  - rnxrx 9y ago
    Operational complexity. For a carrier the size of Verizon to build an authoritative list of prefixes to advertise they would have to have a registration process for every single customer, peer, etc that was both reliable and up to the minute. That list would have to be compiled and driven on a near constant basis to many tens of thousands of routers and potentially hundreds of thousands of BGP sessions. It's not that this isn't doable, but it touches far more process in far more places than might be obvious and potentially requires additional capacity/capability on a lot of hardware. For a giant network this all means a great deal of time and money. Should they do it? Absolutely. Is it cheap/easy? Absolutely not. More cynically, does the potential cost and disruption of doing the right thing map favorably against their view of the risk of NOT doing it? Unknown.
    - jlgaddis 9y ago

notyourday 9y ago

That's because for all the talk about Google its network engineering drives networks like a buzzed 19 year old drives his fathers Porsche. Google does not believe in BGP filtering. They just don't. When someone brings up a BGP peering with Google that someone announces any prefixes to Google without first registering it. When asked "Huh? How do you ensure that I do not announce someone else's address space to you" Google's response becomes something akin to "We are Google, we have a very complicated system that prevents that from happening. It will detect the issue and address it automatically. We would build your filter lists based on those announcements" At the same time, the same people say that prefixes advertised to Google over PNIs take hours to propagate across the entire Google network. BGP filtering of prefixes to the address space registered to the peer is a basic hygiene, something that Google simply does not believe it has to do.

fach 9y ago
Based on the article, you probably should be s/Google/Verizon here. Yes, Google's export policy was misconfigured towards Verizon but Verizon blindly accepted and propagated these prefixes.
- notyourday 9y ago
  Since Google does not register or filter routes it receives it is impossible for VZ or anyone else to know what are the "correct" routes that Google advertises v. what should be filtered. Look, we have been though this before, in 1994, 1995, 1996, 1997, etc. Sprint (1239) used to filter based on AS_PATH. They stopped after FLIX incident.
  - fach 9y ago
    They don't?: https://pastebin.com/5ngq1cJi https://pastebin.com/5ngq1cJi
    - notyourday 9y ago
      Right now Google announces slightly more than 440 IPv4 routes over PNIs. The list in radb is : whois -h whois.radb.net '!oMAINT-AS15169' | grep ^route | awk '{print $2}' | grep -v "::" | wc -l 6180 I picked four prefixes. 3 were in RADB. 1 was not.

thomas_howland 9y ago

Every time I see basic internet infrastructure experiencing issues, my default is to think about who is testing a new cyberweapon or censorship mechanism. BGP has been known as trust-reliant for a while.

simondedalus 9y ago
(: these aren't the droids you're looking for. taking down the internet (or specific pockets of it) via BGP is not a question of method, it's a question of access. BGP hijacking can be easily done by computer science grad students... state actors would not need to test a "cyberweapon" that messes with BGP routes. edit: the point being, how to do it is not an issue. they're not in the position to do it. ...but if you were thinking of censorship or outright disruptive terrorism via BGP, you'd be looking for infiltrating network operator jobs, not developing an attack. the attacks themselves are trivial, well-documented, and often happen accidentally. edit2: "sorry we broke the internet" http://seclists.org/nanog/1997/Apr/444 http://seclists.org/nanog/1997/Apr/444
- notyourday 9y ago
  Internet Truth #33, as told by Mr. V: "If the interstate highway system ran the way we run internet, then in the middle of the rush hour a giant sinkhole would open in the middle of i495, swallow twenty thousand cars without a trace, a sinkhole would close and we would still call it a good day"
campuscodi 9y ago
"A while" being "two decades"

fundabulousrIII 9y ago

When I was doing routing and switching Cisco was still undisputed king and I never had to do anything more than IGP routing. Still, at the time, using two protocol stacks (IPX/SPX and TCP/IP) and redistribution into IS-IS using NLSP and RIP was painful. Still rather do that then deal with BGP.