4 ms·
Let's say you have three ISPs, Red, Green, and Blue. Red and Green have a cable connecting them somewhere, and Green and Blue have a cable connecting them somew
by geofft 9y ago
Let's say you have three ISPs, Red, Green, and Blue. Red and Green have a cable connecting them somewhere, and Green and Blue have a cable connecting them somewhere. Each of them send notices using BGP to each other saying, "Hi, this is my subnet, if you want to route to that subnet you should send packets to me." But Red and Blue also want to communicate to each other, so Green will also send a message to Red saying "Hey, I know how to reach Blue's subnet, I can indirectly route packets there too," and to Blue saying the same thing about Red's subnet.
Now if Red and Blue get a cable connecting the two of them, they'll start speaking BGP to each other. Green will continue advertising the indirect path, but Red's routers and Blue's routers will see that they have a more direct path to each other, and not go through green. So while network engineers need to tell their routers about their direct connections, BGP will automatically help distant ISPs figure out the best indirect paths.
It looks like what happened is that Google, which is not an ISP but peers directly with a bunch of ISPs (so they get better performance than being behind a single ISP, as most smaller companies do), started advertising routes that looked more efficient than the actual routes between various ISPs. Technically, those packets can flow over Google, although Google doesn't have the capacity to route traffic on behalf of the internet at large (it's not an ISP), and probably has routing rules configured to not actually accept packets that aren't destined for Google sites.
So two things happened. The first is that internet connectivity was disrupted. The second is that we got to see, a bit, what Google's peering relationships look like, because of the routes that Google advertised.
- exikyut 9y agoThanks for that explanation. I'm not 100% on BGP yet, although I do want to learn more about it. It sounds like Google's SDN (software-designed networking) stack had a glitch then. I read an overview of that a while ago, I think I got it from HN. https://www.nextplatform.com/2017/07/17/google-wants-rewire-internet/ https://www.nextplatform.com/2017/07/17/google-wants-rewire-... On a side note, I'm aware that BGP and similar "super-large-scale" infrastructure can be examined using publicly-accessible resources, but the sites out there are mostly geared toward people doing lookups for specific info, not people who want to learn. That's completely understandable, but as someone who learns more easily if something is tangible and "tinkerable", it's difficult for me to look at these sites and make the effort to [get oriented and then] piece everything together. How can I go "oooh, I get it" looking at the data from these sites? I'm also vaguely aware you can take the TCP/IP stacks on Linux (and probably Windows) completely to bits and put everything back together so the OS speaks BGP over Ethernet (or something equivalent) instead. It would be kind of cool if I could do that and then actually use the connection, with BGP setup as a replacement for standard IP addresses... if I could do that I'd actually leave everything configured that way, and thus be able to tinker with it. Sure, people don't generally connect their systems together via BGP - but BGP is used to transfer data (in the sense of "link-layer protocol"), right? I'm going to learn something.
- lmz 9y agoBGP does not transfer data. It is an application layer protocol between routers. You can run routing daemons on Linux if you want.
- geofft 9y agoBGP runs over TCP/IP, as a normal TCP service (port 179): it just tells you how to route packets. On 99% of machines you'll run into (your laptop, your servers at work, cloud VMs, etc.), they have exactly one network connection to one network provider, and so the routes are static, or at best provided by DHCP. For instance, at home my laptop knows that everything in 192.168.0.* can go directly over wifi, and everything else can be sent to 192.168.0.1. My router knows that everything should be sent to the ISP. And so forth. BGP is for network devices that have multiple connections to the Internet and need to decide which connection to use. They'll set up static IP routing to their immediate neighbors, make that TCP connection between their BGP daemons, and then let their BGP daemons configure the rest of the routing table. Most of the time these devices are backbone routers run by ISPs, but you can totally run BGP on your random Linux server. You'll need someone to peer with, though. I've used Quagga http://www.nongnu.org/quagga/ http://www.nongnu.org/quagga/ (although with OSPF, another routing protocol optimized for smaller-scale use) for implementing two separate web servers sharing the same IP address - that is, two servers claiming "Hey, I know how to reach this address, you can route through me," such that the service would stay up if either web server died - and it was mostly just `apt-get install quagga`, a small bit of config, and lots of help from the networking team doing some config on our intranet's routers to peer with my two servers and trust them.
- atmosx 9y agoSounds a lot like what FreeBSD CARP[1] suppose to do. [1] https://www.freebsd.org/doc/handbook/carp.html https://www.freebsd.org/doc/handbook/carp.html
- gerdesj 9y agoNot quite. CARP (VRRP in Ciscoese) has two (or more) systems capable of presenting the same IP address and they agree amongst themselves which gets to be the master through a multicast based advertizing system. Generally each system will be near clones of each other and be capable of doing the full job themselves. I have a pair of rather large pfSense systems running CARP as my office routers. I can update the secondary, test it, fail over the CARP addresses and fifty IPSEC tunnels, five internet connections, 10 OpenVPN servers along with rather a lot of clients, five OpenVPN clients, 30 odd internal VLANs and all the states for the above along with many NAT inbound and outbound sessions will seamlessly switch over thanks to PFSYNC. Even voice calls don't drop - you might notice a sub second wobble if someone is talking at the wrong moment. Rinse/repeat for the other one. BGP, RIPx, OSPF and co. are routing protocols that deal with the route your connections take through a network. There are also things like LACP/LAG for layer two redundancy. You can even play games with DNS round robin and of course reverse proxies. CARP is one tool in the box for coping with failures and maintenance - there are lots of others, each with advantages and disadvantages. That's what makes the job interesting 8)