6 ms·

Thanks for that explanation. I'm not 100% on BGP yet, although I do want to learn more about it. It sounds like Google's SDN (software-designed networking) sta

Thanks for that explanation. I'm not 100% on BGP yet, although I do want to learn more about it. It sounds like Google's SDN (software-designed networking) stack had a glitch then. I read an overview of that a while ago, I think I got it from HN. https://www.nextplatform.com/2017/07/17/google-wants-rewire-internet/ https://www.nextplatform.com/2017/07/17/google-wants-rewire-... On a side note, I'm aware that BGP and similar "super-large-scale" infrastructure can be examined using publicly-accessible resources, but the sites out there are mostly geared toward people doing lookups for specific info, not people who want to learn. That's completely understandable, but as someone who learns more easily if something is tangible and "tinkerable", it's difficult for me to look at these sites and make the effort to [get oriented and then] piece everything together. How can I go "oooh, I get it" looking at the data from these sites? I'm also vaguely aware you can take the TCP/IP stacks on Linux (and probably Windows) completely to bits and put everything back together so the OS speaks BGP over Ethernet (or something equivalent) instead. It would be kind of cool if I could do that and then actually use the connection, with BGP setup as a replacement for standard IP addresses... if I could do that I'd actually leave everything configured that way, and thus be able to tinker with it. Sure, people don't generally connect their systems together via BGP - but BGP is used to transfer data (in the sense of "link-layer protocol"), right? I'm going to learn something.

lmz 9y ago
BGP does not transfer data. It is an application layer protocol between routers. You can run routing daemons on Linux if you want.
geofft 9y ago
BGP runs over TCP/IP, as a normal TCP service (port 179): it just tells you how to route packets. On 99% of machines you'll run into (your laptop, your servers at work, cloud VMs, etc.), they have exactly one network connection to one network provider, and so the routes are static, or at best provided by DHCP. For instance, at home my laptop knows that everything in 192.168.0.* can go directly over wifi, and everything else can be sent to 192.168.0.1. My router knows that everything should be sent to the ISP. And so forth. BGP is for network devices that have multiple connections to the Internet and need to decide which connection to use. They'll set up static IP routing to their immediate neighbors, make that TCP connection between their BGP daemons, and then let their BGP daemons configure the rest of the routing table. Most of the time these devices are backbone routers run by ISPs, but you can totally run BGP on your random Linux server. You'll need someone to peer with, though. I've used Quagga http://www.nongnu.org/quagga/ http://www.nongnu.org/quagga/ (although with OSPF, another routing protocol optimized for smaller-scale use) for implementing two separate web servers sharing the same IP address - that is, two servers claiming "Hey, I know how to reach this address, you can route through me," such that the service would stay up if either web server died - and it was mostly just `apt-get install quagga`, a small bit of config, and lots of help from the networking team doing some config on our intranet's routers to peer with my two servers and trust them.
- atmosx 9y ago
  Sounds a lot like what FreeBSD CARP[1] suppose to do. [1] https://www.freebsd.org/doc/handbook/carp.html https://www.freebsd.org/doc/handbook/carp.html
  - gerdesj 9y ago
    Not quite. CARP (VRRP in Ciscoese) has two (or more) systems capable of presenting the same IP address and they agree amongst themselves which gets to be the master through a multicast based advertizing system. Generally each system will be near clones of each other and be capable of doing the full job themselves. I have a pair of rather large pfSense systems running CARP as my office routers. I can update the secondary, test it, fail over the CARP addresses and fifty IPSEC tunnels, five internet connections, 10 OpenVPN servers along with rather a lot of clients, five OpenVPN clients, 30 odd internal VLANs and all the states for the above along with many NAT inbound and outbound sessions will seamlessly switch over thanks to PFSYNC. Even voice calls don't drop - you might notice a sub second wobble if someone is talking at the wrong moment. Rinse/repeat for the other one. BGP, RIPx, OSPF and co. are routing protocols that deal with the route your connections take through a network. There are also things like LACP/LAG for layer two redundancy. You can even play games with DNS round robin and of course reverse proxies. CARP is one tool in the box for coping with failures and maintenance - there are lots of others, each with advantages and disadvantages. That's what makes the job interesting 8)
  - geofft 9y ago
    Yes, but in my case I wanted the two servers to be located in physically separate parts of our datacenter to guard against a switch failing (ideally it would have been split across multiple datacenters), and I think that OpenBSD's CARP, or similar protocols like Cisco's VRRP, can only handle the case of computers on the same subnet sharing another IP address on that subnet. The two machines were on different subnets, and there was a third /24 we assigned virtual addresses from. By making the user-facing address for the web server its own 32 and having the two servers say "Hey, if you want to reach 10.255.255.1 you can route through me" over OSPF, all the switches on the way from a client to the two servers would be able to handle the case of the next hop failing, and let traffic take the other route to the same IP address. (Never mind that 10.255.255.1 was independently served by two Apache processes on the two machines; for all they know it was actually being routed to a third computer directly hooked up to the first two.)
jlgaddis 9y ago
Download and install GNS3, set up a few virtual routers, configure them to exchange NLRI via BGP and play with it all you want to. You're not going to find anyone who will let you establish a BGP session with them to exchange "real" routes -- there's simply too much risk involved for no valid purpose (and with no benefit to them).
- buzer 9y ago
  There's https://dn42.us/ https://dn42.us/ that's pretty good for studying BGP in slightly more real-world scenarios. Of course, you won't be getting internet transit routes from those peers (usually at least).
  - exikyut 9y ago
    Thanks, this is awesome!! I'm totally going to be checking this out and saying hi/connecting at some point in the future (when I have a server etc).
devdas 9y ago
You can do this on internal subnets. On the public internet, hopefully not without safeguards :)

lmz 9y ago

BGP does not transfer data. It is an application layer protocol between routers. You can run routing daemons on Linux if you want.

geofft 9y ago

BGP runs over TCP/IP, as a normal TCP service (port 179): it just tells you how to route packets. On 99% of machines you'll run into (your laptop, your servers at work, cloud VMs, etc.), they have exactly one network connection to one network provider, and so the routes are static, or at best provided by DHCP. For instance, at home my laptop knows that everything in 192.168.0.* can go directly over wifi, and everything else can be sent to 192.168.0.1. My router knows that everything should be sent to the ISP. And so forth. BGP is for network devices that have multiple connections to the Internet and need to decide which connection to use. They'll set up static IP routing to their immediate neighbors, make that TCP connection between their BGP daemons, and then let their BGP daemons configure the rest of the routing table. Most of the time these devices are backbone routers run by ISPs, but you can totally run BGP on your random Linux server. You'll need someone to peer with, though. I've used Quagga http://www.nongnu.org/quagga/ http://www.nongnu.org/quagga/ (although with OSPF, another routing protocol optimized for smaller-scale use) for implementing two separate web servers sharing the same IP address - that is, two servers claiming "Hey, I know how to reach this address, you can route through me," such that the service would stay up if either web server died - and it was mostly just `apt-get install quagga`, a small bit of config, and lots of help from the networking team doing some config on our intranet's routers to peer with my two servers and trust them.

atmosx 9y ago
Sounds a lot like what FreeBSD CARP[1] suppose to do. [1] https://www.freebsd.org/doc/handbook/carp.html https://www.freebsd.org/doc/handbook/carp.html
- gerdesj 9y ago
  Not quite. CARP (VRRP in Ciscoese) has two (or more) systems capable of presenting the same IP address and they agree amongst themselves which gets to be the master through a multicast based advertizing system. Generally each system will be near clones of each other and be capable of doing the full job themselves. I have a pair of rather large pfSense systems running CARP as my office routers. I can update the secondary, test it, fail over the CARP addresses and fifty IPSEC tunnels, five internet connections, 10 OpenVPN servers along with rather a lot of clients, five OpenVPN clients, 30 odd internal VLANs and all the states for the above along with many NAT inbound and outbound sessions will seamlessly switch over thanks to PFSYNC. Even voice calls don't drop - you might notice a sub second wobble if someone is talking at the wrong moment. Rinse/repeat for the other one. BGP, RIPx, OSPF and co. are routing protocols that deal with the route your connections take through a network. There are also things like LACP/LAG for layer two redundancy. You can even play games with DNS round robin and of course reverse proxies. CARP is one tool in the box for coping with failures and maintenance - there are lots of others, each with advantages and disadvantages. That's what makes the job interesting 8)
- geofft 9y ago
  Yes, but in my case I wanted the two servers to be located in physically separate parts of our datacenter to guard against a switch failing (ideally it would have been split across multiple datacenters), and I think that OpenBSD's CARP, or similar protocols like Cisco's VRRP, can only handle the case of computers on the same subnet sharing another IP address on that subnet. The two machines were on different subnets, and there was a third /24 we assigned virtual addresses from. By making the user-facing address for the web server its own 32 and having the two servers say "Hey, if you want to reach 10.255.255.1 you can route through me" over OSPF, all the switches on the way from a client to the two servers would be able to handle the case of the next hop failing, and let traffic take the other route to the same IP address. (Never mind that 10.255.255.1 was independently served by two Apache processes on the two machines; for all they know it was actually being routed to a third computer directly hooked up to the first two.)

jlgaddis 9y ago

Download and install GNS3, set up a few virtual routers, configure them to exchange NLRI via BGP and play with it all you want to. You're not going to find anyone who will let you establish a BGP session with them to exchange "real" routes -- there's simply too much risk involved for no valid purpose (and with no benefit to them).

buzer 9y ago
There's https://dn42.us/ https://dn42.us/ that's pretty good for studying BGP in slightly more real-world scenarios. Of course, you won't be getting internet transit routes from those peers (usually at least).
- exikyut 9y ago
  Thanks, this is awesome!! I'm totally going to be checking this out and saying hi/connecting at some point in the future (when I have a server etc).

devdas 9y ago

You can do this on internal subnets. On the public internet, hopefully not without safeguards :)