3 ms·

Saying BGP is "web scale" is a bit misleading because it has to be very carefully aggregated for it to route the entire internet. If you do something like adve

Saying BGP is "web scale" is a bit misleading because it has to be very carefully aggregated for it to route the entire internet. If you do something like advertise a /32 for each container you can very quickly fill up TCAMs on your network hardware (in particular cheap top of rack switches that are pervasive in data centers). The entire v4 internet is something like 600k prefixes right now and the routers that can handle that many prefixes at line rate are irritatingly expensive. ToRs as of a couple of years ago when I last tested this would fall over at 1-10k prefixes. So be careful when looking at BGP solutions because it's very easy to have a BGP topology that doesn't scale, despite it being the exchange protocol for the Internet.

SEJeff 9y ago
Sure, that is why you can also implement something like one AS per rack with each rack as its own AS peering with other racks through a set of distribution switches, which then aggregate your routes up to a single range that is broadcast out. You could have each host in the rack advertise 1 prefix maybe 2 and you should be good to go even with some absolutely garbage TOR switches. I read your comment as, "Don't use technology that you can misconfigure, because you can misconfigure it!". Well yeah, the same can be said with anything networking related.
jlgaddis 9y ago
In addition to what SEJeff said, as long as you design your IP addressing correctly you'll be fine. By that, I mean hierarchically. Just divide up whatever IP network you're using (e.g. 10/8) and make sure you allocate "enough" to each rack/whatever. Assuming everything is nice and hierarchical, you can easily aggregate an entire rack to a single prefix. Even the shitty ToR switches can usually handle a couple thousand prefixes, which should be plenty if done correctly. Obviously you shouldn't be advertising /32s. > The entire v4 internet is something like 600k prefixes right now ... Just checked my edge routers and it looks like we're up to ~671k prefixes here and that number is still increasing everyday.
- X-Istence 9y ago
  You should be advertising /32's and /128's. From the hypervisor to the TOR, then the TOR aggregates if possible and advertises to a spine. At least, that's what you do if you use Calico and want to be able to use hypervisor migration when using it with OpenStack. Just make sure your TOR's can handle the amount of routes necessary, have a default route from the hypervisor to the internet, and from the TOR to the spine, and have the spine advertise a 0's route. So now the spine is the only place where you need beefier routers that can support more than the TOR's in terms of routes. With some intelligence in the IPAM solution your host will get a /26 (or a /64) and will advertise that entire range, and only a single /32 is advertised if the VM/container moves to another hypervisor host (to support things like live-migration). That being said, most TOR's can handle quite a large amount of routes these days. When I was at a telco we had some gear that did up to 128k routes, so splitting between IPv4/IPv6 we had 64k routes each. Which is plenty, even for larger clusters.
  - jlgaddis 9y ago
    > You should be advertising /32's and /128's. From the hypervisor to the TOR, then the TOR aggregates if possible and advertises to a spine. Yeah, that's what I meant. I thought that was clear from "aggregate an entire rack to a single prefix" but perhaps not. When I said don't advertise /32s I meant "globally" and as a general rule (I didn't realize what I said would be taken so literally). There can be exceptions, of course, such as in the live-migration case you mentioned.
- chris_marino 9y ago
  > Just divide up whatever IP network you're using (e.g. 10/8) and make sure you allocate "enough" to each rack/whatever. Easier said than done. Most datacenters are bit more deliberate about allocating addresses and hand them out in non-contiguous CIDRs. The VLAN mentality is still very prevalent. Getting a /20 at a time is pretty common. Using overlapping IPs puts you right back into the overlay model. >Assuming everything is nice and hierarchical, you can easily aggregate an entire rack to a single prefix. Yes, exactly. The trick then becomes how to you ensure that endpoints that get created within the rack get an IP from the prefix? Romana (the project I work on) does this. It lets you capture your network topology for exactly this reason. This is especially important if/when you must filter routes at ToR.

SEJeff 9y ago

Sure, that is why you can also implement something like one AS per rack with each rack as its own AS peering with other racks through a set of distribution switches, which then aggregate your routes up to a single range that is broadcast out. You could have each host in the rack advertise 1 prefix maybe 2 and you should be good to go even with some absolutely garbage TOR switches. I read your comment as, "Don't use technology that you can misconfigure, because you can misconfigure it!". Well yeah, the same can be said with anything networking related.

jlgaddis 9y ago

In addition to what SEJeff said, as long as you design your IP addressing correctly you'll be fine. By that, I mean hierarchically. Just divide up whatever IP network you're using (e.g. 10/8) and make sure you allocate "enough" to each rack/whatever. Assuming everything is nice and hierarchical, you can easily aggregate an entire rack to a single prefix. Even the shitty ToR switches can usually handle a couple thousand prefixes, which should be plenty if done correctly. Obviously you shouldn't be advertising /32s. > The entire v4 internet is something like 600k prefixes right now ... Just checked my edge routers and it looks like we're up to ~671k prefixes here and that number is still increasing everyday.

X-Istence 9y ago
You should be advertising /32's and /128's. From the hypervisor to the TOR, then the TOR aggregates if possible and advertises to a spine. At least, that's what you do if you use Calico and want to be able to use hypervisor migration when using it with OpenStack. Just make sure your TOR's can handle the amount of routes necessary, have a default route from the hypervisor to the internet, and from the TOR to the spine, and have the spine advertise a 0's route. So now the spine is the only place where you need beefier routers that can support more than the TOR's in terms of routes. With some intelligence in the IPAM solution your host will get a /26 (or a /64) and will advertise that entire range, and only a single /32 is advertised if the VM/container moves to another hypervisor host (to support things like live-migration). That being said, most TOR's can handle quite a large amount of routes these days. When I was at a telco we had some gear that did up to 128k routes, so splitting between IPv4/IPv6 we had 64k routes each. Which is plenty, even for larger clusters.
- jlgaddis 9y ago
  > You should be advertising /32's and /128's. From the hypervisor to the TOR, then the TOR aggregates if possible and advertises to a spine. Yeah, that's what I meant. I thought that was clear from "aggregate an entire rack to a single prefix" but perhaps not. When I said don't advertise /32s I meant "globally" and as a general rule (I didn't realize what I said would be taken so literally). There can be exceptions, of course, such as in the live-migration case you mentioned.
chris_marino 9y ago
> Just divide up whatever IP network you're using (e.g. 10/8) and make sure you allocate "enough" to each rack/whatever. Easier said than done. Most datacenters are bit more deliberate about allocating addresses and hand them out in non-contiguous CIDRs. The VLAN mentality is still very prevalent. Getting a /20 at a time is pretty common. Using overlapping IPs puts you right back into the overlay model. >Assuming everything is nice and hierarchical, you can easily aggregate an entire rack to a single prefix. Yes, exactly. The trick then becomes how to you ensure that endpoints that get created within the rack get an IP from the prefix? Romana (the project I work on) does this. It lets you capture your network topology for exactly this reason. This is especially important if/when you must filter routes at ToR.