3 ms·

In addition to what SEJeff said, as long as you design your IP addressing correctly you'll be fine. By that, I mean hierarchically. Just divide up whatever IP n

In addition to what SEJeff said, as long as you design your IP addressing correctly you'll be fine. By that, I mean hierarchically. Just divide up whatever IP network you're using (e.g. 10/8) and make sure you allocate "enough" to each rack/whatever. Assuming everything is nice and hierarchical, you can easily aggregate an entire rack to a single prefix. Even the shitty ToR switches can usually handle a couple thousand prefixes, which should be plenty if done correctly. Obviously you shouldn't be advertising /32s. > The entire v4 internet is something like 600k prefixes right now ... Just checked my edge routers and it looks like we're up to ~671k prefixes here and that number is still increasing everyday.

X-Istence 9y ago
You should be advertising /32's and /128's. From the hypervisor to the TOR, then the TOR aggregates if possible and advertises to a spine. At least, that's what you do if you use Calico and want to be able to use hypervisor migration when using it with OpenStack. Just make sure your TOR's can handle the amount of routes necessary, have a default route from the hypervisor to the internet, and from the TOR to the spine, and have the spine advertise a 0's route. So now the spine is the only place where you need beefier routers that can support more than the TOR's in terms of routes. With some intelligence in the IPAM solution your host will get a /26 (or a /64) and will advertise that entire range, and only a single /32 is advertised if the VM/container moves to another hypervisor host (to support things like live-migration). That being said, most TOR's can handle quite a large amount of routes these days. When I was at a telco we had some gear that did up to 128k routes, so splitting between IPv4/IPv6 we had 64k routes each. Which is plenty, even for larger clusters.
- jlgaddis 9y ago
  > You should be advertising /32's and /128's. From the hypervisor to the TOR, then the TOR aggregates if possible and advertises to a spine. Yeah, that's what I meant. I thought that was clear from "aggregate an entire rack to a single prefix" but perhaps not. When I said don't advertise /32s I meant "globally" and as a general rule (I didn't realize what I said would be taken so literally). There can be exceptions, of course, such as in the live-migration case you mentioned.
chris_marino 9y ago
> Just divide up whatever IP network you're using (e.g. 10/8) and make sure you allocate "enough" to each rack/whatever. Easier said than done. Most datacenters are bit more deliberate about allocating addresses and hand them out in non-contiguous CIDRs. The VLAN mentality is still very prevalent. Getting a /20 at a time is pretty common. Using overlapping IPs puts you right back into the overlay model. >Assuming everything is nice and hierarchical, you can easily aggregate an entire rack to a single prefix. Yes, exactly. The trick then becomes how to you ensure that endpoints that get created within the rack get an IP from the prefix? Romana (the project I work on) does this. It lets you capture your network topology for exactly this reason. This is especially important if/when you must filter routes at ToR.

X-Istence 9y ago

You should be advertising /32's and /128's. From the hypervisor to the TOR, then the TOR aggregates if possible and advertises to a spine. At least, that's what you do if you use Calico and want to be able to use hypervisor migration when using it with OpenStack. Just make sure your TOR's can handle the amount of routes necessary, have a default route from the hypervisor to the internet, and from the TOR to the spine, and have the spine advertise a 0's route. So now the spine is the only place where you need beefier routers that can support more than the TOR's in terms of routes. With some intelligence in the IPAM solution your host will get a /26 (or a /64) and will advertise that entire range, and only a single /32 is advertised if the VM/container moves to another hypervisor host (to support things like live-migration). That being said, most TOR's can handle quite a large amount of routes these days. When I was at a telco we had some gear that did up to 128k routes, so splitting between IPv4/IPv6 we had 64k routes each. Which is plenty, even for larger clusters.

jlgaddis 9y ago
> You should be advertising /32's and /128's. From the hypervisor to the TOR, then the TOR aggregates if possible and advertises to a spine. Yeah, that's what I meant. I thought that was clear from "aggregate an entire rack to a single prefix" but perhaps not. When I said don't advertise /32s I meant "globally" and as a general rule (I didn't realize what I said would be taken so literally). There can be exceptions, of course, such as in the live-migration case you mentioned.

chris_marino 9y ago

> Just divide up whatever IP network you're using (e.g. 10/8) and make sure you allocate "enough" to each rack/whatever. Easier said than done. Most datacenters are bit more deliberate about allocating addresses and hand them out in non-contiguous CIDRs. The VLAN mentality is still very prevalent. Getting a /20 at a time is pretty common. Using overlapping IPs puts you right back into the overlay model. >Assuming everything is nice and hierarchical, you can easily aggregate an entire rack to a single prefix. Yes, exactly. The trick then becomes how to you ensure that endpoints that get created within the rack get an IP from the prefix? Romana (the project I work on) does this. It lets you capture your network topology for exactly this reason. This is especially important if/when you must filter routes at ToR.