4 ms·

> 10.0.0.0/8 runs out pretty quick 16,777,216 containers, wow.

sumtechguy 2y ago
I am going to give them the benefit of the doubt. Maybe they are thinking it would probably run out due to poor segmentation? It would be quite the interesting problem of network segmentation to run your whole service provider with thousands of customers and network out of a single 10 network. Realistically a better option was each customer gets their own locked off network and vlan level.
ReK_ 2y ago
Except it's not 16 million containers. You need addressing for every link in the network. Want 10 racks of 20 servers with redundant switching in each, redundant spines in the network rack? That's 60 physical links, each of which need two addresses, and that's just the fabric underlay network. You haven't created the actual overlay networks yet, which you need multiple of (front end, back end, database, storage...). Consider that 200 servers is a drop in the bucket at some scales, you can see why data centre is moving to V6 only.
throw0101b 2y ago
>> 10.0.0.0/8 runs out pretty quick > 16,777,216 containers, wow. Have you ever been involved in a corporate merger? IP conflicts are a huge pain point. Quite often you have to NAT with-in the company itself because the acquirer and acquiree are both using 10/8.
- Volundr 2y ago
  This was literally why we went IPv6 in a previous company. We were acquiring other companies like crazy and it and the conflicts were constant pain where we'd either have to re-IP a location (Active Directory does not like) or do internal facing NATs and DNS weirdness (AD also not a big fan). We quickly discovered it was easier to get the new location up and running on IPv6 and mesh that so all inter-office traffic was IPv6 rather than resolving the conflicts. Sure you couldn't reach the printer in Boise from New York because it was IPv4 only, but for the stuff normal users were doing it worked great.
KaiserPro 2y ago
So, if you have a sane ipv4 network, where everything is dhcp, and nothing apart from a few key things are statically assigned, then yeah _technically_ you can have 16million addresses all at once. But, subnets need to be located next to each other physically, otherwise performance suffers. subnets have affinity. but once you have subnets, you then start loosing packing efficiency. for example, in the batshit world of K8s, you give each node its own /24 to dish out. Not only cant that limit the number of containers you can host, it also is really inefficient. (eating 256k addresses) More over, it also means that you need to reuse addresses. in a large cluster of say 1000 nodes, each hosting 40 containers, starting/stopping anything up to 30 containers a second isn't unreasonable. Its not inconceivable that you'll end up trying to connect to a stale address (either because its not propagated yet, or your brand of service discovery isn't that fast). This can cause hilarious transitory errors. but if you could assign an IP per container, and have enough space to not re-use that address for at least a few hours then that goes away. so instead of getting weird fuzzing errors(or misc 404/401), you get a connection timed out.

I am going to give them the benefit of the doubt. Maybe they are thinking it would probably run out due to poor segmentation? It would be quite the interesting problem of network segmentation to run your whole service provider with thousands of customers and network out of a single 10 network. Realistically a better option was each customer gets their own locked off network and vlan level.

ReK_ 2y ago

Except it's not 16 million containers. You need addressing for every link in the network. Want 10 racks of 20 servers with redundant switching in each, redundant spines in the network rack? That's 60 physical links, each of which need two addresses, and that's just the fabric underlay network. You haven't created the actual overlay networks yet, which you need multiple of (front end, back end, database, storage...). Consider that 200 servers is a drop in the bucket at some scales, you can see why data centre is moving to V6 only.

throw0101b 2y ago

>> 10.0.0.0/8 runs out pretty quick > 16,777,216 containers, wow. Have you ever been involved in a corporate merger? IP conflicts are a huge pain point. Quite often you have to NAT with-in the company itself because the acquirer and acquiree are both using 10/8.

Volundr 2y ago
This was literally why we went IPv6 in a previous company. We were acquiring other companies like crazy and it and the conflicts were constant pain where we'd either have to re-IP a location (Active Directory does not like) or do internal facing NATs and DNS weirdness (AD also not a big fan). We quickly discovered it was easier to get the new location up and running on IPv6 and mesh that so all inter-office traffic was IPv6 rather than resolving the conflicts. Sure you couldn't reach the printer in Boise from New York because it was IPv4 only, but for the stuff normal users were doing it worked great.

KaiserPro 2y ago

So, if you have a sane ipv4 network, where everything is dhcp, and nothing apart from a few key things are statically assigned, then yeah _technically_ you can have 16million addresses all at once. But, subnets need to be located next to each other physically, otherwise performance suffers. subnets have affinity. but once you have subnets, you then start loosing packing efficiency. for example, in the batshit world of K8s, you give each node its own /24 to dish out. Not only cant that limit the number of containers you can host, it also is really inefficient. (eating 256k addresses) More over, it also means that you need to reuse addresses. in a large cluster of say 1000 nodes, each hosting 40 containers, starting/stopping anything up to 30 containers a second isn't unreasonable. Its not inconceivable that you'll end up trying to connect to a stale address (either because its not propagated yet, or your brand of service discovery isn't that fast). This can cause hilarious transitory errors. but if you could assign an IP per container, and have enough space to not re-use that address for at least a few hours then that goes away. so instead of getting weird fuzzing errors(or misc 404/401), you get a connection timed out.