9 ms·

IPvlan overlay-free Kubernetes Networking in AWS

144 points by theatrus2 9y ago

muxator 9y ago
For those wondering what's the difference between macvlan and ipvlan, the main ipvlan paper [0] summarizes its raison d'être: > This is especially problematic where the connected next-hop e.g. switch is expecting frames from a specific mac from a specific port. e.g.: if the host is attached to a managed switch with a strict security policy, macvlan would not work. [0] https://www.netdevconf.org/0.1/sessions/28.html https://www.netdevconf.org/0.1/sessions/28.html
- puzzle 9y ago
  If you need distinct MAC addresses, though, for e.g. DHCP, usually not the case with containers, then you have to use macvlan.
- KaiserPro 9y ago
  This is what I'm trying to understand. Macvlan appears to be a much better solution as it allows 1-1 mapping and piggybacking onto all the automatic/set&forget mechanisms that AWS provides. Obviously it needs a switch at the otherside that can handle a huge and quick changing arp table. Also if you have mac address limiting typical on edge switches, its a non flyer
deepakjois 9y ago
Not directly related, but can someone recommend a beginners resource to understand Kubernetes networking? There are some good ones out there that explain basic Kubernetes concepts like pods, replicas etc. But networking seems to be a more complicated topic, and most intro guides skip over it.
- andrewstuart2 9y ago
  The networking model itself is amazingly simple, and part of what makes Kubernetes so much easier to use. The rules are as follows: All containers can communicate with all other containers without NAT. All nodes can communicate with all containers (and vice-versa) without NAT. The IP that a container sees itself as is the same IP that others see it as. When using Docker by itself, you get into all sorts of complicated situations because most running containers have an IP address that's host-specific and not routable for any other machines. This makes networking across hosts a giant pain. Kubernetes takes that away by making things behave exactly how you'd hope they'd behave. My IP as I see it is reachable by anybody in the cluster who has it (policy permitting). The simplicity of working in this networking model means that there's a little more work for the networking infrastructure to handle, making sure that IPs are allocated without collision and that routes are known across many hosts. Several technologies exist to build these bridges, including old-school tech that has solved these exact problems for decades like BGP (see Calico/canal). Ultimately, there's no silver bullet. I'd recommend giving the k8s networking page a read. [1] [1] https://kubernetes.io/docs/concepts/cluster-administration/networking/ https://kubernetes.io/docs/concepts/cluster-administration/n...
  - scurvy 9y ago
    That's amazingly simple? I notice that you conveniently left out the "ingress" component. Stuff in K8s talking with other K8s stuff is easy. Getting the flows into K8s apps from outside the K8 network is amazingly clunky in its current state.
    - atombender 9y ago
      Network is simple from the container's point of view. It's less simple outside the container. But outside the container, the strategy is still much simpler than other solutions (most of which predate Kubernetes). Kubernetes chooses to give every pod its own IP. This means choosing an internal network such as 10.x.x.x, and giving each machine a slice of it. This way, one single cluster shares the same big, flat space of IP addresses; not only do pods have the same IP inside the container, but they can talk to other pods using the other pod's IP, too. But a key point is that Kubernetes is designed to take care of most of it. One part of it is the iptables proxy magic that it does to allow services to have dynamically assigned IPs, too, with simple load-balancing between them. The second part is the many built-in plugins for different, more complicated overlay strategies. Kubernetes' automatic configuration works out of the box on, say, AWS, without anything magical — Kubernetes natively talks to AWS to set up a routing table so that packets end up where they should. You don't need more complex overlay networking stacks such as Calico, Flannel or Weave right away. As for ingress, it has absolutely been Kubernetes' weakest point for several years, and the Kubernetes team knows this perfectly well. That said, it's not complicated, thanks to the above. Once you have, say, Nginx listening on a port, routing traffic into the cluster is a matter of setting up a load balancer (at least on clouds like GCP, DigitalOcean and AWS), something which Kubernetes even can do automatically for you. The weak links are the ingress controller — the Nginx one is popular because it's stable and supports common features such as TLS, whereas others such as Voyager and Traefik are lagging — as well as the impedance mismatch with cloud LBs such as the Google Load Balancer. So far, Kubernetes' ingress support has been generic: One ingress object can be used to "drive" different HTTP servers. The problem being, of course, that all HTTP implementations which have different settings (timeouts, TLS certs, CDN functionality) and concerns that the current, simple ingress format cannot support. I'm expecting this to change soon. Ingress portability really isn't an important concern, and the generic ingress format is a bottleneck for the ingress functionality to mature.
- markbnj 9y ago
  At the risk of self-promotion :). https://medium.com/google-cloud/understanding-kubernetes-networking-pods-7117dd28727 https://medium.com/google-cloud/understanding-kubernetes-net... This is the first of a two-part series, the first dealing with pod networking and the second with services. I plan a third on ingress after kubecon. It's a little GKE-specific in the implementation details, and the whole thing is pluggable and can be configured in different ways (as the OP shows), but I think it covers the fundamentals pretty well.
- manigandham 9y ago
  Try this: https://www.youtube.com/watch?v=WwQ62OyCNz4 https://www.youtube.com/watch?v=WwQ62OyCNz4 And this: https://www.youtube.com/watch?v=y2bhV81MfKQ https://www.youtube.com/watch?v=y2bhV81MfKQ
- Artemis2 9y ago
  Try this: https://jvns.ca/blog/2017/10/10/operating-a-kubernetes-network/ https://jvns.ca/blog/2017/10/10/operating-a-kubernetes-netwo...
- deepakjois 9y ago
  All the resources suggested in replies to the parent thread seem great. I also found that somebody put out an illustrated guide to Kubernetes networking on Medium: * Part 1: https://medium.com/@ApsOps/an-illustrated-guide-to-kubernetes-networking-part-1-d1ede3322727 https://medium.com/@ApsOps/an-illustrated-guide-to-kubernete... * Part 2: https://medium.com/@ApsOps/an-illustrated-guide-to-kubernetes-networking-part-2-13fdc6c4e24c https://medium.com/@ApsOps/an-illustrated-guide-to-kubernete...
tamalsaha001 9y ago
How does it compare to AWS' own CNI plugin? https://github.com/aws/amazon-vpc-cni-k8s https://github.com/aws/amazon-vpc-cni-k8s
- lambda 9y ago
  If you read the article, you'll see: > Lincoln Stoll’s k8s-vpcnet, and more recently, Amazon’s amazon-vpc-cni-k8s CNI stacks use Elastic Network Interfaces (ENIs) and secondary private IPs to achieve an overlay-free AWS VPC-native solutions for Kubernetes networking. While both of these solutions achieve the same base goal of drastically simplifying the network complexity of deploying Kubernetes at scale on AWS, they do not focus on minimizing network latency and kernel overhead as part of implementing a compliant networking stack.
  - scurvy 9y ago
    How does it compare with Romana? They added a VPC router specifically for large K8 clusters on AWS. https://github.com/romana/vpc-router https://github.com/romana/vpc-router
    - chris_marino 9y ago
      Both the Lyft and AWS CNIs use ENIs, Romana's CNI does not. But more specifically, vpc-router works along with Romana's IPAM to aggregate routes so that each VPC route can forward traffic for multiple instances. So, instead of one route per instance, you need only 1 routes per n instances. Where n is set by how much aggregation you want (configurable). The net effect is that you can build large clusters without running out of VPC routes and no overlay is needed when traffic crosses AZs. When a route is used to forward traffic for multiple instances, the target instance acts as router and forwards traffic to the final destination instance. This works because instances within an AZ have routes installed on them to the pod CIDRs on the other instances in the zone, so any one of them can perform this forwarding function. Romana only piggybacks routes when there are no more VPC routes available, so for small cluster it's just like kubenet. For large clusters routes it uses all the instances to forward traffic so that none of them become a bottleneck.
deleted 9y ago
[deleted]
SEJeff 9y ago
Just wanted to give a shout out to kube-router[1], a really fantastic solution if you want to use BGP, that will soon support not needing bgp by implementing a featureset similar to flannel's hostgw support. They are really good about addressing things in the open on their github[2]. BGP is, by definition, "web scale" as it runs most routing for the internet. Lower latency and much higher throughput than any sort of overlay network. [1] https://www.kube-router.io https://www.kube-router.io [2] https://github.com/cloudnativelabs/kube-router https://github.com/cloudnativelabs/kube-router
- hueving 9y ago
  Saying BGP is "web scale" is a bit misleading because it has to be very carefully aggregated for it to route the entire internet. If you do something like advertise a /32 for each container you can very quickly fill up TCAMs on your network hardware (in particular cheap top of rack switches that are pervasive in data centers). The entire v4 internet is something like 600k prefixes right now and the routers that can handle that many prefixes at line rate are irritatingly expensive. ToRs as of a couple of years ago when I last tested this would fall over at 1-10k prefixes. So be careful when looking at BGP solutions because it's very easy to have a BGP topology that doesn't scale, despite it being the exchange protocol for the Internet.
  - SEJeff 9y ago
    Sure, that is why you can also implement something like one AS per rack with each rack as its own AS peering with other racks through a set of distribution switches, which then aggregate your routes up to a single range that is broadcast out. You could have each host in the rack advertise 1 prefix maybe 2 and you should be good to go even with some absolutely garbage TOR switches. I read your comment as, "Don't use technology that you can misconfigure, because you can misconfigure it!". Well yeah, the same can be said with anything networking related.
  - jlgaddis 9y ago
    In addition to what SEJeff said, as long as you design your IP addressing correctly you'll be fine. By that, I mean hierarchically. Just divide up whatever IP network you're using (e.g. 10/8) and make sure you allocate "enough" to each rack/whatever. Assuming everything is nice and hierarchical, you can easily aggregate an entire rack to a single prefix. Even the shitty ToR switches can usually handle a couple thousand prefixes, which should be plenty if done correctly. Obviously you shouldn't be advertising /32s. > The entire v4 internet is something like 600k prefixes right now ... Just checked my edge routers and it looks like we're up to ~671k prefixes here and that number is still increasing everyday.
paxy 9y ago
> Announcing cni-ipvlan-vpc-k8s Rolls right off the tongue, doesn't it?
- andrewstuart2 9y ago
  CIVK, pronounced civic?
  - frogperson 9y ago
    Are doing acronyms of acronyms now?
    - andrewstuart2 9y ago
      Have you looked up PHP or GNU lately? :D At least these terminate when resolved.
ggm 9y ago
I know this invites some eye-rolling, but can somebody explain to me why the k8s people insist on ignoring IPv6 and the possibilities of large address fields? Down the bottom, which is where 'things we probably will never do' is when IPv6 comes in the door. Azure (for instance) is a fully IPv6 enabled fabric. Microsoft "get" IPv6. They are all over it. They understand it, its baked into the DNA. So how come K8s people just kind of think "yea.. nah.. not right now"? Because proxy Ipv6 at the edge is really sucky. We should be using native IPv6, preserve e2e under whatever routing model we need for reliability, and gateway the V4 through proxies in the longer term. (serious Q btw)
- deleted 9y ago
  [deleted]
- andrewstuart2 9y ago
  They're not ignoring it. It's being actively worked on, and is expected to be in alpha for the 1.9 release [1]. The issue [2] has existed for over 3 years, so it's not a new suggestion. [1] https://github.com/kubernetes/features/issues/508 https://github.com/kubernetes/features/issues/508 [2] https://github.com/kubernetes/kubernetes/issues/1443 https://github.com/kubernetes/kubernetes/issues/1443
  - jamiesonbecker 9y ago
    > "not ignoring it" .. "actively worked on" .. "for over 3 years" .. Not to diminish the very real challenges in getting IPv6 implemented, but this is an interesting turn of phrase.. especially because rolling out IPv6 would actually solve a whole class of problems (and I'm not even a particularly big advocate of the need for IPv6, since most things should still be NATed anyway.) (And especially considering parent's phrase "baked into the DNA" at Azure.)
    - ggm 9y ago
      (And especially considering parent's phrase "baked into the DNA" at Azure.) S/Azure/Microsoft/ -Azure has IPv6 but I think its Immature. https://azure.microsoft.com/en-au/updates/ipv6-for-azure-vms/ https://azure.microsoft.com/en-au/updates/ipv6-for-azure-vms... Is about the underlying VM architecture, not support for Kubernetes network models.
netingle 9y ago
IIUC ENIs are limited to 2 per host on small instances, 15 per host on larger ones. Doesn't this approach limit the number of Pods per host? I'm already running about 20 pods per host, and I don't more containers per host is atypical.
bogomipz 9y ago
The author states: >"Unfortunately, AWS’s VPC product has a default maximum of 50 non-propagated routes per route table, which can be increased up to a hard limit of 100 routes at the cost of potentially reducing network performance." Could someone explain why increasing from 50 to 100 non-propagated routes in a VPC results in network performance degradation?
chris_marino 9y ago
It all about trade offs. We've built a CNI for k8s and have looked into all of the techniques described. It seems that Lyft's design is a direct reflection of their requirements. To the extent your requirement match theirs, this could be a good alternative. The most significant in my mind is that it's meant to be used in conjunction with Envoy. Envoy itself has its own set of design tradeoffs as well. For example, Lyft currently uses 'service-assigned EC2 instances'. Not hard to see how this starting point would influence the design. The Envoy/Istio model of proxy per pod also reflects this kind of workload partitioning. Obviously, a design for a small number of pods (each with their own proxy) per instance is going to be very different from one that needs to handle 100 pods (and their IPs), or more, per instance. Another is that k8s network policy can't be applied since the 'Kubernetes Services see connections from a node’s source IP instead of the Pod’s source IP'. But I don't think this CNI is intended to work with any other network policy API enforcement mechanism. Romana (the project I work on) and the other CNI providers that use iptables to enforce network policy rely on seeing the pod's source IP. Again, this might be fine if you're running Envoy. On the other hand, L3 filtering on the host might be important. Also, this design requires that 'CNI plugins communicate with AWS networking APIs to provision network resources for Pods'. This may or may not be something you want your instances to do. FWIW, Romana lets you build clusters larger than 50 nodes without an overlay or more 'exotic networking techniques' or 'massive' complexity. It does it via simple route aggregation, completely standard networking.
- warp_factor 9y ago
  Not all NetworkPolicy implementations base themselves on Source//Destination IPs. I can think specifically of Trireme//Cilium that are using metadata in order to enable policies.
  - chris_marino 9y ago
    I knew that. What I didn't know was if either of these could apply network policy to these endpoints. Guessing that since they each require their own CNI, there will be probs. So, whether the CNI uses iptables, or not, not clear how network policy API can be enforced.

144 points by theatrus2 9y ago

For those wondering what's the difference between macvlan and ipvlan, the main ipvlan paper [0] summarizes its raison d'être: > This is especially problematic where the connected next-hop e.g. switch is expecting frames from a specific mac from a specific port. e.g.: if the host is attached to a managed switch with a strict security policy, macvlan would not work. [0] https://www.netdevconf.org/0.1/sessions/28.html https://www.netdevconf.org/0.1/sessions/28.html

puzzle 9y ago
If you need distinct MAC addresses, though, for e.g. DHCP, usually not the case with containers, then you have to use macvlan.
KaiserPro 9y ago
This is what I'm trying to understand. Macvlan appears to be a much better solution as it allows 1-1 mapping and piggybacking onto all the automatic/set&forget mechanisms that AWS provides. Obviously it needs a switch at the otherside that can handle a huge and quick changing arp table. Also if you have mac address limiting typical on edge switches, its a non flyer

deepakjois 9y ago

Not directly related, but can someone recommend a beginners resource to understand Kubernetes networking? There are some good ones out there that explain basic Kubernetes concepts like pods, replicas etc. But networking seems to be a more complicated topic, and most intro guides skip over it.

andrewstuart2 9y ago
The networking model itself is amazingly simple, and part of what makes Kubernetes so much easier to use. The rules are as follows: All containers can communicate with all other containers without NAT. All nodes can communicate with all containers (and vice-versa) without NAT. The IP that a container sees itself as is the same IP that others see it as. When using Docker by itself, you get into all sorts of complicated situations because most running containers have an IP address that's host-specific and not routable for any other machines. This makes networking across hosts a giant pain. Kubernetes takes that away by making things behave exactly how you'd hope they'd behave. My IP as I see it is reachable by anybody in the cluster who has it (policy permitting). The simplicity of working in this networking model means that there's a little more work for the networking infrastructure to handle, making sure that IPs are allocated without collision and that routes are known across many hosts. Several technologies exist to build these bridges, including old-school tech that has solved these exact problems for decades like BGP (see Calico/canal). Ultimately, there's no silver bullet. I'd recommend giving the k8s networking page a read. [1] [1] https://kubernetes.io/docs/concepts/cluster-administration/networking/ https://kubernetes.io/docs/concepts/cluster-administration/n...
- scurvy 9y ago
  That's amazingly simple? I notice that you conveniently left out the "ingress" component. Stuff in K8s talking with other K8s stuff is easy. Getting the flows into K8s apps from outside the K8 network is amazingly clunky in its current state.
  - atombender 9y ago
    Network is simple from the container's point of view. It's less simple outside the container. But outside the container, the strategy is still much simpler than other solutions (most of which predate Kubernetes). Kubernetes chooses to give every pod its own IP. This means choosing an internal network such as 10.x.x.x, and giving each machine a slice of it. This way, one single cluster shares the same big, flat space of IP addresses; not only do pods have the same IP inside the container, but they can talk to other pods using the other pod's IP, too. But a key point is that Kubernetes is designed to take care of most of it. One part of it is the iptables proxy magic that it does to allow services to have dynamically assigned IPs, too, with simple load-balancing between them. The second part is the many built-in plugins for different, more complicated overlay strategies. Kubernetes' automatic configuration works out of the box on, say, AWS, without anything magical — Kubernetes natively talks to AWS to set up a routing table so that packets end up where they should. You don't need more complex overlay networking stacks such as Calico, Flannel or Weave right away. As for ingress, it has absolutely been Kubernetes' weakest point for several years, and the Kubernetes team knows this perfectly well. That said, it's not complicated, thanks to the above. Once you have, say, Nginx listening on a port, routing traffic into the cluster is a matter of setting up a load balancer (at least on clouds like GCP, DigitalOcean and AWS), something which Kubernetes even can do automatically for you. The weak links are the ingress controller — the Nginx one is popular because it's stable and supports common features such as TLS, whereas others such as Voyager and Traefik are lagging — as well as the impedance mismatch with cloud LBs such as the Google Load Balancer. So far, Kubernetes' ingress support has been generic: One ingress object can be used to "drive" different HTTP servers. The problem being, of course, that all HTTP implementations which have different settings (timeouts, TLS certs, CDN functionality) and concerns that the current, simple ingress format cannot support. I'm expecting this to change soon. Ingress portability really isn't an important concern, and the generic ingress format is a bottleneck for the ingress functionality to mature.
markbnj 9y ago
At the risk of self-promotion :). https://medium.com/google-cloud/understanding-kubernetes-networking-pods-7117dd28727 https://medium.com/google-cloud/understanding-kubernetes-net... This is the first of a two-part series, the first dealing with pod networking and the second with services. I plan a third on ingress after kubecon. It's a little GKE-specific in the implementation details, and the whole thing is pluggable and can be configured in different ways (as the OP shows), but I think it covers the fundamentals pretty well.
manigandham 9y ago
Try this: https://www.youtube.com/watch?v=WwQ62OyCNz4 https://www.youtube.com/watch?v=WwQ62OyCNz4 And this: https://www.youtube.com/watch?v=y2bhV81MfKQ https://www.youtube.com/watch?v=y2bhV81MfKQ
Artemis2 9y ago
Try this: https://jvns.ca/blog/2017/10/10/operating-a-kubernetes-network/ https://jvns.ca/blog/2017/10/10/operating-a-kubernetes-netwo...
deepakjois 9y ago
All the resources suggested in replies to the parent thread seem great. I also found that somebody put out an illustrated guide to Kubernetes networking on Medium: * Part 1: https://medium.com/@ApsOps/an-illustrated-guide-to-kubernetes-networking-part-1-d1ede3322727 https://medium.com/@ApsOps/an-illustrated-guide-to-kubernete... * Part 2: https://medium.com/@ApsOps/an-illustrated-guide-to-kubernetes-networking-part-2-13fdc6c4e24c https://medium.com/@ApsOps/an-illustrated-guide-to-kubernete...

tamalsaha001 9y ago

How does it compare to AWS' own CNI plugin? https://github.com/aws/amazon-vpc-cni-k8s https://github.com/aws/amazon-vpc-cni-k8s

lambda 9y ago
If you read the article, you'll see: > Lincoln Stoll’s k8s-vpcnet, and more recently, Amazon’s amazon-vpc-cni-k8s CNI stacks use Elastic Network Interfaces (ENIs) and secondary private IPs to achieve an overlay-free AWS VPC-native solutions for Kubernetes networking. While both of these solutions achieve the same base goal of drastically simplifying the network complexity of deploying Kubernetes at scale on AWS, they do not focus on minimizing network latency and kernel overhead as part of implementing a compliant networking stack.
- scurvy 9y ago
  How does it compare with Romana? They added a VPC router specifically for large K8 clusters on AWS. https://github.com/romana/vpc-router https://github.com/romana/vpc-router
  - chris_marino 9y ago
    Both the Lyft and AWS CNIs use ENIs, Romana's CNI does not. But more specifically, vpc-router works along with Romana's IPAM to aggregate routes so that each VPC route can forward traffic for multiple instances. So, instead of one route per instance, you need only 1 routes per n instances. Where n is set by how much aggregation you want (configurable). The net effect is that you can build large clusters without running out of VPC routes and no overlay is needed when traffic crosses AZs. When a route is used to forward traffic for multiple instances, the target instance acts as router and forwards traffic to the final destination instance. This works because instances within an AZ have routes installed on them to the pod CIDRs on the other instances in the zone, so any one of them can perform this forwarding function. Romana only piggybacks routes when there are no more VPC routes available, so for small cluster it's just like kubenet. For large clusters routes it uses all the instances to forward traffic so that none of them become a bottleneck.

deleted 9y ago

[deleted]

SEJeff 9y ago

Just wanted to give a shout out to kube-router[1], a really fantastic solution if you want to use BGP, that will soon support not needing bgp by implementing a featureset similar to flannel's hostgw support. They are really good about addressing things in the open on their github[2]. BGP is, by definition, "web scale" as it runs most routing for the internet. Lower latency and much higher throughput than any sort of overlay network. [1] https://www.kube-router.io https://www.kube-router.io [2] https://github.com/cloudnativelabs/kube-router https://github.com/cloudnativelabs/kube-router

hueving 9y ago
Saying BGP is "web scale" is a bit misleading because it has to be very carefully aggregated for it to route the entire internet. If you do something like advertise a /32 for each container you can very quickly fill up TCAMs on your network hardware (in particular cheap top of rack switches that are pervasive in data centers). The entire v4 internet is something like 600k prefixes right now and the routers that can handle that many prefixes at line rate are irritatingly expensive. ToRs as of a couple of years ago when I last tested this would fall over at 1-10k prefixes. So be careful when looking at BGP solutions because it's very easy to have a BGP topology that doesn't scale, despite it being the exchange protocol for the Internet.
- SEJeff 9y ago
  Sure, that is why you can also implement something like one AS per rack with each rack as its own AS peering with other racks through a set of distribution switches, which then aggregate your routes up to a single range that is broadcast out. You could have each host in the rack advertise 1 prefix maybe 2 and you should be good to go even with some absolutely garbage TOR switches. I read your comment as, "Don't use technology that you can misconfigure, because you can misconfigure it!". Well yeah, the same can be said with anything networking related.
- jlgaddis 9y ago
  In addition to what SEJeff said, as long as you design your IP addressing correctly you'll be fine. By that, I mean hierarchically. Just divide up whatever IP network you're using (e.g. 10/8) and make sure you allocate "enough" to each rack/whatever. Assuming everything is nice and hierarchical, you can easily aggregate an entire rack to a single prefix. Even the shitty ToR switches can usually handle a couple thousand prefixes, which should be plenty if done correctly. Obviously you shouldn't be advertising /32s. > The entire v4 internet is something like 600k prefixes right now ... Just checked my edge routers and it looks like we're up to ~671k prefixes here and that number is still increasing everyday.

paxy 9y ago

> Announcing cni-ipvlan-vpc-k8s Rolls right off the tongue, doesn't it?

andrewstuart2 9y ago
CIVK, pronounced civic?
- frogperson 9y ago
  Are doing acronyms of acronyms now?
  - andrewstuart2 9y ago
    Have you looked up PHP or GNU lately? :D At least these terminate when resolved.

ggm 9y ago

I know this invites some eye-rolling, but can somebody explain to me why the k8s people insist on ignoring IPv6 and the possibilities of large address fields? Down the bottom, which is where 'things we probably will never do' is when IPv6 comes in the door. Azure (for instance) is a fully IPv6 enabled fabric. Microsoft "get" IPv6. They are all over it. They understand it, its baked into the DNA. So how come K8s people just kind of think "yea.. nah.. not right now"? Because proxy Ipv6 at the edge is really sucky. We should be using native IPv6, preserve e2e under whatever routing model we need for reliability, and gateway the V4 through proxies in the longer term. (serious Q btw)

deleted 9y ago
[deleted]
andrewstuart2 9y ago
They're not ignoring it. It's being actively worked on, and is expected to be in alpha for the 1.9 release [1]. The issue [2] has existed for over 3 years, so it's not a new suggestion. [1] https://github.com/kubernetes/features/issues/508 https://github.com/kubernetes/features/issues/508 [2] https://github.com/kubernetes/kubernetes/issues/1443 https://github.com/kubernetes/kubernetes/issues/1443
- jamiesonbecker 9y ago
  > "not ignoring it" .. "actively worked on" .. "for over 3 years" .. Not to diminish the very real challenges in getting IPv6 implemented, but this is an interesting turn of phrase.. especially because rolling out IPv6 would actually solve a whole class of problems (and I'm not even a particularly big advocate of the need for IPv6, since most things should still be NATed anyway.) (And especially considering parent's phrase "baked into the DNA" at Azure.)
  - ggm 9y ago
    (And especially considering parent's phrase "baked into the DNA" at Azure.) S/Azure/Microsoft/ -Azure has IPv6 but I think its Immature. https://azure.microsoft.com/en-au/updates/ipv6-for-azure-vms/ https://azure.microsoft.com/en-au/updates/ipv6-for-azure-vms... Is about the underlying VM architecture, not support for Kubernetes network models.

netingle 9y ago

IIUC ENIs are limited to 2 per host on small instances, 15 per host on larger ones. Doesn't this approach limit the number of Pods per host? I'm already running about 20 pods per host, and I don't more containers per host is atypical.

bogomipz 9y ago

The author states: >"Unfortunately, AWS’s VPC product has a default maximum of 50 non-propagated routes per route table, which can be increased up to a hard limit of 100 routes at the cost of potentially reducing network performance." Could someone explain why increasing from 50 to 100 non-propagated routes in a VPC results in network performance degradation?

chris_marino 9y ago

It all about trade offs. We've built a CNI for k8s and have looked into all of the techniques described. It seems that Lyft's design is a direct reflection of their requirements. To the extent your requirement match theirs, this could be a good alternative. The most significant in my mind is that it's meant to be used in conjunction with Envoy. Envoy itself has its own set of design tradeoffs as well. For example, Lyft currently uses 'service-assigned EC2 instances'. Not hard to see how this starting point would influence the design. The Envoy/Istio model of proxy per pod also reflects this kind of workload partitioning. Obviously, a design for a small number of pods (each with their own proxy) per instance is going to be very different from one that needs to handle 100 pods (and their IPs), or more, per instance. Another is that k8s network policy can't be applied since the 'Kubernetes Services see connections from a node’s source IP instead of the Pod’s source IP'. But I don't think this CNI is intended to work with any other network policy API enforcement mechanism. Romana (the project I work on) and the other CNI providers that use iptables to enforce network policy rely on seeing the pod's source IP. Again, this might be fine if you're running Envoy. On the other hand, L3 filtering on the host might be important. Also, this design requires that 'CNI plugins communicate with AWS networking APIs to provision network resources for Pods'. This may or may not be something you want your instances to do. FWIW, Romana lets you build clusters larger than 50 nodes without an overlay or more 'exotic networking techniques' or 'massive' complexity. It does it via simple route aggregation, completely standard networking.

warp_factor 9y ago
Not all NetworkPolicy implementations base themselves on Source//Destination IPs. I can think specifically of Trireme//Cilium that are using metadata in order to enable policies.
- chris_marino 9y ago
  I knew that. What I didn't know was if either of these could apply network policy to these endpoints. Guessing that since they each require their own CNI, there will be probs. So, whether the CNI uses iptables, or not, not clear how network policy API can be enforced.