4 ms·

Not directly related, but can someone recommend a beginners resource to understand Kubernetes networking? There are some good ones out there that explain basic

Not directly related, but can someone recommend a beginners resource to understand Kubernetes networking? There are some good ones out there that explain basic Kubernetes concepts like pods, replicas etc. But networking seems to be a more complicated topic, and most intro guides skip over it.

andrewstuart2 9y ago
The networking model itself is amazingly simple, and part of what makes Kubernetes so much easier to use. The rules are as follows: All containers can communicate with all other containers without NAT. All nodes can communicate with all containers (and vice-versa) without NAT. The IP that a container sees itself as is the same IP that others see it as. When using Docker by itself, you get into all sorts of complicated situations because most running containers have an IP address that's host-specific and not routable for any other machines. This makes networking across hosts a giant pain. Kubernetes takes that away by making things behave exactly how you'd hope they'd behave. My IP as I see it is reachable by anybody in the cluster who has it (policy permitting). The simplicity of working in this networking model means that there's a little more work for the networking infrastructure to handle, making sure that IPs are allocated without collision and that routes are known across many hosts. Several technologies exist to build these bridges, including old-school tech that has solved these exact problems for decades like BGP (see Calico/canal). Ultimately, there's no silver bullet. I'd recommend giving the k8s networking page a read. [1] [1] https://kubernetes.io/docs/concepts/cluster-administration/networking/ https://kubernetes.io/docs/concepts/cluster-administration/n...
- scurvy 9y ago
  That's amazingly simple? I notice that you conveniently left out the "ingress" component. Stuff in K8s talking with other K8s stuff is easy. Getting the flows into K8s apps from outside the K8 network is amazingly clunky in its current state.
  - atombender 9y ago
    Network is simple from the container's point of view. It's less simple outside the container. But outside the container, the strategy is still much simpler than other solutions (most of which predate Kubernetes). Kubernetes chooses to give every pod its own IP. This means choosing an internal network such as 10.x.x.x, and giving each machine a slice of it. This way, one single cluster shares the same big, flat space of IP addresses; not only do pods have the same IP inside the container, but they can talk to other pods using the other pod's IP, too. But a key point is that Kubernetes is designed to take care of most of it. One part of it is the iptables proxy magic that it does to allow services to have dynamically assigned IPs, too, with simple load-balancing between them. The second part is the many built-in plugins for different, more complicated overlay strategies. Kubernetes' automatic configuration works out of the box on, say, AWS, without anything magical — Kubernetes natively talks to AWS to set up a routing table so that packets end up where they should. You don't need more complex overlay networking stacks such as Calico, Flannel or Weave right away. As for ingress, it has absolutely been Kubernetes' weakest point for several years, and the Kubernetes team knows this perfectly well. That said, it's not complicated, thanks to the above. Once you have, say, Nginx listening on a port, routing traffic into the cluster is a matter of setting up a load balancer (at least on clouds like GCP, DigitalOcean and AWS), something which Kubernetes even can do automatically for you. The weak links are the ingress controller — the Nginx one is popular because it's stable and supports common features such as TLS, whereas others such as Voyager and Traefik are lagging — as well as the impedance mismatch with cloud LBs such as the Google Load Balancer. So far, Kubernetes' ingress support has been generic: One ingress object can be used to "drive" different HTTP servers. The problem being, of course, that all HTTP implementations which have different settings (timeouts, TLS certs, CDN functionality) and concerns that the current, simple ingress format cannot support. I'm expecting this to change soon. Ingress portability really isn't an important concern, and the generic ingress format is a bottleneck for the ingress functionality to mature.
    - scurvy 9y ago
      >This way, one single cluster shares the same big, flat space of IP addresses; not only do pods have the same IP inside the container, but they can talk to other pods using the other pod's IP, too.< Why is having a big, flat namespace important? Routers route. Clos L3 networks are no longer a fancy thing. They're commonplace now. I don't see any advantage of having a flat network. > One part of it is the iptables proxy magic that it does to allow services to have dynamically assigned IPs, too, with simple load-balancing between them.< Ah yes, the iptables "magic". We call this, slowness and obfuscation. People who understand how to run networks don't like handwavy magic. We like simple, elegant concepts. Kubernetes networking is very far from simple and elegant. It's blackbox "magic". >You don't need more complex overlay networking stacks such as Calico, Flannel or Weave right away.< I run a native L3 network so have no need for an overlay network on top of it. That said, I'd argue that the overlay junk is probably easier for non-networking-fluent developers to setup and run compared to routing in AWS. Kubernetes networking can be summed up thusly: Great for developers who know nothing about networking but want to run at hyperscale. Terrible for people who actually know how to run networks properly. Kubernetes ingress is garbage. Stop apologizing for it. IPv6 would also get rid of 99% of these overly complicated hand-wavy solutions that Kubernetes proponents constantly tout as strong points. Give each node a /64, and you're set.
- cpuguy83 9y ago
  This is an overly simplified view of the differences between Docker networking and Kunernetes networking. The main difference is that Kunernetes assumes that all IP's are routable and Docker does not. When using bridge networking this means the admin must ensure routes are properly configured in the host for cross-host communication on Kunernetes. Docker does not provide cross-host service discovery for bridge networking out of the box. This does not prevent admins from setting this up themselves. For overlay networking solutions (e.g. Weave), the cross-host networking is handled for you and typically still even uses bridge networking to provide container connectivity, with service discovery also working cross-host. ipvlan and macvlan are "underlay" solutions (i.e. attached directly to the host networking interfaces). For these it is expected that the admin has configured the networking and that containers on different hosts are routable. Service discovery should work across hosts with these solutions, but actual networking is dependent on the how the host networking is setup because the containers will be assigned IP's from the host's network and are bound to a particular host network interface. When using ipvlan or macvlan (or overlay networking for that matter), Docker effectively makes the same assumptions as Kunernetes does for its networking.
  - cpuguy83 9y ago
    wow, thanks for some reason I got auto-corrected from Kubernetes to Kunernetes.... that's both frustrating and kind of funny.... maybe my experiment using swiftkey shall end soon...
markbnj 9y ago
At the risk of self-promotion :). https://medium.com/google-cloud/understanding-kubernetes-networking-pods-7117dd28727 https://medium.com/google-cloud/understanding-kubernetes-net... This is the first of a two-part series, the first dealing with pod networking and the second with services. I plan a third on ingress after kubecon. It's a little GKE-specific in the implementation details, and the whole thing is pluggable and can be configured in different ways (as the OP shows), but I think it covers the fundamentals pretty well.
manigandham 9y ago
Try this: https://www.youtube.com/watch?v=WwQ62OyCNz4 https://www.youtube.com/watch?v=WwQ62OyCNz4 And this: https://www.youtube.com/watch?v=y2bhV81MfKQ https://www.youtube.com/watch?v=y2bhV81MfKQ
Artemis2 9y ago
Try this: https://jvns.ca/blog/2017/10/10/operating-a-kubernetes-network/ https://jvns.ca/blog/2017/10/10/operating-a-kubernetes-netwo...
deepakjois 9y ago
All the resources suggested in replies to the parent thread seem great. I also found that somebody put out an illustrated guide to Kubernetes networking on Medium: * Part 1: https://medium.com/@ApsOps/an-illustrated-guide-to-kubernetes-networking-part-1-d1ede3322727 https://medium.com/@ApsOps/an-illustrated-guide-to-kubernete... * Part 2: https://medium.com/@ApsOps/an-illustrated-guide-to-kubernetes-networking-part-2-13fdc6c4e24c https://medium.com/@ApsOps/an-illustrated-guide-to-kubernete...

andrewstuart2 9y ago

The networking model itself is amazingly simple, and part of what makes Kubernetes so much easier to use. The rules are as follows: All containers can communicate with all other containers without NAT. All nodes can communicate with all containers (and vice-versa) without NAT. The IP that a container sees itself as is the same IP that others see it as. When using Docker by itself, you get into all sorts of complicated situations because most running containers have an IP address that's host-specific and not routable for any other machines. This makes networking across hosts a giant pain. Kubernetes takes that away by making things behave exactly how you'd hope they'd behave. My IP as I see it is reachable by anybody in the cluster who has it (policy permitting). The simplicity of working in this networking model means that there's a little more work for the networking infrastructure to handle, making sure that IPs are allocated without collision and that routes are known across many hosts. Several technologies exist to build these bridges, including old-school tech that has solved these exact problems for decades like BGP (see Calico/canal). Ultimately, there's no silver bullet. I'd recommend giving the k8s networking page a read. [1] [1] https://kubernetes.io/docs/concepts/cluster-administration/networking/ https://kubernetes.io/docs/concepts/cluster-administration/n...

scurvy 9y ago
That's amazingly simple? I notice that you conveniently left out the "ingress" component. Stuff in K8s talking with other K8s stuff is easy. Getting the flows into K8s apps from outside the K8 network is amazingly clunky in its current state.
- atombender 9y ago
  Network is simple from the container's point of view. It's less simple outside the container. But outside the container, the strategy is still much simpler than other solutions (most of which predate Kubernetes). Kubernetes chooses to give every pod its own IP. This means choosing an internal network such as 10.x.x.x, and giving each machine a slice of it. This way, one single cluster shares the same big, flat space of IP addresses; not only do pods have the same IP inside the container, but they can talk to other pods using the other pod's IP, too. But a key point is that Kubernetes is designed to take care of most of it. One part of it is the iptables proxy magic that it does to allow services to have dynamically assigned IPs, too, with simple load-balancing between them. The second part is the many built-in plugins for different, more complicated overlay strategies. Kubernetes' automatic configuration works out of the box on, say, AWS, without anything magical — Kubernetes natively talks to AWS to set up a routing table so that packets end up where they should. You don't need more complex overlay networking stacks such as Calico, Flannel or Weave right away. As for ingress, it has absolutely been Kubernetes' weakest point for several years, and the Kubernetes team knows this perfectly well. That said, it's not complicated, thanks to the above. Once you have, say, Nginx listening on a port, routing traffic into the cluster is a matter of setting up a load balancer (at least on clouds like GCP, DigitalOcean and AWS), something which Kubernetes even can do automatically for you. The weak links are the ingress controller — the Nginx one is popular because it's stable and supports common features such as TLS, whereas others such as Voyager and Traefik are lagging — as well as the impedance mismatch with cloud LBs such as the Google Load Balancer. So far, Kubernetes' ingress support has been generic: One ingress object can be used to "drive" different HTTP servers. The problem being, of course, that all HTTP implementations which have different settings (timeouts, TLS certs, CDN functionality) and concerns that the current, simple ingress format cannot support. I'm expecting this to change soon. Ingress portability really isn't an important concern, and the generic ingress format is a bottleneck for the ingress functionality to mature.
  - scurvy 9y ago
    >This way, one single cluster shares the same big, flat space of IP addresses; not only do pods have the same IP inside the container, but they can talk to other pods using the other pod's IP, too.< Why is having a big, flat namespace important? Routers route. Clos L3 networks are no longer a fancy thing. They're commonplace now. I don't see any advantage of having a flat network. > One part of it is the iptables proxy magic that it does to allow services to have dynamically assigned IPs, too, with simple load-balancing between them.< Ah yes, the iptables "magic". We call this, slowness and obfuscation. People who understand how to run networks don't like handwavy magic. We like simple, elegant concepts. Kubernetes networking is very far from simple and elegant. It's blackbox "magic". >You don't need more complex overlay networking stacks such as Calico, Flannel or Weave right away.< I run a native L3 network so have no need for an overlay network on top of it. That said, I'd argue that the overlay junk is probably easier for non-networking-fluent developers to setup and run compared to routing in AWS. Kubernetes networking can be summed up thusly: Great for developers who know nothing about networking but want to run at hyperscale. Terrible for people who actually know how to run networks properly. Kubernetes ingress is garbage. Stop apologizing for it. IPv6 would also get rid of 99% of these overly complicated hand-wavy solutions that Kubernetes proponents constantly tout as strong points. Give each node a /64, and you're set.
cpuguy83 9y ago
This is an overly simplified view of the differences between Docker networking and Kunernetes networking. The main difference is that Kunernetes assumes that all IP's are routable and Docker does not. When using bridge networking this means the admin must ensure routes are properly configured in the host for cross-host communication on Kunernetes. Docker does not provide cross-host service discovery for bridge networking out of the box. This does not prevent admins from setting this up themselves. For overlay networking solutions (e.g. Weave), the cross-host networking is handled for you and typically still even uses bridge networking to provide container connectivity, with service discovery also working cross-host. ipvlan and macvlan are "underlay" solutions (i.e. attached directly to the host networking interfaces). For these it is expected that the admin has configured the networking and that containers on different hosts are routable. Service discovery should work across hosts with these solutions, but actual networking is dependent on the how the host networking is setup because the containers will be assigned IP's from the host's network and are bound to a particular host network interface. When using ipvlan or macvlan (or overlay networking for that matter), Docker effectively makes the same assumptions as Kunernetes does for its networking.
- cpuguy83 9y ago
  wow, thanks for some reason I got auto-corrected from Kubernetes to Kunernetes.... that's both frustrating and kind of funny.... maybe my experiment using swiftkey shall end soon...

markbnj 9y ago

At the risk of self-promotion :). https://medium.com/google-cloud/understanding-kubernetes-networking-pods-7117dd28727 https://medium.com/google-cloud/understanding-kubernetes-net... This is the first of a two-part series, the first dealing with pod networking and the second with services. I plan a third on ingress after kubecon. It's a little GKE-specific in the implementation details, and the whole thing is pluggable and can be configured in different ways (as the OP shows), but I think it covers the fundamentals pretty well.

manigandham 9y ago

Try this: https://www.youtube.com/watch?v=WwQ62OyCNz4 https://www.youtube.com/watch?v=WwQ62OyCNz4 And this: https://www.youtube.com/watch?v=y2bhV81MfKQ https://www.youtube.com/watch?v=y2bhV81MfKQ

Artemis2 9y ago

Try this: https://jvns.ca/blog/2017/10/10/operating-a-kubernetes-network/ https://jvns.ca/blog/2017/10/10/operating-a-kubernetes-netwo...

deepakjois 9y ago

All the resources suggested in replies to the parent thread seem great. I also found that somebody put out an illustrated guide to Kubernetes networking on Medium: * Part 1: https://medium.com/@ApsOps/an-illustrated-guide-to-kubernetes-networking-part-1-d1ede3322727 https://medium.com/@ApsOps/an-illustrated-guide-to-kubernete... * Part 2: https://medium.com/@ApsOps/an-illustrated-guide-to-kubernetes-networking-part-2-13fdc6c4e24c https://medium.com/@ApsOps/an-illustrated-guide-to-kubernete...