3 ms·

> Networking in kubernetes was wonky. Can you elaborate? Does Kubernetes add some thrills to the relatively simple Docker network configuration?

krab 5y ago
Yes, Kubernetes (actually its add-ons) provide a virtual network that unifies communication within the cluster so you don't need to care on which computer your service runs.
jrockway 5y ago
You can make it as complicated as you want it to be; part of setting up a cluster is picking the networking system ("CNI"). Cloud providers often have their own IPAM (i.e. on Amazon, you get this: https://docs.aws.amazon.com/eks/latest/userguide/pod-networking.html; https://docs.aws.amazon.com/eks/latest/userguide/pod-network... each Pod gets an IP from your VPC, resulting in weird limits like 17 pods per instance because that's how many IP addresses you can have for that particular instance type throughout EC2).
kaidon 5y ago
Please take a seat and let the joys of k8s networking overwhelm your senses: https://kubernetes.io/docs/concepts/cluster-administration/networking/ https://kubernetes.io/docs/concepts/cluster-administration/n... And yes... Kubernetes network configuration is on a whole different level from docker networking.
- theptip 5y ago
  To be fair, multi-node networking of any sort is on a different level than single-host docker networking. If you ever tried to use Docker Swarm to network multiple nodes, god help you. Also worth noting that almost all users of K8s don't actually need to operate a cluster, the hosted offerings handle all of that for you. You just need to understand the Service object, and maybe Ingress if you're trying to do some more advanced cert management or API gateway stuff. It's a common meme around here to point in horror to the complexity that is abstracted away under the K8s cluster API, and claim that k8s is really hard to use. I think that's mostly misguided, the hosted offerings like GKE really do a good job of hiding away all that complexity from you. Honestly I think that it's defensible to say that the k8s networking model is in most cases _simpler_ than what you'd end up configuring in AWS / GCP to route traffic from the internet to multiple VM nodes.
  - kazen44 5y ago
    > Honestly I think that it's defensible to say that the k8s networking model is in most cases _simpler_ than what you'd end up configuring in AWS / GCP to route traffic from the internet to multiple VM nodes. How is routing from the internet to multiple servers a problem? usually, you have either one of these setups: - you run a loadbalancer that distributes traffic across your nodes. (This loadbalancer could even be distributed thanks to BGP). - you either run your own firewall or have a managed one, in which you either announce your IP prefix yourself, or they are announced for you by your uplink provider. - you run an anycast setup (for, for example, globally distributed DNS). and announce multiples of the same prefix across the globe. Routing in the DFZ does the rest for you. Streched L2 across the globe/internet is also possible (although not very performant) either by doing IPsec tunneling, or by buying/setting up L2VPN services. (either MPLS or VXLAN based).
    - theptip 5y ago
      I didn't say it was a problem. My claim was just that it's easier in GKE than in GCE/EC2. I only mentioned multi-node because exposing a single VM to the internet is trivial -- just give it a public IP -- and thus is not an apples-to-apples comparison with the multi-node load balancing that you get from the entry-level k8s configuration of Service > Pod < Deployment.
kazen44 5y ago
the largest issue with kubernetes networking seems to be the lack of integration with modern datacenter networking technology. Things like VXLAN-EVPN are supported on paper, but are no where near mature compared to offerings from normal networking vendors. Heck, even the BGP support inside kubernetes is lacking. Which is a great shame because it creates a barrier between pods and the physical world. (Getting a VXLAN VTEP mapped to a kubernetes node is a major PITA for instance). Most major cloud providers seem to have fixed this by building even more overlay networks (with the included inefficiencies).

krab 5y ago

Yes, Kubernetes (actually its add-ons) provide a virtual network that unifies communication within the cluster so you don't need to care on which computer your service runs.

jrockway 5y ago

You can make it as complicated as you want it to be; part of setting up a cluster is picking the networking system ("CNI"). Cloud providers often have their own IPAM (i.e. on Amazon, you get this: https://docs.aws.amazon.com/eks/latest/userguide/pod-networking.html; https://docs.aws.amazon.com/eks/latest/userguide/pod-network... each Pod gets an IP from your VPC, resulting in weird limits like 17 pods per instance because that's how many IP addresses you can have for that particular instance type throughout EC2).

kaidon 5y ago

Please take a seat and let the joys of k8s networking overwhelm your senses: https://kubernetes.io/docs/concepts/cluster-administration/networking/ https://kubernetes.io/docs/concepts/cluster-administration/n... And yes... Kubernetes network configuration is on a whole different level from docker networking.

theptip 5y ago
To be fair, multi-node networking of any sort is on a different level than single-host docker networking. If you ever tried to use Docker Swarm to network multiple nodes, god help you. Also worth noting that almost all users of K8s don't actually need to operate a cluster, the hosted offerings handle all of that for you. You just need to understand the Service object, and maybe Ingress if you're trying to do some more advanced cert management or API gateway stuff. It's a common meme around here to point in horror to the complexity that is abstracted away under the K8s cluster API, and claim that k8s is really hard to use. I think that's mostly misguided, the hosted offerings like GKE really do a good job of hiding away all that complexity from you. Honestly I think that it's defensible to say that the k8s networking model is in most cases _simpler_ than what you'd end up configuring in AWS / GCP to route traffic from the internet to multiple VM nodes.
- kazen44 5y ago
  > Honestly I think that it's defensible to say that the k8s networking model is in most cases _simpler_ than what you'd end up configuring in AWS / GCP to route traffic from the internet to multiple VM nodes. How is routing from the internet to multiple servers a problem? usually, you have either one of these setups: - you run a loadbalancer that distributes traffic across your nodes. (This loadbalancer could even be distributed thanks to BGP). - you either run your own firewall or have a managed one, in which you either announce your IP prefix yourself, or they are announced for you by your uplink provider. - you run an anycast setup (for, for example, globally distributed DNS). and announce multiples of the same prefix across the globe. Routing in the DFZ does the rest for you. Streched L2 across the globe/internet is also possible (although not very performant) either by doing IPsec tunneling, or by buying/setting up L2VPN services. (either MPLS or VXLAN based).
  - theptip 5y ago
    I didn't say it was a problem. My claim was just that it's easier in GKE than in GCE/EC2. I only mentioned multi-node because exposing a single VM to the internet is trivial -- just give it a public IP -- and thus is not an apples-to-apples comparison with the multi-node load balancing that you get from the entry-level k8s configuration of Service > Pod < Deployment.

kazen44 5y ago

the largest issue with kubernetes networking seems to be the lack of integration with modern datacenter networking technology. Things like VXLAN-EVPN are supported on paper, but are no where near mature compared to offerings from normal networking vendors. Heck, even the BGP support inside kubernetes is lacking. Which is a great shame because it creates a barrier between pods and the physical world. (Getting a VXLAN VTEP mapped to a kubernetes node is a major PITA for instance). Most major cloud providers seem to have fixed this by building even more overlay networks (with the included inefficiencies).