3 ms·

K8s itself doesn’t introduce any real additional network complexity, at least not vanilla k8s. At the end of the day, K8s only takes care of scheduling contain

K8s itself doesn’t introduce any real additional network complexity, at least not vanilla k8s. At the end of the day, K8s only takes care of scheduling containers, and provides a super basic networking proxy layer for convenience. But there’s absolutely nothing in k8s that requires you use that proxy layer, or any other network overlay. You can easily setup pods that directly expose their ports on the node they’re running on, and have k8s services just provide the IPs of nodes running associated pods as a list. Then rely on either on clients to handle multiple addresses themselves (by picking an address at random, and failing over to another random address if needed), configure k8s DNS to provide DNS round robin, or put an NLB or something in front of it all. Everyone uses network overlays with k8s because it makes it easy for services in k8s to talk to other services in k8s. But there’s no requirement to force all your external inbound traffic through that layer. You can just use k8s to handle nodes, and collect needed meta-data for upstream clients to connect directly to services running on nodes with nothing but the container layer between the client and the running service.

andrewfromx 1y ago
| Aspect | Direct EC2 (No K8s) | Kubernetes (K8s Pods) | |-------------------------|-------------------------------------------------------|-------------------------------------------------------------------------------------| | Networking Layers | Direct connection to EC2 instance (optional load balancer). | Service VIP → kube-proxy → CNI → pod (plus optional external load balancer). | | Load Balancing | Optional, handled by ELB/ALB or application. | Built-in via kube-proxy (iptables/IPVS) and Service. | | IP Addressing | Static or dynamic EC2 instance IP. | Pod IPs are dynamic, abstracted by Service VIP. | | Connection Persistence | Depends on application and OS TCP stack. | Depends on session affinity, graceful termination, and application reconnection logic. | | Overhead | Minimal (direct TCP). | Additional latency from kube-proxy, CNI, and load balancer. | | Resilience | Connection drops if instance fails. | Connection may drop if pod is rescheduled, but Kubernetes can reroute to new pods. | | Configuration Complexity| Simple (OS-level TCP tuning). | Complex (session affinity, PDBs, graceful termination, CNI tuning). |
- avianlyric 1y ago
  If you read my reply again, you’ll notice that I explicitly highlight that K8s does not require the use of a CNI. There’s a reason CNIs are plugins, and not core parts of k8s. How do you think external network traffic gets routed into a CNIs front proxy? It’s not via kube-proxy, kube-proxy isn’t designed for use in proper production systems, it’s only a stop gap to provide a functioning cluster to enable bootstrapping of a proper network management layer. There is absolutely nothing preventing a network layer directly routing external traffic to pods, with the only translation being a basic iptable rule to enable routing of data sent to a nodes network interface with a pod IP to be accepted by the node and routed to the pod. Given it’s just basic Linux network interface bridging, happening entirely in the kernel with zero copies, the impact of this layer is practically zero. Indeed the k8s services setup with external load balancers basically handle all of this setup for you. There are plenty of reasons not to use k8s, but arguing that a k8s cluster must inherently introduce multiple additional network components and complexity is simply incorrect.
  - andrewfromx 1y ago
    While Kubernetes theoretically allows for simple iptables-based routing, in practice, very few production environments stop there. Most clusters do use CNIs, kube-proxy, Service abstraction, and often external load balancers or Ingress controllers, which together form a nontrivial networking stack. The claim that kube-proxy is “not designed for use in proper production systems” is simply incorrect. It is widely used in production and supported by Kubernetes core. While it has limitations—especially with high-connection-load environments—it is still the default in most distros, and even advanced CNIs like Calico and Cilium often interact with or replace kube-proxy functionality rather than ignore it. If kube-proxy is just a stopgap, why do so many managed Kubernetes platforms like GKE, EKS, and AKS still ship with it? While it’s true CNIs are plugins and not core to Kubernetes, this outsources complexity rather than eliminates it. Different CNIs (e.g., Calico, Cilium, Flannel, etc.) use different overlay models, BPF implementations, or routing approaches. Even in a CNI with kernel-fast path routing, pod churn, rolling updates, or horizontal scaling still introduce issues. Service IPs are stable, but pod IPs are ephemeral, which means Long-lived TCP clients pinned to pod IPs. You can design a lean Kubernetes network path with minimal abstractions, but: You lose things like dynamic service discovery, load balancing, and readiness-based traffic shifting. You must manage more infra manually (e.g., configure iptables or direct routing rules yourself). You’re fighting against the grain of what Kubernetes is designed to do.

andrewfromx 1y ago

| Aspect | Direct EC2 (No K8s) | Kubernetes (K8s Pods) | |-------------------------|-------------------------------------------------------|-------------------------------------------------------------------------------------| | Networking Layers | Direct connection to EC2 instance (optional load balancer). | Service VIP → kube-proxy → CNI → pod (plus optional external load balancer). | | Load Balancing | Optional, handled by ELB/ALB or application. | Built-in via kube-proxy (iptables/IPVS) and Service. | | IP Addressing | Static or dynamic EC2 instance IP. | Pod IPs are dynamic, abstracted by Service VIP. | | Connection Persistence | Depends on application and OS TCP stack. | Depends on session affinity, graceful termination, and application reconnection logic. | | Overhead | Minimal (direct TCP). | Additional latency from kube-proxy, CNI, and load balancer. | | Resilience | Connection drops if instance fails. | Connection may drop if pod is rescheduled, but Kubernetes can reroute to new pods. | | Configuration Complexity| Simple (OS-level TCP tuning). | Complex (session affinity, PDBs, graceful termination, CNI tuning). |

avianlyric 1y ago
If you read my reply again, you’ll notice that I explicitly highlight that K8s does not require the use of a CNI. There’s a reason CNIs are plugins, and not core parts of k8s. How do you think external network traffic gets routed into a CNIs front proxy? It’s not via kube-proxy, kube-proxy isn’t designed for use in proper production systems, it’s only a stop gap to provide a functioning cluster to enable bootstrapping of a proper network management layer. There is absolutely nothing preventing a network layer directly routing external traffic to pods, with the only translation being a basic iptable rule to enable routing of data sent to a nodes network interface with a pod IP to be accepted by the node and routed to the pod. Given it’s just basic Linux network interface bridging, happening entirely in the kernel with zero copies, the impact of this layer is practically zero. Indeed the k8s services setup with external load balancers basically handle all of this setup for you. There are plenty of reasons not to use k8s, but arguing that a k8s cluster must inherently introduce multiple additional network components and complexity is simply incorrect.
- andrewfromx 1y ago
  While Kubernetes theoretically allows for simple iptables-based routing, in practice, very few production environments stop there. Most clusters do use CNIs, kube-proxy, Service abstraction, and often external load balancers or Ingress controllers, which together form a nontrivial networking stack. The claim that kube-proxy is “not designed for use in proper production systems” is simply incorrect. It is widely used in production and supported by Kubernetes core. While it has limitations—especially with high-connection-load environments—it is still the default in most distros, and even advanced CNIs like Calico and Cilium often interact with or replace kube-proxy functionality rather than ignore it. If kube-proxy is just a stopgap, why do so many managed Kubernetes platforms like GKE, EKS, and AKS still ship with it? While it’s true CNIs are plugins and not core to Kubernetes, this outsources complexity rather than eliminates it. Different CNIs (e.g., Calico, Cilium, Flannel, etc.) use different overlay models, BPF implementations, or routing approaches. Even in a CNI with kernel-fast path routing, pod churn, rolling updates, or horizontal scaling still introduce issues. Service IPs are stable, but pod IPs are ephemeral, which means Long-lived TCP clients pinned to pod IPs. You can design a lean Kubernetes network path with minimal abstractions, but: You lose things like dynamic service discovery, load balancing, and readiness-based traffic shifting. You must manage more infra manually (e.g., configure iptables or direct routing rules yourself). You’re fighting against the grain of what Kubernetes is designed to do.