3 ms·

While Kubernetes theoretically allows for simple iptables-based routing, in practice, very few production environments stop there. Most clusters do use CNIs, ku

While Kubernetes theoretically allows for simple iptables-based routing, in practice, very few production environments stop there. Most clusters do use CNIs, kube-proxy, Service abstraction, and often external load balancers or Ingress controllers, which together form a nontrivial networking stack. The claim that kube-proxy is “not designed for use in proper production systems” is simply incorrect. It is widely used in production and supported by Kubernetes core. While it has limitations—especially with high-connection-load environments—it is still the default in most distros, and even advanced CNIs like Calico and Cilium often interact with or replace kube-proxy functionality rather than ignore it. If kube-proxy is just a stopgap, why do so many managed Kubernetes platforms like GKE, EKS, and AKS still ship with it? While it’s true CNIs are plugins and not core to Kubernetes, this outsources complexity rather than eliminates it. Different CNIs (e.g., Calico, Cilium, Flannel, etc.) use different overlay models, BPF implementations, or routing approaches. Even in a CNI with kernel-fast path routing, pod churn, rolling updates, or horizontal scaling still introduce issues. Service IPs are stable, but pod IPs are ephemeral, which means Long-lived TCP clients pinned to pod IPs. You can design a lean Kubernetes network path with minimal abstractions, but: You lose things like dynamic service discovery, load balancing, and readiness-based traffic shifting. You must manage more infra manually (e.g., configure iptables or direct routing rules yourself). You’re fighting against the grain of what Kubernetes is designed to do.

avianlyric 1y ago
Not entirely sure what your point is. You state that Kubernetes must come with a complex networking stack, and I point out that simply isn’t true, which apparently you agree with. But because other people use complex networking stacks in Kubernetes, that means for a bunch of good reasons, that apparently means it can’t be used without that stack? If you don’t need that functionality, and you have different requirements, then why would you implement a k8s stack that includes all that functionality? That would be foolish. Everyone ships with kube-proxy because it’s the simplest way to provide an unopinionated network stack that will give you a fully functional k8s cluster with expected bells and whistles like simple service discovery. But most people quickly replace it with a CNI for various reasons. My understanding is that your application is all about managing high throughput, direct to application TCP connections. So I have no idea why you’re talking about the need for fully fledged service discovery, or long lived TCP connections pinned to a single pod (I mean, that just the nature of TCP, got nothing to do with k8s or any of its network stacks). > You can design a lean Kubernetes network path with minimal abstractions, but: You lose things like dynamic service discovery, load balancing, and readiness-based traffic shifting. You must manage more infra manually (e.g., configure iptables or direct routing rules yourself). You’re fighting against the grain of what Kubernetes is designed to do. If you’re building a stack without k8s, then you don’t get any of those feature either, so not sure what the fuss is about. Doesn’t mean that’s k8s core competency as a workload schedular, and its general framework for managing stateful and stateless resources doesn’t provide a lot of value. You just need to learn how to take advantage of it, rather than following some cookie cutter setup for a stack that doesn’t meet your needs.
- andrewfromx 1y ago
  My point is this: Having worked at multiple IoT companies with many millions of connections. This is the way. People tend to overcomplicate things with K8S. I have never once seen a massively distributed IoT system run without a TON of headache and outages with k8s. Sure, it can be done, but it requires spending 4-8x the amount of of development time and has many more outages due to random things. It's not just the network, its also the amount of config you have to do to get a deterministic system. For IoT, you dont need as much bursting (for most workloads). Its a bunch of devices that are connected 24/7 with fairly deterministic workloads, that are usually using some type of TCP connection that is not HTTP, and trying to shove it into an HTTP paradigm costs more money and more complexity and is not reliable. https://news.ycombinator.com/item?id=44032250 https://news.ycombinator.com/item?id=44032250
  - avianlyric 1y ago
    > People tend to overcomplicate things with K8S. I have never once seen a massively distributed IoT system run without a TON of headache and outages with k8s I don’t dispute that. I’m simply saying that people overcomplicating their K8s setup isn’t evidence that you can’t build simple K8s setups with simple networking. It’s just that requires people to actually think about their k8s design, and not just copy common k8s setups which are usually geared towards different workloads. > that are usually using some type of TCP connection that is not HTTP, and trying to shove it into an HTTP paradigm costs more money and more complexity and is not reliable. What has HTTP got to do with k8s? There’s absolutely nothing in k8s that requires the use of HTTP for network. I’ve run plenty of workloads that used long lived TCP connections with binary protocols that shared nothing with HTTP.

avianlyric 1y ago

Not entirely sure what your point is. You state that Kubernetes must come with a complex networking stack, and I point out that simply isn’t true, which apparently you agree with. But because other people use complex networking stacks in Kubernetes, that means for a bunch of good reasons, that apparently means it can’t be used without that stack? If you don’t need that functionality, and you have different requirements, then why would you implement a k8s stack that includes all that functionality? That would be foolish. Everyone ships with kube-proxy because it’s the simplest way to provide an unopinionated network stack that will give you a fully functional k8s cluster with expected bells and whistles like simple service discovery. But most people quickly replace it with a CNI for various reasons. My understanding is that your application is all about managing high throughput, direct to application TCP connections. So I have no idea why you’re talking about the need for fully fledged service discovery, or long lived TCP connections pinned to a single pod (I mean, that just the nature of TCP, got nothing to do with k8s or any of its network stacks). > You can design a lean Kubernetes network path with minimal abstractions, but: You lose things like dynamic service discovery, load balancing, and readiness-based traffic shifting. You must manage more infra manually (e.g., configure iptables or direct routing rules yourself). You’re fighting against the grain of what Kubernetes is designed to do. If you’re building a stack without k8s, then you don’t get any of those feature either, so not sure what the fuss is about. Doesn’t mean that’s k8s core competency as a workload schedular, and its general framework for managing stateful and stateless resources doesn’t provide a lot of value. You just need to learn how to take advantage of it, rather than following some cookie cutter setup for a stack that doesn’t meet your needs.

andrewfromx 1y ago
My point is this: Having worked at multiple IoT companies with many millions of connections. This is the way. People tend to overcomplicate things with K8S. I have never once seen a massively distributed IoT system run without a TON of headache and outages with k8s. Sure, it can be done, but it requires spending 4-8x the amount of of development time and has many more outages due to random things. It's not just the network, its also the amount of config you have to do to get a deterministic system. For IoT, you dont need as much bursting (for most workloads). Its a bunch of devices that are connected 24/7 with fairly deterministic workloads, that are usually using some type of TCP connection that is not HTTP, and trying to shove it into an HTTP paradigm costs more money and more complexity and is not reliable. https://news.ycombinator.com/item?id=44032250 https://news.ycombinator.com/item?id=44032250
- avianlyric 1y ago
  > People tend to overcomplicate things with K8S. I have never once seen a massively distributed IoT system run without a TON of headache and outages with k8s I don’t dispute that. I’m simply saying that people overcomplicating their K8s setup isn’t evidence that you can’t build simple K8s setups with simple networking. It’s just that requires people to actually think about their k8s design, and not just copy common k8s setups which are usually geared towards different workloads. > that are usually using some type of TCP connection that is not HTTP, and trying to shove it into an HTTP paradigm costs more money and more complexity and is not reliable. What has HTTP got to do with k8s? There’s absolutely nothing in k8s that requires the use of HTTP for network. I’ve run plenty of workloads that used long lived TCP connections with binary protocols that shared nothing with HTTP.