6 ms·
The open is weak (mutable pods is considered an inherent anti-pattern in k8s), but I think he’s got a lot of good points about networking. Every time I look at
by dijit 6y ago
The open is weak (mutable pods is considered an inherent anti-pattern in k8s), but I think he’s got a lot of good points about networking.
Every time I look at k8s networking seriously it gives me great pause on whether I should continue to run such a complex system. IPv6+EtcD would solve this matter really well.
- joshuak 6y agoYeah the value of being able to eliminate pod mutation as a source of trouble is pretty hard to overcome by any feature I can imagine mutability offering. Certainly not the example given. Actually the example must be over my head, because I do the equivalent of SIGTERM restarts with new configs on pods with a single command all day long when building services.
- jmillikin 6y agoYou don't need to use the complex stateful overlay networks. At Stripe we have a network overlay using IPv6 and the Linux kernel's built-in stateless tunnel device, so there's effectively unlimited addresses with no coordination between worker machines and no iptables port remapping.
- eric_khun 6y agoI'm curious about your practises regarding CPU Limits at Stripe. Do you noticed severe CPU throttling? What's your guidelines on this?
- jmillikin 6y agoProduct teams deploying code to our Kubernetes clusters are strongly recommended to use resource limits, and we're going to make that a hard requirement at some point. We haven't noticed unusual CPU throttling, though we do have some workloads that turned out to be burstier than expected and had to adjust their CPU limits to match. Note that when it comes to subtle Linux thread scheduling behavior, your experience will depend on which runtime you use, and if using runc then which version of the Linux kernel your workers run. We weren't affected by the CFS bug introduced in Linux v4.18 because we never ran Kubernetes workloads on a machine with the affected kernel, and if a similar bug occurs in the future it might not affect workloads running within gVisor or Firecracker. Additionally, Stripe has historically cared more about security than efficiency. This lead to an architecture where services run on dedicated VMs, which naturally strands capacity and reduces the impact of bugs that appear at high utilization and/or high core count.
- xyzzy_plugh 6y agoI believe you did run into the CFS bug in non-Kubernetes workloads, though, specifically with Hadoop tasks. An engineer told me about a workaround he devised using cpuset.
- jmillikin 6y agoThat's likely a different bug. From what I understand the CFS bug being discussed was introduced in Linux v4.18 and fixed in v5.3, and we have not used a kernel within in that range in our Hadoop clusters.
- mnahkies 6y agoNot from stripe, but I've seen pretty bad CPU throttling. Often see quota's get exhausted through short bursts that don't show up in metrics that then causes CFS throttling to occur even though it looks like the pod is no where near its limit. Also struggled with application startup requiring far more CPU than at runtime leading to ridiculously slow startup times if you had a low limit. So far our solution has been to just remove CPU limits, but hoping things will get better. Removing the limits really improved our latency tail, and so far hasn't resulted in CPU saturation at the node level but your mileage may vary
- tbrock 6y agoDid you know that the Linux kernel has a bug that makes CPU limits for containers extra costly? https://github.com/kubernetes/kubernetes/issues/67577 https://github.com/kubernetes/kubernetes/issues/67577 https://github.com/torvalds/linux/commit/512ac999d2755d2b7109e996a76b6fb8b888631d https://github.com/torvalds/linux/commit/512ac999d2755d2b710... If I recall correctly you need 4.18+ to get the fix.
- dilyevsky 6y agoWe’ve seen just regular throttling too especially with erlang vm and go workloads
- fesc 6y agoCould you elaborate on that a little bit? This sounds interesting and I'm not sure where to start researching about this.
- jmillikin 6y agoOn my mobile so hard to go.into detail, but take a look at 6rd and the Linux 6to4 tunnel driver. You can assign a prefix to the overlay, then each machine's IPv4 becomes a subnet, and packets can be routed by the kernel knowing only the destination pod's IPv6 address.
- bogomipz 6y agoIndeed and if you bring up a Kubernetes cluster today on AWS using their EKS there is no overlay network. There's a CNI but no overlay.
- oneplane 6y agoIn some deployments we remove the VPC CNI and use an overlay anyway because of the integrations we need. You do lose Security Groups but that's not a big deal if you aren't using them in your deployment anyway.
- bogomipz 6y agoOh sure there's lots of good reasons for not using their CNI not least of which is needing to worry IP space utilization. I was just trying to point out overlays aren't a requirement.
- AlphaSite 6y agoI would need to double check, but isn’t it still running an overlay? Just one which is transparent to you?
- bogomipz 6y agoWell an overlay network by definition is always going to be transparent to you. But no the AWS VPC CNI does not create an overlay. It's just layer 3. It works by adding ENIs to your worker nodes with secondary IP addresses on them. And those secondary IPs are from your VPC's address space. See: https://github.com/aws/amazon-vpc-cni-k8s/blob/master/docs/cni-proposal.md https://github.com/aws/amazon-vpc-cni-k8s/blob/master/docs/c...
- oneplane 6y agoYep, another reason it does that is to spread the risk and limit the need for tons of unchecked (no src/dst check) addresses in your VPC. In the end there is no ideal scenario; boils down to what works best for the use case (or what is the 'least worst' solution). Sometimes it gets you down, but those imperfections can turn a churn job into an interesting one.
- ignoramous 6y ago> At Stripe we have a network overlay using IPv6 and the Linux kernel's built-in stateless tunnel device... Can you please expand on this and if possible point to some references that'd help in case I want to go down this route myself? Thx.
- jmillikin 6y agoI posted briefly elsewhere in the thread, but am now back at a compute with a keyboard so can give a more detailed answer. First, background reading: https://en.wikipedia.org/wiki/6to4 https://en.wikipedia.org/wiki/6to4 https://en.wikipedia.org/wiki/IPv6_rapid_deployment https://en.wikipedia.org/wiki/IPv6_rapid_deployment The basic idea is you create a SIT tunnel device and assign it an IPv6 /64 composed of two parts: 1. A network prefix between 32 and 56 bits long. This prefix is the same for all machines in the network. 2. A subnet derived from the machine's IPv4 address, minus the netmask. For example, if your IPv4 addresses are allocated from 192.168.1.0/24 and the machine has 192.168.1.155, then the network prefix should be 56 bits long (64 - (32 - 24)) and the machine's prefix is `xxxx:xxxx:xxxx:xx9B::/64`. The Linux kernel knows how to wrap the IPv6 with IPv4 so it can route within your local network to any other machine with a similarly configured tunnel device. If you want to send packets to 192.168.1.200 then they get addressed to `xxxx:xxxx:xxxx:xxC8::1` or whatever, they'll transit the IPv4 network like normal, and on arrival the receiving machine's kernel will strip off the IPv4 wrapper and route the IPv6 locally. How's this useful? Well, if each machine has a /64 prefix then each pod can be allocated an IPv6 within that prefix without coordinating with other machines. Let's say the pod gets `xxxx:xxxx:xxxx:xxC8::aaaa:bbbb:cccc`. Anything with a correctly configured tunnel and that pod IP can route it traffic, no proxy or iptables needed.
- simonebrunozzi 6y agoThis deserves a blog post in itself. Please be kind and share it with the world! I bet you will hire a few engineers as a result of this blog post :)
- rualca 6y agoOutstanding post. Thank you for taking the time to share this gem.
- bboreham 6y agoHe’s overestimating how much money is made by selling Kubernetes network addons.
- throwaway894345 6y agoHe addresses that pretty directly in his post, unless i’m misunderstanding you somehow?
- bboreham 6y agoI think you must have misunderstood one or the other.
- nvarsj 6y agoI came here to post pretty much the same thing. Networking in k8s is the poorest designed aspect of Kubernetes. So much so, I'm honestly surprised it didn't kill off K8s early on. You still can't get proper service layer load balancing out of the box w/ iptables. I'm not sure the history - it's the simplest thing possible, I guess, but doesn't perform very well at the most basic of tasks. To try and solve this, we have the current mess of service meshes, CNI, etc. It's going back to the middleware days of yore - which I think anyone with operational experience should know we really need to avoid. Having multiple layers of network proxies and masquerading between service calls, on an internal network, is just ridiculous and difficult to debug or operate at scale.
- intricatedetail 6y agoWhat rescued kubernetes was adoption by AWS and Google. If you want to host Kubernetes yourself it is still a waste of time broadly speaking.
- joana035 6y agoKubernetes didn't need to be rescued in the first place. If you know how to manage any Linux box, setting up kubernetes is as easy as "# apt install kubelet kubectl". Broadly speaking kubernetes saves time by providing an api for existing Linux technology. For example, iptables, ipvs, storage, self-healing, virtual ips, etc via yaml. Kubernetes was created to give anyone "Google like, production grade" infrastructure without the vendor lock-in of the cloud. And kubernetes on physical machines is where it shines most. EDIT: it is very easy to complain that kubernetes is complex, but try to do virtual ips with keepalived, corosync, pacemaker to understand how much time kubernetes saves by providing the same capability in a generic way which is easy to use. I have the feeling that people take kubernetes for granted without knowing how much manual work one has to go through in order to build a similar experience using the standard building blocks available. EDIT 2: straight out of kubernetes home page: "Kubernetes builds upon 15 years of experience running production workloads at Google" that is a lot of saved time just by using kubernetes.
- deleted 6y ago
- fulafel 6y agoHear, hear. The unquestioning attitude towards the complex system of application level proxies is disquieting.
- stingraycharles 6y agoYeah I’ve been dying to have a “simple” k8s solution that either uses just a VLAN or ipv6. It’s been one of the few components of k8s that I simply do not understand the rationale for, it seems so much engineering and complexity for what, exactly?
- dilyevsky 6y agoCilium can be configured to use vxlan and etcd to coordinate machine subnets (it does routing using ebpf)
- arianvanp 6y agoI've used calico to set up a flat IPv6 cluster without overlays. It was pretty pleasant. https://github.com/arianvp/packet-ipv6-kubernetes https://github.com/arianvp/packet-ipv6-kubernetes
- dijit 6y agoSeems GKE uses calico, though not with ipv6 (as GCP doesn't support it internally), so at least this networking solution has some buy-in. (broadly speaking)
- elFarto 6y agoThere's Bridget[1], a non-overlay network for k8s that can use/create a VLAN over a network adapter (or just use the network adapter directly). [1] https://github.com/kvaps/bridget https://github.com/kvaps/bridget
- hodgesrm 6y ago> IPv6+EtcD would solve this matter really well. Like others on this thread I completely agree that k8s networking is over-the-top complex. Still, it is hard to understand how this addresses relatively common use cases like providing intelligent load balancing for clients outside the cluster or secure multitenancy within it. Solving this requires a step back to think through simplifying networking itself. I thought amazon did the world a favor by removing L2 networking from their model. We need a better set of primitives that abstracts out as much of the low level implementation as possible. The current situation feels like data management before the advent of the relational model.
- deleted 6y ago[deleted]