4 ms·
I feel like k8s networking is 10x more complex than it needs to be... I think it's because they still want to use the socket API, but to "pretend" you can open
by londons_explore 7y ago
I feel like k8s networking is 10x more complex than it needs to be...
I think it's because they still want to use the socket API, but to "pretend" you can open a socket to a pod, or to a service, etc.
Instead, they should have defined a new API, like "connect('k8s:service:foobar?selectionstrategy=leastloaded')"
Then they could have made simple LD_PRELOAD shims to get their magic API to work with existing software without modification.
There would be no need for millions of IPtables rules, virtual IP addresses, port remapping, proxy processes, etc.
- justinsaccount 7y agoThat is basically how consul connect works: https://www.consul.io/docs/connect/native/go.html https://www.consul.io/docs/connect/native/go.html
- atombender 7y agoNot as I understand it. It uses DNS resolution, so you connect to something like foo.web.service.consul. Kubernetes's KubeDNS works exactly the same way with Kubernetes service objects.
- justinsaccount 7y agoFrom that page: // Create an instance representing this service. "my-service" is the // name of _this_ service. The service should be cleaned up via Close. svc, _ := connect.NewService("my-service", client) and when http is used: > The hostname used in the request URL is used to identify the logical service discovery mechanism for the target. It's not actually resolved via DNS but used as a logical identifier for a Consul service discovery mechanism. It has the following specific limitations:
- atombender 7y agoAh, I see. That's presumably possible because Go has its own DNS resolver, and it (net.DefaultResolver) can be overridden by libraries; no LD_PRELOAD needed.
- justinsaccount 7y agoit's because connect.NewService is the consul client, not something like net.DialTCP
- atombender 7y agoOh, they require all TCP connections to go through that. That's a bit heavy-handed — requires rewriting all client code to adapt to that pattern. The nice thing about KubeDNS is that you don't have to. Same with Istio.
- peterwwillis 7y agoIntegrating with app code is optional. The default is to use a sidecar.
- justinsaccount 7y agoI brought up the native client because you mentioned "they should have defined a new API".
- atombender 7y agoThat wasn't me. I'd argue the opposite. Though I'd also maybe argue that SRV records is what we should have standardized on for this stuff a long time ago.
- atombender 7y agoHow is that different from today's use of KubeDNS? Kubernetes' networking would arguably have been less complex (multiple CIDR subnets, proxying, etc.) if they'd embraced IPv6. But Google doesn't seem big on IPv6.
- 0xEFF 7y agoAPI incompatibility is largely why AppEngine failed. Cloud adoption suffers greatly when the platform is not compatible with the data center.
- pas 7y agoIt's complex because K8S is used in a wide variety of complex environments. Secure dynamic overlay networking, plus egress is inherently not a trivial thing. Then add customization/extendability on top. The important thing to get is that it uses really simple building blocks, very UNIX-y. Every time a new container starts it starts a program (the CNI driver) and pipes in a JSON. And the driver does what it wants (executes iptables, BGP announces a prefix, commits something to its internal DB) and spits out some JSON that tells what IP the container got. And .. that's it. No magic. Everything has to be configured, arguments passed to some daemon at cluster setup time, and so on. Yes, kubelet and apiserver both have hundreds of command line args, but [almost] all of them are very clearly associated with tuning a functionality (or passing some data forward to some daemon/driver/component) - and thus if you don't touch it but it works, yaay, you don't have to care about it (though of course it might turn out that you'll need to scale up/down/faster/lighter). The socket API is fine, even with a new shiny API the network fundamentals would have to be managed just the same way. The Linux kernel would be the same, just as Ethernet, IP and so on. Reinventing all of those wheels just to provide transparent pod-to-pod networking doesn't seem to ever worth it.