51 ms·

So I agree with that phrase -- I'd say that the phrase CNI complexity generally refers to at least these things for me: - the complexity of networking in gener

So I agree with that phrase -- I'd say that the phrase CNI complexity generally refers to at least these things for me: - the complexity of networking in general, which is exposed. If I have to know what a CIDR/IPAM/etc are, then that complexity has leaked through. - the near-oneness of kubernetes and CNI -- CNI seems like a small facet of kubernetes (especially early on) but choosing the right provider, knowing what that JSON means etc is a big source of complexity -- combining that with the (usually) new world of complexities that k8s wrangles can be too much. - deciding that every pod must be addressable was a mistake IMO. It would have been harder to design but I think some sort of connection/network proxy should have been developed up front and iterated on (maybe I'm unaware of the history and this did indeed happen). When I spin up two containers, if they ought to talk to each other, I would have preferred so make some `NetworkConnection` object over the ports they need to speak over, or some virtual network they should be a part of. Security would be improved and the explicitness of connection/being present in a network could have obscured the underlying networking complexity. - debugging CNI brings you back into the regular world of linux networking -- tcpdump/iptables/ipvs-lvs/conntrack/etc when things break down in those layers people are even more confused than regular containers since just about everyone knows top/ps maybe even strace but far less people know how tools like ip/ifconfig/etc work. CNI is for the most part really simple and really easy to use these days, but I definitely can't say I'm confident with it -- I've been running kube-router since I found it because it was simpler than kube-proxy + flannel + calico (there's also canal). If you use some of the newer distributions (k3s, k0s) you don't even think about it. Also not securing node<->node communication was a pretty rough decision as well IMO. The Konnectivity service is a pain to think about and deploy, but luckily for me I can sidestep the whole thing with wireguard -- and calico will do it for me. There's also Cilium out there which does similar things. CNI is a smashing success -- people are building innovative things, competing, and clusters share the benefits -- but I still don't know that I'd call it simple yet, I just hope it never breaks.

bboreham 6y ago
Most of what you describe there is Kubernetes. > the near-oneness of kubernetes and CNI This is not something that can be blamed on CNI. We keep strictly even-handed.
- hardwaresofton 6y ago
  True there is a certain bit of mis-attribution but I think others are lumping them together as well right now. It's easy for me to sit back and say it now, but I think CNI could have done more to obscure/shift away from the underlying complexity. To lean off my armchair a bit and offer some parallel work -- I was pretty impressed with the design goals/interface of Ouroboros[0][1]. There just needs to be more projects (nomad is one of them) that are adopting CNI and simpler than k8s. [0]: https://archive.fosdem.org/2018/schedule/event/ipc/ https://archive.fosdem.org/2018/schedule/event/ipc/ [1]: https://ouroboros.rocks/ https://ouroboros.rocks/
  - bboreham 6y ago
    Yeah, the project could have been completely different. And the CNI marketing department sucks :-) Like most things in Open Source, it’s a product of whoever showed up to do some work.
    - hardwaresofton 6y ago
      Do want to take this change to thank you for you and your teams' work on this throughout the years though, I am very grateful for the CNI existing and the foundational work ya'll have done over the years. Weaveworks has also been supporting/crafting/pushing the limits in what networking (and the CNI of course) on Kubernetes could be from near the beginning. Was a treat to hear Alexis talk about the company on the Kubernetes podcast[0] (I just saw that part 2 is out, listening to it now!). I looked back in my YT history and you show up there three times (lots more videos with other folks from weaveworks): Cortex - Infinitely Scalable Prometheus[1] How We Used Jaeger and Prometheus to Deliver Lightning-Fast User Queries[2] Golang UK Conference 2016 - Bryan Boreham - An Actor Model in Go[3] Thanks for all the knowledge over the years. [0]: https://kubernetespodcast.com/ https://kubernetespodcast.com/ [1]: https://www.youtube.com/watch?v=iyN40FsRQEo https://www.youtube.com/watch?v=iyN40FsRQEo [2]: https://www.youtube.com/watch?v=qg0ENOdP1Lo https://www.youtube.com/watch?v=qg0ENOdP1Lo [3]: https://www.youtube.com/watch?v=yCbon_9yGVs https://www.youtube.com/watch?v=yCbon_9yGVs

bboreham 6y ago

Most of what you describe there is Kubernetes. > the near-oneness of kubernetes and CNI This is not something that can be blamed on CNI. We keep strictly even-handed.

hardwaresofton 6y ago
True there is a certain bit of mis-attribution but I think others are lumping them together as well right now. It's easy for me to sit back and say it now, but I think CNI could have done more to obscure/shift away from the underlying complexity. To lean off my armchair a bit and offer some parallel work -- I was pretty impressed with the design goals/interface of Ouroboros[0][1]. There just needs to be more projects (nomad is one of them) that are adopting CNI and simpler than k8s. [0]: https://archive.fosdem.org/2018/schedule/event/ipc/ https://archive.fosdem.org/2018/schedule/event/ipc/ [1]: https://ouroboros.rocks/ https://ouroboros.rocks/
- bboreham 6y ago
  Yeah, the project could have been completely different. And the CNI marketing department sucks :-) Like most things in Open Source, it’s a product of whoever showed up to do some work.
  - hardwaresofton 6y ago
    Do want to take this change to thank you for you and your teams' work on this throughout the years though, I am very grateful for the CNI existing and the foundational work ya'll have done over the years. Weaveworks has also been supporting/crafting/pushing the limits in what networking (and the CNI of course) on Kubernetes could be from near the beginning. Was a treat to hear Alexis talk about the company on the Kubernetes podcast[0] (I just saw that part 2 is out, listening to it now!). I looked back in my YT history and you show up there three times (lots more videos with other folks from weaveworks): Cortex - Infinitely Scalable Prometheus[1] How We Used Jaeger and Prometheus to Deliver Lightning-Fast User Queries[2] Golang UK Conference 2016 - Bryan Boreham - An Actor Model in Go[3] Thanks for all the knowledge over the years. [0]: https://kubernetespodcast.com/ https://kubernetespodcast.com/ [1]: https://www.youtube.com/watch?v=iyN40FsRQEo https://www.youtube.com/watch?v=iyN40FsRQEo [2]: https://www.youtube.com/watch?v=qg0ENOdP1Lo https://www.youtube.com/watch?v=qg0ENOdP1Lo [3]: https://www.youtube.com/watch?v=yCbon_9yGVs https://www.youtube.com/watch?v=yCbon_9yGVs