3 ms·

[a CNI maintainer writes] I have to say I’m a bit puzzled at the phrase “fighting CNI complexity”. CNI says you exec a process to connect your network and giv

[a CNI maintainer writes] I have to say I’m a bit puzzled at the phrase “fighting CNI complexity”. CNI says you exec a process to connect your network and give it json on stdin, encoding what you want it to do. I get that a function call to code you wrote is simpler, but is json+exec really “complex”? EDIT: add link to docs: https://www.cni.dev/ https://www.cni.dev/

tptacek 5y ago
I don't know if it's simpler, and I don't have a dog in this fight, but: our users get direct access to a standard IPv4/IPv6 network that behaves the way you'd expect those networks to work, with a private IPv6 address/hostname they can bind to for services that should only be available to applications in their own organization, and a DNS server that quickly gives you all the available private addresses of an app as `my-app-name.internal`, with no mTLS and no sidecar proxies, and we implemented this in about a thousand lines of Rust and a couple hundred lines of eBPF C. I hear "CNI" and I think "Istio". Is there a CNI plugin that would have gotten us this stuff for less effort? Thanks! Super interested in this.
- bboreham 5y ago
  The CNI project manages the spec, a Go library to help parse the json, and some basic plugins that set up a loopback device, bridge, etc. Intentionally, people can use the spec to write whatever they want. Istio is a vastly bigger tool that uses CNI as a hook to get in between a container and the cluster.
hardwaresofton 5y ago
So I agree with that phrase -- I'd say that the phrase CNI complexity generally refers to at least these things for me: - the complexity of networking in general, which is exposed. If I have to know what a CIDR/IPAM/etc are, then that complexity has leaked through. - the near-oneness of kubernetes and CNI -- CNI seems like a small facet of kubernetes (especially early on) but choosing the right provider, knowing what that JSON means etc is a big source of complexity -- combining that with the (usually) new world of complexities that k8s wrangles can be too much. - deciding that every pod must be addressable was a mistake IMO. It would have been harder to design but I think some sort of connection/network proxy should have been developed up front and iterated on (maybe I'm unaware of the history and this did indeed happen). When I spin up two containers, if they ought to talk to each other, I would have preferred so make some `NetworkConnection` object over the ports they need to speak over, or some virtual network they should be a part of. Security would be improved and the explicitness of connection/being present in a network could have obscured the underlying networking complexity. - debugging CNI brings you back into the regular world of linux networking -- tcpdump/iptables/ipvs-lvs/conntrack/etc when things break down in those layers people are even more confused than regular containers since just about everyone knows top/ps maybe even strace but far less people know how tools like ip/ifconfig/etc work. CNI is for the most part really simple and really easy to use these days, but I definitely can't say I'm confident with it -- I've been running kube-router since I found it because it was simpler than kube-proxy + flannel + calico (there's also canal). If you use some of the newer distributions (k3s, k0s) you don't even think about it. Also not securing node<->node communication was a pretty rough decision as well IMO. The Konnectivity service is a pain to think about and deploy, but luckily for me I can sidestep the whole thing with wireguard -- and calico will do it for me. There's also Cilium out there which does similar things. CNI is a smashing success -- people are building innovative things, competing, and clusters share the benefits -- but I still don't know that I'd call it simple yet, I just hope it never breaks.
- bboreham 5y ago
  Most of what you describe there is Kubernetes. > the near-oneness of kubernetes and CNI This is not something that can be blamed on CNI. We keep strictly even-handed.
  - hardwaresofton 5y ago
    True there is a certain bit of mis-attribution but I think others are lumping them together as well right now. It's easy for me to sit back and say it now, but I think CNI could have done more to obscure/shift away from the underlying complexity. To lean off my armchair a bit and offer some parallel work -- I was pretty impressed with the design goals/interface of Ouroboros[0][1]. There just needs to be more projects (nomad is one of them) that are adopting CNI and simpler than k8s. [0]: https://archive.fosdem.org/2018/schedule/event/ipc/ https://archive.fosdem.org/2018/schedule/event/ipc/ [1]: https://ouroboros.rocks/ https://ouroboros.rocks/
    - bboreham 5y ago
      Yeah, the project could have been completely different. And the CNI marketing department sucks :-) Like most things in Open Source, it’s a product of whoever showed up to do some work.
      - hardwaresofton 5y ago
        Do want to take this change to thank you for you and your teams' work on this throughout the years though, I am very grateful for the CNI existing and the foundational work ya'll have done over the years. Weaveworks has also been supporting/crafting/pushing the limits in what networking (and the CNI of course) on Kubernetes could be from near the beginning. Was a treat to hear Alexis talk about the company on the Kubernetes podcast[0] (I just saw that part 2 is out, listening to it now!). I looked back in my YT history and you show up there three times (lots more videos with other folks from weaveworks): Cortex - Infinitely Scalable Prometheus[1] How We Used Jaeger and Prometheus to Deliver Lightning-Fast User Queries[2] Golang UK Conference 2016 - Bryan Boreham - An Actor Model in Go[3] Thanks for all the knowledge over the years. [0]: https://kubernetespodcast.com/ https://kubernetespodcast.com/ [1]: https://www.youtube.com/watch?v=iyN40FsRQEo https://www.youtube.com/watch?v=iyN40FsRQEo [2]: https://www.youtube.com/watch?v=qg0ENOdP1Lo https://www.youtube.com/watch?v=qg0ENOdP1Lo [3]: https://www.youtube.com/watch?v=yCbon_9yGVs https://www.youtube.com/watch?v=yCbon_9yGVs

tptacek 5y ago

I don't know if it's simpler, and I don't have a dog in this fight, but: our users get direct access to a standard IPv4/IPv6 network that behaves the way you'd expect those networks to work, with a private IPv6 address/hostname they can bind to for services that should only be available to applications in their own organization, and a DNS server that quickly gives you all the available private addresses of an app as `my-app-name.internal`, with no mTLS and no sidecar proxies, and we implemented this in about a thousand lines of Rust and a couple hundred lines of eBPF C. I hear "CNI" and I think "Istio". Is there a CNI plugin that would have gotten us this stuff for less effort? Thanks! Super interested in this.

bboreham 5y ago
The CNI project manages the spec, a Go library to help parse the json, and some basic plugins that set up a loopback device, bridge, etc. Intentionally, people can use the spec to write whatever they want. Istio is a vastly bigger tool that uses CNI as a hook to get in between a container and the cluster.

hardwaresofton 5y ago

So I agree with that phrase -- I'd say that the phrase CNI complexity generally refers to at least these things for me: - the complexity of networking in general, which is exposed. If I have to know what a CIDR/IPAM/etc are, then that complexity has leaked through. - the near-oneness of kubernetes and CNI -- CNI seems like a small facet of kubernetes (especially early on) but choosing the right provider, knowing what that JSON means etc is a big source of complexity -- combining that with the (usually) new world of complexities that k8s wrangles can be too much. - deciding that every pod must be addressable was a mistake IMO. It would have been harder to design but I think some sort of connection/network proxy should have been developed up front and iterated on (maybe I'm unaware of the history and this did indeed happen). When I spin up two containers, if they ought to talk to each other, I would have preferred so make some `NetworkConnection` object over the ports they need to speak over, or some virtual network they should be a part of. Security would be improved and the explicitness of connection/being present in a network could have obscured the underlying networking complexity. - debugging CNI brings you back into the regular world of linux networking -- tcpdump/iptables/ipvs-lvs/conntrack/etc when things break down in those layers people are even more confused than regular containers since just about everyone knows top/ps maybe even strace but far less people know how tools like ip/ifconfig/etc work. CNI is for the most part really simple and really easy to use these days, but I definitely can't say I'm confident with it -- I've been running kube-router since I found it because it was simpler than kube-proxy + flannel + calico (there's also canal). If you use some of the newer distributions (k3s, k0s) you don't even think about it. Also not securing node<->node communication was a pretty rough decision as well IMO. The Konnectivity service is a pain to think about and deploy, but luckily for me I can sidestep the whole thing with wireguard -- and calico will do it for me. There's also Cilium out there which does similar things. CNI is a smashing success -- people are building innovative things, competing, and clusters share the benefits -- but I still don't know that I'd call it simple yet, I just hope it never breaks.

bboreham 5y ago
Most of what you describe there is Kubernetes. > the near-oneness of kubernetes and CNI This is not something that can be blamed on CNI. We keep strictly even-handed.
- hardwaresofton 5y ago
  True there is a certain bit of mis-attribution but I think others are lumping them together as well right now. It's easy for me to sit back and say it now, but I think CNI could have done more to obscure/shift away from the underlying complexity. To lean off my armchair a bit and offer some parallel work -- I was pretty impressed with the design goals/interface of Ouroboros[0][1]. There just needs to be more projects (nomad is one of them) that are adopting CNI and simpler than k8s. [0]: https://archive.fosdem.org/2018/schedule/event/ipc/ https://archive.fosdem.org/2018/schedule/event/ipc/ [1]: https://ouroboros.rocks/ https://ouroboros.rocks/
  - bboreham 5y ago
    Yeah, the project could have been completely different. And the CNI marketing department sucks :-) Like most things in Open Source, it’s a product of whoever showed up to do some work.
    - hardwaresofton 5y ago
      Do want to take this change to thank you for you and your teams' work on this throughout the years though, I am very grateful for the CNI existing and the foundational work ya'll have done over the years. Weaveworks has also been supporting/crafting/pushing the limits in what networking (and the CNI of course) on Kubernetes could be from near the beginning. Was a treat to hear Alexis talk about the company on the Kubernetes podcast[0] (I just saw that part 2 is out, listening to it now!). I looked back in my YT history and you show up there three times (lots more videos with other folks from weaveworks): Cortex - Infinitely Scalable Prometheus[1] How We Used Jaeger and Prometheus to Deliver Lightning-Fast User Queries[2] Golang UK Conference 2016 - Bryan Boreham - An Actor Model in Go[3] Thanks for all the knowledge over the years. [0]: https://kubernetespodcast.com/ https://kubernetespodcast.com/ [1]: https://www.youtube.com/watch?v=iyN40FsRQEo https://www.youtube.com/watch?v=iyN40FsRQEo [2]: https://www.youtube.com/watch?v=qg0ENOdP1Lo https://www.youtube.com/watch?v=qg0ENOdP1Lo [3]: https://www.youtube.com/watch?v=yCbon_9yGVs https://www.youtube.com/watch?v=yCbon_9yGVs