4 ms·

That's exactly why we used nomad. It's focused and easy to understand, and we can build on top of it. We've even patched it on occasion. We manage LVM ourselve

by mrkurt 6y ago

That's exactly why we used nomad. It's focused and easy to understand, and we can build on top of it. We've even patched it on occasion. We manage LVM ourselves, yes. We don't do any live migration. Our disk offering is relatively new so we haven't exposed much snapshot based functionality to them. Most of them just want to know backups are happening and disks are encrypted, though. Network isolation was much simpler with a little bit of bpf and a ipv6 tricks. Some of that is detailed here: https://fly.io/blog/ipv6-wireguard-peering/ https://fly.io/blog/ipv6-wireguard-peering/ Letting customers connect to their networks with wireguard is something we probably couldn't have done well with k8s and CNI. Or at least, we'd have been fighting cni complexity to make it work.

bboreham 6y ago
[a CNI maintainer writes] I have to say I’m a bit puzzled at the phrase “fighting CNI complexity”. CNI says you exec a process to connect your network and give it json on stdin, encoding what you want it to do. I get that a function call to code you wrote is simpler, but is json+exec really “complex”? EDIT: add link to docs: https://www.cni.dev/ https://www.cni.dev/
- tptacek 6y ago
  I don't know if it's simpler, and I don't have a dog in this fight, but: our users get direct access to a standard IPv4/IPv6 network that behaves the way you'd expect those networks to work, with a private IPv6 address/hostname they can bind to for services that should only be available to applications in their own organization, and a DNS server that quickly gives you all the available private addresses of an app as `my-app-name.internal`, with no mTLS and no sidecar proxies, and we implemented this in about a thousand lines of Rust and a couple hundred lines of eBPF C. I hear "CNI" and I think "Istio". Is there a CNI plugin that would have gotten us this stuff for less effort? Thanks! Super interested in this.
  - bboreham 6y ago
    The CNI project manages the spec, a Go library to help parse the json, and some basic plugins that set up a loopback device, bridge, etc. Intentionally, people can use the spec to write whatever they want. Istio is a vastly bigger tool that uses CNI as a hook to get in between a container and the cluster.
- hardwaresofton 6y ago
  So I agree with that phrase -- I'd say that the phrase CNI complexity generally refers to at least these things for me: - the complexity of networking in general, which is exposed. If I have to know what a CIDR/IPAM/etc are, then that complexity has leaked through. - the near-oneness of kubernetes and CNI -- CNI seems like a small facet of kubernetes (especially early on) but choosing the right provider, knowing what that JSON means etc is a big source of complexity -- combining that with the (usually) new world of complexities that k8s wrangles can be too much. - deciding that every pod must be addressable was a mistake IMO. It would have been harder to design but I think some sort of connection/network proxy should have been developed up front and iterated on (maybe I'm unaware of the history and this did indeed happen). When I spin up two containers, if they ought to talk to each other, I would have preferred so make some `NetworkConnection` object over the ports they need to speak over, or some virtual network they should be a part of. Security would be improved and the explicitness of connection/being present in a network could have obscured the underlying networking complexity. - debugging CNI brings you back into the regular world of linux networking -- tcpdump/iptables/ipvs-lvs/conntrack/etc when things break down in those layers people are even more confused than regular containers since just about everyone knows top/ps maybe even strace but far less people know how tools like ip/ifconfig/etc work. CNI is for the most part really simple and really easy to use these days, but I definitely can't say I'm confident with it -- I've been running kube-router since I found it because it was simpler than kube-proxy + flannel + calico (there's also canal). If you use some of the newer distributions (k3s, k0s) you don't even think about it. Also not securing node<->node communication was a pretty rough decision as well IMO. The Konnectivity service is a pain to think about and deploy, but luckily for me I can sidestep the whole thing with wireguard -- and calico will do it for me. There's also Cilium out there which does similar things. CNI is a smashing success -- people are building innovative things, competing, and clusters share the benefits -- but I still don't know that I'd call it simple yet, I just hope it never breaks.
  - bboreham 6y ago
    Most of what you describe there is Kubernetes. > the near-oneness of kubernetes and CNI This is not something that can be blamed on CNI. We keep strictly even-handed.
hardwaresofton 6y ago
> We manage LVM ourselves, yes. We don't do any live migration. Our disk offering is relatively new so we haven't exposed much snapshot based functionality to them. Most of them just want to know backups are happening and disks are encrypted, though. Reasonable -- if all the workloads run in containers to start with and are service-y probably not a real need, and certainly not a real need if no customer has asked for it/had it be a deal breaker.. > Network isolation was much simpler with a little bit of bpf and a ipv6 tricks. Some of that is detailed here: https://fly.io/blog/ipv6-wireguard-peering/ https://fly.io/blog/ipv6-wireguard-peering/ > we'd have been fighting cni complexity to make it work. Appreciate the candid responses, thanks for taking the time. That ipv6 wireguard peering post was really fascinating I read that too. Wireguard has been quite the a game-changer in it's space as well and a lot of value IMO is just in the simplicity and difficulty of misconfiguration, even though the performance is also fantastic. Grateful that ya'll are sharing what you're doing right/finding interesting. Since ya'll might appreciate this, I think there's an ultimate form of all these orchestrators out there that boils everything down to the "operator pattern" -- I call it "buhzaar" but I tried to get my thoughts out of the notebook a while ago[0]. It's almost like a completely normalized DB might be -- to strip an orchestrator down to it's bare minimum, which facilitates other processes that do resource provisioning and management. Then let people bring their own things that provision resources (and maybe you some "officially supported" ones but they all live separately and iterate separately). I didn't quite put down all the thoughts I had but you think this is too much normalization (in the same way no one wants to do 7 joins)? You could argue that both nomad and k8s are denormalized (they intrinsically "know" how to provision/manage certain things) to a certain extent, and nomad just "bundles" less. [0]: https://gitlab.com/buhzaar/design https://gitlab.com/buhzaar/design
e12e 6y ago
Did you investigate zfs vs LVM? Reading some these comments I almost started thinking about a docker2zfs tool that would take an image and build/dump a (possibly encrypted) zfs snapshot. And a service that takes a zfs snapshot, mounts it and runs it like a vm/container/jail...

by mrkurt 6y ago

bboreham 6y ago

[a CNI maintainer writes] I have to say I’m a bit puzzled at the phrase “fighting CNI complexity”. CNI says you exec a process to connect your network and give it json on stdin, encoding what you want it to do. I get that a function call to code you wrote is simpler, but is json+exec really “complex”? EDIT: add link to docs: https://www.cni.dev/ https://www.cni.dev/

tptacek 6y ago
I don't know if it's simpler, and I don't have a dog in this fight, but: our users get direct access to a standard IPv4/IPv6 network that behaves the way you'd expect those networks to work, with a private IPv6 address/hostname they can bind to for services that should only be available to applications in their own organization, and a DNS server that quickly gives you all the available private addresses of an app as `my-app-name.internal`, with no mTLS and no sidecar proxies, and we implemented this in about a thousand lines of Rust and a couple hundred lines of eBPF C. I hear "CNI" and I think "Istio". Is there a CNI plugin that would have gotten us this stuff for less effort? Thanks! Super interested in this.
- bboreham 6y ago
  The CNI project manages the spec, a Go library to help parse the json, and some basic plugins that set up a loopback device, bridge, etc. Intentionally, people can use the spec to write whatever they want. Istio is a vastly bigger tool that uses CNI as a hook to get in between a container and the cluster.
hardwaresofton 6y ago
So I agree with that phrase -- I'd say that the phrase CNI complexity generally refers to at least these things for me: - the complexity of networking in general, which is exposed. If I have to know what a CIDR/IPAM/etc are, then that complexity has leaked through. - the near-oneness of kubernetes and CNI -- CNI seems like a small facet of kubernetes (especially early on) but choosing the right provider, knowing what that JSON means etc is a big source of complexity -- combining that with the (usually) new world of complexities that k8s wrangles can be too much. - deciding that every pod must be addressable was a mistake IMO. It would have been harder to design but I think some sort of connection/network proxy should have been developed up front and iterated on (maybe I'm unaware of the history and this did indeed happen). When I spin up two containers, if they ought to talk to each other, I would have preferred so make some `NetworkConnection` object over the ports they need to speak over, or some virtual network they should be a part of. Security would be improved and the explicitness of connection/being present in a network could have obscured the underlying networking complexity. - debugging CNI brings you back into the regular world of linux networking -- tcpdump/iptables/ipvs-lvs/conntrack/etc when things break down in those layers people are even more confused than regular containers since just about everyone knows top/ps maybe even strace but far less people know how tools like ip/ifconfig/etc work. CNI is for the most part really simple and really easy to use these days, but I definitely can't say I'm confident with it -- I've been running kube-router since I found it because it was simpler than kube-proxy + flannel + calico (there's also canal). If you use some of the newer distributions (k3s, k0s) you don't even think about it. Also not securing node<->node communication was a pretty rough decision as well IMO. The Konnectivity service is a pain to think about and deploy, but luckily for me I can sidestep the whole thing with wireguard -- and calico will do it for me. There's also Cilium out there which does similar things. CNI is a smashing success -- people are building innovative things, competing, and clusters share the benefits -- but I still don't know that I'd call it simple yet, I just hope it never breaks.
- bboreham 6y ago
  Most of what you describe there is Kubernetes. > the near-oneness of kubernetes and CNI This is not something that can be blamed on CNI. We keep strictly even-handed.

hardwaresofton 6y ago

> We manage LVM ourselves, yes. We don't do any live migration. Our disk offering is relatively new so we haven't exposed much snapshot based functionality to them. Most of them just want to know backups are happening and disks are encrypted, though. Reasonable -- if all the workloads run in containers to start with and are service-y probably not a real need, and certainly not a real need if no customer has asked for it/had it be a deal breaker.. > Network isolation was much simpler with a little bit of bpf and a ipv6 tricks. Some of that is detailed here: https://fly.io/blog/ipv6-wireguard-peering/ https://fly.io/blog/ipv6-wireguard-peering/ > we'd have been fighting cni complexity to make it work. Appreciate the candid responses, thanks for taking the time. That ipv6 wireguard peering post was really fascinating I read that too. Wireguard has been quite the a game-changer in it's space as well and a lot of value IMO is just in the simplicity and difficulty of misconfiguration, even though the performance is also fantastic. Grateful that ya'll are sharing what you're doing right/finding interesting. Since ya'll might appreciate this, I think there's an ultimate form of all these orchestrators out there that boils everything down to the "operator pattern" -- I call it "buhzaar" but I tried to get my thoughts out of the notebook a while ago[0]. It's almost like a completely normalized DB might be -- to strip an orchestrator down to it's bare minimum, which facilitates other processes that do resource provisioning and management. Then let people bring their own things that provision resources (and maybe you some "officially supported" ones but they all live separately and iterate separately). I didn't quite put down all the thoughts I had but you think this is too much normalization (in the same way no one wants to do 7 joins)? You could argue that both nomad and k8s are denormalized (they intrinsically "know" how to provision/manage certain things) to a certain extent, and nomad just "bundles" less. [0]: https://gitlab.com/buhzaar/design https://gitlab.com/buhzaar/design

e12e 6y ago

Did you investigate zfs vs LVM? Reading some these comments I almost started thinking about a docker2zfs tool that would take an image and build/dump a (possibly encrypted) zfs snapshot. And a service that takes a zfs snapshot, mounts it and runs it like a vm/container/jail...