4 ms·

When we started with Firecracker, there weren't really any projects built on top of it. Even if they had been, though, most things that touch kubernetes are in

by mrkurt 6y ago

When we started with Firecracker, there weren't really any projects built on top of it. Even if they had been, though, most things that touch kubernetes are incredibly complex. We have to be very careful how we manage complexity. For example, we decided not to use any CSI or CNI projects for our storage and networking because those are designed to be pluggable. We don't need pluggable, and it's simpler to just build the very small things we did need.

hardwaresofton 6y ago
Reasonable -- thanks for expanding on this. No idea what your management infrastructure is like -- would you mind expanding a tiny bit on how you would power some of the things that CSI/CNI give you for the complexity? For example taking a disk snapshot or enforcing workload-specific network firewalling? If I think of the choices for pre-built orchestration being: - LXD - Kubernetes + CSI/CNI - Nomad + a little bit of orchestration I think I know the answer for some of the other options, but not Nomad -- how I'd get some of the features that CSI/CNI provide. Do ya'll just run jobs that take and ship LVM volume snapshots? Is live migration supported (to be fair this isn't really easy/possible/built-in anywhere except LXD AFAIK yet), or do you do a rolling traffic-shifting and application-replica-shooting process? Looks like I need to read up on nomad this weekend -- looks like it's got just enough for you all to productively use it and build just what you need on top. I have to admit I haven't given it enough of a read/examination.
- mrkurt 6y ago
  That's exactly why we used nomad. It's focused and easy to understand, and we can build on top of it. We've even patched it on occasion. We manage LVM ourselves, yes. We don't do any live migration. Our disk offering is relatively new so we haven't exposed much snapshot based functionality to them. Most of them just want to know backups are happening and disks are encrypted, though. Network isolation was much simpler with a little bit of bpf and a ipv6 tricks. Some of that is detailed here: https://fly.io/blog/ipv6-wireguard-peering/ https://fly.io/blog/ipv6-wireguard-peering/ Letting customers connect to their networks with wireguard is something we probably couldn't have done well with k8s and CNI. Or at least, we'd have been fighting cni complexity to make it work.
  - bboreham 6y ago
    [a CNI maintainer writes] I have to say I’m a bit puzzled at the phrase “fighting CNI complexity”. CNI says you exec a process to connect your network and give it json on stdin, encoding what you want it to do. I get that a function call to code you wrote is simpler, but is json+exec really “complex”? EDIT: add link to docs: https://www.cni.dev/ https://www.cni.dev/
    - tptacek 6y ago
      I don't know if it's simpler, and I don't have a dog in this fight, but: our users get direct access to a standard IPv4/IPv6 network that behaves the way you'd expect those networks to work, with a private IPv6 address/hostname they can bind to for services that should only be available to applications in their own organization, and a DNS server that quickly gives you all the available private addresses of an app as `my-app-name.internal`, with no mTLS and no sidecar proxies, and we implemented this in about a thousand lines of Rust and a couple hundred lines of eBPF C. I hear "CNI" and I think "Istio". Is there a CNI plugin that would have gotten us this stuff for less effort? Thanks! Super interested in this.
      - bboreham 6y ago
        The CNI project manages the spec, a Go library to help parse the json, and some basic plugins that set up a loopback device, bridge, etc. Intentionally, people can use the spec to write whatever they want. Istio is a vastly bigger tool that uses CNI as a hook to get in between a container and the cluster.
    - hardwaresofton 6y ago
      So I agree with that phrase -- I'd say that the phrase CNI complexity generally refers to at least these things for me: - the complexity of networking in general, which is exposed. If I have to know what a CIDR/IPAM/etc are, then that complexity has leaked through. - the near-oneness of kubernetes and CNI -- CNI seems like a small facet of kubernetes (especially early on) but choosing the right provider, knowing what that JSON means etc is a big source of complexity -- combining that with the (usually) new world of complexities that k8s wrangles can be too much. - deciding that every pod must be addressable was a mistake IMO. It would have been harder to design but I think some sort of connection/network proxy should have been developed up front and iterated on (maybe I'm unaware of the history and this did indeed happen). When I spin up two containers, if they ought to talk to each other, I would have preferred so make some `NetworkConnection` object over the ports they need to speak over, or some virtual network they should be a part of. Security would be improved and the explicitness of connection/being present in a network could have obscured the underlying networking complexity. - debugging CNI brings you back into the regular world of linux networking -- tcpdump/iptables/ipvs-lvs/conntrack/etc when things break down in those layers people are even more confused than regular containers since just about everyone knows top/ps maybe even strace but far less people know how tools like ip/ifconfig/etc work. CNI is for the most part really simple and really easy to use these days, but I definitely can't say I'm confident with it -- I've been running kube-router since I found it because it was simpler than kube-proxy + flannel + calico (there's also canal). If you use some of the newer distributions (k3s, k0s) you don't even think about it. Also not securing node<->node communication was a pretty rough decision as well IMO. The Konnectivity service is a pain to think about and deploy, but luckily for me I can sidestep the whole thing with wireguard -- and calico will do it for me. There's also Cilium out there which does similar things. CNI is a smashing success -- people are building innovative things, competing, and clusters share the benefits -- but I still don't know that I'd call it simple yet, I just hope it never breaks.
- sofixa 6y ago
  > think I know the answer for some of the other options, but not Nomad -- how I'd get some of the features that CSI/CNI provide What do you mean, Nomad does both CSI and CNI, optionally.
  - hardwaresofton 6y ago
    TIL - I did not know this about Nomad, but I assumed they weren't using CSI or CNI because they mentioned managing LVM themselves in the article (and in this/previous article they mention bpf for networking so definitely not using CNI), so with full context the question would be "how are you providing the functionality without relying on any of the pre-built ecosystem stuff listed above?".
    - sofixa 6y ago
      Nomad allows you to rely on the host OS'+task driver primitives for storage and networking ( e.g. mount a docker volume, use the docker bridge network, mount a host folder, get an IPv6 etc.) or pass through CSI and CNI for those things for mose complex scenarios.

by mrkurt 6y ago

hardwaresofton 6y ago

Reasonable -- thanks for expanding on this. No idea what your management infrastructure is like -- would you mind expanding a tiny bit on how you would power some of the things that CSI/CNI give you for the complexity? For example taking a disk snapshot or enforcing workload-specific network firewalling? If I think of the choices for pre-built orchestration being: - LXD - Kubernetes + CSI/CNI - Nomad + a little bit of orchestration I think I know the answer for some of the other options, but not Nomad -- how I'd get some of the features that CSI/CNI provide. Do ya'll just run jobs that take and ship LVM volume snapshots? Is live migration supported (to be fair this isn't really easy/possible/built-in anywhere except LXD AFAIK yet), or do you do a rolling traffic-shifting and application-replica-shooting process? Looks like I need to read up on nomad this weekend -- looks like it's got just enough for you all to productively use it and build just what you need on top. I have to admit I haven't given it enough of a read/examination.

mrkurt 6y ago
That's exactly why we used nomad. It's focused and easy to understand, and we can build on top of it. We've even patched it on occasion. We manage LVM ourselves, yes. We don't do any live migration. Our disk offering is relatively new so we haven't exposed much snapshot based functionality to them. Most of them just want to know backups are happening and disks are encrypted, though. Network isolation was much simpler with a little bit of bpf and a ipv6 tricks. Some of that is detailed here: https://fly.io/blog/ipv6-wireguard-peering/ https://fly.io/blog/ipv6-wireguard-peering/ Letting customers connect to their networks with wireguard is something we probably couldn't have done well with k8s and CNI. Or at least, we'd have been fighting cni complexity to make it work.
- bboreham 6y ago
  [a CNI maintainer writes] I have to say I’m a bit puzzled at the phrase “fighting CNI complexity”. CNI says you exec a process to connect your network and give it json on stdin, encoding what you want it to do. I get that a function call to code you wrote is simpler, but is json+exec really “complex”? EDIT: add link to docs: https://www.cni.dev/ https://www.cni.dev/
  - tptacek 6y ago
    I don't know if it's simpler, and I don't have a dog in this fight, but: our users get direct access to a standard IPv4/IPv6 network that behaves the way you'd expect those networks to work, with a private IPv6 address/hostname they can bind to for services that should only be available to applications in their own organization, and a DNS server that quickly gives you all the available private addresses of an app as `my-app-name.internal`, with no mTLS and no sidecar proxies, and we implemented this in about a thousand lines of Rust and a couple hundred lines of eBPF C. I hear "CNI" and I think "Istio". Is there a CNI plugin that would have gotten us this stuff for less effort? Thanks! Super interested in this.
    - bboreham 6y ago
      The CNI project manages the spec, a Go library to help parse the json, and some basic plugins that set up a loopback device, bridge, etc. Intentionally, people can use the spec to write whatever they want. Istio is a vastly bigger tool that uses CNI as a hook to get in between a container and the cluster.
  - hardwaresofton 6y ago
    So I agree with that phrase -- I'd say that the phrase CNI complexity generally refers to at least these things for me: - the complexity of networking in general, which is exposed. If I have to know what a CIDR/IPAM/etc are, then that complexity has leaked through. - the near-oneness of kubernetes and CNI -- CNI seems like a small facet of kubernetes (especially early on) but choosing the right provider, knowing what that JSON means etc is a big source of complexity -- combining that with the (usually) new world of complexities that k8s wrangles can be too much. - deciding that every pod must be addressable was a mistake IMO. It would have been harder to design but I think some sort of connection/network proxy should have been developed up front and iterated on (maybe I'm unaware of the history and this did indeed happen). When I spin up two containers, if they ought to talk to each other, I would have preferred so make some `NetworkConnection` object over the ports they need to speak over, or some virtual network they should be a part of. Security would be improved and the explicitness of connection/being present in a network could have obscured the underlying networking complexity. - debugging CNI brings you back into the regular world of linux networking -- tcpdump/iptables/ipvs-lvs/conntrack/etc when things break down in those layers people are even more confused than regular containers since just about everyone knows top/ps maybe even strace but far less people know how tools like ip/ifconfig/etc work. CNI is for the most part really simple and really easy to use these days, but I definitely can't say I'm confident with it -- I've been running kube-router since I found it because it was simpler than kube-proxy + flannel + calico (there's also canal). If you use some of the newer distributions (k3s, k0s) you don't even think about it. Also not securing node<->node communication was a pretty rough decision as well IMO. The Konnectivity service is a pain to think about and deploy, but luckily for me I can sidestep the whole thing with wireguard -- and calico will do it for me. There's also Cilium out there which does similar things. CNI is a smashing success -- people are building innovative things, competing, and clusters share the benefits -- but I still don't know that I'd call it simple yet, I just hope it never breaks.
sofixa 6y ago
> think I know the answer for some of the other options, but not Nomad -- how I'd get some of the features that CSI/CNI provide What do you mean, Nomad does both CSI and CNI, optionally.
- hardwaresofton 6y ago
  TIL - I did not know this about Nomad, but I assumed they weren't using CSI or CNI because they mentioned managing LVM themselves in the article (and in this/previous article they mention bpf for networking so definitely not using CNI), so with full context the question would be "how are you providing the functionality without relying on any of the pre-built ecosystem stuff listed above?".
  - sofixa 6y ago
    Nomad allows you to rely on the host OS'+task driver primitives for storage and networking ( e.g. mount a docker volume, use the docker bridge network, mount a host folder, get an IPv6 etc.) or pass through CSI and CNI for those things for mose complex scenarios.