4 ms·

I'm really impressed by fly.io, and the candidness with which they share some of their really awesome technology. Being container-first is the next step for Paa

I'm really impressed by fly.io, and the candidness with which they share some of their really awesome technology. Being container-first is the next step for PaaS IMO and they are ahead of the pack. I aim to build a platform like theirs someday (probably not any time soon) but I don't think I'd do any of what they're doing -- it feels unnecessary, because I feel like I know lots of projects that have actually already done the hard work for this. I probably only think this because I haven't actually done it, but think it's worth sharing the tech anyway. Bear with me as I recently learned that they use nomad[0] and some of these suggestions are kubernetes projects but I'd love to hear why the following technologies were decided against (if they were): - kata-containers[1] (it does the whole container -> VM flow for you, automatically) with multiple VMM options[2] - linuxkit[3] (let's say you didn't go with kata-containers, this is another container->VM path) - firecracker-containerd[4] (very minimal keep-your-container-but-run-it-as-a-VM) - kubevirt[5] (if you just want to actually run VMs, regardless of how you built them) - Ceph[6] for storage -- make LVM pools and just give them to Ceph, you'll get blocks, distributed filesystems (CephFS), and object gateways (S3/Swift) out of it (in the k8s space Rook manages this) As an aside to all this, there's also LXD, which supports running "system" (user namespace isolated) containers, VMs (somewhat recent[7][8]), live migration via criu[9], management/migration of underlying filesystems, runs on LVM or zfs[10], it's basically all-in-one, but does fall behind in terms of ecosystem since everyone else is aboard the "cloud native"/"works-with-kubernetes" train. I've basically how I plan to run a service like fly.io if I ever did -- so maybe my secret is out, but I sure would like to know just how much of this fly.io got built on (if any of it), and/or what was turned down. [0]: https://news.ycombinator.com/item?id=26745514 https://news.ycombinator.com/item?id=26745514 [1]: https://github.com/kata-containers/kata-containers https://github.com/kata-containers/kata-containers [2]: https://github.com/kata-containers/kata-containers/blob/2fc7f75724ac9e18e60f63dcc9aa395dc51c184d/docs/design/virtualization.md https://github.com/kata-containers/kata-containers/blob/2fc7... [3]: https://github.com/linuxkit/linuxkit https://github.com/linuxkit/linuxkit [4]: https://github.com/firecracker-microvm/firecracker-containerd/ https://github.com/firecracker-microvm/firecracker-container... [5]: https://github.com/kubevirt/kubevirt https://github.com/kubevirt/kubevirt [6]: https://docs.ceph.com/ https://docs.ceph.com/ [7]: https://discuss.linuxcontainers.org/t/running-virtual-machines-with-lxd-4-0/7519 https://discuss.linuxcontainers.org/t/running-virtual-machin... [8]: https://github.com/lxc/lxd/issues/6205 https://github.com/lxc/lxd/issues/6205 [9]: https://criu.org/Main_Page https://criu.org/Main_Page [10]: https://linuxcontainers.org/lxd/docs/master/storage https://linuxcontainers.org/lxd/docs/master/storage

mrkurt 6y ago
When we started with Firecracker, there weren't really any projects built on top of it. Even if they had been, though, most things that touch kubernetes are incredibly complex. We have to be very careful how we manage complexity. For example, we decided not to use any CSI or CNI projects for our storage and networking because those are designed to be pluggable. We don't need pluggable, and it's simpler to just build the very small things we did need.
- hardwaresofton 6y ago
  Reasonable -- thanks for expanding on this. No idea what your management infrastructure is like -- would you mind expanding a tiny bit on how you would power some of the things that CSI/CNI give you for the complexity? For example taking a disk snapshot or enforcing workload-specific network firewalling? If I think of the choices for pre-built orchestration being: - LXD - Kubernetes + CSI/CNI - Nomad + a little bit of orchestration I think I know the answer for some of the other options, but not Nomad -- how I'd get some of the features that CSI/CNI provide. Do ya'll just run jobs that take and ship LVM volume snapshots? Is live migration supported (to be fair this isn't really easy/possible/built-in anywhere except LXD AFAIK yet), or do you do a rolling traffic-shifting and application-replica-shooting process? Looks like I need to read up on nomad this weekend -- looks like it's got just enough for you all to productively use it and build just what you need on top. I have to admit I haven't given it enough of a read/examination.
  - mrkurt 6y ago
    That's exactly why we used nomad. It's focused and easy to understand, and we can build on top of it. We've even patched it on occasion. We manage LVM ourselves, yes. We don't do any live migration. Our disk offering is relatively new so we haven't exposed much snapshot based functionality to them. Most of them just want to know backups are happening and disks are encrypted, though. Network isolation was much simpler with a little bit of bpf and a ipv6 tricks. Some of that is detailed here: https://fly.io/blog/ipv6-wireguard-peering/ https://fly.io/blog/ipv6-wireguard-peering/ Letting customers connect to their networks with wireguard is something we probably couldn't have done well with k8s and CNI. Or at least, we'd have been fighting cni complexity to make it work.

mrkurt 6y ago

When we started with Firecracker, there weren't really any projects built on top of it. Even if they had been, though, most things that touch kubernetes are incredibly complex. We have to be very careful how we manage complexity. For example, we decided not to use any CSI or CNI projects for our storage and networking because those are designed to be pluggable. We don't need pluggable, and it's simpler to just build the very small things we did need.

hardwaresofton 6y ago
Reasonable -- thanks for expanding on this. No idea what your management infrastructure is like -- would you mind expanding a tiny bit on how you would power some of the things that CSI/CNI give you for the complexity? For example taking a disk snapshot or enforcing workload-specific network firewalling? If I think of the choices for pre-built orchestration being: - LXD - Kubernetes + CSI/CNI - Nomad + a little bit of orchestration I think I know the answer for some of the other options, but not Nomad -- how I'd get some of the features that CSI/CNI provide. Do ya'll just run jobs that take and ship LVM volume snapshots? Is live migration supported (to be fair this isn't really easy/possible/built-in anywhere except LXD AFAIK yet), or do you do a rolling traffic-shifting and application-replica-shooting process? Looks like I need to read up on nomad this weekend -- looks like it's got just enough for you all to productively use it and build just what you need on top. I have to admit I haven't given it enough of a read/examination.
- mrkurt 6y ago
  That's exactly why we used nomad. It's focused and easy to understand, and we can build on top of it. We've even patched it on occasion. We manage LVM ourselves, yes. We don't do any live migration. Our disk offering is relatively new so we haven't exposed much snapshot based functionality to them. Most of them just want to know backups are happening and disks are encrypted, though. Network isolation was much simpler with a little bit of bpf and a ipv6 tricks. Some of that is detailed here: https://fly.io/blog/ipv6-wireguard-peering/ https://fly.io/blog/ipv6-wireguard-peering/ Letting customers connect to their networks with wireguard is something we probably couldn't have done well with k8s and CNI. Or at least, we'd have been fighting cni complexity to make it work.