8 ms·

Designing Our Serverless Engine: From Kubernetes to Nomad, Firecracker, and Kuma

148 points by eric_khun 5y ago

rockyluke 5y ago
Thanks for sharing! That's currently an uncommon change but I must admit you explained it well. Quick question for the team: did you consider or try Consul as a service mesh ?
- yann_eu 5y ago
  Hi! Yann, co-founder at Koyeb, here. Yes, we considered Consul as a service mesh but found that it was too strongly coupled with the network and task layer of Nomad. We were looking for something highly customizable which wouldn't get into the way. Also, the multi-tenancy features are paid features, which might have been difficult to sustain economically for us.
  - tsujp 5y ago
    If things like multi-tenancy are required and the project (Consul) is open-source would adding that functionality yourselves not be in-scope given the amount of work you've already put into your setup? I ask as a theoretical what-if.
jimaek 5y ago
Great post and a great tech stack. As far as I know Fly.io also went the same path of using the Nomad stack and tools. We did something similar at appfleet.com and also decided against Kubernetes. We opted to write a lightweight manager of Firecracker VMs.
lloydatkinson 5y ago
I'm hoping that this is the start of a trend away from the overly complex, overly engineered, big ball of mud that is Kubernetes towards simpler and easier deployment/hosting/container runtimes that are easier for smaller and large teams alike to manage. Don't get me started on how "devops" has become a meme. I want to deploy services and features. I wouldn't want to spend my full time dealing with Kubernetes and containers and all manner of complexity. I think cloud providers have much better models generally (e.g. Azure App Service). Setup pipeline, deploy, done. Containers are an optional thing. What I'm saying is, I don't think it's fair on anyone that essentially the only choices are 1) run servers and VM's yourself and manage deploying to them 2) kubernetes 3) cloud providers. There needs to be an option between them.
- kfk 5y ago
  Kubernetes is complex ok but I disagree on cloud providers offering better alternatives, especially Azure. Azure is bug ridden and support depends on what type of price tier you are in. The thing with Cloud providers is when their services go down or change you are on the hook for supporting your app, not them. At least Kubernetes (or Nomad) lets you mange your own dumpster fire around which you can build safer deployment workflows.
- dmitriid 5y ago
  > I'm hoping that this is the start of a trend away from the overly complex, overly engineered, big ball of mud that is Kubernetes towards simpler and easier deployment/hosting/container runtimes that are easier for smaller and large teams alike to manage. But is it easier to manage? Is it simpler? So they went from kubernetes to - Control Pane running in - Container running on - Firecracker with Kuma running on - Nomad with CoreOS running on - Some servers No idea what half of those words are, how they all are ocnfigured, interact with each other, and fail in spectacular ways.
  - gravypod 5y ago
    To be fair many of these words would be needed for running a bare metal kube cluster. - Some servers: you need servers to run a bare metal cluster. - CoreOS: This is just a minimal OS that's designed to be PXE booted. Think of it as a docker container for a bare metal machine. - Kuma: Multi-cluster mesh network. This makes it easier to write services that talk to `foobar:8080` and have it automatically fail over to other pods in other regions. I wouldn't use kuma, there's a few other options, but this is very helpful. Good service meshs will allow you to do Canary deployments. If you are multi-cluster than this sort of reliability is probably important to you. But, an important part of this is that you don't need these if you're ok with running on another cloud's infrastructure.
    - dmitriid 5y ago
      > But, an important part of this is that you don't need these if you're ok with running on another cloud's infrastructure. Same with kubernetes ;)
JanMa 5y ago
Nice post, thanks for sharing it. From personal experience I can only agree with their choice to pick Nomad. At the place where I work we have been running Nomad as our main container orchestrator for around 2.5 years now. It's rock solid, very easy to set-up and maintain and overall not too complex to understand in depth.
pm90 5y ago
> Global and multi-zone deployments: User workloads on Koyeb need to be able to run in multiple zones. Kubernetes doesn't support multi-zone out of the box. Implementing multi-zone with Kubernetes requires deploying a full cluster per zone, with a dedicated control plane for each data center. I’m having trouble understanding this. K8s worker nodes should be deployable across zones, regions etc. You can label the nodes with the zone id and use taints/tolerations to ensure workloads are deployed in specific zones/regions (if that’s what you want).
- bproven 5y ago
  yeah i am not sure if they mean "region" (as in AWS region) instead of zone. if that is the case then I agree with their assessment.
  - dilyevsky 5y ago
    Yeah pretty sure it’s that bc k8s 100% supports multi-az on every major cloud provider out of the box. It also supports federation via kubefed but it needs a separate control plane in each region. In theory nothing should be stopping you from deploying your etcd/master nodes in different regions but you’d probably need to tweak cloud resource provider to handle that and if one of regions is partitioned away from quorum those master become unavailable
- rileymichael 5y ago
  You certainly can deploy workers across regions, however the latency to the control plane makes it quite unpleasant.
  - johntash 5y ago
    IIRC the only real issue I ran into with workers in multiple datacenters was etcd latency. If you have a fast enough network, things will mostly be fine.
tantalor 5y ago
> Kubelet uses between 10% and 25% of RAM ... We're more around 100MB with our new architecture These figures are not comparable.
- dilyevsky 5y ago
  Highly questionable figures. Our kubelets have a 2% memory limit which I’ve never seen hit as it’s a hugely over-provisioned.
ngrilly 5y ago
Seems similar to fly.io, but fly.io seems better: uses WireGuard instead of service mesh, supports custom domains, supports any TCP or UDP service, provides volumes, etc. But that’s great seeing more options in that market!
tkiolp4 5y ago
I’m waiting the day someone discovers a simpler alternative to k8s. It will be like that day in which we realized that http verbs >>> Corba/rmi for web services.
staticassertion 5y ago
I'm curious - when a user deploys their new code to your product, does that kick off a new Nomad job, or is that managed internally, kicking off a koyeb-managed Firecracker with Kuma for service discovery?
icythere 5y ago
(Off topic) I maintain this small repo [1] and I have added our thread today to the section (#infrastructure). It's to learn "What/Why people move from this to that." It's not an _awesome_-like repo but I've found it's useful to learn others' decisions. Feel free to keep them up-to-date (but if you know there is better place / resource, I'm happy to work with them too.) Thanks a lot. [1] https://github.com/icy/w2w https://github.com/icy/w2w

148 points by eric_khun 5y ago

Thanks for sharing! That's currently an uncommon change but I must admit you explained it well. Quick question for the team: did you consider or try Consul as a service mesh ?

yann_eu 5y ago
Hi! Yann, co-founder at Koyeb, here. Yes, we considered Consul as a service mesh but found that it was too strongly coupled with the network and task layer of Nomad. We were looking for something highly customizable which wouldn't get into the way. Also, the multi-tenancy features are paid features, which might have been difficult to sustain economically for us.
- tsujp 5y ago
  If things like multi-tenancy are required and the project (Consul) is open-source would adding that functionality yourselves not be in-scope given the amount of work you've already put into your setup? I ask as a theoretical what-if.

jimaek 5y ago

Great post and a great tech stack. As far as I know Fly.io also went the same path of using the Nomad stack and tools. We did something similar at appfleet.com and also decided against Kubernetes. We opted to write a lightweight manager of Firecracker VMs.

lloydatkinson 5y ago

I'm hoping that this is the start of a trend away from the overly complex, overly engineered, big ball of mud that is Kubernetes towards simpler and easier deployment/hosting/container runtimes that are easier for smaller and large teams alike to manage. Don't get me started on how "devops" has become a meme. I want to deploy services and features. I wouldn't want to spend my full time dealing with Kubernetes and containers and all manner of complexity. I think cloud providers have much better models generally (e.g. Azure App Service). Setup pipeline, deploy, done. Containers are an optional thing. What I'm saying is, I don't think it's fair on anyone that essentially the only choices are 1) run servers and VM's yourself and manage deploying to them 2) kubernetes 3) cloud providers. There needs to be an option between them.

kfk 5y ago
Kubernetes is complex ok but I disagree on cloud providers offering better alternatives, especially Azure. Azure is bug ridden and support depends on what type of price tier you are in. The thing with Cloud providers is when their services go down or change you are on the hook for supporting your app, not them. At least Kubernetes (or Nomad) lets you mange your own dumpster fire around which you can build safer deployment workflows.
dmitriid 5y ago
> I'm hoping that this is the start of a trend away from the overly complex, overly engineered, big ball of mud that is Kubernetes towards simpler and easier deployment/hosting/container runtimes that are easier for smaller and large teams alike to manage. But is it easier to manage? Is it simpler? So they went from kubernetes to - Control Pane running in - Container running on - Firecracker with Kuma running on - Nomad with CoreOS running on - Some servers No idea what half of those words are, how they all are ocnfigured, interact with each other, and fail in spectacular ways.
- gravypod 5y ago
  To be fair many of these words would be needed for running a bare metal kube cluster. - Some servers: you need servers to run a bare metal cluster. - CoreOS: This is just a minimal OS that's designed to be PXE booted. Think of it as a docker container for a bare metal machine. - Kuma: Multi-cluster mesh network. This makes it easier to write services that talk to `foobar:8080` and have it automatically fail over to other pods in other regions. I wouldn't use kuma, there's a few other options, but this is very helpful. Good service meshs will allow you to do Canary deployments. If you are multi-cluster than this sort of reliability is probably important to you. But, an important part of this is that you don't need these if you're ok with running on another cloud's infrastructure.
  - dmitriid 5y ago
    > But, an important part of this is that you don't need these if you're ok with running on another cloud's infrastructure. Same with kubernetes ;)

JanMa 5y ago

Nice post, thanks for sharing it. From personal experience I can only agree with their choice to pick Nomad. At the place where I work we have been running Nomad as our main container orchestrator for around 2.5 years now. It's rock solid, very easy to set-up and maintain and overall not too complex to understand in depth.

pm90 5y ago

> Global and multi-zone deployments: User workloads on Koyeb need to be able to run in multiple zones. Kubernetes doesn't support multi-zone out of the box. Implementing multi-zone with Kubernetes requires deploying a full cluster per zone, with a dedicated control plane for each data center. I’m having trouble understanding this. K8s worker nodes should be deployable across zones, regions etc. You can label the nodes with the zone id and use taints/tolerations to ensure workloads are deployed in specific zones/regions (if that’s what you want).

bproven 5y ago
yeah i am not sure if they mean "region" (as in AWS region) instead of zone. if that is the case then I agree with their assessment.
- dilyevsky 5y ago
  Yeah pretty sure it’s that bc k8s 100% supports multi-az on every major cloud provider out of the box. It also supports federation via kubefed but it needs a separate control plane in each region. In theory nothing should be stopping you from deploying your etcd/master nodes in different regions but you’d probably need to tweak cloud resource provider to handle that and if one of regions is partitioned away from quorum those master become unavailable
rileymichael 5y ago
You certainly can deploy workers across regions, however the latency to the control plane makes it quite unpleasant.
- johntash 5y ago
  IIRC the only real issue I ran into with workers in multiple datacenters was etcd latency. If you have a fast enough network, things will mostly be fine.

tantalor 5y ago

> Kubelet uses between 10% and 25% of RAM ... We're more around 100MB with our new architecture These figures are not comparable.

dilyevsky 5y ago
Highly questionable figures. Our kubelets have a 2% memory limit which I’ve never seen hit as it’s a hugely over-provisioned.

ngrilly 5y ago

Seems similar to fly.io, but fly.io seems better: uses WireGuard instead of service mesh, supports custom domains, supports any TCP or UDP service, provides volumes, etc. But that’s great seeing more options in that market!

tkiolp4 5y ago

I’m waiting the day someone discovers a simpler alternative to k8s. It will be like that day in which we realized that http verbs >>> Corba/rmi for web services.

staticassertion 5y ago

I'm curious - when a user deploys their new code to your product, does that kick off a new Nomad job, or is that managed internally, kicking off a koyeb-managed Firecracker with Kuma for service discovery?

icythere 5y ago

(Off topic) I maintain this small repo [1] and I have added our thread today to the section (#infrastructure). It's to learn "What/Why people move from this to that." It's not an _awesome_-like repo but I've found it's useful to learn others' decisions. Feel free to keep them up-to-date (but if you know there is better place / resource, I'm happy to work with them too.) Thanks a lot. [1] https://github.com/icy/w2w https://github.com/icy/w2w