5 ms·

We did. Our use case is spinning up containers on demand to user actions, giving them ephemeral, internet-routable hostnames, and shutting them down when all in

by paulgb 4y ago

We did. Our use case is spinning up containers on demand to user actions, giving them ephemeral, internet-routable hostnames, and shutting them down when all inbound connections have dropped. Because users are waiting to interact with these containers, we found the start times with Kubernetes too slow and its architecture to be a bad fit. We ended up writing our own control plane that uses NATS as a message bus. We are in the process of open sourcing it here: https://github.com/drifting-in-space/spawner https://github.com/drifting-in-space/spawner

cbanek 4y ago
> we found the start times with Kubernetes too slow Just curious if you could elaborate here? I work with k8s on docker, and we're also going to be spinning up ephemeral containers (and most of the other things you say) with jupyter notebooks. We're all in on k8s, but since you might be ahead of me, just wondering what hurdles you have faced? Our big problem was fetching containers took too long since we have kitchen sink containers that are like 10 GB (!) each. They seem to spin up pretty fast though if the image is already pulled. I've worked on a service that lives in the k8s cluster to pull images to make sure they are fresh (https://github.com/lsst-sqre/cachemachine https://github.com/lsst-sqre/cachemachine) but curious if you are talking about that or the networking? From what it looks like in your repo it might be that you need to do session timing (like ms) response time from a browser?
- paulgb 4y ago
  Jupyter notebooks are actually a use case we think about a lot, you can try a live demo with a Jupyter notebook here: https://jamsocket.com/tmpenv/ https://jamsocket.com/tmpenv/ It wasn't really one thing with Kubernetes that was slow, but that the more we tried to optimize it the less of core Kubernetes we were using and so the less value we were getting for the complexity tax we were paying. The image pulling you mention is a good example of that; having pre-pulled images is a big factor, but we have too many images to push every image to every node, instead we'd like the scheduler to be aware of which node has which image. We could do that with node affinity, but what we'd end up building would be more work than if we wrote our own scheduler to support it from day one. > From what it looks like in your repo it might be that you need to do session timing (like ms) response time from a browser? Our goal is subsecond container starts. We're not there yet, and might not get there with Docker, but we have a POC that is there with WebAssembly-based workloads. Too bad those are rare :) (By the way, I'm always happy to chat about this stuff, my email is in my profile)
  - justinsb 4y ago
    > we'd like the scheduler to be aware of which node has which image The kubernetes scheduler should be aware of which node has which image, that is why the Node object has the status.images field: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#nodestatus-v1-core https://kubernetes.io/docs/reference/generated/kubernetes-ap.... It turned out to be somewhat tricky, because it increased the size of the Node object, and colocating node heartbeats onto the same object meant that a bigger object was changing relatively often. But that was addressed by moving heartbeats to a different object: https://github.com/kubernetes/enhancements/issues/589 https://github.com/kubernetes/enhancements/issues/589
    - paulgb 4y ago
      TIL, thanks. Looks like there's a corresponding ImageLocality score used by the scheduler: https://kubernetes.io/docs/reference/scheduling/config/#scheduling-plugins https://kubernetes.io/docs/reference/scheduling/config/#sche... It doesn't get all the way to what we want, but it could be used to build a piece of it.
      - cbanek 4y ago
        Very cool, I didn't know about this either. I feel like so many of these features are coming in which is great, but also part of the drag of k8s is the kind of constant upgrade churn and having to keep your yaml fresh.
  - jjoonathan 4y ago
    AWS has put work into fast-starting containers [1] using tricks like lazy loading container storage, profiling container startup, non-lazily priming critical blocks, and caching shared blocks. IIRC parts of it are open source. I don't know if enough of it is open source to be helpful, but it's cool stuff! [1] Gigabytes in milliseconds: Bringing container support to AWS Lambda without adding latency. https://www.youtube.com/watch?v=A-7j0QlGwFk https://www.youtube.com/watch?v=A-7j0QlGwFk
    - 4y ago
- tecleandor 4y ago
  If you're pulling big images you could try kube-fledged (it's the simplest option, a CRD that works like a pre-puller for your images), or if you have a big cluster you can try a p2p distributor, like kraken or dragonfly2. Also there's that project called Nydus that allows starting up big containers way faster. IIRC, starts the container before pulling the whole image, and begins to pull data as needed from the registry. https://github.com/senthilrch/kube-fledged https://github.com/senthilrch/kube-fledged https://github.com/dragonflyoss/Dragonfly2 https://github.com/dragonflyoss/Dragonfly2 https://github.com/uber/kraken https://github.com/uber/kraken https://nydus.dev/ https://nydus.dev/
  - badLiveware 4y ago
    Lazy pulling is already supported by a lot of container runtimes, most notably containerd with estargz https://github.com/containerd/stargz-snapshotter/blob/main/docs/estargz.md https://github.com/containerd/stargz-snapshotter/blob/main/d...
    - tecleandor 4y ago
      Ah thanks! "Lazy pulling" is what I was looking for. I was trying to find estargz (didn't remember the name) and I couldn't find a proper keyword to do it :P :D
  - cbanek 4y ago
    Yeah I think we considered this, but we want the container to actually run as the user and have all the permissions set up so they can have all the right access on the cluster (kind of like a PaaS), although I think we are doing some of the stuff with the starting the container while the data is still streaming down. Black magic.
samsquire 4y ago
This is really awesome. Thank you for sharing this. One of my ideas lately has been to upgrade FaaS to a full on server after a set amount of traffic. Or said differently, a dedicated server spin up that serves the same app as callable functions ala scalable RPC and upgrade to a dedicated instance composed of said functions. The best of both worlds. Combine the scale to zero of Serverless combined with the scalability and capacity of a dedicated server.
baryphonic 4y ago
Wow, this is excellent! At a previous job, we had been using k8s + knative to spin up containers on demand, and likewise were unhappy with the delays. Spawner seems excellent. One question: have you had to do any custom container builds on demand, and if so, have you had to deal with large "kitchen sink" containers (e.g. a Python base image with a few larger packages installed from PyPI, plus some system packages like Postgres client)? We would run up against extremely long build image times using tools like kaniko, and caching would typically have only a limited benefit. I was experimenting using Nix to maybe solve some of these problems, but never got far enough to run a speed test, and then left the job before finishing. But it seems to me some sort of algorithm like Nixery uses (https://nixery.dev https://nixery.dev) to generate cacheable layers with completely repeatable builds and nothing extraneous would help. Maybe that's not a problem you had to solve, but if it is, I'd love your thoughts.
no_circuit 4y ago
Kind of curious what made it too slow for your use case? I'm guessing you did not users to wait for something like kube-dns to update or the workload scheduler? Of course things like spinning up a Pod can be slow. Or non-Kubernetes things like doing DNS ACME challenges could affect things. But on other hand, I can't quite figure out why something would prevent, you, yourself, from running the service that hosts the VMs that hosts the containers on demand on Kubernetes.
- paulgb 4y ago
  Our goal is sub-second container starts (admittedly, we're not there yet), and with Kubernetes we'd have to create Pods, create Services, wait for the scheduler to update, etc. We didn't go down the rabbit hole of profiling where the slowness was, but it was clear that Kubernetes just wasn't built with the type of speed we wanted. We realized we'd have to contend with a lot of design decisions that were the right choice for the things Kubernetes optimizes for (replication, resiliency), but not the right choice for us (fast launches of ephemeral containers). > But on other hand, I can't quite figure out why something would prevent, you, yourself, from running the service that hosts the VMs that hosts the containers on demand on Kubernetes. I'm not sure I understand this part, I guess we could use Kubernetes operators to scale up the underlying compute resources and manage the containers ourselves? This adds a lot of complexity for our use case.
wvh 4y ago
I just wrote a controller that does pretty much that – spawn containers on demand and report back status changes. While this solution does require some knowledge, it so far has been perfectly reliable and reasonably fast. I can fathom the need for processes to spawn and tear down faster in specific use cases than the Kubernetes scheduler would allow for, but for us a few seconds of wait time has been perfectly reasonable.
ehutch79 4y ago
It's always been my understanding that with things like k8s and other orchestration stuff, you're supposed to spin up before you need the capacity? You set a threshold, like 75% capacity, and if you're over that for a bit, you spin up a new container(s) to get you back to under effectively 75% capacity. Is that not how this works?
- paulgb 4y ago
  Yes, that's the scaling model that works best for Kubernetes if the use case supports it. Our use case precludes it, because we are focused on uses where containers need to be spun up on a per-user (or per-group-of-users) basis as they use an application.
  - swid 4y ago
    Precludes feels like the word word here. Nothing prevents you from satisfying your use case and spinning up vms before they are needed. I wrote the student vm system for udacity, and I spun up student vms before they needed them, with some last mile loading to finalize the files they need. The student VMs were not using k8s, although a small piece of the infrastructure did. I worried most about untrusted users working in a complex environment with the ability to harm the experience of other users, and just used GCE. For me, boot time was < 5 minutes, so if you can predict the next five minutes of demand, you can boot those machines early. If you are wrong you will pay extra or someone will wait extra time, but still less than the full boot time. Generally it takes less than 10 seconds to access a vm with your coursework on in.
  - blincoln 4y ago
    Can you not have a queue of whatever type of containers the users are likely to be using already ready to go, like the GP suggests?
  - acoard 4y ago
    Keep a pool of eg 5 pods that are warm and good to go. When a user needs one, the pod pulls in necessary config data which should be quick as it's already initialized.
osigurdson 4y ago
Is there a fundamental reason why Kubernetes cannot start pods and services fast (outside of pulling images of course!)?
- unity1001 4y ago
  There isnt. K8 does start pods and services fast. Im able to launch an entire stateful WordPress pod (multi-container) in just ~10 seconds. Including the provisioning and attaching times of PVs from scratch. This is at Digitalocean. You can easily run stateful things like WP if you build your pods well and use PVCs - even without needing to make them stateful sets. It ends up being a neatly constructed, integral VM living on virtualization. Everything is taken care by K8. When using K8, if you use the most basic K8 features and concepts, things generally work out pretty ok.
  - osigurdson 4y ago
    10s isn’t terribly long but, honestly I think 1s should be achievable and would open up more use cases for K8s in general (cloud provider physical machine latency not withstanding of course).
    - GauntletWizard 4y ago
      1s is very achievable iff you have spare capacity and images already pulled. There's good reasons the latter is hard to achieved, and the former is a tradeoff few are willing to make as it increases costs.

by paulgb 4y ago

cbanek 4y ago

> we found the start times with Kubernetes too slow Just curious if you could elaborate here? I work with k8s on docker, and we're also going to be spinning up ephemeral containers (and most of the other things you say) with jupyter notebooks. We're all in on k8s, but since you might be ahead of me, just wondering what hurdles you have faced? Our big problem was fetching containers took too long since we have kitchen sink containers that are like 10 GB (!) each. They seem to spin up pretty fast though if the image is already pulled. I've worked on a service that lives in the k8s cluster to pull images to make sure they are fresh (https://github.com/lsst-sqre/cachemachine https://github.com/lsst-sqre/cachemachine) but curious if you are talking about that or the networking? From what it looks like in your repo it might be that you need to do session timing (like ms) response time from a browser?

paulgb 4y ago
Jupyter notebooks are actually a use case we think about a lot, you can try a live demo with a Jupyter notebook here: https://jamsocket.com/tmpenv/ https://jamsocket.com/tmpenv/ It wasn't really one thing with Kubernetes that was slow, but that the more we tried to optimize it the less of core Kubernetes we were using and so the less value we were getting for the complexity tax we were paying. The image pulling you mention is a good example of that; having pre-pulled images is a big factor, but we have too many images to push every image to every node, instead we'd like the scheduler to be aware of which node has which image. We could do that with node affinity, but what we'd end up building would be more work than if we wrote our own scheduler to support it from day one. > From what it looks like in your repo it might be that you need to do session timing (like ms) response time from a browser? Our goal is subsecond container starts. We're not there yet, and might not get there with Docker, but we have a POC that is there with WebAssembly-based workloads. Too bad those are rare :) (By the way, I'm always happy to chat about this stuff, my email is in my profile)
- justinsb 4y ago
  > we'd like the scheduler to be aware of which node has which image The kubernetes scheduler should be aware of which node has which image, that is why the Node object has the status.images field: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#nodestatus-v1-core https://kubernetes.io/docs/reference/generated/kubernetes-ap.... It turned out to be somewhat tricky, because it increased the size of the Node object, and colocating node heartbeats onto the same object meant that a bigger object was changing relatively often. But that was addressed by moving heartbeats to a different object: https://github.com/kubernetes/enhancements/issues/589 https://github.com/kubernetes/enhancements/issues/589
  - paulgb 4y ago
    TIL, thanks. Looks like there's a corresponding ImageLocality score used by the scheduler: https://kubernetes.io/docs/reference/scheduling/config/#scheduling-plugins https://kubernetes.io/docs/reference/scheduling/config/#sche... It doesn't get all the way to what we want, but it could be used to build a piece of it.
    - cbanek 4y ago
      Very cool, I didn't know about this either. I feel like so many of these features are coming in which is great, but also part of the drag of k8s is the kind of constant upgrade churn and having to keep your yaml fresh.
- jjoonathan 4y ago
  AWS has put work into fast-starting containers [1] using tricks like lazy loading container storage, profiling container startup, non-lazily priming critical blocks, and caching shared blocks. IIRC parts of it are open source. I don't know if enough of it is open source to be helpful, but it's cool stuff! [1] Gigabytes in milliseconds: Bringing container support to AWS Lambda without adding latency. https://www.youtube.com/watch?v=A-7j0QlGwFk https://www.youtube.com/watch?v=A-7j0QlGwFk
  - 4y ago
tecleandor 4y ago
If you're pulling big images you could try kube-fledged (it's the simplest option, a CRD that works like a pre-puller for your images), or if you have a big cluster you can try a p2p distributor, like kraken or dragonfly2. Also there's that project called Nydus that allows starting up big containers way faster. IIRC, starts the container before pulling the whole image, and begins to pull data as needed from the registry. https://github.com/senthilrch/kube-fledged https://github.com/senthilrch/kube-fledged https://github.com/dragonflyoss/Dragonfly2 https://github.com/dragonflyoss/Dragonfly2 https://github.com/uber/kraken https://github.com/uber/kraken https://nydus.dev/ https://nydus.dev/
- badLiveware 4y ago
  Lazy pulling is already supported by a lot of container runtimes, most notably containerd with estargz https://github.com/containerd/stargz-snapshotter/blob/main/docs/estargz.md https://github.com/containerd/stargz-snapshotter/blob/main/d...
  - tecleandor 4y ago
    Ah thanks! "Lazy pulling" is what I was looking for. I was trying to find estargz (didn't remember the name) and I couldn't find a proper keyword to do it :P :D
- cbanek 4y ago
  Yeah I think we considered this, but we want the container to actually run as the user and have all the permissions set up so they can have all the right access on the cluster (kind of like a PaaS), although I think we are doing some of the stuff with the starting the container while the data is still streaming down. Black magic.

samsquire 4y ago

This is really awesome. Thank you for sharing this. One of my ideas lately has been to upgrade FaaS to a full on server after a set amount of traffic. Or said differently, a dedicated server spin up that serves the same app as callable functions ala scalable RPC and upgrade to a dedicated instance composed of said functions. The best of both worlds. Combine the scale to zero of Serverless combined with the scalability and capacity of a dedicated server.

baryphonic 4y ago

Wow, this is excellent! At a previous job, we had been using k8s + knative to spin up containers on demand, and likewise were unhappy with the delays. Spawner seems excellent. One question: have you had to do any custom container builds on demand, and if so, have you had to deal with large "kitchen sink" containers (e.g. a Python base image with a few larger packages installed from PyPI, plus some system packages like Postgres client)? We would run up against extremely long build image times using tools like kaniko, and caching would typically have only a limited benefit. I was experimenting using Nix to maybe solve some of these problems, but never got far enough to run a speed test, and then left the job before finishing. But it seems to me some sort of algorithm like Nixery uses (https://nixery.dev https://nixery.dev) to generate cacheable layers with completely repeatable builds and nothing extraneous would help. Maybe that's not a problem you had to solve, but if it is, I'd love your thoughts.

no_circuit 4y ago

Kind of curious what made it too slow for your use case? I'm guessing you did not users to wait for something like kube-dns to update or the workload scheduler? Of course things like spinning up a Pod can be slow. Or non-Kubernetes things like doing DNS ACME challenges could affect things. But on other hand, I can't quite figure out why something would prevent, you, yourself, from running the service that hosts the VMs that hosts the containers on demand on Kubernetes.

paulgb 4y ago
Our goal is sub-second container starts (admittedly, we're not there yet), and with Kubernetes we'd have to create Pods, create Services, wait for the scheduler to update, etc. We didn't go down the rabbit hole of profiling where the slowness was, but it was clear that Kubernetes just wasn't built with the type of speed we wanted. We realized we'd have to contend with a lot of design decisions that were the right choice for the things Kubernetes optimizes for (replication, resiliency), but not the right choice for us (fast launches of ephemeral containers). > But on other hand, I can't quite figure out why something would prevent, you, yourself, from running the service that hosts the VMs that hosts the containers on demand on Kubernetes. I'm not sure I understand this part, I guess we could use Kubernetes operators to scale up the underlying compute resources and manage the containers ourselves? This adds a lot of complexity for our use case.

wvh 4y ago

I just wrote a controller that does pretty much that – spawn containers on demand and report back status changes. While this solution does require some knowledge, it so far has been perfectly reliable and reasonably fast. I can fathom the need for processes to spawn and tear down faster in specific use cases than the Kubernetes scheduler would allow for, but for us a few seconds of wait time has been perfectly reasonable.

ehutch79 4y ago

It's always been my understanding that with things like k8s and other orchestration stuff, you're supposed to spin up before you need the capacity? You set a threshold, like 75% capacity, and if you're over that for a bit, you spin up a new container(s) to get you back to under effectively 75% capacity. Is that not how this works?

paulgb 4y ago
Yes, that's the scaling model that works best for Kubernetes if the use case supports it. Our use case precludes it, because we are focused on uses where containers need to be spun up on a per-user (or per-group-of-users) basis as they use an application.
- swid 4y ago
  Precludes feels like the word word here. Nothing prevents you from satisfying your use case and spinning up vms before they are needed. I wrote the student vm system for udacity, and I spun up student vms before they needed them, with some last mile loading to finalize the files they need. The student VMs were not using k8s, although a small piece of the infrastructure did. I worried most about untrusted users working in a complex environment with the ability to harm the experience of other users, and just used GCE. For me, boot time was < 5 minutes, so if you can predict the next five minutes of demand, you can boot those machines early. If you are wrong you will pay extra or someone will wait extra time, but still less than the full boot time. Generally it takes less than 10 seconds to access a vm with your coursework on in.
- blincoln 4y ago
  Can you not have a queue of whatever type of containers the users are likely to be using already ready to go, like the GP suggests?
- acoard 4y ago
  Keep a pool of eg 5 pods that are warm and good to go. When a user needs one, the pod pulls in necessary config data which should be quick as it's already initialized.

osigurdson 4y ago

Is there a fundamental reason why Kubernetes cannot start pods and services fast (outside of pulling images of course!)?

unity1001 4y ago
There isnt. K8 does start pods and services fast. Im able to launch an entire stateful WordPress pod (multi-container) in just ~10 seconds. Including the provisioning and attaching times of PVs from scratch. This is at Digitalocean. You can easily run stateful things like WP if you build your pods well and use PVCs - even without needing to make them stateful sets. It ends up being a neatly constructed, integral VM living on virtualization. Everything is taken care by K8. When using K8, if you use the most basic K8 features and concepts, things generally work out pretty ok.
- osigurdson 4y ago
  10s isn’t terribly long but, honestly I think 1s should be achievable and would open up more use cases for K8s in general (cloud provider physical machine latency not withstanding of course).
  - GauntletWizard 4y ago
    1s is very achievable iff you have spare capacity and images already pulled. There's good reasons the latter is hard to achieved, and the former is a tradeoff few are willing to make as it increases costs.