3 ms·

> we found the start times with Kubernetes too slow Just curious if you could elaborate here? I work with k8s on docker, and we're also going to be spinning u

by cbanek 4y ago

> we found the start times with Kubernetes too slow Just curious if you could elaborate here? I work with k8s on docker, and we're also going to be spinning up ephemeral containers (and most of the other things you say) with jupyter notebooks. We're all in on k8s, but since you might be ahead of me, just wondering what hurdles you have faced? Our big problem was fetching containers took too long since we have kitchen sink containers that are like 10 GB (!) each. They seem to spin up pretty fast though if the image is already pulled. I've worked on a service that lives in the k8s cluster to pull images to make sure they are fresh (https://github.com/lsst-sqre/cachemachine https://github.com/lsst-sqre/cachemachine) but curious if you are talking about that or the networking? From what it looks like in your repo it might be that you need to do session timing (like ms) response time from a browser?

paulgb 4y ago
Jupyter notebooks are actually a use case we think about a lot, you can try a live demo with a Jupyter notebook here: https://jamsocket.com/tmpenv/ https://jamsocket.com/tmpenv/ It wasn't really one thing with Kubernetes that was slow, but that the more we tried to optimize it the less of core Kubernetes we were using and so the less value we were getting for the complexity tax we were paying. The image pulling you mention is a good example of that; having pre-pulled images is a big factor, but we have too many images to push every image to every node, instead we'd like the scheduler to be aware of which node has which image. We could do that with node affinity, but what we'd end up building would be more work than if we wrote our own scheduler to support it from day one. > From what it looks like in your repo it might be that you need to do session timing (like ms) response time from a browser? Our goal is subsecond container starts. We're not there yet, and might not get there with Docker, but we have a POC that is there with WebAssembly-based workloads. Too bad those are rare :) (By the way, I'm always happy to chat about this stuff, my email is in my profile)
- justinsb 4y ago
  > we'd like the scheduler to be aware of which node has which image The kubernetes scheduler should be aware of which node has which image, that is why the Node object has the status.images field: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#nodestatus-v1-core https://kubernetes.io/docs/reference/generated/kubernetes-ap.... It turned out to be somewhat tricky, because it increased the size of the Node object, and colocating node heartbeats onto the same object meant that a bigger object was changing relatively often. But that was addressed by moving heartbeats to a different object: https://github.com/kubernetes/enhancements/issues/589 https://github.com/kubernetes/enhancements/issues/589
  - paulgb 4y ago
    TIL, thanks. Looks like there's a corresponding ImageLocality score used by the scheduler: https://kubernetes.io/docs/reference/scheduling/config/#scheduling-plugins https://kubernetes.io/docs/reference/scheduling/config/#sche... It doesn't get all the way to what we want, but it could be used to build a piece of it.
    - cbanek 4y ago
      Very cool, I didn't know about this either. I feel like so many of these features are coming in which is great, but also part of the drag of k8s is the kind of constant upgrade churn and having to keep your yaml fresh.
- jjoonathan 4y ago
  AWS has put work into fast-starting containers [1] using tricks like lazy loading container storage, profiling container startup, non-lazily priming critical blocks, and caching shared blocks. IIRC parts of it are open source. I don't know if enough of it is open source to be helpful, but it's cool stuff! [1] Gigabytes in milliseconds: Bringing container support to AWS Lambda without adding latency. https://www.youtube.com/watch?v=A-7j0QlGwFk https://www.youtube.com/watch?v=A-7j0QlGwFk
  - TurningCanadian 4y ago
    On the Google side, Artifact Registry supports image streaming https://cloud.google.com/kubernetes-engine/docs/how-to/image-streaming https://cloud.google.com/kubernetes-engine/docs/how-to/image...
- hosh 4y ago
  Doesn’t the latest version of k8s let you use your own custom scheduler?
  - paulgb 4y ago
    You can, but that falls into this bucket: > the more we tried to optimize it the less of core Kubernetes we were using and so the less value we were getting for the complexity tax we were paying Since we were headed down that path, we took a step back and asked what we were really getting out of Kubernetes, and most of it was things that were orthogonal to our intended use case. The way Kubernetes is architected around control loops works great for its intended use case, but we wanted a more event-driven system.
    - hosh 4y ago
      Event driven ... like a streaming data pipeline? Given your comment about Jupyter notebooks, that makes sense. It might be the Mesos project is better architected for your use-case. Then again, I think Mesos ported some of their schedulers to Kubernetes.
tecleandor 4y ago
If you're pulling big images you could try kube-fledged (it's the simplest option, a CRD that works like a pre-puller for your images), or if you have a big cluster you can try a p2p distributor, like kraken or dragonfly2. Also there's that project called Nydus that allows starting up big containers way faster. IIRC, starts the container before pulling the whole image, and begins to pull data as needed from the registry. https://github.com/senthilrch/kube-fledged https://github.com/senthilrch/kube-fledged https://github.com/dragonflyoss/Dragonfly2 https://github.com/dragonflyoss/Dragonfly2 https://github.com/uber/kraken https://github.com/uber/kraken https://nydus.dev/ https://nydus.dev/
- badLiveware 4y ago
  Lazy pulling is already supported by a lot of container runtimes, most notably containerd with estargz https://github.com/containerd/stargz-snapshotter/blob/main/docs/estargz.md https://github.com/containerd/stargz-snapshotter/blob/main/d...
  - tecleandor 4y ago
    Ah thanks! "Lazy pulling" is what I was looking for. I was trying to find estargz (didn't remember the name) and I couldn't find a proper keyword to do it :P :D
- cbanek 4y ago
  Yeah I think we considered this, but we want the container to actually run as the user and have all the permissions set up so they can have all the right access on the cluster (kind of like a PaaS), although I think we are doing some of the stuff with the starting the container while the data is still streaming down. Black magic.

by cbanek 4y ago

paulgb 4y ago

Jupyter notebooks are actually a use case we think about a lot, you can try a live demo with a Jupyter notebook here: https://jamsocket.com/tmpenv/ https://jamsocket.com/tmpenv/ It wasn't really one thing with Kubernetes that was slow, but that the more we tried to optimize it the less of core Kubernetes we were using and so the less value we were getting for the complexity tax we were paying. The image pulling you mention is a good example of that; having pre-pulled images is a big factor, but we have too many images to push every image to every node, instead we'd like the scheduler to be aware of which node has which image. We could do that with node affinity, but what we'd end up building would be more work than if we wrote our own scheduler to support it from day one. > From what it looks like in your repo it might be that you need to do session timing (like ms) response time from a browser? Our goal is subsecond container starts. We're not there yet, and might not get there with Docker, but we have a POC that is there with WebAssembly-based workloads. Too bad those are rare :) (By the way, I'm always happy to chat about this stuff, my email is in my profile)

justinsb 4y ago
> we'd like the scheduler to be aware of which node has which image The kubernetes scheduler should be aware of which node has which image, that is why the Node object has the status.images field: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#nodestatus-v1-core https://kubernetes.io/docs/reference/generated/kubernetes-ap.... It turned out to be somewhat tricky, because it increased the size of the Node object, and colocating node heartbeats onto the same object meant that a bigger object was changing relatively often. But that was addressed by moving heartbeats to a different object: https://github.com/kubernetes/enhancements/issues/589 https://github.com/kubernetes/enhancements/issues/589
- paulgb 4y ago
  TIL, thanks. Looks like there's a corresponding ImageLocality score used by the scheduler: https://kubernetes.io/docs/reference/scheduling/config/#scheduling-plugins https://kubernetes.io/docs/reference/scheduling/config/#sche... It doesn't get all the way to what we want, but it could be used to build a piece of it.
  - cbanek 4y ago
    Very cool, I didn't know about this either. I feel like so many of these features are coming in which is great, but also part of the drag of k8s is the kind of constant upgrade churn and having to keep your yaml fresh.
jjoonathan 4y ago
AWS has put work into fast-starting containers [1] using tricks like lazy loading container storage, profiling container startup, non-lazily priming critical blocks, and caching shared blocks. IIRC parts of it are open source. I don't know if enough of it is open source to be helpful, but it's cool stuff! [1] Gigabytes in milliseconds: Bringing container support to AWS Lambda without adding latency. https://www.youtube.com/watch?v=A-7j0QlGwFk https://www.youtube.com/watch?v=A-7j0QlGwFk
- TurningCanadian 4y ago
  On the Google side, Artifact Registry supports image streaming https://cloud.google.com/kubernetes-engine/docs/how-to/image-streaming https://cloud.google.com/kubernetes-engine/docs/how-to/image...
hosh 4y ago
Doesn’t the latest version of k8s let you use your own custom scheduler?
- paulgb 4y ago
  You can, but that falls into this bucket: > the more we tried to optimize it the less of core Kubernetes we were using and so the less value we were getting for the complexity tax we were paying Since we were headed down that path, we took a step back and asked what we were really getting out of Kubernetes, and most of it was things that were orthogonal to our intended use case. The way Kubernetes is architected around control loops works great for its intended use case, but we wanted a more event-driven system.
  - hosh 4y ago
    Event driven ... like a streaming data pipeline? Given your comment about Jupyter notebooks, that makes sense. It might be the Mesos project is better architected for your use-case. Then again, I think Mesos ported some of their schedulers to Kubernetes.

tecleandor 4y ago

If you're pulling big images you could try kube-fledged (it's the simplest option, a CRD that works like a pre-puller for your images), or if you have a big cluster you can try a p2p distributor, like kraken or dragonfly2. Also there's that project called Nydus that allows starting up big containers way faster. IIRC, starts the container before pulling the whole image, and begins to pull data as needed from the registry. https://github.com/senthilrch/kube-fledged https://github.com/senthilrch/kube-fledged https://github.com/dragonflyoss/Dragonfly2 https://github.com/dragonflyoss/Dragonfly2 https://github.com/uber/kraken https://github.com/uber/kraken https://nydus.dev/ https://nydus.dev/

badLiveware 4y ago
Lazy pulling is already supported by a lot of container runtimes, most notably containerd with estargz https://github.com/containerd/stargz-snapshotter/blob/main/docs/estargz.md https://github.com/containerd/stargz-snapshotter/blob/main/d...
- tecleandor 4y ago
  Ah thanks! "Lazy pulling" is what I was looking for. I was trying to find estargz (didn't remember the name) and I couldn't find a proper keyword to do it :P :D
cbanek 4y ago
Yeah I think we considered this, but we want the container to actually run as the user and have all the permissions set up so they can have all the right access on the cluster (kind of like a PaaS), although I think we are doing some of the stuff with the starting the container while the data is still streaming down. Black magic.