4 ms·

> Above I alluded to the fact that we briefly ran ephemeral, interactive, session-lived processes on Kubernetes. We quickly realized that Kubernetes is designed

> Above I alluded to the fact that we briefly ran ephemeral, interactive, session-lived processes on Kubernetes. We quickly realized that Kubernetes is designed for robustness and modularity over container start times. Is there a clear example of this? E.g. is kubernetes inherently unable to start a pod (assuming the same sequence of events, e.g. warm/cold image with streaming enabled) under 500ms, 1s etc? I am asking this as someone who spent quite a bit of time and wasn't able to bring it down 2s< mark, which eventually led us to rewrite the latency sensitive parts to use Nomad. But we are currently in a state where we are re-considering kubernetes for its auxilary tooling benefits and would love to learn more if anyone had experiences with starting and stopping thousands of pods with the lowest possible latencies without caring for utilization or placement but just observable boot latencies.

p_l 3y ago
You'd have to ensure that a) preload all images of course b) there's enough of nodes with enough capacity c) the pods don't use anything that has possible longer latency (high latency CSI etc.) d) you might want to write custom scheduler for your workloads (it could take into account what images are preloaded where, etc)
paulgb 3y ago
I do believe that with the right knowledge of Kubernetes internals it's probably possible to get k8s cold start times competitive with where we landed without Kubernetes (generally subsecond, often under 0.5s depending on how much the container does before passing a health check), but we'd have to understand k8s internals really well and would have ended up throwing out much of what already existed. And we'd probably end up breaking most of the reasons for using Kubernetes in the first place in the process.
- p_l 3y ago
  Not much internals needed, but actual in depth understanding of Pod kube-api plus at least basics of how scheduler, kubelet, and kubelet drivers interact. Big possible win is custom scheduling, but barely anyone seems to know it exists
  - paulgb 3y ago
    Yeah, looking into writing a scheduler was basically where we stepped back and said “if we write this ourselves, why not the rest, too”. As I see it, the biggest gains that we were able to get were by making things happen in parallel that would by default happen in sequence, and optimizing for the happy path instead of optimizing for reducing failure. In Kubernetes it's reasonable to have to wait for a dozen things to serially go through RAFT consensus in etcd before the pod runs, but we don't want that. (I made up the dozen number, but my point is that that design would be perfectly acceptable given Kubernetes' design constraints)
  - cogman10 3y ago
    Not surprising to me. People are complaining about how difficult it is to know k8s when you talk about the basic default objects. Getting into the weeds of how the api and control plane work (especially since it has little impact on day to day dev) is something devs tend to just avoid.
    - p_l 3y ago
      Honestly, devs of the applications that run on top probably should not have to worry about it. Instead have a platform team provide the necessary features.
- hobofan 3y ago
  Yeah, with plain Kubernetes I'd also see the practical limit around ~0.5s. If you are on GKE Autopilot where you also have little control over node startup there is likely also a lot more unpredictability. Something like Knative can allow for faster startup times if you follow the common best-practices (pre-fetching images, etc.), but I'm not sure if it supports enough of the session-related feature that you were probably looking for to be a stand-in for Plane.

p_l 3y ago

You'd have to ensure that a) preload all images of course b) there's enough of nodes with enough capacity c) the pods don't use anything that has possible longer latency (high latency CSI etc.) d) you might want to write custom scheduler for your workloads (it could take into account what images are preloaded where, etc)

paulgb 3y ago

I do believe that with the right knowledge of Kubernetes internals it's probably possible to get k8s cold start times competitive with where we landed without Kubernetes (generally subsecond, often under 0.5s depending on how much the container does before passing a health check), but we'd have to understand k8s internals really well and would have ended up throwing out much of what already existed. And we'd probably end up breaking most of the reasons for using Kubernetes in the first place in the process.

p_l 3y ago
Not much internals needed, but actual in depth understanding of Pod kube-api plus at least basics of how scheduler, kubelet, and kubelet drivers interact. Big possible win is custom scheduling, but barely anyone seems to know it exists
- paulgb 3y ago
  Yeah, looking into writing a scheduler was basically where we stepped back and said “if we write this ourselves, why not the rest, too”. As I see it, the biggest gains that we were able to get were by making things happen in parallel that would by default happen in sequence, and optimizing for the happy path instead of optimizing for reducing failure. In Kubernetes it's reasonable to have to wait for a dozen things to serially go through RAFT consensus in etcd before the pod runs, but we don't want that. (I made up the dozen number, but my point is that that design would be perfectly acceptable given Kubernetes' design constraints)
- cogman10 3y ago
  Not surprising to me. People are complaining about how difficult it is to know k8s when you talk about the basic default objects. Getting into the weeds of how the api and control plane work (especially since it has little impact on day to day dev) is something devs tend to just avoid.
  - p_l 3y ago
    Honestly, devs of the applications that run on top probably should not have to worry about it. Instead have a platform team provide the necessary features.
hobofan 3y ago
Yeah, with plain Kubernetes I'd also see the practical limit around ~0.5s. If you are on GKE Autopilot where you also have little control over node startup there is likely also a lot more unpredictability. Something like Knative can allow for faster startup times if you follow the common best-practices (pre-fetching images, etc.), but I'm not sure if it supports enough of the session-related feature that you were probably looking for to be a stand-in for Plane.