5 ms·

Sounds more to me like they need a new CTO. And that they're desperate to tell customers that they've fixed their problems. Kubernetes is absolutely the wrong

Sounds more to me like they need a new CTO. And that they're desperate to tell customers that they've fixed their problems. Kubernetes is absolutely the wrong tool for this use case, and I argue that this should be obvious to someone in a CTO-level position, or their immediate advisors. Kubernetes excels as a microservices platform, running reasonably trustworthy workloads. The key features of Kubernetes are rollout (highly available upgrades), elasticity (horizontal scaleout), bin packing (resource limits), CSI (dynamically mounted block storage), and so on. All this relates to a highly dynamic environment. This is not at all what Gitpod needs. They need high performance disks, ballooning memory, live migrations, and isolated workloads. Kubernetes does not provide you sufficient security boundaries for untrusted workloads. You need virtualization for that, and ideally physically separate machines. Another major mistake they made was trying to build this on public cloud infrastructure. Of course the performance will be ridiculous. However, one major reason for using Kubernetes is sharing the GPU. That is, to my knowledge, not possible with virtualization. But again, do you want to risk sharing your data, on a shared GPU?

dilyevsky 2y ago
I agree on the cloud thing. Don't agree that "high performance disks, ballooning memory, live migrations, and isolated workloads" preclude from using k8s - you can still run it as base layer. You get some central configuration storage, machine management and some other niceties for free and you can push your VM-specific features into your application pod. In fact, that's how Google Cloud is designed (except they use Borg not k8s but same idea).
- concerndc1tizen 2y ago
  True! I love the idea of using K8s to orchestrate the running of VMs. With graceful shutdown and distributed storage, it makes it even more trivial to semi-live migrate VMs. Are you aware of the limits? It must run as root and privileged?
  - dilyevsky 2y ago
    In this scenario k8s is orchestrating the hypervisor, not VMs themselves. Hypervisor then orchestrates VMs + network (eg OVS) + other supporting functions (logs shipping, etc) on each individual “worker” node. VM scheduling/migration component needs to be completely decoupled from k8s apiserver (but itself can still run as normal k8s deployment) bc scaling kube api with unbound users is challenging. And yes, hypervisor will need to run privileged but you can limit it to worker nodes only
ed_mercer 2y ago
Why would you say that performance is bad on public cloud infrastructure?
- concerndc1tizen 2y ago
  There are things that public cloud is great for. Cost efficiency at high performance is not it. For Gitpod, performance is critical to their product offering, because any latency in a dev environment is terrible UX. Example: What performance do you get out of your NVMe disks? Because these days you can build storage that delivers 100-200 GB/s. https://www.graidtech.com/wp-content/uploads/2023/04/Results-Summary-Tables_Sequential-1024x566.jpg https://www.graidtech.com/wp-content/uploads/2023/04/Results... I bet few public cloud customers are seeing that kind of performance.
  - ed_mercer 2y ago
    This is also my personal experience. I am finding that building out our own high-performance cluster of (second-hand) servers is orders of magnitude cheaper than having the same on GCP, even though we have to maintain/configure everything ourselves.
deleted 2y ago
[deleted]
bittermandel 2y ago
I consider Kubernetes to be an excellent framework to build these kinds of applications. The difference here is Gitpod being stateful, which is notoriously hard on Kubernetes, though easier now than ever before! To clarify on one of your points, Kubernetes itself has nothing to do with actually setting the security boundaries. It only providers a schema to describe resources and policies, and then an underlying system (perhaps Cilium for networking, or Kata Containers for micro VMs) can ensure that the resources created actually follow those schemas and policies. For example, Neon have built https://github.com/neondatabase/autoscaling https://github.com/neondatabase/autoscaling which manages Neon Instances with Kubernetes by running them with QEMU instead. This allows them to do live migrations and resource (de)allocation while the service is running, without having to replace Kubernetes. These workloads are, as far as I understand it, stateless.
- concerndc1tizen 2y ago
  You make an excellent point, and it emphasizes the need to distinguish between a typical Kubernetes setup (containers, pod/service mesh, and so on), and what Kubernetes can do in the abstract. In the extreme, the API server is just an HTTP interface for a KV store with a bit of RBAC and validation-mutation extensions. What Neon is doing is quite a feat: Live migration (of a VM) while preserving TCP connections. It also took a lot of customization to achieve that. But I agree that Kubernetes can indeed be used this way. If anything, it further cements my original point about the Gitpod leadership. The problem was never Kubernetes, but the dimwitted notion of using containers. And then blaming Kubernetes for it: We're leaving you.
- santiagobasulto 2y ago
  > The difference here is Gitpod being stateful, which is notoriously hard on Kubernetes, though easier now than ever before! We've always had issues with stateful kubernetes setups. Can you share what makes it easier today than before? Genuinely interested.

dilyevsky 2y ago

I agree on the cloud thing. Don't agree that "high performance disks, ballooning memory, live migrations, and isolated workloads" preclude from using k8s - you can still run it as base layer. You get some central configuration storage, machine management and some other niceties for free and you can push your VM-specific features into your application pod. In fact, that's how Google Cloud is designed (except they use Borg not k8s but same idea).

concerndc1tizen 2y ago
True! I love the idea of using K8s to orchestrate the running of VMs. With graceful shutdown and distributed storage, it makes it even more trivial to semi-live migrate VMs. Are you aware of the limits? It must run as root and privileged?
- dilyevsky 2y ago
  In this scenario k8s is orchestrating the hypervisor, not VMs themselves. Hypervisor then orchestrates VMs + network (eg OVS) + other supporting functions (logs shipping, etc) on each individual “worker” node. VM scheduling/migration component needs to be completely decoupled from k8s apiserver (but itself can still run as normal k8s deployment) bc scaling kube api with unbound users is challenging. And yes, hypervisor will need to run privileged but you can limit it to worker nodes only

ed_mercer 2y ago

Why would you say that performance is bad on public cloud infrastructure?

concerndc1tizen 2y ago
There are things that public cloud is great for. Cost efficiency at high performance is not it. For Gitpod, performance is critical to their product offering, because any latency in a dev environment is terrible UX. Example: What performance do you get out of your NVMe disks? Because these days you can build storage that delivers 100-200 GB/s. https://www.graidtech.com/wp-content/uploads/2023/04/Results-Summary-Tables_Sequential-1024x566.jpg https://www.graidtech.com/wp-content/uploads/2023/04/Results... I bet few public cloud customers are seeing that kind of performance.
- ed_mercer 2y ago
  This is also my personal experience. I am finding that building out our own high-performance cluster of (second-hand) servers is orders of magnitude cheaper than having the same on GCP, even though we have to maintain/configure everything ourselves.

deleted 2y ago

[deleted]

bittermandel 2y ago

I consider Kubernetes to be an excellent framework to build these kinds of applications. The difference here is Gitpod being stateful, which is notoriously hard on Kubernetes, though easier now than ever before! To clarify on one of your points, Kubernetes itself has nothing to do with actually setting the security boundaries. It only providers a schema to describe resources and policies, and then an underlying system (perhaps Cilium for networking, or Kata Containers for micro VMs) can ensure that the resources created actually follow those schemas and policies. For example, Neon have built https://github.com/neondatabase/autoscaling https://github.com/neondatabase/autoscaling which manages Neon Instances with Kubernetes by running them with QEMU instead. This allows them to do live migrations and resource (de)allocation while the service is running, without having to replace Kubernetes. These workloads are, as far as I understand it, stateless.

concerndc1tizen 2y ago
You make an excellent point, and it emphasizes the need to distinguish between a typical Kubernetes setup (containers, pod/service mesh, and so on), and what Kubernetes can do in the abstract. In the extreme, the API server is just an HTTP interface for a KV store with a bit of RBAC and validation-mutation extensions. What Neon is doing is quite a feat: Live migration (of a VM) while preserving TCP connections. It also took a lot of customization to achieve that. But I agree that Kubernetes can indeed be used this way. If anything, it further cements my original point about the Gitpod leadership. The problem was never Kubernetes, but the dimwitted notion of using containers. And then blaming Kubernetes for it: We're leaving you.
santiagobasulto 2y ago
> The difference here is Gitpod being stateful, which is notoriously hard on Kubernetes, though easier now than ever before! We've always had issues with stateful kubernetes setups. Can you share what makes it easier today than before? Genuinely interested.