34 ms·

We're Leaving Kubernetes

517 points by filiptronicek 2y ago

javier_e06 2y ago
The article is an excellent cautionary tale. Debugging an app in a container is one thing. Debugging and app running inside a Kubernetes node is a rabbit hole that demands more hours and expertise.
deleted 2y ago
[deleted]
lolinder 2y ago
> This is not a story of whether or not to use Kubernetes for production workloads that’s a whole separate conversation. As is the topic of how to build a comprehensive soup-to-nuts developer experience for shipping applications on Kubernetes. > This is the story of how (not) to build development environments in the cloud. I'd like to request that the comment thread not turn into a bunch of generic k8s complaints. This is a legitimately interesting article about complicated engineering trade-offs faced by an organization with a very unique workload. Let's talk about that instead of talking about the title!
- kitd 2y ago
  Agreed. It's actually a very interesting use case and I can easily see that K8s wouldn't be the answer. My dev env is very definitely my "pet", thank you very much!
  - ethbr1 2y ago
    It'd be nice to editorialize the title a bit with "... (for dev envs)" for clarity. Super useful negative example, and the lengths they pursued to make it fit! And no knock on the initial choice or impressive engineering, as many of the k8s problems they hit likely weren't understood gaps at the time they chose k8s. Which makes sense, given k8s roots in (a) not being a security isolation tool & (b) targeting up-front configurability over runtime flexibility. Neither of which mesh well with the co-hosted dev environment use case.
- preommr 2y ago
  Can someone clarify if they mean development environments, or if they're talking about a service that they sell that's related to development environments. Because I don't understand most of the article if it's the former. How are things like performance are a concern for internal development environments? And why are so many things stateful - ideally there should be some kind of configuration/secret management solution so that deployments are consistent. If it's the latter, then this is incredibly niche and maybe interesting, but unlikely to be applicable to anyone else.
  - wutwutwat 2y ago
    4th paragraph in if you read the article… > This is not a story of whether or not to use Kubernetes for production workloads that’s a whole separate conversation. As is the topic of how to build a comprehensive soup-to-nuts developer experience for shipping applications on Kubernetes. > This is the story of how (not) to build development environments in the cloud.
    - epgui 2y ago
      I'm not sure that this really answers their question.
      - wutwutwat 2y ago
        > Can someone clarify if they mean development environments, or if they're talking about a service that they sell that's related to development environments. their question isn't asking anything. It's both about development environments AND a service they sell, which is dev envs. Even so > This is the story of how (not) to build development environments in the cloud. that is what the article is about. Their words. The person asked what the article is about, this it it, from the authors themselves. read the damn article
  - bittermandel 2y ago
    It's for running their commercial products, which are stateful and long-lived developer environments.
clvx 2y ago
I tried doing a dev environment on Kubernetes but the fact you have to be dealing with a set of containers that could change if the base layer changed meant instability in certain cases which threw me off. I ended up with a mix of nix and it's vm build system which is based on qemu. The issue is too tied to NixOS and all services run in the same place which forces you to manage ports and other things. How I wish it could work is having a flake that defines certain services, these services could or could not run in different µVMs sharing an isolated linux network layer. Your flake could define your versions, your commands to interact and manage the lifecyle of those µVM's. As the nix store can be cached/shared, it can be provide fast and reproducible builds after the first build.
- candiddevmike 2y ago
  > the fact you have to be dealing with a set of containers that could change if the base layer changed meant instability Can you expand on this? Are you talking about containers you create?
- eptcyka 2y ago
  Have you tried https://github.com/astro/microvm.nix https://github.com/astro/microvm.nix ? You can use the same NixOS module for both declarative VMs and imperatively configured and spawned VMs.
bhouston 2y ago
I also recently left Kubernetes. It was a huge waste of time and money. I've replaced it with just a series of services on Google Cloud Run and then using Google's Cloud Run Tasks services for longer running tasks. The infrastructure now incredibly understandable and simple and cost effective. Kubernetes cost us >$million in both DevOps time and actually Google Cloud costs unnecessarily, and even worse it cost us time to market. Stay off of Kubernetes as long as you can in your company, unless you are basically forced onto it. You should view it as an unnecessary evil that comes with massive downsides in terms of complexity and cost.
- elcomet 2y ago
  Aren't you afraid of being now stuck with GCP?
  - bhouston 2y ago
    It is just a bunch of docker containers. Some run in tasks and some run as auto-scaling services. Would probably take a week to switch to AWS as there are equivalent managed services there. But this is really a spurious concern. I myself used to care about it years ago. But in practice, rarely do people switch between cloud providers because the incremental benefits are minor, they are nearly equivalent, there is nothing much to be gained by moving from one to the other unless politics are involved (e.g. someone high up wants a specific provider.)
    - spwa4 2y ago
      How does the orchestration work? How do you share storage? How do the docker containers know how to find each other? How does security work? I feel like Kubernetes' downfall, for me, is the number of "enterprise" features it (got convinced into) supporting and enterprise features doing what they do best: turning the simplest of operations into a disaster.
      - bhouston 2y ago
        > How does the orchestration work? Github Actions CI. Take this and make a few more dependencies and a matrix strategy and you are good to go: https://github.com/bhouston/template-typescript-monorepo/blob/main/.github/workflows/ci.yml https://github.com/bhouston/template-typescript-monorepo/blo... For dev environments, you can add post-fixes to the services based on branches. > How do you share storage? I use managed DBs and Cloud Storage for shared storage. I think that provisioning your own SSDs/HDs to the cloud is indicative of an anti-pattern in your architecture. > How do the docker containers know how to find each other? I try to avoid too much communication between services directly, rather try to go through pub-sub or similar. But you can set up each service with a domain name and access them that way. With https://web3dsurvey.com https://web3dsurvey.com, I have an api on https://api.web3dsurvey.com https://api.web3dsurvey.com and then a review environment (connected to the main branch) with https://preview.web3dsurvey.com https://preview.web3dsurvey.com / https://api.preview.web3dsurvey.com https://api.preview.web3dsurvey.com. > How does security work? You can configure Cloud Run services to be internal only and not to accept outside connections. Otherwise one can just use JWT or whatever is normal on your routes in your web server.
ensignavenger 2y ago
The article does a great job of explaining the challenges they ran into with Kubernetes, and some of the things they tried... but I feel like it drops the ball at the end by not telling us at least a little what they chose instead. The article mentions they call their new solution "Gitpod Flex" but there is nothing about what Gitpod Flex is. They said they tried microVMs and decided against them, and of course Kubernetes, the focus of the article. So is GitpodFlex based on full VM's? Docker? Some other container runtime?? Perhaps a followup article will go into detail about their replacement.
- loujaybee 2y ago
  Yeah, that's fair. The blog was getting quite long, so we need to do some deeper dives in follow-ups. Gitpod Flex is runner-based. The runner interface is intentionally generic so that we can support different clouds, on-prem or just Linux in future. The first implemented runner is built around AWS primitives like EC2, EBS and ECS. But because of the more generic interface Gitpod now supports local / desktop environments on MacOS. And again, future OS support will come. There’s a bit more information in the docs, but we will do some follow ups! - https://www.gitpod.io/docs/flex/runners/aws/setup-aws-runners https://www.gitpod.io/docs/flex/runners/aws/setup-aws-runner... - https://www.gitpod.io/docs/flex/gitpod-desktop https://www.gitpod.io/docs/flex/gitpod-desktop (I work at Gitpod)
  - nickstinemates 2y ago
    Echoing the parent you're replying to. You built up all of the context and missed they payoff.
    - ethbr1 2y ago
      I thought it was fair. >> We’ll be posting a lot more about Gitpod Flex architecture in the coming weeks or months. Cramming more detail into this post would have exceeded the average user read time ceiling.
  - Bombthecat 2y ago
datadeft 2y ago
The original k8s paper mentioned that the only use case was a low latency and a high latency workflow combination and the resource allocation is based on that. The generic idea is that you can easily move low latency work between nodes and there are no serios repercussions when a high latency job fails. Based on this information, it is hard to justify to even consider k8s for the problem that gitpod has.
- junkaccount 2y ago
  Thanks for reading the paper!
  - datadeft 2y ago
    For those who are interested: https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43438.pdf https://static.googleusercontent.com/media/research.google.c... I am not sure what differences k8s has compare to Borg. At the concept level these are pretty comparable.
xyst 2y ago
I do agree with the points in article that k8s is not a good fit for development environments. In my opinion, k8s is great for stable and consistent deployment/orchestration of applications. Dev environments by default are in a constant state of flux. I don’t understand the need for “cloud development environments” though. Isn’t the point of containerized apps is to avoid the need for synchronizing dev envs amongst teams? Or maybe this product is supposed to decrease onboarding friction?
- sofixa 2y ago
  It's to ensure a consistent environment for all developers, with the resources required. E.g. they mention GPUs, for developers working with GPU-intensive workloads. You can ship all developers gaming laptops with 64GB RAM and proper GPUs, and have them fight the environment to get the correct libraries as you have in prod (even with containers that's not trivial), or you can ship them Macbook Airs and similar, and have them run consistent (the same) dev environments remotely (you can self-host gitpod, it's not only a cloud service, it's more the API/environment to get consistent remote dev enviornments).
  - loujaybee 2y ago
    Yeah, exactly. Containers locally are a basic foundation. But usually those containers or services need to talk to one another, they need some form of auth and credentials, they need some networking setup. There's a lot of configuration in all of that. The more devs swap projects or the more complex the thing you're working on the more the challenge grows. Automating depedencies, secret access, ensuring projects have the right memory, cpu, gpu etc. Also security - moving source code off your laptop and devices and standardizing your setups helps if you need to do a lot of audit and compliance as you can automate it.
- dikei 2y ago
  Sarcastically, CDE is one way to move cost from CAPEX (get your developer a Mac Book Pro) to OPEX (a monthly subscription that you only need to pay as long as the dev has not been lay off) It's also much cheaper to hire contractors and give them the CDE that can be terminated on a moment notice.
lmeyerov 2y ago
I was intrigued because the development environment problem is similar to the data scientist one - data gravity, GPU sharing, etc - but I'm confused on the solution? Oddly, I left with a funny alternate takeaway: One by one, their clever inhouse tweaks & scheduling preferences were recognized by the community and turned into standard k8s knobs So I'm back to the original question... What is fundamentally left? It sounds like one part is maintaining a clean container path to simplify a local deploy, which a lot of k8s teams do (ex: most of our enterprise customers prefer our docker compose & AMIs over k8s). But more importantly, something fundamental architecturally about how envs run that k8s cannot do, but they do not identify?
- thenaturalist 2y ago
  > We’ll be posting a lot more about Gitpod Flex architecture in the coming weeks or months. I’d love to invite you on November the 6th to a virtual event where I’ll be giving a demo of Gitpod Flex and I’ll deep-dive into the architecture and security model at length. Bottom of the post.
- csweichel 2y ago
  OP here. The Kubernetes community has been fantastic at evolving the platform, and we've greatly enjoyed being in the middle of it. Indeed, many of the things we had to build next to Kubernetes have now become part of k8s itself. Still, some of the core challenges remain: - the flexibility Kubernetes affords makes it hard to build and distribute a product with such specific requirements across the broad swath of differently set up Kubernetes installations. Managed Kubernetes services help, but come with their own restrictions (e.g. Kernel versions on GKE). - state handling and storage remains unsolved. PVCs are not reliable enough, subject to a lot of variance (see point above), and depending on the backing storage have vastly different behaviour. Local disks (which we use to this day), make workspace startup and backup expensive from a resource perspective and hard to predict timing wise. - user namespaces have come a long way in Kubernetes, but by themselves are not enough. /proc is still masked, FUSE is still not usable. - startup times, specifically container pulls and backup restoration, are hard to optimize because they depend on a lot of factors outside of our control (image homogeneity, cluster configuration) Fundamentally, Kubernetes simply isn't the right choice here. It's possible to make it work, but at some point the ROI of running on Kubernetes simply isn't there.
debarshri 2y ago
Phew, it is absolutely true. Building dev environments on k8s become wasteful. To add to this complexity, if you are building a product that is self hosted on customer's infrastructure. Debugging and support also become non homogeneous and difficult. What we have seen works especially when you are building developer centric product is expose these native issues around network, memory, compute and storage to engineers and they are more willing to work around it. Abstracting those issues leads to shift in responsibility on the product. Having said that, I still think k8s is an upgrade when you have a large team.
horsawlarway 2y ago
Personally - just let the developer own the machine they use for development. If you really need consistency for the environment - Let them own the machine, and then give them a stable base VM image, and pay for decent virtualization tooling that they run... on their own machine. I have seen several attempts to move dev environments to a remote host. They invariably suck. Yes - that means you need to pay for decent hardware for your devs, it's usually cheaper than remote resources (for a lot of reasons). Yes - that means you need to support running your stack locally. This is a good constraint (and a place where containers are your friend for consistency). Yes - that means you need data generation tooling to populate a local env. This can be automated relatively well, and it's something you need with a remote env anyways. --- The only real downside is data control (ie - the company has less control over how a developer manages assets like source code). I'm my experience, the vast majority of companies should worry less about this - your value as a company isn't your source code in 99.5% of cases, it's the team that executes that source code in production. If you're in the 0.5% of other cases... you know it and you should be in an air-gapped closed room anyways (and I've worked in those too...)
- dangsux 2y ago
  [dead]
- shriek 2y ago
  And the reason they suck is the feedback loop is just too high as compared to running it locally. You have to jump through hoops to debug/troubleshoot your code or any issues that you come across between your code and output of your code. And it's almost impossible to work on things when you have spotty internet. I haven't worked on extremely sensitive data but for PII data from prod to dev, scrubbing is a good practice to follow. This will vary based on the project/team you're on of course.
  - ethbr1 2y ago
    Aka 'if a developer knew beforehand everything they needed, it wouldn't be development'
  - 2y ago
pphysch 2y ago
The problem with "development environments", like other interactive workloads, is that there is a human at the other end that desires a good interactive experience with every keypress. It's a radically different problem space than what k8s was designed for. From a resource provider productive, the only way to squeeze a margin out of that space would be to reverse engineer 100% of human developer behavior so that you can ~perfectly predict "slack" in the system that could be reallocated to other users. Otherwise it's just a worse DX, like TFA gives examples of. Not a business I'm envious too be in... Just give everyone a dedicated VM or desktop, and make sure there's a batch system for big workloads.
rekoros 2y ago
We've been using Nix flakes and direnv (https://direnv.net/ https://direnv.net/) for developer environments and NixOS with https://github.com/serokell/deploy-rs https://github.com/serokell/deploy-rs for prod/deploys - takes serious digging and time to set up, but excellent experience with it so far.
- andreweggleston 2y ago
  I’ve been using Nix for the past year and it really feels like the holy grail for stable development environments. Like you said—it takes serious time to set up, but it seems like that’s an unavoidable reality of easily sharable dev envs.
- aliasxneo 2y ago
  Serious time to set up _and_ maintain as the project changes. At least, that was my experience. I really _want_ to have Nix-powered development environments, but I do _not_ want to spend the rest of my career maintaining them because developers refuse to "seriously dig" to understand how it works and why it decided to randomly break when they added a new dependency. I think this approach works best in small teams where everyone agrees to drink the Nix juice. Otherwise, it's caused nothing but strife in my company.
  - rekoros 2y ago
    This may be the one area where some form of autocracy has merit :-)
    - imp0cat 2y ago
      What's that saying about developers and herding cats? :)
geoctl 2y ago
I've worked on something similar to gitpod in a slightly different context that's part of a much bigger personal project related to secure remote access that I've actually spent a few years building now and hope to open source in a few months from now. While I agree on many of the points in the article, I just don't understand how using micro VMs by itself replaces K8s unless they actually start building their own K8s that orchestrates their micro VMs (as opposed to containers in the case of k8s) ending up with the same thing basically when k8s itself can be used to orchestrate the outer containers that run the micro VMs used to run the dev containers. Yes, k8s has many challenges when it comes to nesting containers, cgroups, creating rootless containers inside the outer k8s containers and other stuff such as multi-region scaling, but actually the biggest challenge that I've faced so far isn't related to networkPolicies or cgroups but is actually by far related to storage, both when it comes to (lazily) pulling big OCI images which are extremely unready to be used for dev containers whose sizes are typically in the GBs or 10s of GBs as well as also when it comes to storage virtualization over the underlying k8s node storage. There are serious attempts to accelerate image pulling (e.g. Nydus) but such solutions would still probably be needed whether you use micro VMs or rootless/userns containers in order to load and run your dev containers.
cheptsov 2y ago
I can completely relate to anyone abandoning K8s. I'm working with dstack, an open-source alternative to K8s for AI infra [1]. We talk to many people who are frustrated with K8s, especially for GPU and AI workloads. [1] https://github.com/dstackai/dstack https://github.com/dstackai/dstack
- Muhtasham 2y ago
  I really like dstack, keep up the great work
rohitghumare 2y ago
You just simplified Kubernetes Management System
myestery 2y ago
Leaving this comment here so I'll always come back to read this as someone who was considering kubernetes for a platform like gitpod
- teach 2y ago
  Remember that you can favorite posts.
alecfong 2y ago
Our first implementation of brev.dev was built on top of kubernetes. We were also building a remote dev environment tool at the time. Treating dev environments like cattle seemed to be the wrong assumption. Turning kubernetes into a pet manager was a huge endeavor with long tail of issues. We rewrote our platform against vms and were immediately able to provide a better experience. Lots of tradeoffs but makes sense for dev envs.
junkaccount 2y ago
The real reason for this shift is that kubernetes moved to containerd which they cannot handle. Docker was much easier. Differential workloads is not correct to blame. Also, there is a long tail of issues to be fixed if you do it with Kubernetes. Kubernetes does not just give you scaling, it gives you many things: run on any architecture, be close to your deployment etc.
- moondev 2y ago
  https://github.com/Mirantis/cri-dockerd https://github.com/Mirantis/cri-dockerd
  - junkaccount 2y ago
    Most of the kubernetes providers (GKE, EKS) do not support this new shim. Even on baremetal it is possibly hard to run.
trimen 2y ago
[flagged]
tacone 2y ago
On a side note: has anybody experience with MicroK8s? I'd love to learn stories about it. I'm interested in both dev and production experiences.
- redrove 2y ago
  Microk8s is nothing but a kubernetes distro from canonical. Personally I would use k3s because it’s a little more widespread and less opinionated in a good way. Anyway, as always it depends on what you want to use it for.
concerndc1tizen 2y ago
Sounds more to me like they need a new CTO. And that they're desperate to tell customers that they've fixed their problems. Kubernetes is absolutely the wrong tool for this use case, and I argue that this should be obvious to someone in a CTO-level position, or their immediate advisors. Kubernetes excels as a microservices platform, running reasonably trustworthy workloads. The key features of Kubernetes are rollout (highly available upgrades), elasticity (horizontal scaleout), bin packing (resource limits), CSI (dynamically mounted block storage), and so on. All this relates to a highly dynamic environment. This is not at all what Gitpod needs. They need high performance disks, ballooning memory, live migrations, and isolated workloads. Kubernetes does not provide you sufficient security boundaries for untrusted workloads. You need virtualization for that, and ideally physically separate machines. Another major mistake they made was trying to build this on public cloud infrastructure. Of course the performance will be ridiculous. However, one major reason for using Kubernetes is sharing the GPU. That is, to my knowledge, not possible with virtualization. But again, do you want to risk sharing your data, on a shared GPU?
- dilyevsky 2y ago
  I agree on the cloud thing. Don't agree that "high performance disks, ballooning memory, live migrations, and isolated workloads" preclude from using k8s - you can still run it as base layer. You get some central configuration storage, machine management and some other niceties for free and you can push your VM-specific features into your application pod. In fact, that's how Google Cloud is designed (except they use Borg not k8s but same idea).
  - concerndc1tizen 2y ago
    True! I love the idea of using K8s to orchestrate the running of VMs. With graceful shutdown and distributed storage, it makes it even more trivial to semi-live migrate VMs. Are you aware of the limits? It must run as root and privileged?
    - dilyevsky 2y ago
      In this scenario k8s is orchestrating the hypervisor, not VMs themselves. Hypervisor then orchestrates VMs + network (eg OVS) + other supporting functions (logs shipping, etc) on each individual “worker” node. VM scheduling/migration component needs to be completely decoupled from k8s apiserver (but itself can still run as normal k8s deployment) bc scaling kube api with unbound users is challenging. And yes, hypervisor will need to run privileged but you can limit it to worker nodes only
hintymad 2y ago
I was wondering if there's productivity angle too. Take Ceph vs Rook for example. If a Ceph cluster needs all the resources on its machines and the cluster manages its resources too, then moving to Rook does not give any additional features. All the 50K additional lines of code in Rook is to set up CSIs and statefulsets and whatnot just to get Ceph working on Kubernetes.
I_LIKE_TO_HATE 2y ago
[flagged]
I_LIKE_TO_HATE 2y ago
[flagged]
vbezhenar 2y ago
I read this article and I still don't understand what's wrong with Kubernetes for this task. Everything you would do with virtual machines could be done with Kubernetes with very similar results. I guess team just wants to rewrite everything, it happens. Manager should prevent that.
dwroberts 2y ago
> Autoscaler plugins: In June 2022, we switched to using cluster-autoscaler plugins when they were introduced. Does anyone have any links for cluster-autoscaler plugins? Searching drawing a blank, even in the cluster-autoscaler repo itself. Did this concept get ditched/removed?
rahen 2y ago
Kubernetes works great for stateless workloads. For anything stateful, monolithic, or that doesn't require autoscaling, I find LXC more appropriate: - it can be clusterized (LXD/Incus), like K8S but unlike Compose - it exposes some tooling to the data plane, especially a load balancer, like K8S - it offers system instances with a complete distribution and a init system, like a VM but unlike a Docker container - it can orchestrate both VMs (including Windows VMs) and LXC containers at the same time in the same cluster - LXC containers have the same performance as Docker containers unlike a VM - it uses a declarative syntax - it can be used as a foundation layer for anything stateful or stateless, including the Kubernetes cluster LXD/Incus sits somewhere between Docker Swarm and a vCenter cluster, which makes it one of the most versatile platform. Nomad is also a nice contender, it cannot orchestrate LXC containers but can autoscale a variety of workloads, including Java apps and qemu VMs.
- belthesar 2y ago
  I too am rallying quickly to the Incus way of doing things. Also of note, there's an effort to build a utility to write Compose manifests for Incus workloads that I'm following very closely. https://github.com/bketelsen/incus-compose https://github.com/bketelsen/incus-compose
  - eddyg 2y ago
    Thanks for pointing out `incus-compose`!
- OneCricketeer 2y ago
  Nomad cannot orchestrate what, exactly? https://developer.hashicorp.com/nomad/tutorials/plugins/plugin-lxc https://developer.hashicorp.com/nomad/tutorials/plugins/plug...
deepsun 2y ago
> SSD RAID 0 > A simpler version of this setup is to use a single SSD attached to the node. This approach provides lower IOPS and bandwidth, and still binds the data to individual nodes. Are you sure SSD is that slow? NVMe devices are so fast that I hardly believe there's any need for RAID 0.
- mikeshi42 2y ago
  In AWS iirc NVMe max out at 2GB/s - I'm not sure why that's the case. I know there were issues with the PCIe controller in the past being the bottleneck, but I suspect there's something more to it than that.
abofh 2y ago
I feel like anyone who was building a CI solution to sell to others and chose kubernetes didn't really understand the problem. You're running hot pods for crypto miners and against people who really want to see the rest of the code that box has ever seen. You should be isolating with something purpose built like firecracker, and do your own dispatch & shred for security.
- geoctl 2y ago
  Firecracker is more comparable to container runtimes than to orchestrators such as K8s. You still need an orchestrator to schedule, manage and garbage-collect all your uVMs on top of your infrastructure exactly like you would do with containers via k8s. In other words, you will probably have to either use k8s or build your own k8s to run "supervisor" containers/processes that launch uVMs which in turn launch the customer dev containers.
  - abofh 2y ago
    For sure, but that's the point - containers aren't really good for an adversarial CI solution. You can run that shit in house on kubernetes on a VM in a simulated VR if you want. But if you have adversarial builds, you have a) builds that may well need close to root, and b) customers who may well want to break your shit. Containers are not the right solution for that, VM's get you mostly there, and the right answer is burning bare metal instances with fire after every change-of-tenant - but nobody does that (anymore), because VM's are close enough and it's faster to zero out a virtual disk than a real one. So if you started with kubernetes and fought the whole process of why it's not a great solution to the problem, I have to assume you didn't understand the problem. I :heart: kubernetes, its complexity pays my bills - but it's barely a good CI solution when you trust everyone involved, it's definitely not a good one where you're trying to be general-purpose to everyone with a makefile.
    - geoctl 2y ago
      I would argue that dev containers are more complicated than CI even though they share many of the challenges (e.g. devcontainers might need to load 10s or 100s of GBs to start and are write heavy). I would also argue that userns/rootless containers provide "enough" isolation when it comes to isolating CPU/memory/networking as well as access to the host's syscalls if you're careful enough; however when it comes to storage (e.g. max disk size that a container can use and write to, max opened files, completely hiding the host's fs from the container's, etc...), it's unfortunately still extremely limited,fs-dependent for some features, even though modern solutions (e.g. vDPA and ublk) can be used to fix that and virtualize the storage for containers.
eYrKEC2 2y ago
Have folks seen success with https://earthly.dev/ https://earthly.dev/ as a tool in their dev cycle?
riiii 2y ago
> development environments Kubernetes has never ever struck me as a good idea for a development environment. I'm surprised it took the author this long to figure out. K8s can be a lifesaver for production, staging, testing, ... depending on your requirements and infrastructure.
- sethammons 2y ago
  Our operations team is planning to build dev envs in k8s, but only the networked dependencies. Like a personal testing/staging where you have full control of the thing(s) you are developing and can simply leverage the rest of the stack. Sounds sane. Am i missing anything?
Andkeith 2y ago
[dead]
cryptica 2y ago
Kubernetes is awesome but I understand what the article is getting at. K8s was designed for a mostly homogeneous architecture when your platform requirements end with "deploy this service to my cluster" and you don't really care about the specifics of how it's scheduled. A heterogeneous architecture with multi-tenancy poses some unique challenges because, as mentioned in the article, you get highly inconsistent usage patterns across different services. Also, arbitrary code execution (with sandboxing) can present a signifiant challenge. For security, you ideally need full isolation between services which belong to different users; this isolation wasn't a primary design goal of Kubernetes. That said, you can probably still use K8s, but in a different way. For smaller customers, you could co-locate on the same cluster, but for larger customers which have high scalability requirements, you could have a separate K8s cluster for each one. Surely for such customers, it's worth the extra effort. So in conclusion, I don't think the problems which were identified necessarily warrant abandoning K8s entirely, but maybe just a rethinking of how K8s is used. K8s still provides a lot of value in treating a whole cluster of computers as a single machine, especially if all your architecture is already set up for it. In addition to scheduling/orchestration, K8s offers a lot of very nice-to-have features like performance monitoring, dashboards, aggregated logs, ingress, health checks, ...
ncrmro 2y ago
We started having a few developers have constant VSCode timeouts. We switched to GitHub devcontainers which have been great.
eichi 2y ago
Kubernetes is just combined infra admin practices. Whether we use it or not, we need to do the same things by local oriented way or vendor specific way . 1. Some operations on remote in local oriented way are time consuming and unmanageable. 2. With vendor specific way, our skill would be deprecated, having dependency to the vendors. 3. Kubernetes is not the best tools but it it popular. As always, custom solution is the most powerful but should be replaced with more unified way for the stability of the development.
Jack008 2y ago
Make sure you need microservices-based architecture because it comes with its own complexity - a load balancer, container networking, distributed tracing, etc. If you application does not need to scale its sub-components independently, you are better off using a VM-based application. It's 10X cheaper to maintain/troubleshoot and is high performance/resources.
deleted 2y ago
[deleted]
vrnvu 2y ago
For development, I made the switch to nix/flox and it’s been a game-changer.
- mstrangfeld 2y ago
  How well does Flox work out of the box? I would really like to introduce Nix to the dev environments in my company but the struggle of maintaining nix files and flakes is too large. I've looked at DevBox and it looks quite accessible but Flox also looks like a nice way to sneak some of the Nix goodness into the company.
  - vrnvu 2y ago
    It works pretty well for most common packages, if a rare package/dependency fails is mostly nix problem to solve upstream.
bluelightning2k 2y ago
The debate in the comments about whether you should run locally is fascinating. To the people saying ultra modern hardware could handle it: worth remembering the companies on question started on this path X years ago with Y set of technologies and Z set of experiences. Because it made sense for Google in 2012 or whatever doesn't necessarily mean they would choose it again --or not-- given a do over (but there's basically no way back).
gloosx 2y ago
>Kubernetes seems like the obvious choice for building out remote, standardized and automated development environments - Is it really Obvious Choice™ though Fred? - Hmm, let's consult the graphs. >Kubernetes is a container orchestration system for automating software deployment. - It's about automating deployment Carl, not development environments! >Kubernetes is not the right choice for building development environments, as we’ve found.
linuxftw 2y ago
The article offers toward the end that now self-hosted customers can run their app on something other than k8s. I think this is a mistake. We're a k8s enterprise shop, and I don't want to support any more VMs. If it's not on k8s, I'm not running it. I don't want to be responsible for golden images, patching, and all the fun that comes with managing workloads outside of k8s. That's why I have k8s. All the problems in the article also seem self-imposed. k8s can run stateful workloads just fine. Don't start and stop them. Figure out the math on how much it costs to run a container 24/7, add your margin, and pass that cost to the customer. Customer can decide to stop the containers to save $$, so the latency won't hurt, they'll accept it because they know they're saving money.
eadem 2y ago
The cloud maker is the answer of all this. qbo.io
- remram 2y ago
  Hi Alex Diaz from qbo, can you stop spamming links to your website all over the net? At least elaborate.
eadem 2y ago
The cloud Maker is the answer to all this. qbo.io
mrbluecoat 2y ago
> Kubernetes is immensely challenging as a development environment platform Glad someone said it out loud. So true. Apptainer has been a far better development experience for us.
remram 2y ago
Damn those are really good features they could have contributed to Kubernetes.
4WIW 2y ago
Regardless how you edit/compile your code, you still need to debug/troubleshoot problems in production, and that is very likely to use Kubernetes. So the more reasonable approach seems to be: first, figure out how do you troubleshoot/identify/mitigate a problem in production, then reproduce it in development environment and work to fix it at daytime. When you have instrumented your app for reasonable debugging experience then using these tools on development machine becomes much easier problem, K8s or not.
dangobanned 2y ago
[flagged]
mleonhard 2y ago
Why don't they describe their new system? I feel disappointed. :(

517 points by filiptronicek 2y ago

The article is an excellent cautionary tale. Debugging an app in a container is one thing. Debugging and app running inside a Kubernetes node is a rabbit hole that demands more hours and expertise.

deleted 2y ago

[deleted]

lolinder 2y ago

> This is not a story of whether or not to use Kubernetes for production workloads that’s a whole separate conversation. As is the topic of how to build a comprehensive soup-to-nuts developer experience for shipping applications on Kubernetes. > This is the story of how (not) to build development environments in the cloud. I'd like to request that the comment thread not turn into a bunch of generic k8s complaints. This is a legitimately interesting article about complicated engineering trade-offs faced by an organization with a very unique workload. Let's talk about that instead of talking about the title!

kitd 2y ago
Agreed. It's actually a very interesting use case and I can easily see that K8s wouldn't be the answer. My dev env is very definitely my "pet", thank you very much!
- ethbr1 2y ago
  It'd be nice to editorialize the title a bit with "... (for dev envs)" for clarity. Super useful negative example, and the lengths they pursued to make it fit! And no knock on the initial choice or impressive engineering, as many of the k8s problems they hit likely weren't understood gaps at the time they chose k8s. Which makes sense, given k8s roots in (a) not being a security isolation tool & (b) targeting up-front configurability over runtime flexibility. Neither of which mesh well with the co-hosted dev environment use case.
preommr 2y ago
Can someone clarify if they mean development environments, or if they're talking about a service that they sell that's related to development environments. Because I don't understand most of the article if it's the former. How are things like performance are a concern for internal development environments? And why are so many things stateful - ideally there should be some kind of configuration/secret management solution so that deployments are consistent. If it's the latter, then this is incredibly niche and maybe interesting, but unlikely to be applicable to anyone else.
- wutwutwat 2y ago
  4th paragraph in if you read the article… > This is not a story of whether or not to use Kubernetes for production workloads that’s a whole separate conversation. As is the topic of how to build a comprehensive soup-to-nuts developer experience for shipping applications on Kubernetes. > This is the story of how (not) to build development environments in the cloud.
  - epgui 2y ago
    I'm not sure that this really answers their question.
    - wutwutwat 2y ago
      > Can someone clarify if they mean development environments, or if they're talking about a service that they sell that's related to development environments. their question isn't asking anything. It's both about development environments AND a service they sell, which is dev envs. Even so > This is the story of how (not) to build development environments in the cloud. that is what the article is about. Their words. The person asked what the article is about, this it it, from the authors themselves. read the damn article
- bittermandel 2y ago
  It's for running their commercial products, which are stateful and long-lived developer environments.

clvx 2y ago

I tried doing a dev environment on Kubernetes but the fact you have to be dealing with a set of containers that could change if the base layer changed meant instability in certain cases which threw me off. I ended up with a mix of nix and it's vm build system which is based on qemu. The issue is too tied to NixOS and all services run in the same place which forces you to manage ports and other things. How I wish it could work is having a flake that defines certain services, these services could or could not run in different µVMs sharing an isolated linux network layer. Your flake could define your versions, your commands to interact and manage the lifecyle of those µVM's. As the nix store can be cached/shared, it can be provide fast and reproducible builds after the first build.

candiddevmike 2y ago
> the fact you have to be dealing with a set of containers that could change if the base layer changed meant instability Can you expand on this? Are you talking about containers you create?
eptcyka 2y ago
Have you tried https://github.com/astro/microvm.nix https://github.com/astro/microvm.nix ? You can use the same NixOS module for both declarative VMs and imperatively configured and spawned VMs.

bhouston 2y ago

I also recently left Kubernetes. It was a huge waste of time and money. I've replaced it with just a series of services on Google Cloud Run and then using Google's Cloud Run Tasks services for longer running tasks. The infrastructure now incredibly understandable and simple and cost effective. Kubernetes cost us >$million in both DevOps time and actually Google Cloud costs unnecessarily, and even worse it cost us time to market. Stay off of Kubernetes as long as you can in your company, unless you are basically forced onto it. You should view it as an unnecessary evil that comes with massive downsides in terms of complexity and cost.

elcomet 2y ago
Aren't you afraid of being now stuck with GCP?
- bhouston 2y ago
  It is just a bunch of docker containers. Some run in tasks and some run as auto-scaling services. Would probably take a week to switch to AWS as there are equivalent managed services there. But this is really a spurious concern. I myself used to care about it years ago. But in practice, rarely do people switch between cloud providers because the incremental benefits are minor, they are nearly equivalent, there is nothing much to be gained by moving from one to the other unless politics are involved (e.g. someone high up wants a specific provider.)
  - spwa4 2y ago
    How does the orchestration work? How do you share storage? How do the docker containers know how to find each other? How does security work? I feel like Kubernetes' downfall, for me, is the number of "enterprise" features it (got convinced into) supporting and enterprise features doing what they do best: turning the simplest of operations into a disaster.
    - bhouston 2y ago
      > How does the orchestration work? Github Actions CI. Take this and make a few more dependencies and a matrix strategy and you are good to go: https://github.com/bhouston/template-typescript-monorepo/blob/main/.github/workflows/ci.yml https://github.com/bhouston/template-typescript-monorepo/blo... For dev environments, you can add post-fixes to the services based on branches. > How do you share storage? I use managed DBs and Cloud Storage for shared storage. I think that provisioning your own SSDs/HDs to the cloud is indicative of an anti-pattern in your architecture. > How do the docker containers know how to find each other? I try to avoid too much communication between services directly, rather try to go through pub-sub or similar. But you can set up each service with a domain name and access them that way. With https://web3dsurvey.com https://web3dsurvey.com, I have an api on https://api.web3dsurvey.com https://api.web3dsurvey.com and then a review environment (connected to the main branch) with https://preview.web3dsurvey.com https://preview.web3dsurvey.com / https://api.preview.web3dsurvey.com https://api.preview.web3dsurvey.com. > How does security work? You can configure Cloud Run services to be internal only and not to accept outside connections. Otherwise one can just use JWT or whatever is normal on your routes in your web server.

ensignavenger 2y ago

The article does a great job of explaining the challenges they ran into with Kubernetes, and some of the things they tried... but I feel like it drops the ball at the end by not telling us at least a little what they chose instead. The article mentions they call their new solution "Gitpod Flex" but there is nothing about what Gitpod Flex is. They said they tried microVMs and decided against them, and of course Kubernetes, the focus of the article. So is GitpodFlex based on full VM's? Docker? Some other container runtime?? Perhaps a followup article will go into detail about their replacement.

loujaybee 2y ago
Yeah, that's fair. The blog was getting quite long, so we need to do some deeper dives in follow-ups. Gitpod Flex is runner-based. The runner interface is intentionally generic so that we can support different clouds, on-prem or just Linux in future. The first implemented runner is built around AWS primitives like EC2, EBS and ECS. But because of the more generic interface Gitpod now supports local / desktop environments on MacOS. And again, future OS support will come. There’s a bit more information in the docs, but we will do some follow ups! - https://www.gitpod.io/docs/flex/runners/aws/setup-aws-runners https://www.gitpod.io/docs/flex/runners/aws/setup-aws-runner... - https://www.gitpod.io/docs/flex/gitpod-desktop https://www.gitpod.io/docs/flex/gitpod-desktop (I work at Gitpod)
- nickstinemates 2y ago
  Echoing the parent you're replying to. You built up all of the context and missed they payoff.
  - ethbr1 2y ago
    I thought it was fair. >> We’ll be posting a lot more about Gitpod Flex architecture in the coming weeks or months. Cramming more detail into this post would have exceeded the average user read time ceiling.
- Bombthecat 2y ago

datadeft 2y ago

The original k8s paper mentioned that the only use case was a low latency and a high latency workflow combination and the resource allocation is based on that. The generic idea is that you can easily move low latency work between nodes and there are no serios repercussions when a high latency job fails. Based on this information, it is hard to justify to even consider k8s for the problem that gitpod has.

junkaccount 2y ago
Thanks for reading the paper!
- datadeft 2y ago
  For those who are interested: https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43438.pdf https://static.googleusercontent.com/media/research.google.c... I am not sure what differences k8s has compare to Borg. At the concept level these are pretty comparable.

xyst 2y ago

I do agree with the points in article that k8s is not a good fit for development environments. In my opinion, k8s is great for stable and consistent deployment/orchestration of applications. Dev environments by default are in a constant state of flux. I don’t understand the need for “cloud development environments” though. Isn’t the point of containerized apps is to avoid the need for synchronizing dev envs amongst teams? Or maybe this product is supposed to decrease onboarding friction?

sofixa 2y ago
It's to ensure a consistent environment for all developers, with the resources required. E.g. they mention GPUs, for developers working with GPU-intensive workloads. You can ship all developers gaming laptops with 64GB RAM and proper GPUs, and have them fight the environment to get the correct libraries as you have in prod (even with containers that's not trivial), or you can ship them Macbook Airs and similar, and have them run consistent (the same) dev environments remotely (you can self-host gitpod, it's not only a cloud service, it's more the API/environment to get consistent remote dev enviornments).
- loujaybee 2y ago
  Yeah, exactly. Containers locally are a basic foundation. But usually those containers or services need to talk to one another, they need some form of auth and credentials, they need some networking setup. There's a lot of configuration in all of that. The more devs swap projects or the more complex the thing you're working on the more the challenge grows. Automating depedencies, secret access, ensuring projects have the right memory, cpu, gpu etc. Also security - moving source code off your laptop and devices and standardizing your setups helps if you need to do a lot of audit and compliance as you can automate it.
dikei 2y ago
Sarcastically, CDE is one way to move cost from CAPEX (get your developer a Mac Book Pro) to OPEX (a monthly subscription that you only need to pay as long as the dev has not been lay off) It's also much cheaper to hire contractors and give them the CDE that can be terminated on a moment notice.

lmeyerov 2y ago

I was intrigued because the development environment problem is similar to the data scientist one - data gravity, GPU sharing, etc - but I'm confused on the solution? Oddly, I left with a funny alternate takeaway: One by one, their clever inhouse tweaks & scheduling preferences were recognized by the community and turned into standard k8s knobs So I'm back to the original question... What is fundamentally left? It sounds like one part is maintaining a clean container path to simplify a local deploy, which a lot of k8s teams do (ex: most of our enterprise customers prefer our docker compose & AMIs over k8s). But more importantly, something fundamental architecturally about how envs run that k8s cannot do, but they do not identify?

thenaturalist 2y ago
> We’ll be posting a lot more about Gitpod Flex architecture in the coming weeks or months. I’d love to invite you on November the 6th to a virtual event where I’ll be giving a demo of Gitpod Flex and I’ll deep-dive into the architecture and security model at length. Bottom of the post.
csweichel 2y ago
OP here. The Kubernetes community has been fantastic at evolving the platform, and we've greatly enjoyed being in the middle of it. Indeed, many of the things we had to build next to Kubernetes have now become part of k8s itself. Still, some of the core challenges remain: - the flexibility Kubernetes affords makes it hard to build and distribute a product with such specific requirements across the broad swath of differently set up Kubernetes installations. Managed Kubernetes services help, but come with their own restrictions (e.g. Kernel versions on GKE). - state handling and storage remains unsolved. PVCs are not reliable enough, subject to a lot of variance (see point above), and depending on the backing storage have vastly different behaviour. Local disks (which we use to this day), make workspace startup and backup expensive from a resource perspective and hard to predict timing wise. - user namespaces have come a long way in Kubernetes, but by themselves are not enough. /proc is still masked, FUSE is still not usable. - startup times, specifically container pulls and backup restoration, are hard to optimize because they depend on a lot of factors outside of our control (image homogeneity, cluster configuration) Fundamentally, Kubernetes simply isn't the right choice here. It's possible to make it work, but at some point the ROI of running on Kubernetes simply isn't there.

debarshri 2y ago

Phew, it is absolutely true. Building dev environments on k8s become wasteful. To add to this complexity, if you are building a product that is self hosted on customer's infrastructure. Debugging and support also become non homogeneous and difficult. What we have seen works especially when you are building developer centric product is expose these native issues around network, memory, compute and storage to engineers and they are more willing to work around it. Abstracting those issues leads to shift in responsibility on the product. Having said that, I still think k8s is an upgrade when you have a large team.

horsawlarway 2y ago

Personally - just let the developer own the machine they use for development. If you really need consistency for the environment - Let them own the machine, and then give them a stable base VM image, and pay for decent virtualization tooling that they run... on their own machine. I have seen several attempts to move dev environments to a remote host. They invariably suck. Yes - that means you need to pay for decent hardware for your devs, it's usually cheaper than remote resources (for a lot of reasons). Yes - that means you need to support running your stack locally. This is a good constraint (and a place where containers are your friend for consistency). Yes - that means you need data generation tooling to populate a local env. This can be automated relatively well, and it's something you need with a remote env anyways. --- The only real downside is data control (ie - the company has less control over how a developer manages assets like source code). I'm my experience, the vast majority of companies should worry less about this - your value as a company isn't your source code in 99.5% of cases, it's the team that executes that source code in production. If you're in the 0.5% of other cases... you know it and you should be in an air-gapped closed room anyways (and I've worked in those too...)

dangsux 2y ago
[dead]
shriek 2y ago
And the reason they suck is the feedback loop is just too high as compared to running it locally. You have to jump through hoops to debug/troubleshoot your code or any issues that you come across between your code and output of your code. And it's almost impossible to work on things when you have spotty internet. I haven't worked on extremely sensitive data but for PII data from prod to dev, scrubbing is a good practice to follow. This will vary based on the project/team you're on of course.
- ethbr1 2y ago
  Aka 'if a developer knew beforehand everything they needed, it wouldn't be development'
- 2y ago

pphysch 2y ago

The problem with "development environments", like other interactive workloads, is that there is a human at the other end that desires a good interactive experience with every keypress. It's a radically different problem space than what k8s was designed for. From a resource provider productive, the only way to squeeze a margin out of that space would be to reverse engineer 100% of human developer behavior so that you can ~perfectly predict "slack" in the system that could be reallocated to other users. Otherwise it's just a worse DX, like TFA gives examples of. Not a business I'm envious too be in... Just give everyone a dedicated VM or desktop, and make sure there's a batch system for big workloads.

rekoros 2y ago

We've been using Nix flakes and direnv (https://direnv.net/ https://direnv.net/) for developer environments and NixOS with https://github.com/serokell/deploy-rs https://github.com/serokell/deploy-rs for prod/deploys - takes serious digging and time to set up, but excellent experience with it so far.

andreweggleston 2y ago
I’ve been using Nix for the past year and it really feels like the holy grail for stable development environments. Like you said—it takes serious time to set up, but it seems like that’s an unavoidable reality of easily sharable dev envs.
aliasxneo 2y ago
Serious time to set up _and_ maintain as the project changes. At least, that was my experience. I really _want_ to have Nix-powered development environments, but I do _not_ want to spend the rest of my career maintaining them because developers refuse to "seriously dig" to understand how it works and why it decided to randomly break when they added a new dependency. I think this approach works best in small teams where everyone agrees to drink the Nix juice. Otherwise, it's caused nothing but strife in my company.
- rekoros 2y ago
  This may be the one area where some form of autocracy has merit :-)
  - imp0cat 2y ago
    What's that saying about developers and herding cats? :)

geoctl 2y ago

I've worked on something similar to gitpod in a slightly different context that's part of a much bigger personal project related to secure remote access that I've actually spent a few years building now and hope to open source in a few months from now. While I agree on many of the points in the article, I just don't understand how using micro VMs by itself replaces K8s unless they actually start building their own K8s that orchestrates their micro VMs (as opposed to containers in the case of k8s) ending up with the same thing basically when k8s itself can be used to orchestrate the outer containers that run the micro VMs used to run the dev containers. Yes, k8s has many challenges when it comes to nesting containers, cgroups, creating rootless containers inside the outer k8s containers and other stuff such as multi-region scaling, but actually the biggest challenge that I've faced so far isn't related to networkPolicies or cgroups but is actually by far related to storage, both when it comes to (lazily) pulling big OCI images which are extremely unready to be used for dev containers whose sizes are typically in the GBs or 10s of GBs as well as also when it comes to storage virtualization over the underlying k8s node storage. There are serious attempts to accelerate image pulling (e.g. Nydus) but such solutions would still probably be needed whether you use micro VMs or rootless/userns containers in order to load and run your dev containers.

cheptsov 2y ago

I can completely relate to anyone abandoning K8s. I'm working with dstack, an open-source alternative to K8s for AI infra [1]. We talk to many people who are frustrated with K8s, especially for GPU and AI workloads. [1] https://github.com/dstackai/dstack https://github.com/dstackai/dstack

Muhtasham 2y ago
I really like dstack, keep up the great work

rohitghumare 2y ago

You just simplified Kubernetes Management System

myestery 2y ago

Leaving this comment here so I'll always come back to read this as someone who was considering kubernetes for a platform like gitpod

teach 2y ago
Remember that you can favorite posts.

alecfong 2y ago

Our first implementation of brev.dev was built on top of kubernetes. We were also building a remote dev environment tool at the time. Treating dev environments like cattle seemed to be the wrong assumption. Turning kubernetes into a pet manager was a huge endeavor with long tail of issues. We rewrote our platform against vms and were immediately able to provide a better experience. Lots of tradeoffs but makes sense for dev envs.

junkaccount 2y ago

The real reason for this shift is that kubernetes moved to containerd which they cannot handle. Docker was much easier. Differential workloads is not correct to blame. Also, there is a long tail of issues to be fixed if you do it with Kubernetes. Kubernetes does not just give you scaling, it gives you many things: run on any architecture, be close to your deployment etc.

moondev 2y ago
https://github.com/Mirantis/cri-dockerd https://github.com/Mirantis/cri-dockerd
- junkaccount 2y ago
  Most of the kubernetes providers (GKE, EKS) do not support this new shim. Even on baremetal it is possibly hard to run.

trimen 2y ago

[flagged]

tacone 2y ago

On a side note: has anybody experience with MicroK8s? I'd love to learn stories about it. I'm interested in both dev and production experiences.

redrove 2y ago
Microk8s is nothing but a kubernetes distro from canonical. Personally I would use k3s because it’s a little more widespread and less opinionated in a good way. Anyway, as always it depends on what you want to use it for.

concerndc1tizen 2y ago

Sounds more to me like they need a new CTO. And that they're desperate to tell customers that they've fixed their problems. Kubernetes is absolutely the wrong tool for this use case, and I argue that this should be obvious to someone in a CTO-level position, or their immediate advisors. Kubernetes excels as a microservices platform, running reasonably trustworthy workloads. The key features of Kubernetes are rollout (highly available upgrades), elasticity (horizontal scaleout), bin packing (resource limits), CSI (dynamically mounted block storage), and so on. All this relates to a highly dynamic environment. This is not at all what Gitpod needs. They need high performance disks, ballooning memory, live migrations, and isolated workloads. Kubernetes does not provide you sufficient security boundaries for untrusted workloads. You need virtualization for that, and ideally physically separate machines. Another major mistake they made was trying to build this on public cloud infrastructure. Of course the performance will be ridiculous. However, one major reason for using Kubernetes is sharing the GPU. That is, to my knowledge, not possible with virtualization. But again, do you want to risk sharing your data, on a shared GPU?

dilyevsky 2y ago
I agree on the cloud thing. Don't agree that "high performance disks, ballooning memory, live migrations, and isolated workloads" preclude from using k8s - you can still run it as base layer. You get some central configuration storage, machine management and some other niceties for free and you can push your VM-specific features into your application pod. In fact, that's how Google Cloud is designed (except they use Borg not k8s but same idea).
- concerndc1tizen 2y ago
  True! I love the idea of using K8s to orchestrate the running of VMs. With graceful shutdown and distributed storage, it makes it even more trivial to semi-live migrate VMs. Are you aware of the limits? It must run as root and privileged?
  - dilyevsky 2y ago
    In this scenario k8s is orchestrating the hypervisor, not VMs themselves. Hypervisor then orchestrates VMs + network (eg OVS) + other supporting functions (logs shipping, etc) on each individual “worker” node. VM scheduling/migration component needs to be completely decoupled from k8s apiserver (but itself can still run as normal k8s deployment) bc scaling kube api with unbound users is challenging. And yes, hypervisor will need to run privileged but you can limit it to worker nodes only

hintymad 2y ago

I was wondering if there's productivity angle too. Take Ceph vs Rook for example. If a Ceph cluster needs all the resources on its machines and the cluster manages its resources too, then moving to Rook does not give any additional features. All the 50K additional lines of code in Rook is to set up CSIs and statefulsets and whatnot just to get Ceph working on Kubernetes.

I_LIKE_TO_HATE 2y ago

[flagged]

I_LIKE_TO_HATE 2y ago

[flagged]

vbezhenar 2y ago

I read this article and I still don't understand what's wrong with Kubernetes for this task. Everything you would do with virtual machines could be done with Kubernetes with very similar results. I guess team just wants to rewrite everything, it happens. Manager should prevent that.

dwroberts 2y ago

> Autoscaler plugins: In June 2022, we switched to using cluster-autoscaler plugins when they were introduced. Does anyone have any links for cluster-autoscaler plugins? Searching drawing a blank, even in the cluster-autoscaler repo itself. Did this concept get ditched/removed?

rahen 2y ago

Kubernetes works great for stateless workloads. For anything stateful, monolithic, or that doesn't require autoscaling, I find LXC more appropriate: - it can be clusterized (LXD/Incus), like K8S but unlike Compose - it exposes some tooling to the data plane, especially a load balancer, like K8S - it offers system instances with a complete distribution and a init system, like a VM but unlike a Docker container - it can orchestrate both VMs (including Windows VMs) and LXC containers at the same time in the same cluster - LXC containers have the same performance as Docker containers unlike a VM - it uses a declarative syntax - it can be used as a foundation layer for anything stateful or stateless, including the Kubernetes cluster LXD/Incus sits somewhere between Docker Swarm and a vCenter cluster, which makes it one of the most versatile platform. Nomad is also a nice contender, it cannot orchestrate LXC containers but can autoscale a variety of workloads, including Java apps and qemu VMs.

belthesar 2y ago
I too am rallying quickly to the Incus way of doing things. Also of note, there's an effort to build a utility to write Compose manifests for Incus workloads that I'm following very closely. https://github.com/bketelsen/incus-compose https://github.com/bketelsen/incus-compose
- eddyg 2y ago
  Thanks for pointing out `incus-compose`!
OneCricketeer 2y ago
Nomad cannot orchestrate what, exactly? https://developer.hashicorp.com/nomad/tutorials/plugins/plugin-lxc https://developer.hashicorp.com/nomad/tutorials/plugins/plug...

deepsun 2y ago

> SSD RAID 0 > A simpler version of this setup is to use a single SSD attached to the node. This approach provides lower IOPS and bandwidth, and still binds the data to individual nodes. Are you sure SSD is that slow? NVMe devices are so fast that I hardly believe there's any need for RAID 0.

mikeshi42 2y ago
In AWS iirc NVMe max out at 2GB/s - I'm not sure why that's the case. I know there were issues with the PCIe controller in the past being the bottleneck, but I suspect there's something more to it than that.

abofh 2y ago

I feel like anyone who was building a CI solution to sell to others and chose kubernetes didn't really understand the problem. You're running hot pods for crypto miners and against people who really want to see the rest of the code that box has ever seen. You should be isolating with something purpose built like firecracker, and do your own dispatch & shred for security.

geoctl 2y ago
Firecracker is more comparable to container runtimes than to orchestrators such as K8s. You still need an orchestrator to schedule, manage and garbage-collect all your uVMs on top of your infrastructure exactly like you would do with containers via k8s. In other words, you will probably have to either use k8s or build your own k8s to run "supervisor" containers/processes that launch uVMs which in turn launch the customer dev containers.
- abofh 2y ago
  For sure, but that's the point - containers aren't really good for an adversarial CI solution. You can run that shit in house on kubernetes on a VM in a simulated VR if you want. But if you have adversarial builds, you have a) builds that may well need close to root, and b) customers who may well want to break your shit. Containers are not the right solution for that, VM's get you mostly there, and the right answer is burning bare metal instances with fire after every change-of-tenant - but nobody does that (anymore), because VM's are close enough and it's faster to zero out a virtual disk than a real one. So if you started with kubernetes and fought the whole process of why it's not a great solution to the problem, I have to assume you didn't understand the problem. I :heart: kubernetes, its complexity pays my bills - but it's barely a good CI solution when you trust everyone involved, it's definitely not a good one where you're trying to be general-purpose to everyone with a makefile.
  - geoctl 2y ago
    I would argue that dev containers are more complicated than CI even though they share many of the challenges (e.g. devcontainers might need to load 10s or 100s of GBs to start and are write heavy). I would also argue that userns/rootless containers provide "enough" isolation when it comes to isolating CPU/memory/networking as well as access to the host's syscalls if you're careful enough; however when it comes to storage (e.g. max disk size that a container can use and write to, max opened files, completely hiding the host's fs from the container's, etc...), it's unfortunately still extremely limited,fs-dependent for some features, even though modern solutions (e.g. vDPA and ublk) can be used to fix that and virtualize the storage for containers.

eYrKEC2 2y ago

Have folks seen success with https://earthly.dev/ https://earthly.dev/ as a tool in their dev cycle?

riiii 2y ago

> development environments Kubernetes has never ever struck me as a good idea for a development environment. I'm surprised it took the author this long to figure out. K8s can be a lifesaver for production, staging, testing, ... depending on your requirements and infrastructure.

sethammons 2y ago
Our operations team is planning to build dev envs in k8s, but only the networked dependencies. Like a personal testing/staging where you have full control of the thing(s) you are developing and can simply leverage the rest of the stack. Sounds sane. Am i missing anything?

Andkeith 2y ago

[dead]

cryptica 2y ago

Kubernetes is awesome but I understand what the article is getting at. K8s was designed for a mostly homogeneous architecture when your platform requirements end with "deploy this service to my cluster" and you don't really care about the specifics of how it's scheduled. A heterogeneous architecture with multi-tenancy poses some unique challenges because, as mentioned in the article, you get highly inconsistent usage patterns across different services. Also, arbitrary code execution (with sandboxing) can present a signifiant challenge. For security, you ideally need full isolation between services which belong to different users; this isolation wasn't a primary design goal of Kubernetes. That said, you can probably still use K8s, but in a different way. For smaller customers, you could co-locate on the same cluster, but for larger customers which have high scalability requirements, you could have a separate K8s cluster for each one. Surely for such customers, it's worth the extra effort. So in conclusion, I don't think the problems which were identified necessarily warrant abandoning K8s entirely, but maybe just a rethinking of how K8s is used. K8s still provides a lot of value in treating a whole cluster of computers as a single machine, especially if all your architecture is already set up for it. In addition to scheduling/orchestration, K8s offers a lot of very nice-to-have features like performance monitoring, dashboards, aggregated logs, ingress, health checks, ...

ncrmro 2y ago

We started having a few developers have constant VSCode timeouts. We switched to GitHub devcontainers which have been great.

eichi 2y ago

Kubernetes is just combined infra admin practices. Whether we use it or not, we need to do the same things by local oriented way or vendor specific way . 1. Some operations on remote in local oriented way are time consuming and unmanageable. 2. With vendor specific way, our skill would be deprecated, having dependency to the vendors. 3. Kubernetes is not the best tools but it it popular. As always, custom solution is the most powerful but should be replaced with more unified way for the stability of the development.

Jack008 2y ago

Make sure you need microservices-based architecture because it comes with its own complexity - a load balancer, container networking, distributed tracing, etc. If you application does not need to scale its sub-components independently, you are better off using a VM-based application. It's 10X cheaper to maintain/troubleshoot and is high performance/resources.

deleted 2y ago

[deleted]

vrnvu 2y ago

For development, I made the switch to nix/flox and it’s been a game-changer.

mstrangfeld 2y ago
How well does Flox work out of the box? I would really like to introduce Nix to the dev environments in my company but the struggle of maintaining nix files and flakes is too large. I've looked at DevBox and it looks quite accessible but Flox also looks like a nice way to sneak some of the Nix goodness into the company.
- vrnvu 2y ago
  It works pretty well for most common packages, if a rare package/dependency fails is mostly nix problem to solve upstream.

bluelightning2k 2y ago

The debate in the comments about whether you should run locally is fascinating. To the people saying ultra modern hardware could handle it: worth remembering the companies on question started on this path X years ago with Y set of technologies and Z set of experiences. Because it made sense for Google in 2012 or whatever doesn't necessarily mean they would choose it again --or not-- given a do over (but there's basically no way back).

gloosx 2y ago

>Kubernetes seems like the obvious choice for building out remote, standardized and automated development environments - Is it really Obvious Choice™ though Fred? - Hmm, let's consult the graphs. >Kubernetes is a container orchestration system for automating software deployment. - It's about automating deployment Carl, not development environments! >Kubernetes is not the right choice for building development environments, as we’ve found.

linuxftw 2y ago

The article offers toward the end that now self-hosted customers can run their app on something other than k8s. I think this is a mistake. We're a k8s enterprise shop, and I don't want to support any more VMs. If it's not on k8s, I'm not running it. I don't want to be responsible for golden images, patching, and all the fun that comes with managing workloads outside of k8s. That's why I have k8s. All the problems in the article also seem self-imposed. k8s can run stateful workloads just fine. Don't start and stop them. Figure out the math on how much it costs to run a container 24/7, add your margin, and pass that cost to the customer. Customer can decide to stop the containers to save $$, so the latency won't hurt, they'll accept it because they know they're saving money.

eadem 2y ago

The cloud maker is the answer of all this. qbo.io

remram 2y ago
Hi Alex Diaz from qbo, can you stop spamming links to your website all over the net? At least elaborate.

eadem 2y ago

The cloud Maker is the answer to all this. qbo.io

mrbluecoat 2y ago

> Kubernetes is immensely challenging as a development environment platform Glad someone said it out loud. So true. Apptainer has been a far better development experience for us.

remram 2y ago

Damn those are really good features they could have contributed to Kubernetes.

4WIW 2y ago

Regardless how you edit/compile your code, you still need to debug/troubleshoot problems in production, and that is very likely to use Kubernetes. So the more reasonable approach seems to be: first, figure out how do you troubleshoot/identify/mitigate a problem in production, then reproduce it in development environment and work to fix it at daytime. When you have instrumented your app for reasonable debugging experience then using these tools on development machine becomes much easier problem, K8s or not.

dangobanned 2y ago

[flagged]

mleonhard 2y ago

Why don't they describe their new system? I feel disappointed. :(