7 ms·

Dear friend, you have built a Kubernetes (2024)

143 points by Wingy 5mo ago

zdw 5mo ago
IMO, Kubernetes isn't inevitable, and this seems to paint it as such. K8s is well suited to dynamically scaling a SaaS product delivered over the web. When you get outside this scenario - for example, on-prem or single node "clusters" that are running K8s just for API compatibility, it seems like either overkill or a bad choice. Even when cloud deployed, K8s mostly functions as a batteries-not-included wrapper around the underlying cloud provider services and APIs. There are also folks who understand the innards of K8s very well that have legitimate criticisms of it - for example, this one from the MetalLB developer: https://blog.dave.tf/post/new-kubernetes/ https://blog.dave.tf/post/new-kubernetes/ Before you deploy something, actually understand what the pros/cons are, and what problem it was made to solve, and if your problem isn't at least mostly a match, keep looking.
- zbentley 5mo ago
  > K8s is well suited to dynamically scaling a SaaS product delivered over the web It’s well suited to other things as well, people are just in denial about some of them. “I need to run more than two containers and have a googleable way to manage their behavior” is a very common need.
  - capitalhilbilly 5mo ago
    This is a need it fails at miserably. k8s reminds me of the raid recentralization anti pattern problem where you fix a hardware failure that never occurs in exchange for knowing simple higher level mistakes or security problems will tank something now too large to fail again.
- antonvs 5mo ago
  Kubernetes, in the form of k3s, was a critical success factor for us with the onprem deployment of our SaaS product. What's the problem with a single-node cluster? We use that for e.g. dev environments, as well as some small onprem deployments. > Even when cloud deployed, K8s mostly functions as a batteries-not-included wrapper around the underlying cloud provider services and APIs. Which batteries are not included? The "wrapper around the underlying cloud provider services and APIs" is enormously important. Why would you prefer to use a less well-designed, more vendor-specific set of APIs? I seriously don't get these criticisms of k8s. K8s abstracts away, and standardizes, an enormous amount of system complexity. The people who object to it just don't have the requirements where it starts making sense, that's all.
  - subhobroto 5mo ago
    > Kubernetes, in the form of k3s, was a critical success factor for us with the onprem deployment of our SaaS product. What surprises and gotchas did you have to deal with using k3s as a Kubernetes implementation? Did you use an LB? Which one? I'm assuming all your onprem nodes were just linux servers with very basic equipment (the fanciest networking equipment you used were 10GbE PCIe cards, nothing more special than that?)
    - antonvs 5mo ago
      We sell to enterprise customers. All of them deploy our solution on internal cloud-style VM clusters. We use the Traefik ingress controller by default. There really weren't any particular surprises or gotchas at that level. In this context, I've never had to deal with anything at the level of the type of Ethernet card. That's kind of the point: platforms like k8s abstract away from that.
- physicsguy 5mo ago
  It's also difficult for data pipelines or data intensive things. At several companies we've run into the "Need to put ML model behind API and pods get killed because health checks via API are basically not compatible with container fully under load but still working"
isodev 5mo ago
Unless you’re in Erlang world (Elixir, Gleam..) and all that is already baked into OTP and the BEAM. You can go on holiday knowing it will be a while longer before you need to break out the pods (and at that scale, you will be able to afford a colleague or two to help you).
- kirici 5mo ago
  How does Elixir renew my certificates, mount networked storage mounts and create a VIP and internal DNS entries for my valkey instance? Scaling is a sidenote, that it becomes easy is a result of hoisting everything else onto one control plane and a set of coherent APIs.
et1337 5mo ago
The saddest part about Kubernetes is… after you set it all up, you still need a hacky deploy.sh to sed in the image tag to deploy! And pretty soon you’re back to “my dear friend you have built a Helm”. And so the configuration clock continues ticking…
- cassianoleal 5mo ago
  I have been using Kubernetes for 7 or 8 years now, and have nearly 100% stayed away from Helm. Some Kustomize, a little bit of envsubst and we're good to go thank you very much.
  - supriyo-biswas 5mo ago
    How do you handle cleanups and hooks? The best way to do helm, at least for me, seems to be about limiting its use to simple templating use cases; if you end up needing an if, you've probably done something terribly wrong.
    - srcreigh 5mo ago
      Seems to be a case of the XY problem. What do you need cleanups and hooks for?
      - supriyo-biswas 5mo ago
        Cleanups: I want to do a `helm uninstall` and have all the manifests go away at once instead of looking around for N different resources. Hooks: I want to apply my database migrations and populate the database with static datasets before I deploy my application, without having my CI connect to the database cluster (at places I've worked, the CI cluster and K8s cluster were completely separate).
        cassianoleal 5mo ago
        > Cleanups: I want to do a `helm uninstall` and have all the manifests go away at once instead of looking around for N different resources. kubectl delete -f <manifests.yaml> kubectl delete -k <kustomization_directory> > I want to apply my database migrations and populate the database with static datasets before I deploy my application, without having my CI connect to the database cluster A Job feels like a good fit for this. CI deployes the Job without connecting to DB, Job runs migrations using the same connectivity as the application.
jstanley 5mo ago
> I know you wanted to "choose boring tech" to just run some containers. The people advocating for boring tech generally aren't interested in containers. You can just run programs.
- stavros 5mo ago
  Containers are just statically-linked programs for the rest of us.
  - subhobroto 5mo ago
    I am a big fan which is why I am saying this: you're dismissing the kernel and ABI surface is a huge assumption that must hold true for your comment to hold stavros. If you had said "unikernels" I would have had no arguments to make.
    - stavros 5mo ago
      What do you mean? Statically linked programs depend on the kernel too.
      - subhobroto 5mo ago
        right - that's precisely what I meant. I read your comment "Containers are just statically-linked programs for the rest of us." as "containers can be replaced by statically-linked programs". If you didn't imply that, I apologize. If you did mean that, I disagree with you precisely because your point works if you only care about dependency management - it falls apart on system state. A static binary is a process on the host and shares the same process space, network stack, and filesystem. OTOH, a container is a jail (the primary usecase): I can't cgroup a static binary's memory usage or give it a virtual network interface without reimplementing "container lite". Containers aren't just 'statically linked programs' - they allow me to use the kernel as a hypervisor for isolated environments. What they are though, a messy but practical compromise to Unikernels - which was my last point in our GP.
oddurmagnusson 5mo ago
Found this the same day I published this: https://github.com/oddur/yoink https://github.com/oddur/yoink Kubernetes was overkill (I do that all day, 5 days a week); Kamal was too restrictive, so I found myself rolling out Yoink. Just what I need from k8s, but simple enough I can point it to a baremetal machine on Hertzner that can easily run all my workloads.
- subhobroto 5mo ago
  > found myself rolling out Yoink - using Tailscale SSH is brilliant - using caddy-docker-proxy for ingress is brilliant What do you use for: - service discovery - secret store (EDIT: Crap you use Infisical. No shade, I just have this horrible foreboding it will end up like Hashicorp. I use Conjur Secretless Broker but am tracking: https://news.ycombinator.com/item?id=47903690 https://news.ycombinator.com/item?id=47903690) - backing up and restoring state like in a DB PS: Have you been having issues with Hetzner the last few weeks?
  - oddurmagnusson 5mo ago
    Service discovery is basically just Docker's internal DNS. Caddy-docker-proxy can use it to find healthy upstreams. For secrets, I self-host Infisical on the box -- easy to plug in whatever secret manager, should make it pair nicely with https://github.com/tellerops/teller https://github.com/tellerops/teller or something similar Had no problems with Hertzner so far, just enjoying the raw CPU power of bare metal. The plan is to roll out more boxes across different providers, using Tailscale for the backplane network and Cloudflare to load-balance between them. All in due time What issues have you been having ?
    - subhobroto 5mo ago
      I have a suspicion you're using Headscale? If so, I urge you to consider Ionscale. I use it with Authentik as the IdP. Personally commiting to using Tailscale as a core foundation of my infrastructure and Ionscale is my hedge against getting Hashicorped. > Service discovery is basically just Docker's internal DNS. Caddy-docker-proxy can use it to find healthy upstreams Do you have a writeup of this somewhere? I'm unaware of being able to manage Docker's internal DNS over some kind of an API (would appreciate if you know a way to). The only way I know is to manipulate network aliases via Docker Engine API. As a result I use Hickory DNS with RFC 2136. That coupled with Caddy-docker-proxy gets me extremely close.
deleted 5mo ago
[deleted]
heyitsdaad 5mo ago
Kubernetes networking is an inefficient mess.
- wmf 5mo ago
  Some CNIs are definitely better than others. Unfortunately it seems 99% of people want to work against the k8s networking model.
  - zbentley 5mo ago
    Shit just gets really weird when your network isn’t split for k8s in an equivalent way to what GCP/AWS expect. Like, if you have other services running on the nodes that you want things inside k8s to talk to, or if the nodes are in a flat subnet with other stuff in it, things get annoying. Those are worst practices for a reason, but pretty common in environments with home rolled k8s clusters.
whynotmaybe 5mo ago
I need clarifications. I see docker as a way to avoid having a standard dev platform for everyone in the company so that the infra team don't have to worry about patch xyz for library abc, only run docker. But, with all the effort put in place to coordinate docker, k8s and all the shebang, isn't it finally easier to force a platform and let it slowly evolve over time? Is docker another technical tool that tries to solve a non-technical problem?
- whycombinetor 5mo ago
  Porque no los dos. Force a platform. Deploy non containerized. Build a dockerized version of the forced platform for cross-platform local dev.
- esafak 5mo ago
  I do not follow you. Every app has different needs. Containers encode them in a shareable way. You can evolve the image over time. So what more do you want?
- 0xbadcafebee 5mo ago
  Docker is a solution to one specific problem: the need for "a user" to run 10 different potentially conflicting apps, all at the same time, on one machine, and abstract away anything which might make those apps conflict if they ran on a single OS. It provides a dozen different solutions in one package. K8s is a way to take that and make it scale up for a large number of applications on a large number of hosts in a production business in a way that's automated and resilient to failure.
supriyo-biswas 5mo ago
Criticisms of Kubernetes generally come from a few places: - People who would prefer their way of doing this, whether that's deployments on VMs, or use some sort of simpler cloud provider. I had the same opinion a few years ago, but have kind of come to like it, because I can cleanly deploy multiple applications on a cluster in a declarative fashion. I still don't buy the "everything on K8s", and my personal setup is to have a set of VMs bought from a infrastructure provider, setup a primary/replica database on two of them, and use the rest as Kubernetes nodes. - People who run Kubernetes at larger scales and have had issues with them. This usually needs some custom scaling work; the best way to work around this if you're managing your own infra[1] is to split the cluster into many small independent clusters, akin to "cellular deployments"[2]/"bulkhead pattern"[3]. Alternatively, if you are at the point where you have a 500+ node cluster, it may not be a bad idea to start using a hyperscaler's service as they have typically done some of the scaling work for you, typically in form of replacing etcd and the RPC layer through something more stable. - People who need a deep level of orchestration Examples of such use cases may be to run a CI system or a container service like fly.io; for such use cases, I agree that K8s is often overkill, as you need to keep the two datastores in sync and generate huge loads on the kube-apiserver and the cluster datastore in the process, and it might be often better to just bring up Firecracker MicroVMs or similar yourself. Although, I should say that teams writing their first orchestration process almost always run to Kubernetes without realizing this pitfall, though I have learned to keep my mouth shut as I started a small religious war recently at my current workplace by raising this exact point. [1] Notice how I don't say "on-prem", because the hyperscaler marketing teams would rather have you believe in two extremes of either using their service or running around in a datacenter with racks, whereas you can often get bog-standard VMs from Hetzner or Vultr or DigitalOcean and build around that. [2] https://docs.aws.amazon.com/wellarchitected/latest/reducing-scope-of-impact-with-cell-based-architecture/cell-deployment.html https://docs.aws.amazon.com/wellarchitected/latest/reducing-... [3] https://learn.microsoft.com/en-us/azure/architecture/patterns/bulkhead https://learn.microsoft.com/en-us/azure/architecture/pattern...
- rfoo 5mo ago
  Another case: People who want to run workloads that are inherently incompatible with Kubernetes networking model. For example: * For some cursed reasons you want to make sure every single one instance of a large batch job see just one NIC in its container and they are all the same IP and you NAT to the outside world. Ingress? What ingress? This is a batch job! * Like the previous point, except that your "batch job" somehow has multiple containers in one instance now, and they should be able to reach each other by domain.
cortesoft 5mo ago
This is obviously slightly exaggerated, but I do feel like this whenever people dismiss Kubernetes as either too complicated or not needed. The response I always got when suggesting Kubernetes is "you can do all those things without Kubernetes" Sure, of course. There are a million different ways to do everything Kubernetes does, and some of them might be simpler or fit your use case more perfectly. You can make different decisions for each choice Kubernetes makes, and maybe your decisions are more perfect for your workload. However, the big win with Kubernetes is that all of those choices have been made and agreed upon, and now you have an entire ecosystem of tools, expertise, blog posts, AI knowledge, etc, that knows the choices Kubernetes made and can interface with that. This is VERY powerful.
- PunchyHamster 5mo ago
  Honestly the main problem is people using k8s for something that's like... a database, and an app, and maybe a second app, that all could be containers or just a systemd service. And then they hit all the things that make sense in big company with like 40 services but very little in their context and complain that complex thing designed for complex interactions isn't simple
  - jmalicki 5mo ago
    Luckily since I met this guy named Claude most of that complexity has gone away.
    - andai 5mo ago
      A while back when the agents got hyped I was looking into the whole "give it a VM / docker container" I realized the safest and simplest option was just to give it its own machine. Then I realized giving it root on a $3 VPS is functionally equivalent. If it blows it up, you just reset the VM. It sounds bad but I can't see an actual difference.
  - nazcan 5mo ago
    But if you want some redundancy, k8s let's you just say run 4 of this, 6 of this on these 3 machines. At least I find it quite straight forward. The database is more complex since there is storage affinity (I use cockroachDB with local persistent volumes for it) - but stateful is always complicated.
winton 5mo ago
"Except if you quit or go on vacation, who will maintain this custom pile of shell scripts?" LLMs can reason about and fix them quite well.
- zbentley 5mo ago
  Not as well as they can reason (or others can google) something as standardized as kubernetes. There’s just less context (in both senses of the term) needed to understand something running on a common substrate versus something bespoke, even if the bespoke thing is itself comprised of standardized parts.
  - winton 5mo ago
    For a project set up by a qualified engineer, there would be little difference to the end user in practice. The LLM would work out a solution with a negligible difference in speed. Maybe debugging would also be faster for the LLM without the abstraction layers and low level access?
deleted 5mo ago
[deleted]
Dedime 5mo ago
PREACH! I run K8s at home. I used to do docker-compose - and I'd still recommend that to most people - but even for my 1 little NUC with 4vcpu / 16Gi Homelab, I still love deploying with K8s. It's genuinely simpler for me. If anyone's looking for inspiration, my setup: * ArgoCD pointed to my GitLab repos * GitLab repos contain Helm charts * Most of the Helm charts contain open-source charts as subcharts, with versions set like (e.g.) `version: ~0` - meaning I automatically receive updates for all major version until `1` * Updating my apps usually consists of logging into the UI, reviewing the infrastructure and image tag updates, and manually clicking sync. I do this once every few months My next little side project: Autoscaling into the cloud (via a secure WireGuard tunnel) when I want to expand past my current hardware limitations
- wernerb 5mo ago
  A reason not to run k8s is if you want your server to reach C10 idle states. The k8s control plain with its polling and checking are quite heavy on the mostly idle server. I have reverted to just use Nixos and oci podman containers. Everything is declarative and reproducible
  - r_lee 5mo ago
    another one is swap. UnlimitedSwap was deprecated and you can now only use LimitedSwap which restricts how much swap you can use, so you can't take full advantage of zram, which sucks for those looking to run lean
perfunctory 5mo ago
I somehow feel that it's actually the opposite. It should be "Dear kubernetes user, you have just built a shell".
blindlobstar 5mo ago
Why both posts mention docker compose and not mentioning docker swarm. Being using it for my projects for long time. And it's so nice. Similar syntax, easy networking, rollout strategy, easy to add nodes to cluster. You can have one template docker-compose.yaml file and separate deployment files for different envs, like: docker-compose.dev.yaml, docker-compose.prod.yaml I think swarm is really underrated
- PufPufPuf 5mo ago
  I've been there. We still ended up with messy deploy scripts written in Ruby and the only debugging solution was "just comment out everything then run line by line".
  - blindlobstar 5mo ago
    'docker stack deploy' covers most of the cases. But yeah, there is still some problems like: "update a config or a secret", that require manually invoking additional commands (or via scripts)
- vbezhenar 5mo ago
  How do you solve persistence with swarm? Can I deploy postgres with network storage that will mount automatically on node where container is launched?
  - blindlobstar 5mo ago
    > Can I deploy postgres with network storage that will mount automatically on node where container is launched? Yes (if I'm getting your question right). here is an example of nfs volume: https://docs.docker.com/reference/compose-file/volumes/#driver_opts https://docs.docker.com/reference/compose-file/volumes/#driv...
- Havoc 5mo ago
  It’s in a death spiral - not widely used so people aren’t incentivised to put time into it so not widely used
hasyimibhar 5mo ago
I've experienced something like this at work but with data warehouse instead, and it happened multiple times (to be fair, data engineering is still fairly new where I'm from). One example was an engineer wanted to build an API that accepts large CSV (GBs of credit reports) to extract some data and perform some aggregations. He was in the process of discussing with SREs on the best way to process the huge CSV file without using k8s stateful set, and the solution he was about to build was basically writing to S3 and having a worker asynchronously load and process the CSV in chunks, then finally writing the aggregation to db. I stepped in and told him he was about to build a data warehouse. :P
- jeffrallen 5mo ago
  If it was less than 100 gb, he probably should have just loaded the whole thing in RAM on a single machine, and processed it all in a single shot. No S3, no network round trips, no chunking, no data warehouse.
jeffrallen 5mo ago
Some days, it would be better to build a working not-kubernetes than debugging the not-working kubernetes.
whycombinetor 5mo ago
After reading this and remembering an old hobby project, I decided to switch the deploy from a systemd service to PM2, which apparently has rolling deployments without needing Docker engine (for those of us minmaxing instance RAM).
stego-tech 5mo ago
As someone rolling their self-hosted stuff via Compose and shell scripts instead of K8s specifically for the simplicity of the experience, this is 100% why you need to understand what Kubernetes solves before writing it off entirely. I'm not doing overlay networks, I'm using a single bare-metal host, and I value the hands-on Linux administration experience versus the K8s cluster admin experience. All of these are reasons I specifically chose not to use Kubernetes. The second I want HA, or want to shift from local VLANs to multi-cloud overlays, or I don't need the local Linux sysadmin experience anymore? Yeah, it's K8s at the top of the list. Until then, my solution works for exactly what I need.
jFriedensreich 5mo ago
im just about giving OPs premise another go. compose just feels so much better as abstraction especially with small and medium setups looking close to the optimum of expressiveness without boilerplate to describe what is needed. The missing pieces seem to also be in the compose compatible “docker stack” aka new docker swarm, which i ignored for probably too long as i assumed it was the discontinued old swarm. Even if new swarm mode sucks how hard can it be to make something compose shaped vs running k8?
shrubble 5mo ago
I can tell you how vendors deliver a software solution that runs on Kubernetes: very poorly. The needed tweaks, the ability to customize things, basically goes to zero because the support staff is technical about the software, but NOT about Kubernetes. I am not joking: a recent deployment required 3x VMs for Kubernetes, each VM having 256 gigabytes of RAM; then a separate 3x VMs for a different piece. 1.5TB of RAM to manage less than 1200 network devices (routers etc. that run BGP). No one knew, for instance, how to lower the MongoDB (because of course you need it!) resource usage, despite the fact that the clustered VMware install is using a very fast SSD storage solution and thus MongoDB is unlikely accelerate anything; so over 128GB RAM is being burned on caching the results coming back from SSDs that are running at many-GB/s throughput.
- throwaway041207 5mo ago
  Whether this is deployed via Helm charts or a native controller, there's almost certainly some overlay where you can override resource values, unless this is just a very crappy vendor.
  - shrubble 5mo ago
    The setting is there, somewhere, I agree; but no one can tell you how to change it or what the implications might be. So unless I want to dig into their YAML files that are not documented…
justsomehnguy 5mo ago
> Ah, but wait! Inevitably, you find a reason to expand to a second server >> The real problem is that programmers have spent far too much time worrying about efficiency in the wrong places and at the wrong times; premature optimization is the root of all evil (or at least most of it) in programming. -- Donald Knuth, Computer Programming as an Art (1974) EDIT: > Except if you quit or go on vacation, who will maintain this custom pile of shell scripts? Honestly? I don't care. There is a reason why I quit and 99% of time it's the pay. And if the company doesn't pay me enough to bother then why should I? Why should I bother about some company future in the first place?
kube-system 5mo ago
Kubernetes is a powerful tool for complicated problems. If if it seems complicated, you probably don’t have a complicated deployment problem. But really this applies to any powerful tool. If you need to measure a voltage, an 4 channel oscilloscope also probably seems too complicated.
dang 5mo ago
Discussed at the time: Dear friend, you have built a Kubernetes - https://news.ycombinator.com/item?id=42226005 https://news.ycombinator.com/item?id=42226005 - Nov 2024 (277 comments)
drdaeman 5mo ago
They have built an orchestrator, not Kubernetes. There is one key difference: they know this thing, end-to-end, down to every single bolt and piece of duct tape (with possible exception for Docker internals) And that's a very important distinction when it comes to maintaining complex systems. This could've changed with LLMs (I'm still adjusting to what new capabilities mean for various decision-making logic), but before machine intelligence debugging an issue with Kubernetes could've been a whole world of pain.
- chuckadams 5mo ago
  And chances are only they know it. If my role has enough cluster access, I can muddle through pretty much any helm chart (with lots of cursing, yes) but it might take me days to set up whatever elaborate bespoke environment and script invocations are needed to replicate the current production setup maybe.
lowbloodsugar 5mo ago
See also Greenspuns tenth law: > Any sufficiently complicated C or Fortran program contains an ad-hoc, informally-specified, bug-ridden, slow implementation of half of CommonLisp. https://wiki.c2.com/?GreenspunsTenthRuleOfProgramming https://wiki.c2.com/?GreenspunsTenthRuleOfProgramming
deleted 5mo ago
[deleted]
dodu_ 5mo ago
NOOO you have to use my shitpile of nested yaml with the same dependency sprawl cancer as modern javascript. You can't just upload a binary to your own servers and host it there you need to overthink everything and make an extremely simple process overcomplicated just install one more side car and fifty more dependencies on your helm chart bro and then we can move on to figuring out CSI it should only take like a month to get it working properly I promise!!!!!
waterTanuki 5mo ago
I'll just say the quiet part out loud: A pile of shell scripts that no one else understands is job security. If your work is easily googleable/parseable by an AI, why would anyone pay you?
- antonvs 5mo ago
  I worked with a guy who fits this description. If there’s something he can reinvent poorly, he’d do it. A big part of what I did while working on the same team as him was slowly chipping away at his weird domain by introducing standard tools. I always had to advocate for them in such a way that they were solving a different problem, and then slowly work towards a point where some of his homegrown stuff became unnecessary. As to the job security idea: the only people who do this are people who aren’t good at creating real value, so they have to try to create niches where they’re needed.
deleted 5mo ago
[deleted]
Havoc 5mo ago
Literally just finished building a personal orchestrator system I wanted and had this very much in back of my mind. Ended up doing a mix. Built on compose for now but in a manner that’ll lift and shift to k8s easily enough. Its containers talking over network either way
tptacek 5mo ago
All it would take to make this post actually good would be to replace "Kubernetes" with "orchestrator"; that would also keep the symmetry with the post it's riffing on, about building compilers (it's not "Dear friend you have built a GHC").
wg0 5mo ago
Docker swarm is good enough for most cases if the black plague called Microservices isn't in your stack. Basecamp is running successfully with Kamal and surely making more money than all the Kubernetes experts combined on this platform. Small companies should not behave and pretend like big enterprises. Maybe Kubernetes is justified in those scenarios. But for 95% of the population, it's an overkill and a constantly moving target.
deleted 5mo ago
[deleted]
fifilura 5mo ago
Not sure how to describe this, but in a small/medium size company you have a few chips with talent. And somehow k8s tends to use one of the most valuable chips. The smartest guy who from then on only works with maintaining and updating k8s. Instead of building the best and smartest products. Kafka also competes with this.
MrDrMcCoy 5mo ago
So much defense of Kubernetes as the only solution to problems like this. Nomad and Incus exist as solid and less complicated alternatives. They also have a larger scaling ceiling.
gatnoodle 5mo ago
Partially true, I would argue. Try running kubernetes on an on-prem infra and you'll soon realize that it was much easier to run the apps on VMs themselves.
perarneng 5mo ago
k3s (lightweight k8s) is basically almost as easy as docker compose and then you have access to the full open source CNCF ecosystem in form of helm charts.
pjmlp 5mo ago
More like "Dear friend, you have built an Application Container", even more so now with the VCs looking for money in powering WebAssembly containers as pods.
callamdelaney 5mo ago
If you manage to build kubernetes again I recommend you burn it down and start again.
newer_vienna 5mo ago
The rule is to build everything such that there is either 0, 1, or infinite of anything
wachd 5mo ago
[dead]

143 points by Wingy 5mo ago

zdw 5mo ago

IMO, Kubernetes isn't inevitable, and this seems to paint it as such. K8s is well suited to dynamically scaling a SaaS product delivered over the web. When you get outside this scenario - for example, on-prem or single node "clusters" that are running K8s just for API compatibility, it seems like either overkill or a bad choice. Even when cloud deployed, K8s mostly functions as a batteries-not-included wrapper around the underlying cloud provider services and APIs. There are also folks who understand the innards of K8s very well that have legitimate criticisms of it - for example, this one from the MetalLB developer: https://blog.dave.tf/post/new-kubernetes/ https://blog.dave.tf/post/new-kubernetes/ Before you deploy something, actually understand what the pros/cons are, and what problem it was made to solve, and if your problem isn't at least mostly a match, keep looking.

zbentley 5mo ago
> K8s is well suited to dynamically scaling a SaaS product delivered over the web It’s well suited to other things as well, people are just in denial about some of them. “I need to run more than two containers and have a googleable way to manage their behavior” is a very common need.
- capitalhilbilly 5mo ago
  This is a need it fails at miserably. k8s reminds me of the raid recentralization anti pattern problem where you fix a hardware failure that never occurs in exchange for knowing simple higher level mistakes or security problems will tank something now too large to fail again.
antonvs 5mo ago
Kubernetes, in the form of k3s, was a critical success factor for us with the onprem deployment of our SaaS product. What's the problem with a single-node cluster? We use that for e.g. dev environments, as well as some small onprem deployments. > Even when cloud deployed, K8s mostly functions as a batteries-not-included wrapper around the underlying cloud provider services and APIs. Which batteries are not included? The "wrapper around the underlying cloud provider services and APIs" is enormously important. Why would you prefer to use a less well-designed, more vendor-specific set of APIs? I seriously don't get these criticisms of k8s. K8s abstracts away, and standardizes, an enormous amount of system complexity. The people who object to it just don't have the requirements where it starts making sense, that's all.
- subhobroto 5mo ago
  > Kubernetes, in the form of k3s, was a critical success factor for us with the onprem deployment of our SaaS product. What surprises and gotchas did you have to deal with using k3s as a Kubernetes implementation? Did you use an LB? Which one? I'm assuming all your onprem nodes were just linux servers with very basic equipment (the fanciest networking equipment you used were 10GbE PCIe cards, nothing more special than that?)
  - antonvs 5mo ago
    We sell to enterprise customers. All of them deploy our solution on internal cloud-style VM clusters. We use the Traefik ingress controller by default. There really weren't any particular surprises or gotchas at that level. In this context, I've never had to deal with anything at the level of the type of Ethernet card. That's kind of the point: platforms like k8s abstract away from that.
physicsguy 5mo ago
It's also difficult for data pipelines or data intensive things. At several companies we've run into the "Need to put ML model behind API and pods get killed because health checks via API are basically not compatible with container fully under load but still working"

isodev 5mo ago

Unless you’re in Erlang world (Elixir, Gleam..) and all that is already baked into OTP and the BEAM. You can go on holiday knowing it will be a while longer before you need to break out the pods (and at that scale, you will be able to afford a colleague or two to help you).

kirici 5mo ago
How does Elixir renew my certificates, mount networked storage mounts and create a VIP and internal DNS entries for my valkey instance? Scaling is a sidenote, that it becomes easy is a result of hoisting everything else onto one control plane and a set of coherent APIs.

et1337 5mo ago

The saddest part about Kubernetes is… after you set it all up, you still need a hacky deploy.sh to sed in the image tag to deploy! And pretty soon you’re back to “my dear friend you have built a Helm”. And so the configuration clock continues ticking…

cassianoleal 5mo ago
I have been using Kubernetes for 7 or 8 years now, and have nearly 100% stayed away from Helm. Some Kustomize, a little bit of envsubst and we're good to go thank you very much.
- supriyo-biswas 5mo ago
  How do you handle cleanups and hooks? The best way to do helm, at least for me, seems to be about limiting its use to simple templating use cases; if you end up needing an if, you've probably done something terribly wrong.
  - srcreigh 5mo ago
    Seems to be a case of the XY problem. What do you need cleanups and hooks for?
    - supriyo-biswas 5mo ago
      Cleanups: I want to do a `helm uninstall` and have all the manifests go away at once instead of looking around for N different resources. Hooks: I want to apply my database migrations and populate the database with static datasets before I deploy my application, without having my CI connect to the database cluster (at places I've worked, the CI cluster and K8s cluster were completely separate).
      - cassianoleal 5mo ago
        > Cleanups: I want to do a `helm uninstall` and have all the manifests go away at once instead of looking around for N different resources. kubectl delete -f <manifests.yaml> kubectl delete -k <kustomization_directory> > I want to apply my database migrations and populate the database with static datasets before I deploy my application, without having my CI connect to the database cluster A Job feels like a good fit for this. CI deployes the Job without connecting to DB, Job runs migrations using the same connectivity as the application.

jstanley 5mo ago

> I know you wanted to "choose boring tech" to just run some containers. The people advocating for boring tech generally aren't interested in containers. You can just run programs.

stavros 5mo ago
Containers are just statically-linked programs for the rest of us.
- subhobroto 5mo ago
  I am a big fan which is why I am saying this: you're dismissing the kernel and ABI surface is a huge assumption that must hold true for your comment to hold stavros. If you had said "unikernels" I would have had no arguments to make.
  - stavros 5mo ago
    What do you mean? Statically linked programs depend on the kernel too.
    - subhobroto 5mo ago
      right - that's precisely what I meant. I read your comment "Containers are just statically-linked programs for the rest of us." as "containers can be replaced by statically-linked programs". If you didn't imply that, I apologize. If you did mean that, I disagree with you precisely because your point works if you only care about dependency management - it falls apart on system state. A static binary is a process on the host and shares the same process space, network stack, and filesystem. OTOH, a container is a jail (the primary usecase): I can't cgroup a static binary's memory usage or give it a virtual network interface without reimplementing "container lite". Containers aren't just 'statically linked programs' - they allow me to use the kernel as a hypervisor for isolated environments. What they are though, a messy but practical compromise to Unikernels - which was my last point in our GP.

oddurmagnusson 5mo ago

Found this the same day I published this: https://github.com/oddur/yoink https://github.com/oddur/yoink Kubernetes was overkill (I do that all day, 5 days a week); Kamal was too restrictive, so I found myself rolling out Yoink. Just what I need from k8s, but simple enough I can point it to a baremetal machine on Hertzner that can easily run all my workloads.

subhobroto 5mo ago
> found myself rolling out Yoink - using Tailscale SSH is brilliant - using caddy-docker-proxy for ingress is brilliant What do you use for: - service discovery - secret store (EDIT: Crap you use Infisical. No shade, I just have this horrible foreboding it will end up like Hashicorp. I use Conjur Secretless Broker but am tracking: https://news.ycombinator.com/item?id=47903690 https://news.ycombinator.com/item?id=47903690) - backing up and restoring state like in a DB PS: Have you been having issues with Hetzner the last few weeks?
- oddurmagnusson 5mo ago
  Service discovery is basically just Docker's internal DNS. Caddy-docker-proxy can use it to find healthy upstreams. For secrets, I self-host Infisical on the box -- easy to plug in whatever secret manager, should make it pair nicely with https://github.com/tellerops/teller https://github.com/tellerops/teller or something similar Had no problems with Hertzner so far, just enjoying the raw CPU power of bare metal. The plan is to roll out more boxes across different providers, using Tailscale for the backplane network and Cloudflare to load-balance between them. All in due time What issues have you been having ?
  - subhobroto 5mo ago
    I have a suspicion you're using Headscale? If so, I urge you to consider Ionscale. I use it with Authentik as the IdP. Personally commiting to using Tailscale as a core foundation of my infrastructure and Ionscale is my hedge against getting Hashicorped. > Service discovery is basically just Docker's internal DNS. Caddy-docker-proxy can use it to find healthy upstreams Do you have a writeup of this somewhere? I'm unaware of being able to manage Docker's internal DNS over some kind of an API (would appreciate if you know a way to). The only way I know is to manipulate network aliases via Docker Engine API. As a result I use Hickory DNS with RFC 2136. That coupled with Caddy-docker-proxy gets me extremely close.

deleted 5mo ago

[deleted]

heyitsdaad 5mo ago

Kubernetes networking is an inefficient mess.

wmf 5mo ago
Some CNIs are definitely better than others. Unfortunately it seems 99% of people want to work against the k8s networking model.
- zbentley 5mo ago
  Shit just gets really weird when your network isn’t split for k8s in an equivalent way to what GCP/AWS expect. Like, if you have other services running on the nodes that you want things inside k8s to talk to, or if the nodes are in a flat subnet with other stuff in it, things get annoying. Those are worst practices for a reason, but pretty common in environments with home rolled k8s clusters.

whynotmaybe 5mo ago

I need clarifications. I see docker as a way to avoid having a standard dev platform for everyone in the company so that the infra team don't have to worry about patch xyz for library abc, only run docker. But, with all the effort put in place to coordinate docker, k8s and all the shebang, isn't it finally easier to force a platform and let it slowly evolve over time? Is docker another technical tool that tries to solve a non-technical problem?

whycombinetor 5mo ago
Porque no los dos. Force a platform. Deploy non containerized. Build a dockerized version of the forced platform for cross-platform local dev.
esafak 5mo ago
I do not follow you. Every app has different needs. Containers encode them in a shareable way. You can evolve the image over time. So what more do you want?
0xbadcafebee 5mo ago
Docker is a solution to one specific problem: the need for "a user" to run 10 different potentially conflicting apps, all at the same time, on one machine, and abstract away anything which might make those apps conflict if they ran on a single OS. It provides a dozen different solutions in one package. K8s is a way to take that and make it scale up for a large number of applications on a large number of hosts in a production business in a way that's automated and resilient to failure.

supriyo-biswas 5mo ago

Criticisms of Kubernetes generally come from a few places: - People who would prefer their way of doing this, whether that's deployments on VMs, or use some sort of simpler cloud provider. I had the same opinion a few years ago, but have kind of come to like it, because I can cleanly deploy multiple applications on a cluster in a declarative fashion. I still don't buy the "everything on K8s", and my personal setup is to have a set of VMs bought from a infrastructure provider, setup a primary/replica database on two of them, and use the rest as Kubernetes nodes. - People who run Kubernetes at larger scales and have had issues with them. This usually needs some custom scaling work; the best way to work around this if you're managing your own infra[1] is to split the cluster into many small independent clusters, akin to "cellular deployments"[2]/"bulkhead pattern"[3]. Alternatively, if you are at the point where you have a 500+ node cluster, it may not be a bad idea to start using a hyperscaler's service as they have typically done some of the scaling work for you, typically in form of replacing etcd and the RPC layer through something more stable. - People who need a deep level of orchestration Examples of such use cases may be to run a CI system or a container service like fly.io; for such use cases, I agree that K8s is often overkill, as you need to keep the two datastores in sync and generate huge loads on the kube-apiserver and the cluster datastore in the process, and it might be often better to just bring up Firecracker MicroVMs or similar yourself. Although, I should say that teams writing their first orchestration process almost always run to Kubernetes without realizing this pitfall, though I have learned to keep my mouth shut as I started a small religious war recently at my current workplace by raising this exact point. [1] Notice how I don't say "on-prem", because the hyperscaler marketing teams would rather have you believe in two extremes of either using their service or running around in a datacenter with racks, whereas you can often get bog-standard VMs from Hetzner or Vultr or DigitalOcean and build around that. [2] https://docs.aws.amazon.com/wellarchitected/latest/reducing-scope-of-impact-with-cell-based-architecture/cell-deployment.html https://docs.aws.amazon.com/wellarchitected/latest/reducing-... [3] https://learn.microsoft.com/en-us/azure/architecture/patterns/bulkhead https://learn.microsoft.com/en-us/azure/architecture/pattern...

rfoo 5mo ago
Another case: People who want to run workloads that are inherently incompatible with Kubernetes networking model. For example: * For some cursed reasons you want to make sure every single one instance of a large batch job see just one NIC in its container and they are all the same IP and you NAT to the outside world. Ingress? What ingress? This is a batch job! * Like the previous point, except that your "batch job" somehow has multiple containers in one instance now, and they should be able to reach each other by domain.

cortesoft 5mo ago

This is obviously slightly exaggerated, but I do feel like this whenever people dismiss Kubernetes as either too complicated or not needed. The response I always got when suggesting Kubernetes is "you can do all those things without Kubernetes" Sure, of course. There are a million different ways to do everything Kubernetes does, and some of them might be simpler or fit your use case more perfectly. You can make different decisions for each choice Kubernetes makes, and maybe your decisions are more perfect for your workload. However, the big win with Kubernetes is that all of those choices have been made and agreed upon, and now you have an entire ecosystem of tools, expertise, blog posts, AI knowledge, etc, that knows the choices Kubernetes made and can interface with that. This is VERY powerful.

PunchyHamster 5mo ago
Honestly the main problem is people using k8s for something that's like... a database, and an app, and maybe a second app, that all could be containers or just a systemd service. And then they hit all the things that make sense in big company with like 40 services but very little in their context and complain that complex thing designed for complex interactions isn't simple
- jmalicki 5mo ago
  Luckily since I met this guy named Claude most of that complexity has gone away.
  - andai 5mo ago
    A while back when the agents got hyped I was looking into the whole "give it a VM / docker container" I realized the safest and simplest option was just to give it its own machine. Then I realized giving it root on a $3 VPS is functionally equivalent. If it blows it up, you just reset the VM. It sounds bad but I can't see an actual difference.
- nazcan 5mo ago
  But if you want some redundancy, k8s let's you just say run 4 of this, 6 of this on these 3 machines. At least I find it quite straight forward. The database is more complex since there is storage affinity (I use cockroachDB with local persistent volumes for it) - but stateful is always complicated.

winton 5mo ago

"Except if you quit or go on vacation, who will maintain this custom pile of shell scripts?" LLMs can reason about and fix them quite well.

zbentley 5mo ago
Not as well as they can reason (or others can google) something as standardized as kubernetes. There’s just less context (in both senses of the term) needed to understand something running on a common substrate versus something bespoke, even if the bespoke thing is itself comprised of standardized parts.
- winton 5mo ago
  For a project set up by a qualified engineer, there would be little difference to the end user in practice. The LLM would work out a solution with a negligible difference in speed. Maybe debugging would also be faster for the LLM without the abstraction layers and low level access?

deleted 5mo ago

[deleted]

Dedime 5mo ago

PREACH! I run K8s at home. I used to do docker-compose - and I'd still recommend that to most people - but even for my 1 little NUC with 4vcpu / 16Gi Homelab, I still love deploying with K8s. It's genuinely simpler for me. If anyone's looking for inspiration, my setup: * ArgoCD pointed to my GitLab repos * GitLab repos contain Helm charts * Most of the Helm charts contain open-source charts as subcharts, with versions set like (e.g.) `version: ~0` - meaning I automatically receive updates for all major version until `1` * Updating my apps usually consists of logging into the UI, reviewing the infrastructure and image tag updates, and manually clicking sync. I do this once every few months My next little side project: Autoscaling into the cloud (via a secure WireGuard tunnel) when I want to expand past my current hardware limitations

wernerb 5mo ago
A reason not to run k8s is if you want your server to reach C10 idle states. The k8s control plain with its polling and checking are quite heavy on the mostly idle server. I have reverted to just use Nixos and oci podman containers. Everything is declarative and reproducible
- r_lee 5mo ago
  another one is swap. UnlimitedSwap was deprecated and you can now only use LimitedSwap which restricts how much swap you can use, so you can't take full advantage of zram, which sucks for those looking to run lean

perfunctory 5mo ago

I somehow feel that it's actually the opposite. It should be "Dear kubernetes user, you have just built a shell".

blindlobstar 5mo ago

Why both posts mention docker compose and not mentioning docker swarm. Being using it for my projects for long time. And it's so nice. Similar syntax, easy networking, rollout strategy, easy to add nodes to cluster. You can have one template docker-compose.yaml file and separate deployment files for different envs, like: docker-compose.dev.yaml, docker-compose.prod.yaml I think swarm is really underrated

PufPufPuf 5mo ago
I've been there. We still ended up with messy deploy scripts written in Ruby and the only debugging solution was "just comment out everything then run line by line".
- blindlobstar 5mo ago
  'docker stack deploy' covers most of the cases. But yeah, there is still some problems like: "update a config or a secret", that require manually invoking additional commands (or via scripts)
vbezhenar 5mo ago
How do you solve persistence with swarm? Can I deploy postgres with network storage that will mount automatically on node where container is launched?
- blindlobstar 5mo ago
  > Can I deploy postgres with network storage that will mount automatically on node where container is launched? Yes (if I'm getting your question right). here is an example of nfs volume: https://docs.docker.com/reference/compose-file/volumes/#driver_opts https://docs.docker.com/reference/compose-file/volumes/#driv...
Havoc 5mo ago
It’s in a death spiral - not widely used so people aren’t incentivised to put time into it so not widely used

hasyimibhar 5mo ago

I've experienced something like this at work but with data warehouse instead, and it happened multiple times (to be fair, data engineering is still fairly new where I'm from). One example was an engineer wanted to build an API that accepts large CSV (GBs of credit reports) to extract some data and perform some aggregations. He was in the process of discussing with SREs on the best way to process the huge CSV file without using k8s stateful set, and the solution he was about to build was basically writing to S3 and having a worker asynchronously load and process the CSV in chunks, then finally writing the aggregation to db. I stepped in and told him he was about to build a data warehouse. :P

jeffrallen 5mo ago
If it was less than 100 gb, he probably should have just loaded the whole thing in RAM on a single machine, and processed it all in a single shot. No S3, no network round trips, no chunking, no data warehouse.

jeffrallen 5mo ago

Some days, it would be better to build a working not-kubernetes than debugging the not-working kubernetes.

whycombinetor 5mo ago

After reading this and remembering an old hobby project, I decided to switch the deploy from a systemd service to PM2, which apparently has rolling deployments without needing Docker engine (for those of us minmaxing instance RAM).

stego-tech 5mo ago

As someone rolling their self-hosted stuff via Compose and shell scripts instead of K8s specifically for the simplicity of the experience, this is 100% why you need to understand what Kubernetes solves before writing it off entirely. I'm not doing overlay networks, I'm using a single bare-metal host, and I value the hands-on Linux administration experience versus the K8s cluster admin experience. All of these are reasons I specifically chose not to use Kubernetes. The second I want HA, or want to shift from local VLANs to multi-cloud overlays, or I don't need the local Linux sysadmin experience anymore? Yeah, it's K8s at the top of the list. Until then, my solution works for exactly what I need.

jFriedensreich 5mo ago

im just about giving OPs premise another go. compose just feels so much better as abstraction especially with small and medium setups looking close to the optimum of expressiveness without boilerplate to describe what is needed. The missing pieces seem to also be in the compose compatible “docker stack” aka new docker swarm, which i ignored for probably too long as i assumed it was the discontinued old swarm. Even if new swarm mode sucks how hard can it be to make something compose shaped vs running k8?

shrubble 5mo ago

I can tell you how vendors deliver a software solution that runs on Kubernetes: very poorly. The needed tweaks, the ability to customize things, basically goes to zero because the support staff is technical about the software, but NOT about Kubernetes. I am not joking: a recent deployment required 3x VMs for Kubernetes, each VM having 256 gigabytes of RAM; then a separate 3x VMs for a different piece. 1.5TB of RAM to manage less than 1200 network devices (routers etc. that run BGP). No one knew, for instance, how to lower the MongoDB (because of course you need it!) resource usage, despite the fact that the clustered VMware install is using a very fast SSD storage solution and thus MongoDB is unlikely accelerate anything; so over 128GB RAM is being burned on caching the results coming back from SSDs that are running at many-GB/s throughput.

throwaway041207 5mo ago
Whether this is deployed via Helm charts or a native controller, there's almost certainly some overlay where you can override resource values, unless this is just a very crappy vendor.
- shrubble 5mo ago
  The setting is there, somewhere, I agree; but no one can tell you how to change it or what the implications might be. So unless I want to dig into their YAML files that are not documented…

justsomehnguy 5mo ago

> Ah, but wait! Inevitably, you find a reason to expand to a second server >> The real problem is that programmers have spent far too much time worrying about efficiency in the wrong places and at the wrong times; premature optimization is the root of all evil (or at least most of it) in programming. -- Donald Knuth, Computer Programming as an Art (1974) EDIT: > Except if you quit or go on vacation, who will maintain this custom pile of shell scripts? Honestly? I don't care. There is a reason why I quit and 99% of time it's the pay. And if the company doesn't pay me enough to bother then why should I? Why should I bother about some company future in the first place?

kube-system 5mo ago

Kubernetes is a powerful tool for complicated problems. If if it seems complicated, you probably don’t have a complicated deployment problem. But really this applies to any powerful tool. If you need to measure a voltage, an 4 channel oscilloscope also probably seems too complicated.

dang 5mo ago

Discussed at the time: Dear friend, you have built a Kubernetes - https://news.ycombinator.com/item?id=42226005 https://news.ycombinator.com/item?id=42226005 - Nov 2024 (277 comments)

drdaeman 5mo ago

They have built an orchestrator, not Kubernetes. There is one key difference: they know this thing, end-to-end, down to every single bolt and piece of duct tape (with possible exception for Docker internals) And that's a very important distinction when it comes to maintaining complex systems. This could've changed with LLMs (I'm still adjusting to what new capabilities mean for various decision-making logic), but before machine intelligence debugging an issue with Kubernetes could've been a whole world of pain.

chuckadams 5mo ago
And chances are only they know it. If my role has enough cluster access, I can muddle through pretty much any helm chart (with lots of cursing, yes) but it might take me days to set up whatever elaborate bespoke environment and script invocations are needed to replicate the current production setup maybe.

lowbloodsugar 5mo ago

See also Greenspuns tenth law: > Any sufficiently complicated C or Fortran program contains an ad-hoc, informally-specified, bug-ridden, slow implementation of half of CommonLisp. https://wiki.c2.com/?GreenspunsTenthRuleOfProgramming https://wiki.c2.com/?GreenspunsTenthRuleOfProgramming

deleted 5mo ago

[deleted]

dodu_ 5mo ago

NOOO you have to use my shitpile of nested yaml with the same dependency sprawl cancer as modern javascript. You can't just upload a binary to your own servers and host it there you need to overthink everything and make an extremely simple process overcomplicated just install one more side car and fifty more dependencies on your helm chart bro and then we can move on to figuring out CSI it should only take like a month to get it working properly I promise!!!!!

waterTanuki 5mo ago

I'll just say the quiet part out loud: A pile of shell scripts that no one else understands is job security. If your work is easily googleable/parseable by an AI, why would anyone pay you?

antonvs 5mo ago
I worked with a guy who fits this description. If there’s something he can reinvent poorly, he’d do it. A big part of what I did while working on the same team as him was slowly chipping away at his weird domain by introducing standard tools. I always had to advocate for them in such a way that they were solving a different problem, and then slowly work towards a point where some of his homegrown stuff became unnecessary. As to the job security idea: the only people who do this are people who aren’t good at creating real value, so they have to try to create niches where they’re needed.

deleted 5mo ago

[deleted]

Havoc 5mo ago

Literally just finished building a personal orchestrator system I wanted and had this very much in back of my mind. Ended up doing a mix. Built on compose for now but in a manner that’ll lift and shift to k8s easily enough. Its containers talking over network either way

tptacek 5mo ago

All it would take to make this post actually good would be to replace "Kubernetes" with "orchestrator"; that would also keep the symmetry with the post it's riffing on, about building compilers (it's not "Dear friend you have built a GHC").

wg0 5mo ago

Docker swarm is good enough for most cases if the black plague called Microservices isn't in your stack. Basecamp is running successfully with Kamal and surely making more money than all the Kubernetes experts combined on this platform. Small companies should not behave and pretend like big enterprises. Maybe Kubernetes is justified in those scenarios. But for 95% of the population, it's an overkill and a constantly moving target.

deleted 5mo ago

[deleted]

fifilura 5mo ago

Not sure how to describe this, but in a small/medium size company you have a few chips with talent. And somehow k8s tends to use one of the most valuable chips. The smartest guy who from then on only works with maintaining and updating k8s. Instead of building the best and smartest products. Kafka also competes with this.

MrDrMcCoy 5mo ago

So much defense of Kubernetes as the only solution to problems like this. Nomad and Incus exist as solid and less complicated alternatives. They also have a larger scaling ceiling.

gatnoodle 5mo ago

Partially true, I would argue. Try running kubernetes on an on-prem infra and you'll soon realize that it was much easier to run the apps on VMs themselves.

perarneng 5mo ago

k3s (lightweight k8s) is basically almost as easy as docker compose and then you have access to the full open source CNCF ecosystem in form of helm charts.

pjmlp 5mo ago

More like "Dear friend, you have built an Application Container", even more so now with the VCs looking for money in powering WebAssembly containers as pods.

callamdelaney 5mo ago

If you manage to build kubernetes again I recommend you burn it down and start again.

newer_vienna 5mo ago

The rule is to build everything such that there is either 0, 1, or infinite of anything

wachd 5mo ago

[dead]