12 ms·

The hater's guide to Kubernetes

288 points by paulgb 3y ago

t3rabytes 3y ago
My current company is split... maybe 75/25 (at this point) between Kubernetes and a bespoke, Ansible-driven deployment system that manually runs Docker containers on nodes in an AWS ASG and will take care of deregistering/reregistering the nodes with the ALB while the containers on a given node are getting futzed with. The Ansible method works remarkably well for it's age, but the big thing I use to convince teams to move to Kubernetes is that we can take your peak deploy times from, say, a couple hours down to a few minutes, and you can autoscale far faster and more efficiently than you can with CPU-based scaling on an ASG. From service teams that have done the migrations, the things I hear consistently though are: - when a Helm deploy fails, finding the reason why is a PITA (we run with --atomic so it'll rollback on a failed deploy. What failed? Was it bad code causing a pod to crash loop? Failed k8s resource create? who knows! have fun finding out!) - they have to learn a whole new way of operating, particularly around in-the-moment scaling. A team today can go into the AWS Console at 4am during an incident and change the ASG scaling targets, but to do that with a service running in Kubernetes means making sure they have kubectl (and it's deps, for us that's aws-cli) installed and configured, AND remembering the `kubectl scale deployment X --replicas X` syntax. [Both of those things are very much fixable]
- dpflan 3y ago
  HPAs and VPAs are useful k8s concepts for your auto-scaling needs.
  - t3rabytes 3y ago
    HPA is useful until your maxReplicas count is set too low and you're already tapped out.
    - cogman10 3y ago
      Sort of a learning thing though right? Like, if you find maxReplicas is too low you move that number up until it isn't right? This is different from waking people up at 4am frequently to bump up the number of replicas.
      - deleted 3y ago
        [deleted]
    - dpflan 3y ago
      You can edit your HPA live, in maybe as many commands or keystrokes as manually scaling…until you commit the change to your repo of configs.
- makestuff 3y ago
  I haven’t used kubernetes in a few years, but do they have a good UI for operations? Your example of the AWS console where you can just log in and scale something in the UI but for kubernetes. We run something similar on AWS right now, during an incident we log into the account with admin access to modify something and then go back to configure that in the CDK post incident.
  - t3rabytes 3y ago
    AWS has a UI for resources in the cluster but it relies on the IAM role you're using in the console to have configured perms in the cluster, and our AWS SSO setup prevents that from working properly (this isn't usually the case for AWS SSO users, it's a known quirk of our particular auth setup between EKS and IAM -- we'll fix it sometime).
  - adhamsalama 3y ago
    https://k8slens.dev https://k8slens.dev
- cogman10 3y ago
  For scaling, have you tried using either an HPA or keda? We've had pretty good success with simple HPAs.
  - t3rabytes 3y ago
    Yep, I'd say >half of the teams with K8s services have adopted KEDA, but we've got some HPA stragglers for sure.
    - dpflan 3y ago
      I have to say that when you have more buy in from delivery teams and adoption of HPAs your system can become more harmonious overall. Each team can monitor and tweak their services, and many services are usually connected upstream or downstream. When more components can ebb and flow according to the compute context then the system overall ebbs and flows better. #my2cents
- freedomben 3y ago
  Personally, I don't like Helm. I think for the vast majority of usecases where all you need is some simple templating/substitution, it just introduces way more complexity and abstraction than it is worth. I've been really happy with just using `envsubst` and environment variables to generate a manifest at deploy time. It's easy with most CI systems to "archive" the manifest, and it can then be easily read by a human or downloaded/applied manually for debugging with. Deploys are also just `cat k8s/${ENV}/deploy.yaml | envsubt > output.yaml && kubectl apply -f output.yaml` I've also experimented with using terraform. It's actually been a good enough experience that I may go fully with terraform on a new project and see how it goes.
  - linuxftw 3y ago
    You might like kubernetes kustomize if you don't care for helm (IMO, just embrace helm, you can keep your charts very simple and it's straight forward). Kustomize takes a little getting used to, but it's a nice abstraction and widely used. I cannot recommend terraform. I use it daily, and daily I wish I did not. I think Pulumi is the future. Not as battle tested, but terraform is a mountain of bugs anyway, so it can't possibly be worse. Just one example where terraform sucks: You cannot both deploy a kubernetes cluster (say an EKS/AKS cluster) and then use kubernetes_manifest provider in a single workspace. You must do this across two separate terraform runs.
- jonathaneunice 3y ago
  The problem with bespoke, homegrown, and DIY isn't that the solutions are bad. Often, they are quite good—excellent, even, within their particular contexts and constraints. And because they're tailored and limited to your context, they can even be quite a bit simpler. The problem is that they're custom and homegrown. Your organization alone invests in them, trains new staff in them, is responsible for debugging and fixing when they break, has to re-invest when they no longer do all the things you want. DIY frameworks ultimately end up as byzantine and labyrinthine as Kubernetes itself. The virtue of industry platforms like Kubernetes is, however complex and only half-baked they start, over time the entire industry trains on them, invests in them, refines and improves them. They benefit from a long-term economic virtuous cycle that DIY rarely if ever can. Even the longest, strongest, best-funded holdouts for bespoke languages, OSs, and frameworks—aerospace, finance, miltech—have largely come 'round to COTS first and foremost.
roschdal 3y ago
[flagged]
api 3y ago
I understand where most of the complexity in K8S comes from, but it still horrifies and offends me and I hate it. But I don't think it's Kubernetes' fault directly. I think the problem is deeper in the foundation. It comes from the fact that we are trying to build modern, distributed, high availability, incrementally upgradeable, self-regulating systems on a foundation of brittle clunky 1970s operating systems that are not designed for any of that. The whole thing is a bolt-on that has to spend a ton of time working around the limitations of the foundation, and it shows. Unfortunately there seems to be zero interest in fixing that and so much sunk cost in existing Unix/Posix designs that it seems like we are completely stuck with a basic foundation of outdated brittleness. What I think we need: * An OS that runs hardware-independent code (WASM?) natively and permits things like hot updates, state saving and restoration, etc. Abstract away the hardware. * Native built-in support for clustering, hot backups, live process migration between nodes, and generally treating hardware as a pure commodity in a RAIN (redundant array of inexpensive nodes) configuration. * A modern I/O API. Posix I/O APIs are awful. They could be supported for backward compatibility via a compatibility library. * Native built-in support for distributed clustered storage with high availability. Basically a low or zero config equivalent of Ceph or similar built into the OS as a first class citizen. * Immutable OS that installs almost instantly on hardware, can be provisioned entirely with code, and where apps/services can be added and removed with no "OS rot." The concept of installing software "on" the OS needs to be killed with fire. * Shared distributed network stack where multiple machines can have the same virtual network interfaces, IPs, and open TCP connections can migrate. Built-in load balancing. I'm sure people around here can think of more ideas that belong in this list. These are not fringe things that are impossible to build. Basically you should have an immutable image OS that turns many boxes into one box and you don't have to think about it. Storage is automatically clustered. Processes automatically restart or, if a hardware fault is detected in time, automatically migrate. There were efforts to build such things (Mosix, Plan 9, etc.) but they were bulldozed by the viral spread of free Unix-like OSes that were "good enough." Edit: That being said, I'm not saying Kubernetes is good software either. The core engine is actually decent and as the OP said has a lot of complexity that's needed to support what it does. The ugly nasty disgusting parts are the config interface, clunky shit like YAML, and how generally arcane and unapproachable and ugly the thing is to actually use. I just loathe software like this. I feel the same way about Postgres and Systemd. "Algorithmically" they are fine, but the interface and the way you use them is arcane and makes me feel like I'm using a 70s mainframe on a green VT220 monitor. Either these things are designed by the sorts of "hackers" who like complexity and arcane-ness, or they're hacks that went viral and matured into global infrastructure without planning. I think it's a mix of both... though in the case of Postgres it's also that the project is legitimately old. It feels like old-school Unix clunkware because it is.
Spivak 3y ago
This matches our experience as well. As long as you treat your managed k8s cluster as autoscaling-group as-a-service you'll do fine. k8s's worst property is that it's a cleverness trap. You can do anything in k8s whether it's sane to do so or not. The biggest guardrail against falling into is managing your k8s with terraform-ish so that you don't find yourself in a spot where "effort to do it right" >> "effort to hack in YAML" and finding your k8s cluster becoming spaghetti.
- x86x87 3y ago
  Why not just use an autoscaling group? Re: cleverness trap. I feel like this is the tragedy of software development. We like to be seen as clever. We are doing "hard" things. I have way more respect for engineers that do "simple" things that just work using boring tech and factor in whole lifecycle of the product.
  - p_l 3y ago
    > Why not just use an autoscaling group? Not everyone has money to burn, even back in ZIRP era. And before you trot out wages for experienced operations team - I've regularly dealt with it being cheaper to pay for one or two very experienced people than deal with AWS bill. For the very simple reason that cloud provider's prices are scaled to US market and not everyone has US money levels.
  - Spivak 3y ago
    Sorry, I could have explained that better. The biggest value add that k8s has is that it gives you as many or as few autoscaling groups as you need at a given time using only a single pool (or at least fewer pools) of heterogeneous servers. There's lots of fine print here but it really does let you run the same workloads on less hardware and to me that's the first and last reason you should be using it. I wouldn't start with k8s and instead opt for ASGs until you reach the point where you look at your AWS account and see a bunch of EC2 instances sitting underutilized.
treesciencebot 3y ago
> Above I alluded to the fact that we briefly ran ephemeral, interactive, session-lived processes on Kubernetes. We quickly realized that Kubernetes is designed for robustness and modularity over container start times. Is there a clear example of this? E.g. is kubernetes inherently unable to start a pod (assuming the same sequence of events, e.g. warm/cold image with streaming enabled) under 500ms, 1s etc? I am asking this as someone who spent quite a bit of time and wasn't able to bring it down 2s< mark, which eventually led us to rewrite the latency sensitive parts to use Nomad. But we are currently in a state where we are re-considering kubernetes for its auxilary tooling benefits and would love to learn more if anyone had experiences with starting and stopping thousands of pods with the lowest possible latencies without caring for utilization or placement but just observable boot latencies.
- p_l 3y ago
  You'd have to ensure that a) preload all images of course b) there's enough of nodes with enough capacity c) the pods don't use anything that has possible longer latency (high latency CSI etc.) d) you might want to write custom scheduler for your workloads (it could take into account what images are preloaded where, etc)
- paulgb 3y ago
  I do believe that with the right knowledge of Kubernetes internals it's probably possible to get k8s cold start times competitive with where we landed without Kubernetes (generally subsecond, often under 0.5s depending on how much the container does before passing a health check), but we'd have to understand k8s internals really well and would have ended up throwing out much of what already existed. And we'd probably end up breaking most of the reasons for using Kubernetes in the first place in the process.
  - p_l 3y ago
    Not much internals needed, but actual in depth understanding of Pod kube-api plus at least basics of how scheduler, kubelet, and kubelet drivers interact. Big possible win is custom scheduling, but barely anyone seems to know it exists
fifilura 3y ago
There is nothing wrong with k8s it is a nice piece of technology. But the article trending here a couple of days ago describes it well. https://www.theolognion.com/p/company-forgets-why-they-exist-after-11-week-migration-to-kubernetes https://www.theolognion.com/p/company-forgets-why-they-exist... It is complex enough to make the k8s maintainers the heroes of the company. And this is where things tend to go sideways. It has enough knobs and levers to distract the project from what they are actually trying to achieve.
- cedws 3y ago
  I see Kubernetes the same way as git. Elegant fundamental design, but the interface to it is awful. Kubernetes is designed to solve big problems and if you don't have those problems, you're introducing a tonne of complexity for very little benefit. An ideal orchestrator would be more composable and not introduce more complexity than needed for the scale you're running at. I'd really like to see a modern alternative to K8S that learns from some of its mistakes.
  - pphysch 3y ago
    Git is a much more subtle abstraction than k8s though. You can be blissfully unaware that a directory is a git repo, and still read/patch files. You cannot pretend k8s doesn't exist in a k8s system.
PUSH_AX 3y ago
I was once talking to an ex google site reliability engineer. He said there are maybe a handful of companies in the world that _need_ k8s. I tend to agree. A lot of people practice hype driven development.
- x86x87 3y ago
  I tend to agree. K8s makes a lot of sense if you are running your own bare metal servers at scale. If you are already using the cloud, maybe leverage abstraction already available in that context.
  - candiddevmike 3y ago
    You either recreate a less reliable version of kubernetes for workload ops or you go all in on your cloud provider and hope they'll be responsible for your destiny. Vanilla Kubernetes is just enough abstraction to avoid both of those situations.
    - x86x87 3y ago
      You cannot really be cloud agnostic these days - even when using k8s. So learning to use the capabilities the cloud provides is key.
      - p_l 3y ago
        Doesn't really mesh with my experience, especially the longer k8s been out. It can be cheaper to depend on cloud provider to ship some features, but with tools like crossplane you can abstract that out so developers can just "order" a database service etc. for their application.
    - PUSH_AX 3y ago
      Is “hope” the new replacement for SLAs? Or am I missing something with that statement?
      - 3y ago
rwmj 3y ago
> It’s also worth noting that we don’t administer Kubernetes ourselves This is the key point. Even getting to the point where I could install Kubernetes myself on my own hardware took weeks, just understanding what hardware was needed and which of the (far too many) different installers I had to use.
- LegibleCrimson 3y ago
  I found k3s pretty easy to spin up.
hobofan 3y ago
OT: Can something be done about HN commenting culture so that the comments stay more on topic? Some technologies (like Kubernetes) tend to attract discussions where half of the commenters completely ignore the original article, so we end up having a weekly thread about Kubernetes where the points of the article (which are interesting) are not able to be discussed because they are drowned out by the same unstructured OT discussions? At the time of this posting there are ~20 comments with ~2 actually having anything to do with the points of the article rather than Kubernetes in general.
- cogman10 3y ago
  Having read the article, isn't the point of the article kubernetes in general and what the author prescribes you sign up for/avoid? Discussions of k8s pitfalls and successes in general seems to be very much in line with what the article is advocating. And, to that point, there's frankly just not a whole lot interesting in this article for discussion "We avoid yaml and operators"... Neat.
  - hobofan 3y ago
    > Having read the article, isn't the point of the article kubernetes in general and what the author prescribes you sign up for/avoid? Yeah, and I think that provides a good basis to discussion, where people can critique/discuss whether the evaluation that the author has made are correct (which a few comments are doing). At the same time a lot of that discussion is being displaced by what I would roughly characterize as "general technology flaming" which isn't going anywhere productive.
- geodel 3y ago
  Huh, this article has hardly anything deep, technical, thought provoking or unique compared to ten thousand other Kubernetes articles. I am rather happy that people are having general purpose discussion about K8s.
- freedomben 3y ago
  What you're seeing is the early-crowd. With most (not all) posts, comments will eventually rise to the top that are more what you're looking for. IME it usually takes a couple hours. If it's a post where I really want to read the relevant comments, I'll usually come back at least 8 to 12 hours later and there's usually some good ones to choose from. Even topics like Apple that attract the extreme lovers and haters tend to trend this direction
Kab1r 3y ago
I almost feel attacked for using plain yaml, helm, cert-manager AND the ingress api just for personal homelab shenanigans.
- cogman10 3y ago
  Yeah, I disagree with the OP on the dangers there. They work fairly well for us and aren't the source of headache. Though, I still try and teach my dev teams that "just because bitnami puts in variables everywhere, doesn't mean you need to. We aren't trying to make these apps deployable on homelabs."
__MatrixMan__ 3y ago
> But we often do multiple deploys per day, and when our products break, our customer’s products break for their users. Even a minute of downtime is noticed by someone. Kubernetes might be the right tool for the job if we accept that this is a necessary evil. But maybe it's not? The idea that I might fail to collaborate with you because a third party failed because a fourth party failed kind of smells like a recipe for software that breaks all the time.
- paulgb 3y ago
  It really comes down to, I don't ever want to have the conversation “is this a good time to deploy, or should we wait until tonight when there’s less usage”. We have had some periods where our system was more fragile, and planning our days around the least-bad deployment window was a time suck, and didn't scale to our current reality of round-the-clock usage.
  - hellcow 3y ago
    You can achieve this without k8s, though. If your goal is, "I want zero-downtime deploys," that alone is not sufficient reason to reach for something as massively complex as k8s. Set up a reverse proxy and do blue-green deploys behind it.
    - paulgb 3y ago
      > Set up a reverse proxy and do blue-green deploys behind it. That's what I currently use Kubernetes for. What stack are you proposing instead?
      - sureglymop 3y ago
        If you only need zero downtime deployments, compose and traefik/caddy are enough. If you need to replicate storage, share networks and otherwise share resources across multiple hosts, kubernetes is better suited. But you'll also have much less control with compose, e.g. no limiting of egress/ingress and more.
deleted 3y ago
[deleted]
teeray 3y ago
Is there something like a k1s? What I’d love is “run this set of containers on this machine. If the machine goes down, I don’t care—I will fix it.” If it wired into nginx or caddy as well, so much the better. Something like that for homelab use would be wonderful.
- eropple 3y ago
  You've basically described k3s, I think. I run it in my homelab (though I am enough of a tryhard to have multiple control planes) as well as on a couple of cloud servers as container runtimes (trading some overhead for consistency). k3s really hammers home the "kubernetes is a set of behaviors, not a set of tools" stuff when you realize you can ditch etcd entirely and use sqlite if you really want to, and is a good learning environment.
- szszrk 3y ago
  That's basically just a docker-compose. If you want something crazy all-in-one for homelab check out https://github.com/azukaar/Cosmos-Server https://github.com/azukaar/Cosmos-Server
- silverquiet 3y ago
  Docker Compose probably fits the bill for that. They also have a built in minimalist orchestrator called Swarm if you do want to extend to multiple machines. I suppose it's considered "dead" since Kubernetes won mindshare, but it still gets updates.
- morbicer 3y ago
  For homelab, Docker compose should be enough For something more production oriented https://github.com/basecamp/kamal https://github.com/basecamp/kamal
- pheatherlite 3y ago
  Docker bare bones or docker compose. Run as systemd services and have docker run the container as a service account. Manual orchestration is all you need. Anything else like rancher or whatever are just fluff.
- 3y ago
throwaway892238 3y ago
When people call Kubernetes a "great piece of technology", I find it the same as people saying the United States is the "greatest country in the world". Oh yeah? Great in what sense? Large? Absolutely. Powerful? Definitely. But then the adjectives sort of take a turn... Complicated? Expensive? Problematic? Threatening? A quagmire? You betcha. If there were an alternative to Kubernetes that were just 10% less confusing, complicated, opaque, monolithic, clunky, etc, we would all be using it. But because Kubernetes exists, and everyone is using it, there's no point in trying to make an alternative. It would take years to reach feature parity, and until you do, you can't really switch away. It's like you're driving an 18-wheeler, and you think it kinda sucks, but you can't just buy and then drive a completely different 18 wheeler for only a couple of your deliveries. You probably will end up using K8s at some point in the next 20 years. There's not really an alternative that makes sense. As much as it sucks, and as much as it makes some things both more complicated and harder, if you actually need everything it provides, it makes no sense to DIY, and there is no equivalent solution.
- p_l 3y ago
  People forgot just how much of a mess Mesos environment was in comparison. And often pushed Nomad to this day surprises me with randomly missing a feature or two that turns out to be impactful enough to want to deal with more complexity because ultimately the result was less complexity in total.
madduci 3y ago
I don't get most of the blame and reasoning. Sure, everyone has their own product and experience and it's fine to express it, but I don't get the usage of other decisions such as "no to services meshes", "no to helm" and many more. You don't want to ideally reinvent the wheel for every workload you need (say you need a OIDC endpoint, an existing application): you are tempted to write everything from scratch by yourself, which is also fine, but the point is: why? Many products deliver their own Helm package. And if you are sick of writing YAML, I would look for Terraform over Pulumi, for the reason that you use the same tool for bringing up Infrastructure and then workloads. Kubernetes itself isn't easy to be used, in many cases you don't need it, but it might bring you nice things straight out of the box with less pain than other tooling (e.g. zero downtime deployments)
- p_l 3y ago
  The problem with Helm is that it did the one thing you should not do, and refused to fix it even when their promised to. They do text-replacement templating for YAML. I have once spent a month, being quite experienced k8s wrangler, trying to figure out why Helm 2 was timing out, only to finally trace it down to how sometimes we would get wrong number of spaces in some lines.
- eropple 3y ago
  I admit that I use some Helm stuff in my home environment, but for production I'm genuinely worried about the need to support whatever they've thrown into it. At minimum I'm going to have to study the chart and understand exactly what they propose to open-palm slam into my cluster, and for many/most applications at that point it might genuinely be worth just writing a manifest myself. Not always. Some applications are genuinely complex and need to be! But often, this has been the case for me. For all my stuff, though, I use kustomize and I'm pretty happy with it; it's too stupid for me to be clever, and this is good. Service meshes are a different kettle of fish. hey add exciting new points of failure where they need not exist, and while there are definitely use cases for them, I'd default to avoiding them until somebody proves the need for one.
- org3 3y ago
  Can’t Pulumi be used to bring up infra and workloads as well?
pheatherlite 3y ago
Why are people still scared of k8s? At certain jobs thresholds, it is worth every ounce of effort to maintain it. Better yet, go managed.
- cogman10 3y ago
  I honestly don't understand it either. Familiarity? K8s has like, what, 5 big concepts to know and once you are there the other concepts (generally) just build from there. - Containers - Pods - Deployments - Services - Ingresses There are certainly other concepts you can learn, but you aren't often dealing with them (just like you aren't dealing with them when working with something like docker compose).
  - nprateem 3y ago
    Good luck fixing etcd when a major version upgrade breaks. It took all weekend to fight that fire when it happened to us.
    - p_l 3y ago
      Been there, done that, didn't get a t-shirt but got to yell at some people for setting up with undersized VMs and forgetting to note it anywhere. Haven't had an issue once I fixed sizing.
    - Thaxll 3y ago
      Use managed k8s, problem solved.
      - nprateem 3y ago
        That problem solved, but plenty of other things hiding in that 'simple' setup of just 5 concepts.
        cogman10 3y ago
        Which is why you don't lose your SRE/OPS team just because you k8s. I'd say that if you aren't big enough to have dedicated SRE then k8s is not for you. However, it really only takes 1 or 2 people to manage pretty large clusters with 100s or 1000s of deployments. That's where my company currently finds itself.
axpy906 3y ago
I clicked expecting something a bit more detailed here. What are the best resources to learn simple k8 in 2024?
- jakupovic 3y ago
  Try putting a simple https app on a managed k8s cluster and use google/whatever to figure it out, that should get you started.
- bananapub 3y ago
  https://github.com/kelseyhightower/kubernetes-the-hard-way https://github.com/kelseyhightower/kubernetes-the-hard-way
k8sToGo 3y ago
Interesting that they avoid helm. It is the "plug and play" solution for Kubernetes. However, that is only in theory. My experience with most operators out there was clunky, buggy, or very limited and did not expose everything needed. But I still end up using helm itself with the combination of ArgoCD.
- habitue 3y ago
  Helm is just a mess. If you're going to deploy something from helm, you're better off taking it apart and reconstructing it yourself, rather than depending on it to work like a package manager
  - hobofan 3y ago
    In my experience, if you use first-party charts (= published by the same people that publish the packaged software) that are likely also provided to enterprise customers you'll have a good time (or at least a good starting point). For third-party charts, especially for more niche software I'd also rather avoid them.
- szszrk 3y ago
  I think the important detail here is that he mentions he doesn't use it because of operators. That may mean they tried it in previous major version which used teller. That was quite a long time ago. That being said, helm templates are disgusting and I absolutely hate how easily developers complicate their charts. Even the default empty chart has helpers. Why, on Earth, why? I almost fully relate to OPs aproach to k8s but I think with their simplified approach helm (the current one) could work quite well.
- stackskipton 3y ago
  We avoided Helm as well. We found that Kustomize provides enough templating to cover almost all the common use cases and it's very easy for anyone to check their work, kubectl kustomize > compiled.yaml. FluxCD handles postbuild find and replace. At most places, your cluster configuration is probably pretty set in stone and doesn't vary a ton.
neya 3y ago
There were some "hype cycles" (in Gartner's lingo) that I avoided during my career. The first one was the MongoDB/NoSQL hype - "Let's use NoSQL for everything!" trend. I tried it in a medium sized project and burnt my finger and it was right around when HN was flooded with full of "Why we migrated to MongoDB" stories. The next one was Microservices. Everyone was doing something with microservices and I was just on a good 'ole Ruby on Rails monolith. Again, the HN stories came and went "Why we broke down our simple CRUD app into 534 microservices". The final one was Kubernetes. I was a Cloud consultant in my past life and had to work with a lot of my peers who had the freedom to deploy In any architecture they saw fit. A bunch of them were on Kubernetes and I was just on a standard Compute VM for my clients. We had a requirement from our management that all of us had to take some certification courses so they would be easily to pitch to clients. So, I prepped for one and read about Kubernetes and tried deploying a bunch of applications only to realize it was a very complex piece of moving parts - unnecessarily I may add. I was never able to understand why this was pushed on as normal. It made my decision to not use it only stronger. Over the course of the 5 year journey, my peers' apps would randomly fail and they would be sometimes pulled over the weekends to push fixes to avert the P1 situation whilst I would be casually chilling in a bar with my friends. My compute engine VM, till date, to its credit has only had one P1 situation yet. And that was because the client forgot to renew their domain name. Out of all the 3 hype cycles that I avoided in my career, the Kubernetes is the one I really am thankful of evading the most. This sort of complexity should not be normalised. I know this maybe unpopular opinion on HN, but I am willing to bite the bullet and save my time and my clients' money. So, thanks for the hater's guide. But, I prefer to remain one. I'd rather call a spade one.
- dminor 3y ago
  Early on in the container hype cycle we decided to convert some of our services from VMs to ECS. It was easy to manage and the container build times were so much better than AMI build times. Some time down the road we got acquired, and the company that acquired us ran their services in their own Kubernetes cluster. When we were talking with their two person devops team about our architecture, I explained that we deployed some of our services on ECS. "Have you ever used it?" I asked them. "No, thank goodness" one of them said jokingly. By this time it was clear that Kubernetes had won and AWS was planning its managed Kubernetes offering. I assumed that after I became familiar with Kubernetes I'd feel the same way. After a few months though it became clear that all these guys did was babysit their Kubernetes cluster. Upgrading it was a routine chore and every crisis they faced was related to some problem with the cluster. Meanwhile our ECS deploys continued to be relatively hassle free. We didn't even have a devops team. I grew to understand that managing Kubernetes was fun for them, despite the fact that it was overkill for their situation. They had architected for scale that didn't exist. I felt much better about having chosen a technology that didn't "win".
therealfiona 3y ago
Something struck with me here that I've been thinking about. OP says a human should never wait for a pod. Agreed, it is annoying and sometimes means waiting for an EC2 and the pod. We have jobs that users initiate that use 80+GB of memory and a few dozen cores. We run only one pod per node because the next size up EC2 costs a fortune and performance tops out on our current size. These jobs are triggered via a button click that trigger a lambda that submits a job to the cluster. If it is a fresh node, user has to wait for the 1gb container to download from ECR. But it is the same container that the automated jobs that kick off every few minutes also uses, to rarely is there any waiting. But sometimes there is. Should we be running some sort of clustering job scheduler that gets the job request and distributes work amongst long running pods in the cluster? My fear is that we just creat another layer of complexity and still end up waiting for the EC2, waiting for the pod to download, waiting for the agent now running on this pod to join the work distribution cluster. However, we probably could be more proactive with this because we could spin up an extra pod+EC2 when the work cluster is 1:1 job:ec2. Thoughts? We're in the process of moving to Karpenter, so all this may be solved for us very soon with some clever configuration.
- p_l 3y ago
  If you don't want to change the setup too much, consider running your nodes off an AMI with pre-loaded image. Maybe also ensure how exactly the images are layered, so if necessary you can reduce amount of "first boot patch" download.
- Too 3y ago
  There is a difference between waiting and waiting. For an hourly batch job that already takes 10 minutes to run, the extra time for pod scheduling and container downloading is negligible anyway. What you shouldn’t do is put pod scheduling in places where thousands of users per minute expect sub-second latency. In your case, if the time for starting up the EC2 becomes a bigger factor than the job itself, you can add placeholder pods that just sleep, while requiring exactly that machine config but request 0 cpus, just to make sure it stays online.
- hackermeows 3y ago
  Have you looked into using AWS Batch for these jobs, it can handle this no problem
the_duke 3y ago
I know it's fashionable to hate on Kubernetes these days, and it is overly complex and has plenty problems. But what other solution allows you to: * declarative define your infrastructure * gives you load balancing, automatic recovery and scaling * provides great observability into your whole stack (kubectl, k9s, ...) * has a huge amount of pre-packaged software available (helm charts) * and most importantly: allows you to stand up mostly the same infrastructure in the cloud, on your own servers (k3s), and locally (KIND), and thus doesn't tie you into a specific cloud provider The answer is: there isn't any. Kubernetes could have been much simpler, and probably was intentionally built to not be easy to use end to end. But it's still by far the best we've got.
- dijit 3y ago
  Cloud and terraform gives you those. You’re right that kubernetes is a bit batteries included, and for that its tempting to take it off the shelf because it “does a lot of needed things”, but you don’t need one tool to do all of those things. It is ok to have domain specific processes or utilities to solve those.
  - theossuary 3y ago
    You missed what I think is the most important point in OP's list: it does all of the above in a cloud agnostic way. If I want to move clouds with TF I'm rewriting everything to fit into a new cloud's paradigm. With Kubernetes there's a dozen providers built in (storage, loadbalancing, networking, auto scaling, etc.) or easy to pull in (certificates, KMS secrets, DNS); and they make moving clouds (and more importantly) running locally much easier. Kubernetes is currently the best way to wrap up workloads in a cloud agnostic way. I've written dozens of services for K8s using different deployment mechanisms (Helm, Carvel's kapp, Flux, Kustomize) and I can run them just as easily in my home K8s cluster and in GCP. It's honestly incredible; I don't know of any other cloud tech that lets me do that. One thing I think a lot of people miss too, is how good the concepts around Operators in Kubernetes are. It's hard to see unless you've written some yourself, but the theory around how operators work is very reminiscent of reactive coding in front end frameworks (or robotics closed loop control, what they were originally inspired by). When written well they're extremely resilient and incredibly powerful, and a lot of that power comes from etcd and the established patterns they're written with. I think Kubernetes is really painful sometimes, and huge parts of it aren't great due to limitations of the language it's written in; but I also think it's the best thing available that I can run locally and in a cloud with a FOSS license.
liampulles 3y ago
Good article. I used to be a k8s zealot (both CKAD and CKA certified) but have come to think that the good parts of k8s are the bare essentials (deployments, services, configmaps) and the rest should be left for exceptional circumstances. Our team is happy to write raw YAML and use kustomize, because we prefer keeping the config plain and obvious, but we otherwise pretty much follow everything here.
izietto 3y ago
> Hand-writing YAML. YAML has enough foot-guns that I avoid it as much as possible. Instead, our Kubernetes resource definitions are created from TypeScript with Pulumi. LOL so, rather than linting YAML, bring in a whole programming language runtime plus third party library, adding yet another vendor lock, having to maintain versions, project compiling, moving away from K8S, adding mental overhead...
- p_l 3y ago
  Managing structures in programming language is easier than dealing with finicky optional serialization format. I have drastically reduced the amount of errors, mistakes, bugs, plain old wtf-induced hair pulling, by just mandating avoidance of YAML (and Helm) and using Jsonnet. Sure, there was some up-front work to write library code, but afterwards? I had people introduced to JSonnet with example deployment on one day, and shipping production-ready deployment for another app the next day. Something we couldn't get with yaml.
- paulgb 3y ago
  We use Pulumi for IAC of non-k8s cloud resources too, so it doesn't introduce anything extra. In reality all but the smallest Kubernetes services will want something other than hand-written YAML: Helm-style templating, HCL, etc. TypeScript gives us type safety, and composable type safety. E.g. we have a function that encapsulates our best practices for speccing a deployment, and we get type safety for free across that function call boundary. Can't do that with YAML.
- Aurornis 3y ago
  Most devops disaster stories I’ve heard lately are the result of endless addition of new tools. People join the company, see a problem, and then add another layer of tooling to address it, introducing new problems in the process. Then they leave the company, new people join, see the problems from that new tooling, add yet another layer of tooling, continuing the cycle. I was talking to someone from a local startup a couple weeks ago who was trying to explain their devops stack. The number of different tools and platforms they were using was in the range of 50 different things, and they were asking for advice about how to integrate yet another thing to solve yet another self-inflicted problem. It was as though they forgot what the goal was and started trying to collect as much experience with as many different tools as they could.
nusl 3y ago
k8s is really about you and if it makes sense for your use case. It’s not universally bad or universally good, and I don’t feel that there is a minimum team size required for it to make sense. Managing k8s, for me at least, is a lot easier than juggling multiple servers with potentially different hardware, software, or whatever else. It’s rare that businesses will have machines that are all identical. Trying to keep adding machines to a pool that you manage manually and keep them running can be very messy and get out of control if you’re not on top of it. k8s can also get out of control though it’s also easier to reason about and understand in this context. Eg you have eight machines of varying specs but all they really have installed is what’s required to run k8s, so you haven’t got as much divergence there. You can then use k8s to schedule work across them or ask questions about the machines.
liveoneggs 3y ago
We've found kubernetes to be surprisingly fragile, buggy, inflexible, and strictly imperative. People make big claims but then it's not declarative enough to look up a resource or build a dependency tree and then your context deadline is exceeded.
elktown 3y ago
I think an underestimated issue with k8s (et al) is on a cultural level. Once you let in complex generic things, it doesn't stop there. A chain reaction has started, and before you know it, you've got all kinds of components reinforcing each other, that are suddenly required due to some real, or just perceived, problems that are only there in the first place because of a previous step in the chain reaction. I remember back when the Cloud first started getting a foothold that what people was drawn to was that it would enable reducing complexity of managing the most frustrating things like the load-balancer and the database, albeit at a price of course, but it was still worth it. Stateless app servers however, was certainly not a large maintenance problem. But somehow we've managed to squeeze in things like k8s in the there anyway, we just needed to evangelize microservices to create a problem that didn't exist before. Now that this is part of the "culture" it's hard to even get beyond hand-wavy rationalizations that microservices is a must, assumingly because it's the initial spark that triggered the whole chain reaction of complexity.
- jupp0r 3y ago
  Cloud providers automate things like lease renewals, dealing with customs and part time labor contract compliance disputes for that datacenter in that Asian country that you don't know the language of. I'm constantly fascinated how people handwaivingly underestimate the cost and headaches of actually running on prem global infrastructure.
  - robertlagrant 3y ago
    > I'm constantly fascinated To call a halt to your constant fascination: they don't all have that problem. They still get the complexity of cloudy things regardless when they use one.
    - jupp0r 3y ago
      They also get some of the complexity of cloudy things when they run their own datacenter. In the end you find stuff like OpenStack which becomes its own nightmare universe.
aeturnum 3y ago
> if a human is ever waiting for a pod to start, Kubernetes is the wrong choice. As someone who is always working "under" a particular set of infrastructure choices I want people who write this kind of article to understand something: the people who dislike particular infrastructure systems are by-in-large those who are working under sub-optimal uses of them. No one who has the space to think about "if their infrastructure choices will create an effect" in the future hates any infrastructure system. Their life is good. They can choose and most everyone agrees that any system can be done well. The haters come from being in situations where a system has not been done well - where for whatever combination of reasons they are stuck using a system that's the wrong mix of complex / monitorable / fragile / etc. It's true enough that, if that system had been built with more attention to its needs, that people would not hate it - but that's just not how people come to hate k8s (or any other tool).
deleted 3y ago
[deleted]
shrubble 3y ago
If Kubernetes is the answer ... you very likely asked the wrong questions. Reading about JamSocket and what it does, it seems that it essentially lets you run Docker instances inside the Jamsocket infrastructure. Why not just take Caddy in a clustered configuration, add some modules to control Docker startup/shutdown and reduce your services usage by 50%? As one example.
- paulgb 3y ago
  I’m not sure what you mean by that reducing service usage. The earliest version of the product really was just nginx behind some containers, but we outgrew the functionality of existing proxies pretty quickly. See e.g. keys (https://plane.dev/developing/keys https://plane.dev/developing/keys) which would not be possible with clustered Caddy alone.
  - shrubble 3y ago
    My understanding was that K8s itself has overhead, which ultimately has to be paid for, even if using a managed service (it might be included in the cost of what you pay, of course). I did add the caveat of "with modules" and the idea of sharing values around to different servers would be easy to do, since you have Postgres around as a database to hold those values/statuses.
    - paulgb 3y ago
      HTTP proxying is not much of our codebase. I wouldn’t want to shoehorn what we’re doing into being a module of a proxy service just to avoid writing that part. That proxy doesn’t run on Kubernetes currently anyway, so it wouldn’t change anything we currently use Kubernetes for.
siliconc0w 3y ago
IMO the big win with Kubernetes is helm or operators. If you're going to pay the complexity costs you might as well get the wins which is essentially a huge 'app-store' of popular infrastructure components and an entirely programmatic way to manage your operations (deployments, updates, fail-overs, backups, etc). For example if you want to setup something complex like Ceph - Rook is a really nice way to do that. It's a very leaky abstraction so you aren't hiding all the complexity of Ceph but the declarative interface is generally a much nicer way to manage Ceph than a boatload of ansible scripts or generally what we had before. The key to understand is that helm or operators don't magically make infrastructure a managed 'turn-key' appliances, you do generally need to understand how the thing works.
jakupovic 3y ago
This article talks about using k8s but trying not to use it as much as possible. First example being operators, this is the underlying mechanism that makes k8s possible. To me taking a stance not to use operators but use k8s is less than optimal, or plainly stupid. The whole stack is built on operators, which you inherently trust as you use k8s, but choosing not to use them. Sorry but this is hard to read. The only thing I learned is about Caddy as a cert-manager replacement, even though I have used, extended and been pretty happy with cert-manager. The rest is hard to read ;(.
hintymad 3y ago
When I checked out an operator repo for some stateful services, say, elasticsearch, the repo most likely would contain 10s of thousands of lines of YAML and 10s of thousands lines of Go code. Is this due to essential complexity of implementing auto-pilot of a complex service, or is it due to massive integration with k8s' operators framework?
patmcc 3y ago
Missed opportunity to title this the h8r's guide to k8s.
kube-system 3y ago
The people who dislike kubernetes are, in my experience, people who don’t need to do all of the things kubernetes does. If you just need to run an application, it’s not what you want.
erulabs 3y ago
I suppose I’m the guy pushing k8s on midsized companies. If there have been unhappy engineers along the way - they’ve by the vast majority stayed quiet and lied about being happier on surveys. Yes, k8s is complex. The tool matches the problem: complex. But having a standard is so much better than having a somewhat simpler undocumented chaos. “Kubectl explain X” is a thousand times better than even AWS documentation, which in turn was a game changer compared to that-one-whiteboard-above-Dave’s-desk. Standards are tricky, but worth the effort. Personally I’m also very judicious with operators and CRDs - both can be somewhat hidden to beginners. However, the operator pattern is wonderful. Another amazing feature is ultra simple leader election - genuinely difficult outside of k8s, a 5 minute task inside. I agree with Paul’s take here tho of at least being extremely careful about which operators you introduce. At any rate, yes k8s is more complex than your bash deploy script, of course it is. It’s also much more capable and works the same way as it did at all your developers previous jobs. Velocity is the name of the game!
- paulgb 3y ago
  Good point about k8s vs. AWS docs — a lot of the time people say “just use ECS” or the AWS service of the day, and it will invariably be more confusing to me and more vendor-tied than just doing the thing in k8s.
  - p_l 3y ago
    And then if you're unlucky you might hit one of the areas where the AWS documentation has a "teaser" about some functionality that is critical for your project, you spend months looking for the rest of the documentation when initial foray doesn't work, and the highly paid AWS-internal consultants disappear into thin air when asked about the features. So nearly a year later you end up writing the whole feature from scratch yourself.
- pclmulqdq 3y ago
  I have to say that I don't believe the problem is all that complex unless you make it hard. But on the flip side, if you're a competent Kubernetes person, the correct Kubernetes config is also not that complex. I think a lot of the reaction here is a result of the age-old issues of "management is pushing software on me that I don't want" and people adopting it without knowing how to use it because it's considered a "best practice." In other words, the reaction you probably have to an Oracle database is the same reaction that others have to Kubernetes (although Oracle databases are objectively crappy).
throwitaway222 3y ago
Article makes a good point. Allow k8s disallow any service meshes.
bedobi 3y ago
the "best" infra i ever had was a gig where we * built a jar (it was a jvm shop) * put it on a docker image * put that on an ami * then had a regular aws load balancer that just combined the ami with the correctly specced (for each service) ec2 instances to cope with load it was SIMPLE + it meant we could super easily spin up the previous version ami + ec2s in case of any issues on deploys (in fact, when deploying, we could keep the previous ones running and just repoint the load balancer to them) ps putting the jar on a docker image was arguably unnecessary, we did it mostly to avoid "it works on my machine" style problems
racheljaneville 3y ago
[dead]
remram 3y ago
My main issue with Kubernetes is the cluster-scoped Custom Resource Definition (CRD). We are here because we wanted a way to deal with software that each need different versions of libraries or configuration, so we invented containers. Then we wanted to run multiple services, so we invented orchestrators. But now applications are deployed with cluster-scoped operators that depend on cluster-scoped definitions, and we need to run separate services in separate cluster. It's like with need to create containerization for Kubernetes apps again.
vanillax 3y ago
great article and it helped me validate some of my own challenges with k8s

288 points by paulgb 3y ago

My current company is split... maybe 75/25 (at this point) between Kubernetes and a bespoke, Ansible-driven deployment system that manually runs Docker containers on nodes in an AWS ASG and will take care of deregistering/reregistering the nodes with the ALB while the containers on a given node are getting futzed with. The Ansible method works remarkably well for it's age, but the big thing I use to convince teams to move to Kubernetes is that we can take your peak deploy times from, say, a couple hours down to a few minutes, and you can autoscale far faster and more efficiently than you can with CPU-based scaling on an ASG. From service teams that have done the migrations, the things I hear consistently though are: - when a Helm deploy fails, finding the reason why is a PITA (we run with --atomic so it'll rollback on a failed deploy. What failed? Was it bad code causing a pod to crash loop? Failed k8s resource create? who knows! have fun finding out!) - they have to learn a whole new way of operating, particularly around in-the-moment scaling. A team today can go into the AWS Console at 4am during an incident and change the ASG scaling targets, but to do that with a service running in Kubernetes means making sure they have kubectl (and it's deps, for us that's aws-cli) installed and configured, AND remembering the `kubectl scale deployment X --replicas X` syntax. [Both of those things are very much fixable]

dpflan 3y ago
HPAs and VPAs are useful k8s concepts for your auto-scaling needs.
- t3rabytes 3y ago
  HPA is useful until your maxReplicas count is set too low and you're already tapped out.
  - cogman10 3y ago
    Sort of a learning thing though right? Like, if you find maxReplicas is too low you move that number up until it isn't right? This is different from waking people up at 4am frequently to bump up the number of replicas.
    - deleted 3y ago
      [deleted]
  - dpflan 3y ago
    You can edit your HPA live, in maybe as many commands or keystrokes as manually scaling…until you commit the change to your repo of configs.
makestuff 3y ago
I haven’t used kubernetes in a few years, but do they have a good UI for operations? Your example of the AWS console where you can just log in and scale something in the UI but for kubernetes. We run something similar on AWS right now, during an incident we log into the account with admin access to modify something and then go back to configure that in the CDK post incident.
- t3rabytes 3y ago
  AWS has a UI for resources in the cluster but it relies on the IAM role you're using in the console to have configured perms in the cluster, and our AWS SSO setup prevents that from working properly (this isn't usually the case for AWS SSO users, it's a known quirk of our particular auth setup between EKS and IAM -- we'll fix it sometime).
- adhamsalama 3y ago
  https://k8slens.dev https://k8slens.dev
cogman10 3y ago
For scaling, have you tried using either an HPA or keda? We've had pretty good success with simple HPAs.
- t3rabytes 3y ago
  Yep, I'd say >half of the teams with K8s services have adopted KEDA, but we've got some HPA stragglers for sure.
  - dpflan 3y ago
    I have to say that when you have more buy in from delivery teams and adoption of HPAs your system can become more harmonious overall. Each team can monitor and tweak their services, and many services are usually connected upstream or downstream. When more components can ebb and flow according to the compute context then the system overall ebbs and flows better. #my2cents
freedomben 3y ago
Personally, I don't like Helm. I think for the vast majority of usecases where all you need is some simple templating/substitution, it just introduces way more complexity and abstraction than it is worth. I've been really happy with just using `envsubst` and environment variables to generate a manifest at deploy time. It's easy with most CI systems to "archive" the manifest, and it can then be easily read by a human or downloaded/applied manually for debugging with. Deploys are also just `cat k8s/${ENV}/deploy.yaml | envsubt > output.yaml && kubectl apply -f output.yaml` I've also experimented with using terraform. It's actually been a good enough experience that I may go fully with terraform on a new project and see how it goes.
- linuxftw 3y ago
  You might like kubernetes kustomize if you don't care for helm (IMO, just embrace helm, you can keep your charts very simple and it's straight forward). Kustomize takes a little getting used to, but it's a nice abstraction and widely used. I cannot recommend terraform. I use it daily, and daily I wish I did not. I think Pulumi is the future. Not as battle tested, but terraform is a mountain of bugs anyway, so it can't possibly be worse. Just one example where terraform sucks: You cannot both deploy a kubernetes cluster (say an EKS/AKS cluster) and then use kubernetes_manifest provider in a single workspace. You must do this across two separate terraform runs.
jonathaneunice 3y ago
The problem with bespoke, homegrown, and DIY isn't that the solutions are bad. Often, they are quite good—excellent, even, within their particular contexts and constraints. And because they're tailored and limited to your context, they can even be quite a bit simpler. The problem is that they're custom and homegrown. Your organization alone invests in them, trains new staff in them, is responsible for debugging and fixing when they break, has to re-invest when they no longer do all the things you want. DIY frameworks ultimately end up as byzantine and labyrinthine as Kubernetes itself. The virtue of industry platforms like Kubernetes is, however complex and only half-baked they start, over time the entire industry trains on them, invests in them, refines and improves them. They benefit from a long-term economic virtuous cycle that DIY rarely if ever can. Even the longest, strongest, best-funded holdouts for bespoke languages, OSs, and frameworks—aerospace, finance, miltech—have largely come 'round to COTS first and foremost.

roschdal 3y ago

[flagged]

api 3y ago

I understand where most of the complexity in K8S comes from, but it still horrifies and offends me and I hate it. But I don't think it's Kubernetes' fault directly. I think the problem is deeper in the foundation. It comes from the fact that we are trying to build modern, distributed, high availability, incrementally upgradeable, self-regulating systems on a foundation of brittle clunky 1970s operating systems that are not designed for any of that. The whole thing is a bolt-on that has to spend a ton of time working around the limitations of the foundation, and it shows. Unfortunately there seems to be zero interest in fixing that and so much sunk cost in existing Unix/Posix designs that it seems like we are completely stuck with a basic foundation of outdated brittleness. What I think we need: * An OS that runs hardware-independent code (WASM?) natively and permits things like hot updates, state saving and restoration, etc. Abstract away the hardware. * Native built-in support for clustering, hot backups, live process migration between nodes, and generally treating hardware as a pure commodity in a RAIN (redundant array of inexpensive nodes) configuration. * A modern I/O API. Posix I/O APIs are awful. They could be supported for backward compatibility via a compatibility library. * Native built-in support for distributed clustered storage with high availability. Basically a low or zero config equivalent of Ceph or similar built into the OS as a first class citizen. * Immutable OS that installs almost instantly on hardware, can be provisioned entirely with code, and where apps/services can be added and removed with no "OS rot." The concept of installing software "on" the OS needs to be killed with fire. * Shared distributed network stack where multiple machines can have the same virtual network interfaces, IPs, and open TCP connections can migrate. Built-in load balancing. I'm sure people around here can think of more ideas that belong in this list. These are not fringe things that are impossible to build. Basically you should have an immutable image OS that turns many boxes into one box and you don't have to think about it. Storage is automatically clustered. Processes automatically restart or, if a hardware fault is detected in time, automatically migrate. There were efforts to build such things (Mosix, Plan 9, etc.) but they were bulldozed by the viral spread of free Unix-like OSes that were "good enough." Edit: That being said, I'm not saying Kubernetes is good software either. The core engine is actually decent and as the OP said has a lot of complexity that's needed to support what it does. The ugly nasty disgusting parts are the config interface, clunky shit like YAML, and how generally arcane and unapproachable and ugly the thing is to actually use. I just loathe software like this. I feel the same way about Postgres and Systemd. "Algorithmically" they are fine, but the interface and the way you use them is arcane and makes me feel like I'm using a 70s mainframe on a green VT220 monitor. Either these things are designed by the sorts of "hackers" who like complexity and arcane-ness, or they're hacks that went viral and matured into global infrastructure without planning. I think it's a mix of both... though in the case of Postgres it's also that the project is legitimately old. It feels like old-school Unix clunkware because it is.

Spivak 3y ago

This matches our experience as well. As long as you treat your managed k8s cluster as autoscaling-group as-a-service you'll do fine. k8s's worst property is that it's a cleverness trap. You can do anything in k8s whether it's sane to do so or not. The biggest guardrail against falling into is managing your k8s with terraform-ish so that you don't find yourself in a spot where "effort to do it right" >> "effort to hack in YAML" and finding your k8s cluster becoming spaghetti.

x86x87 3y ago
Why not just use an autoscaling group? Re: cleverness trap. I feel like this is the tragedy of software development. We like to be seen as clever. We are doing "hard" things. I have way more respect for engineers that do "simple" things that just work using boring tech and factor in whole lifecycle of the product.
- p_l 3y ago
  > Why not just use an autoscaling group? Not everyone has money to burn, even back in ZIRP era. And before you trot out wages for experienced operations team - I've regularly dealt with it being cheaper to pay for one or two very experienced people than deal with AWS bill. For the very simple reason that cloud provider's prices are scaled to US market and not everyone has US money levels.
- Spivak 3y ago
  Sorry, I could have explained that better. The biggest value add that k8s has is that it gives you as many or as few autoscaling groups as you need at a given time using only a single pool (or at least fewer pools) of heterogeneous servers. There's lots of fine print here but it really does let you run the same workloads on less hardware and to me that's the first and last reason you should be using it. I wouldn't start with k8s and instead opt for ASGs until you reach the point where you look at your AWS account and see a bunch of EC2 instances sitting underutilized.

treesciencebot 3y ago

> Above I alluded to the fact that we briefly ran ephemeral, interactive, session-lived processes on Kubernetes. We quickly realized that Kubernetes is designed for robustness and modularity over container start times. Is there a clear example of this? E.g. is kubernetes inherently unable to start a pod (assuming the same sequence of events, e.g. warm/cold image with streaming enabled) under 500ms, 1s etc? I am asking this as someone who spent quite a bit of time and wasn't able to bring it down 2s< mark, which eventually led us to rewrite the latency sensitive parts to use Nomad. But we are currently in a state where we are re-considering kubernetes for its auxilary tooling benefits and would love to learn more if anyone had experiences with starting and stopping thousands of pods with the lowest possible latencies without caring for utilization or placement but just observable boot latencies.

p_l 3y ago
You'd have to ensure that a) preload all images of course b) there's enough of nodes with enough capacity c) the pods don't use anything that has possible longer latency (high latency CSI etc.) d) you might want to write custom scheduler for your workloads (it could take into account what images are preloaded where, etc)
paulgb 3y ago
I do believe that with the right knowledge of Kubernetes internals it's probably possible to get k8s cold start times competitive with where we landed without Kubernetes (generally subsecond, often under 0.5s depending on how much the container does before passing a health check), but we'd have to understand k8s internals really well and would have ended up throwing out much of what already existed. And we'd probably end up breaking most of the reasons for using Kubernetes in the first place in the process.
- p_l 3y ago
  Not much internals needed, but actual in depth understanding of Pod kube-api plus at least basics of how scheduler, kubelet, and kubelet drivers interact. Big possible win is custom scheduling, but barely anyone seems to know it exists

fifilura 3y ago

There is nothing wrong with k8s it is a nice piece of technology. But the article trending here a couple of days ago describes it well. https://www.theolognion.com/p/company-forgets-why-they-exist-after-11-week-migration-to-kubernetes https://www.theolognion.com/p/company-forgets-why-they-exist... It is complex enough to make the k8s maintainers the heroes of the company. And this is where things tend to go sideways. It has enough knobs and levers to distract the project from what they are actually trying to achieve.

cedws 3y ago
I see Kubernetes the same way as git. Elegant fundamental design, but the interface to it is awful. Kubernetes is designed to solve big problems and if you don't have those problems, you're introducing a tonne of complexity for very little benefit. An ideal orchestrator would be more composable and not introduce more complexity than needed for the scale you're running at. I'd really like to see a modern alternative to K8S that learns from some of its mistakes.
- pphysch 3y ago
  Git is a much more subtle abstraction than k8s though. You can be blissfully unaware that a directory is a git repo, and still read/patch files. You cannot pretend k8s doesn't exist in a k8s system.

PUSH_AX 3y ago

I was once talking to an ex google site reliability engineer. He said there are maybe a handful of companies in the world that _need_ k8s. I tend to agree. A lot of people practice hype driven development.

x86x87 3y ago
I tend to agree. K8s makes a lot of sense if you are running your own bare metal servers at scale. If you are already using the cloud, maybe leverage abstraction already available in that context.
- candiddevmike 3y ago
  You either recreate a less reliable version of kubernetes for workload ops or you go all in on your cloud provider and hope they'll be responsible for your destiny. Vanilla Kubernetes is just enough abstraction to avoid both of those situations.
  - x86x87 3y ago
    You cannot really be cloud agnostic these days - even when using k8s. So learning to use the capabilities the cloud provides is key.
    - p_l 3y ago
      Doesn't really mesh with my experience, especially the longer k8s been out. It can be cheaper to depend on cloud provider to ship some features, but with tools like crossplane you can abstract that out so developers can just "order" a database service etc. for their application.
  - PUSH_AX 3y ago
    Is “hope” the new replacement for SLAs? Or am I missing something with that statement?
    - 3y ago

rwmj 3y ago

> It’s also worth noting that we don’t administer Kubernetes ourselves This is the key point. Even getting to the point where I could install Kubernetes myself on my own hardware took weeks, just understanding what hardware was needed and which of the (far too many) different installers I had to use.

LegibleCrimson 3y ago
I found k3s pretty easy to spin up.

hobofan 3y ago

OT: Can something be done about HN commenting culture so that the comments stay more on topic? Some technologies (like Kubernetes) tend to attract discussions where half of the commenters completely ignore the original article, so we end up having a weekly thread about Kubernetes where the points of the article (which are interesting) are not able to be discussed because they are drowned out by the same unstructured OT discussions? At the time of this posting there are ~20 comments with ~2 actually having anything to do with the points of the article rather than Kubernetes in general.

cogman10 3y ago
Having read the article, isn't the point of the article kubernetes in general and what the author prescribes you sign up for/avoid? Discussions of k8s pitfalls and successes in general seems to be very much in line with what the article is advocating. And, to that point, there's frankly just not a whole lot interesting in this article for discussion "We avoid yaml and operators"... Neat.
- hobofan 3y ago
  > Having read the article, isn't the point of the article kubernetes in general and what the author prescribes you sign up for/avoid? Yeah, and I think that provides a good basis to discussion, where people can critique/discuss whether the evaluation that the author has made are correct (which a few comments are doing). At the same time a lot of that discussion is being displaced by what I would roughly characterize as "general technology flaming" which isn't going anywhere productive.
geodel 3y ago
Huh, this article has hardly anything deep, technical, thought provoking or unique compared to ten thousand other Kubernetes articles. I am rather happy that people are having general purpose discussion about K8s.
freedomben 3y ago
What you're seeing is the early-crowd. With most (not all) posts, comments will eventually rise to the top that are more what you're looking for. IME it usually takes a couple hours. If it's a post where I really want to read the relevant comments, I'll usually come back at least 8 to 12 hours later and there's usually some good ones to choose from. Even topics like Apple that attract the extreme lovers and haters tend to trend this direction

Kab1r 3y ago

I almost feel attacked for using plain yaml, helm, cert-manager AND the ingress api just for personal homelab shenanigans.

cogman10 3y ago
Yeah, I disagree with the OP on the dangers there. They work fairly well for us and aren't the source of headache. Though, I still try and teach my dev teams that "just because bitnami puts in variables everywhere, doesn't mean you need to. We aren't trying to make these apps deployable on homelabs."

__MatrixMan__ 3y ago

> But we often do multiple deploys per day, and when our products break, our customer’s products break for their users. Even a minute of downtime is noticed by someone. Kubernetes might be the right tool for the job if we accept that this is a necessary evil. But maybe it's not? The idea that I might fail to collaborate with you because a third party failed because a fourth party failed kind of smells like a recipe for software that breaks all the time.

paulgb 3y ago
It really comes down to, I don't ever want to have the conversation “is this a good time to deploy, or should we wait until tonight when there’s less usage”. We have had some periods where our system was more fragile, and planning our days around the least-bad deployment window was a time suck, and didn't scale to our current reality of round-the-clock usage.
- hellcow 3y ago
  You can achieve this without k8s, though. If your goal is, "I want zero-downtime deploys," that alone is not sufficient reason to reach for something as massively complex as k8s. Set up a reverse proxy and do blue-green deploys behind it.
  - paulgb 3y ago
    > Set up a reverse proxy and do blue-green deploys behind it. That's what I currently use Kubernetes for. What stack are you proposing instead?
    - sureglymop 3y ago
      If you only need zero downtime deployments, compose and traefik/caddy are enough. If you need to replicate storage, share networks and otherwise share resources across multiple hosts, kubernetes is better suited. But you'll also have much less control with compose, e.g. no limiting of egress/ingress and more.

deleted 3y ago

[deleted]

teeray 3y ago

Is there something like a k1s? What I’d love is “run this set of containers on this machine. If the machine goes down, I don’t care—I will fix it.” If it wired into nginx or caddy as well, so much the better. Something like that for homelab use would be wonderful.

eropple 3y ago
You've basically described k3s, I think. I run it in my homelab (though I am enough of a tryhard to have multiple control planes) as well as on a couple of cloud servers as container runtimes (trading some overhead for consistency). k3s really hammers home the "kubernetes is a set of behaviors, not a set of tools" stuff when you realize you can ditch etcd entirely and use sqlite if you really want to, and is a good learning environment.
szszrk 3y ago
That's basically just a docker-compose. If you want something crazy all-in-one for homelab check out https://github.com/azukaar/Cosmos-Server https://github.com/azukaar/Cosmos-Server
silverquiet 3y ago
Docker Compose probably fits the bill for that. They also have a built in minimalist orchestrator called Swarm if you do want to extend to multiple machines. I suppose it's considered "dead" since Kubernetes won mindshare, but it still gets updates.
morbicer 3y ago
For homelab, Docker compose should be enough For something more production oriented https://github.com/basecamp/kamal https://github.com/basecamp/kamal
pheatherlite 3y ago
Docker bare bones or docker compose. Run as systemd services and have docker run the container as a service account. Manual orchestration is all you need. Anything else like rancher or whatever are just fluff.
3y ago

throwaway892238 3y ago

When people call Kubernetes a "great piece of technology", I find it the same as people saying the United States is the "greatest country in the world". Oh yeah? Great in what sense? Large? Absolutely. Powerful? Definitely. But then the adjectives sort of take a turn... Complicated? Expensive? Problematic? Threatening? A quagmire? You betcha. If there were an alternative to Kubernetes that were just 10% less confusing, complicated, opaque, monolithic, clunky, etc, we would all be using it. But because Kubernetes exists, and everyone is using it, there's no point in trying to make an alternative. It would take years to reach feature parity, and until you do, you can't really switch away. It's like you're driving an 18-wheeler, and you think it kinda sucks, but you can't just buy and then drive a completely different 18 wheeler for only a couple of your deliveries. You probably will end up using K8s at some point in the next 20 years. There's not really an alternative that makes sense. As much as it sucks, and as much as it makes some things both more complicated and harder, if you actually need everything it provides, it makes no sense to DIY, and there is no equivalent solution.

p_l 3y ago
People forgot just how much of a mess Mesos environment was in comparison. And often pushed Nomad to this day surprises me with randomly missing a feature or two that turns out to be impactful enough to want to deal with more complexity because ultimately the result was less complexity in total.

madduci 3y ago

I don't get most of the blame and reasoning. Sure, everyone has their own product and experience and it's fine to express it, but I don't get the usage of other decisions such as "no to services meshes", "no to helm" and many more. You don't want to ideally reinvent the wheel for every workload you need (say you need a OIDC endpoint, an existing application): you are tempted to write everything from scratch by yourself, which is also fine, but the point is: why? Many products deliver their own Helm package. And if you are sick of writing YAML, I would look for Terraform over Pulumi, for the reason that you use the same tool for bringing up Infrastructure and then workloads. Kubernetes itself isn't easy to be used, in many cases you don't need it, but it might bring you nice things straight out of the box with less pain than other tooling (e.g. zero downtime deployments)

p_l 3y ago
The problem with Helm is that it did the one thing you should not do, and refused to fix it even when their promised to. They do text-replacement templating for YAML. I have once spent a month, being quite experienced k8s wrangler, trying to figure out why Helm 2 was timing out, only to finally trace it down to how sometimes we would get wrong number of spaces in some lines.
eropple 3y ago
I admit that I use some Helm stuff in my home environment, but for production I'm genuinely worried about the need to support whatever they've thrown into it. At minimum I'm going to have to study the chart and understand exactly what they propose to open-palm slam into my cluster, and for many/most applications at that point it might genuinely be worth just writing a manifest myself. Not always. Some applications are genuinely complex and need to be! But often, this has been the case for me. For all my stuff, though, I use kustomize and I'm pretty happy with it; it's too stupid for me to be clever, and this is good. Service meshes are a different kettle of fish. hey add exciting new points of failure where they need not exist, and while there are definitely use cases for them, I'd default to avoiding them until somebody proves the need for one.
org3 3y ago
Can’t Pulumi be used to bring up infra and workloads as well?

pheatherlite 3y ago

Why are people still scared of k8s? At certain jobs thresholds, it is worth every ounce of effort to maintain it. Better yet, go managed.

cogman10 3y ago
I honestly don't understand it either. Familiarity? K8s has like, what, 5 big concepts to know and once you are there the other concepts (generally) just build from there. - Containers - Pods - Deployments - Services - Ingresses There are certainly other concepts you can learn, but you aren't often dealing with them (just like you aren't dealing with them when working with something like docker compose).
- nprateem 3y ago
  Good luck fixing etcd when a major version upgrade breaks. It took all weekend to fight that fire when it happened to us.
  - p_l 3y ago
    Been there, done that, didn't get a t-shirt but got to yell at some people for setting up with undersized VMs and forgetting to note it anywhere. Haven't had an issue once I fixed sizing.
  - Thaxll 3y ago
    Use managed k8s, problem solved.
    - nprateem 3y ago
      That problem solved, but plenty of other things hiding in that 'simple' setup of just 5 concepts.
      - cogman10 3y ago
        Which is why you don't lose your SRE/OPS team just because you k8s. I'd say that if you aren't big enough to have dedicated SRE then k8s is not for you. However, it really only takes 1 or 2 people to manage pretty large clusters with 100s or 1000s of deployments. That's where my company currently finds itself.

axpy906 3y ago

I clicked expecting something a bit more detailed here. What are the best resources to learn simple k8 in 2024?

jakupovic 3y ago
Try putting a simple https app on a managed k8s cluster and use google/whatever to figure it out, that should get you started.
bananapub 3y ago
https://github.com/kelseyhightower/kubernetes-the-hard-way https://github.com/kelseyhightower/kubernetes-the-hard-way

k8sToGo 3y ago

Interesting that they avoid helm. It is the "plug and play" solution for Kubernetes. However, that is only in theory. My experience with most operators out there was clunky, buggy, or very limited and did not expose everything needed. But I still end up using helm itself with the combination of ArgoCD.

habitue 3y ago
Helm is just a mess. If you're going to deploy something from helm, you're better off taking it apart and reconstructing it yourself, rather than depending on it to work like a package manager
- hobofan 3y ago
  In my experience, if you use first-party charts (= published by the same people that publish the packaged software) that are likely also provided to enterprise customers you'll have a good time (or at least a good starting point). For third-party charts, especially for more niche software I'd also rather avoid them.
szszrk 3y ago
I think the important detail here is that he mentions he doesn't use it because of operators. That may mean they tried it in previous major version which used teller. That was quite a long time ago. That being said, helm templates are disgusting and I absolutely hate how easily developers complicate their charts. Even the default empty chart has helpers. Why, on Earth, why? I almost fully relate to OPs aproach to k8s but I think with their simplified approach helm (the current one) could work quite well.
stackskipton 3y ago
We avoided Helm as well. We found that Kustomize provides enough templating to cover almost all the common use cases and it's very easy for anyone to check their work, kubectl kustomize > compiled.yaml. FluxCD handles postbuild find and replace. At most places, your cluster configuration is probably pretty set in stone and doesn't vary a ton.

neya 3y ago

There were some "hype cycles" (in Gartner's lingo) that I avoided during my career. The first one was the MongoDB/NoSQL hype - "Let's use NoSQL for everything!" trend. I tried it in a medium sized project and burnt my finger and it was right around when HN was flooded with full of "Why we migrated to MongoDB" stories. The next one was Microservices. Everyone was doing something with microservices and I was just on a good 'ole Ruby on Rails monolith. Again, the HN stories came and went "Why we broke down our simple CRUD app into 534 microservices". The final one was Kubernetes. I was a Cloud consultant in my past life and had to work with a lot of my peers who had the freedom to deploy In any architecture they saw fit. A bunch of them were on Kubernetes and I was just on a standard Compute VM for my clients. We had a requirement from our management that all of us had to take some certification courses so they would be easily to pitch to clients. So, I prepped for one and read about Kubernetes and tried deploying a bunch of applications only to realize it was a very complex piece of moving parts - unnecessarily I may add. I was never able to understand why this was pushed on as normal. It made my decision to not use it only stronger. Over the course of the 5 year journey, my peers' apps would randomly fail and they would be sometimes pulled over the weekends to push fixes to avert the P1 situation whilst I would be casually chilling in a bar with my friends. My compute engine VM, till date, to its credit has only had one P1 situation yet. And that was because the client forgot to renew their domain name. Out of all the 3 hype cycles that I avoided in my career, the Kubernetes is the one I really am thankful of evading the most. This sort of complexity should not be normalised. I know this maybe unpopular opinion on HN, but I am willing to bite the bullet and save my time and my clients' money. So, thanks for the hater's guide. But, I prefer to remain one. I'd rather call a spade one.

dminor 3y ago
Early on in the container hype cycle we decided to convert some of our services from VMs to ECS. It was easy to manage and the container build times were so much better than AMI build times. Some time down the road we got acquired, and the company that acquired us ran their services in their own Kubernetes cluster. When we were talking with their two person devops team about our architecture, I explained that we deployed some of our services on ECS. "Have you ever used it?" I asked them. "No, thank goodness" one of them said jokingly. By this time it was clear that Kubernetes had won and AWS was planning its managed Kubernetes offering. I assumed that after I became familiar with Kubernetes I'd feel the same way. After a few months though it became clear that all these guys did was babysit their Kubernetes cluster. Upgrading it was a routine chore and every crisis they faced was related to some problem with the cluster. Meanwhile our ECS deploys continued to be relatively hassle free. We didn't even have a devops team. I grew to understand that managing Kubernetes was fun for them, despite the fact that it was overkill for their situation. They had architected for scale that didn't exist. I felt much better about having chosen a technology that didn't "win".

therealfiona 3y ago

Something struck with me here that I've been thinking about. OP says a human should never wait for a pod. Agreed, it is annoying and sometimes means waiting for an EC2 and the pod. We have jobs that users initiate that use 80+GB of memory and a few dozen cores. We run only one pod per node because the next size up EC2 costs a fortune and performance tops out on our current size. These jobs are triggered via a button click that trigger a lambda that submits a job to the cluster. If it is a fresh node, user has to wait for the 1gb container to download from ECR. But it is the same container that the automated jobs that kick off every few minutes also uses, to rarely is there any waiting. But sometimes there is. Should we be running some sort of clustering job scheduler that gets the job request and distributes work amongst long running pods in the cluster? My fear is that we just creat another layer of complexity and still end up waiting for the EC2, waiting for the pod to download, waiting for the agent now running on this pod to join the work distribution cluster. However, we probably could be more proactive with this because we could spin up an extra pod+EC2 when the work cluster is 1:1 job:ec2. Thoughts? We're in the process of moving to Karpenter, so all this may be solved for us very soon with some clever configuration.

p_l 3y ago
If you don't want to change the setup too much, consider running your nodes off an AMI with pre-loaded image. Maybe also ensure how exactly the images are layered, so if necessary you can reduce amount of "first boot patch" download.
Too 3y ago
There is a difference between waiting and waiting. For an hourly batch job that already takes 10 minutes to run, the extra time for pod scheduling and container downloading is negligible anyway. What you shouldn’t do is put pod scheduling in places where thousands of users per minute expect sub-second latency. In your case, if the time for starting up the EC2 becomes a bigger factor than the job itself, you can add placeholder pods that just sleep, while requiring exactly that machine config but request 0 cpus, just to make sure it stays online.
hackermeows 3y ago
Have you looked into using AWS Batch for these jobs, it can handle this no problem

the_duke 3y ago

I know it's fashionable to hate on Kubernetes these days, and it is overly complex and has plenty problems. But what other solution allows you to: * declarative define your infrastructure * gives you load balancing, automatic recovery and scaling * provides great observability into your whole stack (kubectl, k9s, ...) * has a huge amount of pre-packaged software available (helm charts) * and most importantly: allows you to stand up mostly the same infrastructure in the cloud, on your own servers (k3s), and locally (KIND), and thus doesn't tie you into a specific cloud provider The answer is: there isn't any. Kubernetes could have been much simpler, and probably was intentionally built to not be easy to use end to end. But it's still by far the best we've got.

dijit 3y ago
Cloud and terraform gives you those. You’re right that kubernetes is a bit batteries included, and for that its tempting to take it off the shelf because it “does a lot of needed things”, but you don’t need one tool to do all of those things. It is ok to have domain specific processes or utilities to solve those.
- theossuary 3y ago
  You missed what I think is the most important point in OP's list: it does all of the above in a cloud agnostic way. If I want to move clouds with TF I'm rewriting everything to fit into a new cloud's paradigm. With Kubernetes there's a dozen providers built in (storage, loadbalancing, networking, auto scaling, etc.) or easy to pull in (certificates, KMS secrets, DNS); and they make moving clouds (and more importantly) running locally much easier. Kubernetes is currently the best way to wrap up workloads in a cloud agnostic way. I've written dozens of services for K8s using different deployment mechanisms (Helm, Carvel's kapp, Flux, Kustomize) and I can run them just as easily in my home K8s cluster and in GCP. It's honestly incredible; I don't know of any other cloud tech that lets me do that. One thing I think a lot of people miss too, is how good the concepts around Operators in Kubernetes are. It's hard to see unless you've written some yourself, but the theory around how operators work is very reminiscent of reactive coding in front end frameworks (or robotics closed loop control, what they were originally inspired by). When written well they're extremely resilient and incredibly powerful, and a lot of that power comes from etcd and the established patterns they're written with. I think Kubernetes is really painful sometimes, and huge parts of it aren't great due to limitations of the language it's written in; but I also think it's the best thing available that I can run locally and in a cloud with a FOSS license.

liampulles 3y ago

Good article. I used to be a k8s zealot (both CKAD and CKA certified) but have come to think that the good parts of k8s are the bare essentials (deployments, services, configmaps) and the rest should be left for exceptional circumstances. Our team is happy to write raw YAML and use kustomize, because we prefer keeping the config plain and obvious, but we otherwise pretty much follow everything here.

izietto 3y ago

> Hand-writing YAML. YAML has enough foot-guns that I avoid it as much as possible. Instead, our Kubernetes resource definitions are created from TypeScript with Pulumi. LOL so, rather than linting YAML, bring in a whole programming language runtime plus third party library, adding yet another vendor lock, having to maintain versions, project compiling, moving away from K8S, adding mental overhead...

p_l 3y ago
Managing structures in programming language is easier than dealing with finicky optional serialization format. I have drastically reduced the amount of errors, mistakes, bugs, plain old wtf-induced hair pulling, by just mandating avoidance of YAML (and Helm) and using Jsonnet. Sure, there was some up-front work to write library code, but afterwards? I had people introduced to JSonnet with example deployment on one day, and shipping production-ready deployment for another app the next day. Something we couldn't get with yaml.
paulgb 3y ago
We use Pulumi for IAC of non-k8s cloud resources too, so it doesn't introduce anything extra. In reality all but the smallest Kubernetes services will want something other than hand-written YAML: Helm-style templating, HCL, etc. TypeScript gives us type safety, and composable type safety. E.g. we have a function that encapsulates our best practices for speccing a deployment, and we get type safety for free across that function call boundary. Can't do that with YAML.
Aurornis 3y ago
Most devops disaster stories I’ve heard lately are the result of endless addition of new tools. People join the company, see a problem, and then add another layer of tooling to address it, introducing new problems in the process. Then they leave the company, new people join, see the problems from that new tooling, add yet another layer of tooling, continuing the cycle. I was talking to someone from a local startup a couple weeks ago who was trying to explain their devops stack. The number of different tools and platforms they were using was in the range of 50 different things, and they were asking for advice about how to integrate yet another thing to solve yet another self-inflicted problem. It was as though they forgot what the goal was and started trying to collect as much experience with as many different tools as they could.

nusl 3y ago

k8s is really about you and if it makes sense for your use case. It’s not universally bad or universally good, and I don’t feel that there is a minimum team size required for it to make sense. Managing k8s, for me at least, is a lot easier than juggling multiple servers with potentially different hardware, software, or whatever else. It’s rare that businesses will have machines that are all identical. Trying to keep adding machines to a pool that you manage manually and keep them running can be very messy and get out of control if you’re not on top of it. k8s can also get out of control though it’s also easier to reason about and understand in this context. Eg you have eight machines of varying specs but all they really have installed is what’s required to run k8s, so you haven’t got as much divergence there. You can then use k8s to schedule work across them or ask questions about the machines.

liveoneggs 3y ago

We've found kubernetes to be surprisingly fragile, buggy, inflexible, and strictly imperative. People make big claims but then it's not declarative enough to look up a resource or build a dependency tree and then your context deadline is exceeded.

elktown 3y ago

I think an underestimated issue with k8s (et al) is on a cultural level. Once you let in complex generic things, it doesn't stop there. A chain reaction has started, and before you know it, you've got all kinds of components reinforcing each other, that are suddenly required due to some real, or just perceived, problems that are only there in the first place because of a previous step in the chain reaction. I remember back when the Cloud first started getting a foothold that what people was drawn to was that it would enable reducing complexity of managing the most frustrating things like the load-balancer and the database, albeit at a price of course, but it was still worth it. Stateless app servers however, was certainly not a large maintenance problem. But somehow we've managed to squeeze in things like k8s in the there anyway, we just needed to evangelize microservices to create a problem that didn't exist before. Now that this is part of the "culture" it's hard to even get beyond hand-wavy rationalizations that microservices is a must, assumingly because it's the initial spark that triggered the whole chain reaction of complexity.

jupp0r 3y ago
Cloud providers automate things like lease renewals, dealing with customs and part time labor contract compliance disputes for that datacenter in that Asian country that you don't know the language of. I'm constantly fascinated how people handwaivingly underestimate the cost and headaches of actually running on prem global infrastructure.
- robertlagrant 3y ago
  > I'm constantly fascinated To call a halt to your constant fascination: they don't all have that problem. They still get the complexity of cloudy things regardless when they use one.
  - jupp0r 3y ago
    They also get some of the complexity of cloudy things when they run their own datacenter. In the end you find stuff like OpenStack which becomes its own nightmare universe.

aeturnum 3y ago

> if a human is ever waiting for a pod to start, Kubernetes is the wrong choice. As someone who is always working "under" a particular set of infrastructure choices I want people who write this kind of article to understand something: the people who dislike particular infrastructure systems are by-in-large those who are working under sub-optimal uses of them. No one who has the space to think about "if their infrastructure choices will create an effect" in the future hates any infrastructure system. Their life is good. They can choose and most everyone agrees that any system can be done well. The haters come from being in situations where a system has not been done well - where for whatever combination of reasons they are stuck using a system that's the wrong mix of complex / monitorable / fragile / etc. It's true enough that, if that system had been built with more attention to its needs, that people would not hate it - but that's just not how people come to hate k8s (or any other tool).

deleted 3y ago

[deleted]

shrubble 3y ago

If Kubernetes is the answer ... you very likely asked the wrong questions. Reading about JamSocket and what it does, it seems that it essentially lets you run Docker instances inside the Jamsocket infrastructure. Why not just take Caddy in a clustered configuration, add some modules to control Docker startup/shutdown and reduce your services usage by 50%? As one example.

paulgb 3y ago
I’m not sure what you mean by that reducing service usage. The earliest version of the product really was just nginx behind some containers, but we outgrew the functionality of existing proxies pretty quickly. See e.g. keys (https://plane.dev/developing/keys https://plane.dev/developing/keys) which would not be possible with clustered Caddy alone.
- shrubble 3y ago
  My understanding was that K8s itself has overhead, which ultimately has to be paid for, even if using a managed service (it might be included in the cost of what you pay, of course). I did add the caveat of "with modules" and the idea of sharing values around to different servers would be easy to do, since you have Postgres around as a database to hold those values/statuses.
  - paulgb 3y ago
    HTTP proxying is not much of our codebase. I wouldn’t want to shoehorn what we’re doing into being a module of a proxy service just to avoid writing that part. That proxy doesn’t run on Kubernetes currently anyway, so it wouldn’t change anything we currently use Kubernetes for.

siliconc0w 3y ago

IMO the big win with Kubernetes is helm or operators. If you're going to pay the complexity costs you might as well get the wins which is essentially a huge 'app-store' of popular infrastructure components and an entirely programmatic way to manage your operations (deployments, updates, fail-overs, backups, etc). For example if you want to setup something complex like Ceph - Rook is a really nice way to do that. It's a very leaky abstraction so you aren't hiding all the complexity of Ceph but the declarative interface is generally a much nicer way to manage Ceph than a boatload of ansible scripts or generally what we had before. The key to understand is that helm or operators don't magically make infrastructure a managed 'turn-key' appliances, you do generally need to understand how the thing works.

jakupovic 3y ago

This article talks about using k8s but trying not to use it as much as possible. First example being operators, this is the underlying mechanism that makes k8s possible. To me taking a stance not to use operators but use k8s is less than optimal, or plainly stupid. The whole stack is built on operators, which you inherently trust as you use k8s, but choosing not to use them. Sorry but this is hard to read. The only thing I learned is about Caddy as a cert-manager replacement, even though I have used, extended and been pretty happy with cert-manager. The rest is hard to read ;(.

hintymad 3y ago

When I checked out an operator repo for some stateful services, say, elasticsearch, the repo most likely would contain 10s of thousands of lines of YAML and 10s of thousands lines of Go code. Is this due to essential complexity of implementing auto-pilot of a complex service, or is it due to massive integration with k8s' operators framework?

patmcc 3y ago

Missed opportunity to title this the h8r's guide to k8s.

kube-system 3y ago

The people who dislike kubernetes are, in my experience, people who don’t need to do all of the things kubernetes does. If you just need to run an application, it’s not what you want.

erulabs 3y ago

I suppose I’m the guy pushing k8s on midsized companies. If there have been unhappy engineers along the way - they’ve by the vast majority stayed quiet and lied about being happier on surveys. Yes, k8s is complex. The tool matches the problem: complex. But having a standard is so much better than having a somewhat simpler undocumented chaos. “Kubectl explain X” is a thousand times better than even AWS documentation, which in turn was a game changer compared to that-one-whiteboard-above-Dave’s-desk. Standards are tricky, but worth the effort. Personally I’m also very judicious with operators and CRDs - both can be somewhat hidden to beginners. However, the operator pattern is wonderful. Another amazing feature is ultra simple leader election - genuinely difficult outside of k8s, a 5 minute task inside. I agree with Paul’s take here tho of at least being extremely careful about which operators you introduce. At any rate, yes k8s is more complex than your bash deploy script, of course it is. It’s also much more capable and works the same way as it did at all your developers previous jobs. Velocity is the name of the game!

paulgb 3y ago
Good point about k8s vs. AWS docs — a lot of the time people say “just use ECS” or the AWS service of the day, and it will invariably be more confusing to me and more vendor-tied than just doing the thing in k8s.
- p_l 3y ago
  And then if you're unlucky you might hit one of the areas where the AWS documentation has a "teaser" about some functionality that is critical for your project, you spend months looking for the rest of the documentation when initial foray doesn't work, and the highly paid AWS-internal consultants disappear into thin air when asked about the features. So nearly a year later you end up writing the whole feature from scratch yourself.
pclmulqdq 3y ago
I have to say that I don't believe the problem is all that complex unless you make it hard. But on the flip side, if you're a competent Kubernetes person, the correct Kubernetes config is also not that complex. I think a lot of the reaction here is a result of the age-old issues of "management is pushing software on me that I don't want" and people adopting it without knowing how to use it because it's considered a "best practice." In other words, the reaction you probably have to an Oracle database is the same reaction that others have to Kubernetes (although Oracle databases are objectively crappy).

throwitaway222 3y ago

Article makes a good point. Allow k8s disallow any service meshes.

bedobi 3y ago

the "best" infra i ever had was a gig where we * built a jar (it was a jvm shop) * put it on a docker image * put that on an ami * then had a regular aws load balancer that just combined the ami with the correctly specced (for each service) ec2 instances to cope with load it was SIMPLE + it meant we could super easily spin up the previous version ami + ec2s in case of any issues on deploys (in fact, when deploying, we could keep the previous ones running and just repoint the load balancer to them) ps putting the jar on a docker image was arguably unnecessary, we did it mostly to avoid "it works on my machine" style problems

racheljaneville 3y ago

[dead]

remram 3y ago

My main issue with Kubernetes is the cluster-scoped Custom Resource Definition (CRD). We are here because we wanted a way to deal with software that each need different versions of libraries or configuration, so we invented containers. Then we wanted to run multiple services, so we invented orchestrators. But now applications are deployed with cluster-scoped operators that depend on cluster-scoped definitions, and we need to run separate services in separate cluster. It's like with need to create containerization for Kubernetes apps again.

vanillax 3y ago

great article and it helped me validate some of my own challenges with k8s