8 ms·

What job interviews taught me about Kubernetes

273 points by chmaynard 4mo ago

mattmatters 4mo ago
A pretty nothing burger of a post with a bunch of ai-isms. Is this written by a real human? K8s is a complicated beast. CTOs hiring for their 10 person company because of its "used everywhere" is a bad reason to adopt a major piece of technology. You can always graduate to it later if need be.
- jbnorth 4mo ago
  Honestly I felt the same way for a while but the more I'm exposed to both Fortune 500 companies and ones who have a handful of employees I see Kubernetes as just a good starting point rather than adopting it later. It removes the overhead of a lot of what sysadmins and devs of yesteryear did by hand or had to have a career's worth of experience to do quickly. That's not to say that people don't need to know what they're getting into when they adopt kubernetes but especially when you're using a managed offering and not on the bleeding edge of what it supports it's pretty easy in terms of overhead and maintenance.
  - WorldMaker 4mo ago
    The big win seems to be "GitOps". There are other tools for "GitOps" than Kubernetes, Kubernetes is just the elephant in the room in terms of size/scale/current adoption patterns. (Certainly not "ease of use", though.) I think one of the themes in comments here is how much people want more "middle ground" "GitOps" tools somewhere between "Serverless" (especially given vendor lock in in that space) and Kubernetes.
mikgp 4mo ago
Kubernetes is so ubiquitous that yeah, as long as you're not trying to run it yourself, a small Kubernetes cluster just isn't all that much to manage. I think it's been so long, people forget how annoying it is to manage servers. All said and done Kubernetes is becoming more the "Boring Technology"
JohnMakin 4mo ago
> I still don't totally get why the shift happened when it did. Five years ago all three camps were doing fine. Now the VM+systemd crowd has basically disappeared from job postings, serverless stayed niche, and K8s just won. > > My best guesses: managed K8s (EKS, GKE, AKS) got mature and the talent pool flipped: enough people learned it that hiring for anything else became the harder choice. And Helm made "just use someone else's chart" a real option. But I'm not certain. If you were there for the shift and have a better theory, I'd genuinely like to know. Pretty much, almost. Have spent a bunch of time in my career working on the "VM + systemd" setups, stuff running on a rack, or in an ec2 on cloud - managed kubernetes is a lot better for me than those cobbled together messes. There's "easier" setups but usually end up costing me a lot more in time and $. To answer simply, it became good + convenient. I could complain about plenty, and people here like to, but honestly you couldn't pay me to go back to the old way. The one legitimate gripe is the upgrade schedule is exhausting, on AWS it's about every 6 months before you go into extended support. I also hate being at the mercy of arbitrary decisions like "ok we know a huge chunk of the web going back a decade has architected off our Ingress API, but recently we decided we dont really like that way anymore and we want you to use Gateway API instead, so, um, like ya we know it just killed off one of the most used open source ingress configs (ingress-nginx) but yea trust us bro this is going to be so much better" kind of thing.
- mschuster91 4mo ago
  I somewhat agree with you... but it's not like you don't need some actual experts who know what they're doing, especially when stuff goes bonkers and it will go bonkers. Even on AWS EKS, you will run into bullshit with their network overlay. Egress policies are a mess (at least half a year ago, you were not able to say something like "allow pod A to egress traffic to service (!) B" despite a service resolving down to an IP address in the end. And that's before going into the unholy mess that is getting connectivity to and from the external world to your cluster. Cloudfront, ACM certificates, ALB, ALB-EKS integration, Route53, Route53-EKS integration, EFS, EFS-EKS integration, EBS, EBS-EKS integration, RDS, RDS-EKS integration, IAM-EKS integration, SSM, SSM-EKS integration, autoscaling... and if you want more pain and don't already wince, try setting that up across regions or, as I had to do once, across account boundaries. Kubernetes is powerful. But do not make the mistake of assuming it's easy to get started with, at least on the admin side. Even if you got prior AWS experience, getting it all integrated into EKS so you don't have to deal with Terraform and helm/k8s for a full deployment of a piece of software will take you an awful lot of time. For users though? It's a breeze, I will admit as much. Everything down to the firewall rules can be encoded in k8s spec files.
clickety_clack 4mo ago
Adopting k8s when you hire your _second_ engineer (first after the CTO)? That’s a red flag that the CTO’s priorities are wrong and he’s just enjoying tinkering with his infra instead of solving the users’ problems.
- Esophagus4 4mo ago
  I thought that was the point of the article, right? That the tech benefits may not be there, but they’re using it for the non-tech benefits
  - clickety_clack 4mo ago
    From the article: > My personal threshold would be the moment the CTO isn't the only engineer anymore. As soon as a second person shows up, the problems K8s solves become real.
    - darkwater 4mo ago
      What a world we live in with people thinking that the problems of a 2 (founder) engineers startup deserve k8s complexity... Even with LLMs, they are going to rack up tech debt if their focus is - as it should in as mall startup - the final product and not the tech stack itself.
      - Esophagus4 4mo ago
        > Even with LLMs, they are going to rack up tech debt And if they don’t?
        darkwater 4mo ago
        Well, congrats then, I guess.
  - mijoharas 4mo ago
    > That the tech benefits may not be there, but they’re using it for the non-tech benefits My read of the article is that this is correct, but that the benefits they're using it for are the operational, and organisational. I think the comment you're replying to is arguing that those benefits don't really matter or outweigh the additional complexity costs when N=2 (engineers). I think I'd probably agree.
xlii 4mo ago
One year ago I might agree that Kubernetes is an overkill but today? Ask your favorite GPT to generate manifests, get primary app into cluster with telepresence or execute straight from container and switch contexts and clusters like it's 90s again. One reason I dislike Docker Compose and Docker is lack of isolation. Yes sure if you put your arm deep enough you can get it, but on local k8s I can spin cluster per workspace and not worry about conflicting ports between PostgreSQL instances. Before LLMs writing consistent YAMLs was PITA but today on low/development scale it's pretty much free lunch.
- iamcreasy 4mo ago
  Interesting. I have just started reading about Kubernetes. Is there an reading material that goes over this process you just described?
  - johnsmith1840 4mo ago
    Don't. Get a chatgpt subscription and spin up a minikube cluster and launch some stuff and play around. K8s is incredibly deep and complex but with AI it's finally easy to just hello world it.
    - bigstrat2003 4mo ago
      This is absolutely terrible advice. You should never ever use LLMs to work on something you don't understand already, because you have no way to catch the machine when it screws up (and it will screw up). Just like with every other form of automation before LLMs, a smart person only automates things he already knows how to do himself.
      - johnsmith1840 4mo ago
        Yeah no. Getting the first hello world up is more important than anything else. Until you physically see it running learning is slow. I learned k8s through many months of study and pain pre AI. Once I actually got it up learning was FAR easier. This is like using a jupyter notebook to learn python and is always the first thing I point to for someone just starting to learn. Only after should you learn venv, pip install, classes ect. 100% use AI to get started on something you don't understand. I will literally never start to learn about a technical system again without first doing a hello world with AI.
Glyptodon 4mo ago
Even as a solo dev there's generally been a yawning gap between k8s and manual infra that nothing has ever filled that well and it's part of why things like Heroku were so popular for a while.
- ghaff 4mo ago
  Well, maybe especially as a solo dev. Things like Heroku and other tools that were largely called PaaS at the time were very popular with individuals and small teams but they had limitations for enterprise development--and even ran into barriers once anyone ran into those limitations.
- soco 4mo ago
  From what I've seen, Azure container apps (ACA) sits exactly there in the middle: a managed Kubernetes, opinionated and full of defaults you may or may not like, but makes the deployment and maintenance a breeze.
mikeocool 4mo ago
I made this decision at a startup (albeit when the eng team was ~30 people, and we had a monolith with ~10 supporting services). I wouldn’t do it again, even for the reasons stated in the article. The uniformity is nice, we were moving from apps running directly ec2 instances provisioned with ansible. Each time we spun up a new service it was a process to get the ec2 instances provisioned just so. But k8s is such a pain in the ass. One thing that I think people new to it don’t realize is that it’s not at all batteries included - to get a basic managed cluster setup, you’re still going to be installing a bunch of additional controllers (ingress, cert-manager, external dns to start). And then you’re on the hook for making sure all those processes stay up (hope the admission webhook controller for a critical resource doesn’t go down!). Then you’ve got to do a major upgrade on not only your cluster, but all of those controllers every ~3 months. And no one is shy about introducing breaking changes. Also you’re introducing a huge amount of complexity with the k8s networking and dns layer that most startups have zero need for (if you’re on EKS, make sure to read about scaling and monitoring CoreDNS). I think there is a real hole in the market for a simple solution that lets you deploy some containers to some instances in a declarative fashion without all of that complexity and does decent LTS versions. I imagine there’s something out there that does this, but k8s has really sucked up all the oxygen.
- embedding-shape 4mo ago
  > I think there is a real hole in the market for a simple solution that lets you deploy some containers to some instances in a declarative fashion without all of that complexity and does decent LTS versions Hashicorp's Nomad basically is just that, supports various way of running stuff too which is neat. Shame about the license change which basically killed all my interest in it, so seems the hole is indeed still unfilled.
  - mikeocool 4mo ago
    Yeah I’ve always meant to check out nomad and never had an opportunity. Though as I recall, it makes heavy use of consul, which I have used in anger, and makes me a little weary (though that experience is likely very out of date).
nitwit005 4mo ago
> First one was uniformity. Every service deploys the same way. My current company makes this claim, but it's not true. They also have serverless apps, and also have some services running directly on EC2. They just think of the Kubernetes deployments as the "standard" way. > Second was shared, hireable knowledge. K8s is basically a lingua franca now. People were demanding experience with Kubernetes, long before it was reasonable to expect it. Everyone added it to their resume, because they had to.
SamuelAdams 4mo ago
This seems to be less about K8’s and more about the infra as code movement. It doesn’t matter if you use K8, CDK, or terraform - you get the same benefits the OP stated across the board. It is nice to be able to have a consistent deployment pattern, with traceability, rollback support, and production approval checks. It’s nice to not have some archaic something stuck in someone’s head. It’s also nice to be able to see how something works by reading the code, which is usually up to date and deployable.
- sshine 4mo ago
  > less about K8’s and more about the infra as code movement. It doesn’t matter if you use K8, CDK, or terraform - you get the same benefits the OP stated I’d like to gently push back on that. ;-D Terraform, when committed to git, provides organisational memory. But less so uniformity, since all providers are different (and you should expect different things when applying). No tracing besides git. And tfstate is hard to share between developers, unlike kube state. Kubernetes is more the same across providers. And it manages drift after something is applied, which is not a direct argument of OP, but a strong reason over other IAC. And yes, I also enjoy how well deploying works. And how things generally fit together. Liking the networking complexity less so.
  - simoncion 4mo ago
    > And tfstate is hard to share between developers... Really? For years and years we put our tfstate files into private S3 buckets at $DAYJOB and it seemed to work just fine. We didn't even take pains to ensure that everyone was on the very same version of the Terraform CLI. What problems did you guys run into?
    - InvertedRhodium 4mo ago
      One person upgrading terraform can break the state format for backwards compatibility. We use direnv with asdf to ensure everyone is in the same version now.
      - 4mo ago
avhception 4mo ago
Well, I totally get the benefits that made those people choose Kubernetes. It's just that those benefits could be had w/o running a massively complicated piece of machinery that is mostly engineered to solve problems I don't have.
shevis 4mo ago
> The CTOs I talked to aren't making a dumb choice. They're solving real problems. Unrelated to the content of the article, this sentence structure is a dead giveaway of LLM writing.
- jcattle 4mo ago
  This blurb gave me the idea to try and quantize this. Scrape the top HN blogs over the last few years and see how occurences of common phrases change. I'd expect to see a huge increase in "solving real problems" over the last months.
  - eamon0989 4mo ago
    I would love to see the results of this if you actually do it!
    - jcattle 4mo ago
      A coding agent should make short work of that. However I'm a bit doubtful if the results would actually be meaningful. I'm thinking that some of the LLM-isms are a bit more complex than just repeated phrases. It's often more that short, punchy writing style with quick setups and punchlines. But would be interesting nonetheless. I really think that some things (like "solving real problems" or "it's not"/"this isn't") would show up.
vasco 4mo ago
It's a bit odd that the author presents no data other than their interviewing and declares that the shift happened recently. It's not true, there's been steady growth of adoption of kubernetes for years. Just reading CNCF surveys from last years before posting would tell them that. Their identified reasons are OK though.
h4kunamata 4mo ago
Taught me that companies follow hype. I worked once at a bankm fully kubernetes, the amount of problems were out of reality from this world. Complexities are being added for no reason at all.
- SJC_Hacker 4mo ago
  Yeah this was ,ynexperienve as Wellfleet. Nó one réaluadar knew what was going on and we had 5 figure cloud spend bills per month.
crefiz 4mo ago
Another complementary approach is what Vasilios shared today[1] (the ex-Attlasian guy that recently got attraction) [1] https://youtu.be/Iv9hoYTQp_8?si=5YsUxYayFUY-RfKC https://youtu.be/Iv9hoYTQp_8?si=5YsUxYayFUY-RfKC
suralind 4mo ago
So I’m personally a huge fan of k8s and while I agree it may be „complicated”, it’s because deploying applications is complicated. (I want to point out that there is no requirement no set up cert manager, ArgoCD, external secrets, etc. - and many people who’d consider a VPS would happily slap a .env with an unencrypted secret then ssh to update, but when they choose Kubernetes they take the long route of doing proper GitOps and complain that there are so many things to configure :) But I found funny that the OP summarized to use Kubernetes when CTO is no longer the only dev.
- vbezhenar 4mo ago
  100% agree with you. You can actually treat kubernetes as a glorified docker compose engine. Deploy pods, deploy nginx instead of ingress controller, deploy certbot cronjob instead of cert-manager, and believe it or not, it'll work! On a single server! People often compare Kubernetes with thousands of additional services to a simple VPS, but that's not apples to apples comparison.
- solatic 4mo ago
  > many people who’d consider a VPS would happily slap a .env with an unencrypted secret then ssh to update I just want to point out that you can totally still do this with Kubernetes. Of course it's not correct, but you can save that unencrypted secret in a .env file right into your container while you're building it - no need to use Kubernetes's support for supplying environment variables from the manifest. And of course, you don't even need a Dockerfile to build that container - you can just exec into a running container, paste it in, and then docker save. Kubernetes doesn't save you from making stupid decisions, it just makes it easier to make better ones.
  - suralind 4mo ago
    Perhaps I wasn't clear enough - that was my point as well. You can do that, but when people switch to Kubernetes a lot of them do a proper (or better) job of avoiding that, but compare to previous experience where they'd just ssh to update the env, etc.
reillyse 4mo ago
The reason this is accelerating recently is agents are really good at spinning up k8s clusters. They've made devops work super super simple. Basically all the annoying stuff you know you should do but it's way too much hassle - using let's encrypt to create unique certs for every app in your cluster to enable zero trust, configuring permissions and security profiles for everything etc etc (never mind just standing them up in the first place) - it's all simple now.
- phailhaus 4mo ago
  The setup may be simple, but what about the maintenance?
  - lawn 4mo ago
    Anecdotal of course but I've found that they're excellent at debugging and analysing Kubernetes issues.
  - kakwa_ 4mo ago
    And the troubleshooting? Given the number of moving parts, I would be terrified to have to look under the hood of what Talos deployed for me.
    - reillyse 4mo ago
      I’ve found it to be excellent at troubleshooting- recently had a hardware incident and it diagnosed the problem and migrated my cluster to a new machine super fast. I just give the agent access to kubectl and I let it investigate bugs and it does an excellent job - I would say way better than it is at normal coding.
  - __turbobrew__ 4mo ago
    Kubernetes MCP
louwrentius 4mo ago
I'm going against the grain but I read: we have a cultural/policy issue and we 'fix' it with tools. I think what you hear is never the whole story, there is much more going on.
stego-tech 4mo ago
OP gets it. Right now, I’m one dinosaur managing a startup’s tech portfolio. Everything lives in my head first, then in my break-glass vault for addressing the bus problem. Our public cloud footprint is a single KMS for backups. We have no VMs, everything is a cloud service. The literal fucking second we have real infrastructure requirements for compute, it’s right to GCE. No ifs, ands, or buts. Here’s our Git Repo, here’s the managed K8s control plane, make it work. If (or when) we need on-prem compute, we add them to the K8s control plane as worker nodes and taint accordingly. It’s just so much more interchangeable, even if the learning curve for non-SDEs can be a little steeper than VMs.
FpUser 4mo ago
I call BS on that. I serve SMB clients and many are happy like a clams with monoliths deployed using those proverbial bash scripts that also does lots of other things. Understanding scripts in the age of AI is trivial for newcomers. I for example fed my own uber script to AI as an experiment and it has produced all encompassing nice documentation with examples and tests.
mianos 4mo ago
This is exactly why I call it 'resume++'. You have to use it to attract talent. People want to use it to expand their employment pool. This is not justification to using it. To use it is a whole different question, and not in any way related to job interviews. I have worked in places that are crazy for not using it and others where using it was even crazier.
siliconc0w 4mo ago
I had a similar experience after a recent job search and started working on a 'kube-lite' that just uses object storage for coordination and normal cloud primitives like auto-scale groups (skiff.pages.dev). I ended up in a different non-SRE role but if you're interested in working on it, please let me know and I'd love to walk you through it.
liampulles 4mo ago
There is a core 20% to Kubernetes which is very nice, mostly being the Deployment and Service management stuff. That along with a very basic GitOps for cluster management (an infra repo for operators using Flux, applying service level yaml from app repos in CI) above a cloud managed Kubernetes cluster, where you still keep your DB and build servers off the cluster, can be quite nice for a small team. Beyond that, there are massive holes of despair to fall down if a novice team starts to engage with extensive operators (starving the control plane), DB operators (distributed persistence) and build operators (spikey, expensive loads). At least, I know that I've had to dig out of those holes. I just hope people don't use k8s in the same way many use microservices: as a way to introduce complexity for complexity's sake.
- zbentley 4mo ago
  Spot on. I have a lot of trouble convincing cloud folks that for durable state, you probably don’t want kubernetes. It’s not that e.g. the CSI drivers and operators for clustered databases aren’t top notch—they are; the era of “avoid stateful kube services” is long behind us—it’s that the cloud provider managed services for e.g. blob stores or databases are so much more reliable. The S3s and Auroras of the world are expensive for a reason: no matter how good your kube native database operator is, it still doesn’t assume responsibility for a ton of the failure points that managed services do. And that’s true even at modest scale (e.g. upgrades are just harder when you’re running your own DB) and in cost conscious environments (sure, the Elasticache bill is steep, but the salary and velocity cost of fixing memory-leak-caused kube memcached crashes is steeper).
  - portly 4mo ago
    Also cluster migrations are required pretty often in my company. Having state on a cluster means migrating that as well, which is a complex and time consuming operation. Having your state in S3 or external database makes migrations a breeze.
chaos_emergent 4mo ago
In addition to all of OP’s points, another reason k8s is getting popular is that LLMs have made them easier to use! It's reasonably well represented in the dataset and there are pretty strong monitoring and observability tools and verification gates to make sure that you've specified your cluster specifications correctly.
zug_zug 4mo ago
Here's my conspiracy theory -- There's a certain type of engineer (maybe 25% of them) who does "hype-driven-development." No matter the technology, they are huge advocates for the technology. The hype may be absolutely real, complete nonsense (e.g. mongodb), or somewhere in between (ai). The vast majority of the time it's hype for a new technology that feels 90% the same from the end-user perspective (react vs vue, docker vs colima, go vs other, whatever vs whatever). These engineers though, only care about something when it's new and trendy enough to be a differentiator. This is because they don't give any hoots about the actual usefulness of anything, they are just trying to differentiate themselves in a market by leveraging vibes rather than raw competence. I think these types of engineer drove kubernetes for companies that don't need it, but tipped the scales enough that it has critical mass. The irony being kubernetes is way too heavy/clumsy an abstraction for most companies. The savings of packing pods onto the same node is usually a tiny fraction of the engineers' salaries who are managing it. The other irony is now that kubernetes isn't the new sexy thing, but a standard tool that AI or a normie can do all the hard work for, the hype driven engineers are off looking for the next thing.
- zbentley 4mo ago
  The linked article discusses very different reasons for preferring kube. CTOs and hiring managers like it for reasons totally different from the cargo cult/hype-driven engineers.
  - zug_zug 4mo ago
    Yeah I read the article and I saw that. And I do think there is a way to use kubernetes with minimal damage, but it requires making firm rules about not focusing on things that aren't needed yet (e.g. istio) and making firm hiring choices about only people who understand that such optimizations are complete wastes of time for a series A startup.
- gib444 4mo ago
  Tangential but IME those engineers do very very well, as they're good at marketing themselves, but more importantly they often align with the CTO/CEO who wants to present the company as the hottest thing since sliced bread – they signal they're on the same page. Who is going to criticise the engineer who is open to new things / appears innovative / loves MVPs / 110% supportive of the latest bs the CTO is spewing? Yeah
deleted 4mo ago
[deleted]
aaronbrethorst 4mo ago
The knowledge is in the YAML Exactly why I hate CloudFormation, K8S, GitHub Actions, etc. yaml is a terrible format for the knowledge encoded in these artifacts.
- rienbdj 4mo ago
  There are compile-to-yaml config languages
  - aweiland 4mo ago
    Any good ones people would recommend?
    - WorldMaker 4mo ago
      In my experience I tried a few and kept returning to just Typescript with a good YAML serializer library. Most of what you need in YAML programming is "Typed JSON" and Typescript is an excellent "Typed JSON" and "Typed JSON modularization and merger tool". Then just a little bit of glue code to pass your finished "Typed JSON" into a YAML serializer, depending on any "complex" YAML features you might need.
      - rienbdj 4mo ago
        Not a bad solution at all. It does require some discipline to write deterministic templates etc. though.
johnsmith1840 4mo ago
AI, managed control plane, minikube That makes it a no brainer for me for basically any sized project. Small project? -> minikube single node deploy it. Tiny project? -> minimum a docker container I cringe watching anyone build and run code on a raw machine even locally without atleast a container. The endless hours of headaches you avoid is obvious k8s is just the natural extension from this.
ritcgab 4mo ago
Multi-cluster federation is still hard.
metaltyphoon 4mo ago
The grass is not always greener. I would change the Service Fabric crap I have to deal with any day to K8s.
globular-toast 4mo ago
I pushed k8s in the medium sized company I work for much the same reasons. We use flux for gitops which works really well. The problem is we now have as many clusters as we did bare metal hosts before. There's production clusters, dev clusters, ones in other regions etc. The idea was to have "one place, one way to deploy" but it's actually many places. Am I doing it wrong? Should it all be one cluster and just have different nodes for different reasons and RBAC etc?
- raesene9 4mo ago
  that probably depends on how much security and resource isolation you need. Multi-Tenant security in Kubernetes is not a simple thing, for a wide variety of reasons, and noisy neighbour problems are also potentially a headache.
codemog 4mo ago
It's not uniformity, it's cargo culting and offloading thinking to group norms. Doesn't help engineers are some of the most arrogant people alive and refuse to admit anything is complicated, as they consider it some kind affront to their intelligence. I would not advise asking the majority of CTOs these questions either. Many got to that position by saying what people want to hear, which is the "average" safe answer. They will parrot whatever is "hot" at that time because it's the least risky response. They are not your friend nor a reliable source.
- quibono 4mo ago
  I think a big part of this whole discussion (and why it's always such a divisive topic) is that there are so many factors to consider that there is no one single golden bullet. An industry standard just makes this issue go away; also it makes the decision making easier since you're not taking risks going for "VM-with-systemd" or "plain docker" or "bare metal" over $STANDARD. > I would not advise asking the majority of CTOs these questions either. Many got to that position by saying what people want to hear, which is the "average" safe answer. Agree; this is the same as asking people why they're not having kids: they either a) don't know or b) don't want to / are not willing to say the truth.
  - codemog 4mo ago
    I believe a better phrasing for this is “standards change but your program will always be crappy”. Meaning if you choose all the popular tools and languages of the era because they’re standard and not because they’re the right fit for the project, the standards will eventually evolve and move on like they always do, but your program will sit fossilized and bad.
dzonga 4mo ago
for smaller b2b startups - serverless is still a win.
hanneshdc 4mo ago
I’ve worked for a few startups now and we’ve stayed away from k8s, instead being perfectly happy with ECS services managed via terraform. It gives most of the benefits the author mentions (traceability of changes, clearly written down infrastructure), without the complexity of k8s.
andrewcamel 4mo ago
I’m not sure it will cause a reversal, but I bet the k8s takeover wouldn’t have happened with costs of today. You inherently end up using far more resources than you need - between HA, built in services, overhead per node, etc. I personally will be using more resource efficient approaches in everything I do. Question is just what provides the closest set of benefits without the full k8s weight.
ralferoo 4mo ago
I should add a disclaimer that I've never used kubernetes or docker, the latter mostly because I prefer the mental model and data encapsulation of a VM per service. That said, the app I'm developing in my startup is designed with scalability from the outset. I have a single setup script for each type of node, and can take a fresh ubuntu install and just "wget -O- $URL | sh" as root and it sets up the node from scratch and/or reconfigures it to the latest configuration. That does all the ancillary stuff, setting up sane firewall defaults, blocking SSH from non-whitelisted IPs, setting up NTP, borg backup and zabbix (both require manual work on the respective backing servers currently), setting sane system configs (e.g. systemd logs limited to 100MB), wireguard (for the backends that distribute sqlite databases using litefs), etc. and installing the relevant packages with my software. The actual backend application is built into a debian package automatically, so it's just a case of adding my private repository to apt sources and installing it. Updating a machine is just "ssh root@$MACHINE 'apt-get update ; apt-get install $APP'". I probably could automate that with ansible, but I prefer to upgrade them piecemeal while I'm testing out an upgrade, so I have a couple of bash scripts that do the ssh in a for loop instead with different targets in each. This has the advantage for me of being able to buy any old VPS from a cheap provider and add it to my pool in minutes. I'm sure I could end up with something that's just as easy to update with kubernetes, but it seems like another big learning curve with dependencies that probably change every few months and require me to keep learning new things just to keep it running. I understand my bash scripts, and know they won't just stop working going forwards (modulo exceptional events like having to migrate to systemd scripts, but that kind of change is usually only required on a very few major OS distribution upgrades). I already have enough pain from some of my tech depending on other people's projects (I have a frontend app written in Flutter, and forced SDK upgrades about every 6 months and then resulting issues with toolchains I haven't even chosen to use, like gradle and kotlin, that seem to break everything every release), that I have no great desire to rebuild everything on someone else's deployment framework. When I get to the point of hiring others to help, I'd hope they'd be clued in enough to understand a simple bash script that sets up everything, and logically follow it through.
lanycrost 4mo ago
I'm not asking about kubernetes in interviews too, may be trying to understand do the interviewer understand how containerization, CNI, kube-proxy working under the hood. For me the best indicator of strong engineer is that he is interested enough to go under the hood, and understand the foundations rather the tools.
jessinra98 4mo ago
I’m personally a huge fan of k8s and while I agree it may be „complicated”, it’s because deploying applications is complicated. (I want to point out that there is no requirement no set up cert manager, ArgoCD, external secrets, etc. - and many people who’d consider a VPS would happily slap a .env with an unencrypted secret then ssh to update, but when they choose Kubernetes they take the long route of doing proper GitOps and complain that there are so many things to configure :)
imglorp 4mo ago
> shared, hireable knowledge So it's the same motivation as enterprise java.
- TZubiri 4mo ago
  But if you just build bash scripts on top of vms, the talent pool is much larger, necessarily. I feel the motivation goes a bit in the opposite direction. Yes a commoditized worker who speaks a common k8s language, but not too common a language, there has to be some selection here, a talent pool of 100K engineers that know linux is too big, but a pool of 5k engineers that know k8s apparently is just right. Gotta filter those resumes somehow!
xylon 4mo ago
I assumed the reason there are so many job offerings for kubernetes engineers is either because these jobs have high churn, or because kubernetes takes more work to maintain. I think linux+nginx+postgres+favourite-lang is still the most sensible way to go. Good engineers don't overcomplicate things.
- esbeeb 4mo ago
  "linux+nginx+postgres+favourite-lang" - I agree here. Yes, don't over-complicate things.
bionsystem 4mo ago
None of those arguments make sense. Uniformity ? Try deploying openbao inside kube, if kube decides to restart your pods, you're in for unsealing them at 3am, waking up everybody who owns a Shamir key. So bao stays out of the cluster, or pinned to certain nodes, defeating the purpose entirely. Also, with the ultra wide variety of tools at every layer of the stack, uniformity is a joke ; there are no 2 kube cluster deployment that are the same really. Standardized knowledge ? The operating system is standardized knowledge. Any competent SRE should be able to login into a Linux box and figure out what's running there. And if you let your previous ops shadow it all you're just a pretty bad CTO. Tracing who does what ? First of all anybody with admin access can run one time jobs just like anybody with sudo can run one time commands. That's like chapter 01 of the kube doc. Also again at the kube layer itself, below the helm chart, the ops who set that up or updates it can and will change stuff that breaks stuff. Kube isn't necessarily bad and has it's purpose but it's not a product. It's like Linux, a complex piece of tech that requires a lot more knowledge than "just push this helm chart" to work.
- bakies 4mo ago
  Sounds like you had specific issues with openbap and your cluster provider. Whatever the tech stack it's all presented the same way and easily discoverable. Kubernetes is a cloud operating system- this is exactly the point about standardized knowledge.
  - bionsystem 4mo ago
    I give a counter example to a general statement, in logic, it is enough to debunk the original statement. My point is, SOME things wouldn't run predictably inside the cluster (in that case without pinning to nodes, which the article says isn't necessary, and which in general defeats the purpose), so you'd need to run some things outside of it. As per standardized knowledge, I can't see how somebody even proficient with kube, could jump into any app and troubleshoot bad behavior. Apps each have their quirks and subtleties, specific components that behave a certain way. The layers still exists, the kube cluster itself (which again has many component options at every layer of the stack ; hard to know them all), and the app (which will require at least some specialty knowledge). If it's just about pushing helm charts we wouldn't need SRE anymore, just a CI.
kh_hk 4mo ago
I am still unsure if the complexity is an inherent trait of any big organization or a byproduct of using kubernetes
zop_10 4mo ago
[flagged]
jarekdy 3mo ago
[flagged]

273 points by chmaynard 4mo ago

A pretty nothing burger of a post with a bunch of ai-isms. Is this written by a real human? K8s is a complicated beast. CTOs hiring for their 10 person company because of its "used everywhere" is a bad reason to adopt a major piece of technology. You can always graduate to it later if need be.

jbnorth 4mo ago
Honestly I felt the same way for a while but the more I'm exposed to both Fortune 500 companies and ones who have a handful of employees I see Kubernetes as just a good starting point rather than adopting it later. It removes the overhead of a lot of what sysadmins and devs of yesteryear did by hand or had to have a career's worth of experience to do quickly. That's not to say that people don't need to know what they're getting into when they adopt kubernetes but especially when you're using a managed offering and not on the bleeding edge of what it supports it's pretty easy in terms of overhead and maintenance.
- WorldMaker 4mo ago
  The big win seems to be "GitOps". There are other tools for "GitOps" than Kubernetes, Kubernetes is just the elephant in the room in terms of size/scale/current adoption patterns. (Certainly not "ease of use", though.) I think one of the themes in comments here is how much people want more "middle ground" "GitOps" tools somewhere between "Serverless" (especially given vendor lock in in that space) and Kubernetes.

mikgp 4mo ago

Kubernetes is so ubiquitous that yeah, as long as you're not trying to run it yourself, a small Kubernetes cluster just isn't all that much to manage. I think it's been so long, people forget how annoying it is to manage servers. All said and done Kubernetes is becoming more the "Boring Technology"

JohnMakin 4mo ago

> I still don't totally get why the shift happened when it did. Five years ago all three camps were doing fine. Now the VM+systemd crowd has basically disappeared from job postings, serverless stayed niche, and K8s just won. > > My best guesses: managed K8s (EKS, GKE, AKS) got mature and the talent pool flipped: enough people learned it that hiring for anything else became the harder choice. And Helm made "just use someone else's chart" a real option. But I'm not certain. If you were there for the shift and have a better theory, I'd genuinely like to know. Pretty much, almost. Have spent a bunch of time in my career working on the "VM + systemd" setups, stuff running on a rack, or in an ec2 on cloud - managed kubernetes is a lot better for me than those cobbled together messes. There's "easier" setups but usually end up costing me a lot more in time and $. To answer simply, it became good + convenient. I could complain about plenty, and people here like to, but honestly you couldn't pay me to go back to the old way. The one legitimate gripe is the upgrade schedule is exhausting, on AWS it's about every 6 months before you go into extended support. I also hate being at the mercy of arbitrary decisions like "ok we know a huge chunk of the web going back a decade has architected off our Ingress API, but recently we decided we dont really like that way anymore and we want you to use Gateway API instead, so, um, like ya we know it just killed off one of the most used open source ingress configs (ingress-nginx) but yea trust us bro this is going to be so much better" kind of thing.

mschuster91 4mo ago
I somewhat agree with you... but it's not like you don't need some actual experts who know what they're doing, especially when stuff goes bonkers and it will go bonkers. Even on AWS EKS, you will run into bullshit with their network overlay. Egress policies are a mess (at least half a year ago, you were not able to say something like "allow pod A to egress traffic to service (!) B" despite a service resolving down to an IP address in the end. And that's before going into the unholy mess that is getting connectivity to and from the external world to your cluster. Cloudfront, ACM certificates, ALB, ALB-EKS integration, Route53, Route53-EKS integration, EFS, EFS-EKS integration, EBS, EBS-EKS integration, RDS, RDS-EKS integration, IAM-EKS integration, SSM, SSM-EKS integration, autoscaling... and if you want more pain and don't already wince, try setting that up across regions or, as I had to do once, across account boundaries. Kubernetes is powerful. But do not make the mistake of assuming it's easy to get started with, at least on the admin side. Even if you got prior AWS experience, getting it all integrated into EKS so you don't have to deal with Terraform and helm/k8s for a full deployment of a piece of software will take you an awful lot of time. For users though? It's a breeze, I will admit as much. Everything down to the firewall rules can be encoded in k8s spec files.

clickety_clack 4mo ago

Adopting k8s when you hire your _second_ engineer (first after the CTO)? That’s a red flag that the CTO’s priorities are wrong and he’s just enjoying tinkering with his infra instead of solving the users’ problems.

Esophagus4 4mo ago
I thought that was the point of the article, right? That the tech benefits may not be there, but they’re using it for the non-tech benefits
- clickety_clack 4mo ago
  From the article: > My personal threshold would be the moment the CTO isn't the only engineer anymore. As soon as a second person shows up, the problems K8s solves become real.
  - darkwater 4mo ago
    What a world we live in with people thinking that the problems of a 2 (founder) engineers startup deserve k8s complexity... Even with LLMs, they are going to rack up tech debt if their focus is - as it should in as mall startup - the final product and not the tech stack itself.
    - Esophagus4 4mo ago
      > Even with LLMs, they are going to rack up tech debt And if they don’t?
      - darkwater 4mo ago
        Well, congrats then, I guess.
- mijoharas 4mo ago
  > That the tech benefits may not be there, but they’re using it for the non-tech benefits My read of the article is that this is correct, but that the benefits they're using it for are the operational, and organisational. I think the comment you're replying to is arguing that those benefits don't really matter or outweigh the additional complexity costs when N=2 (engineers). I think I'd probably agree.

xlii 4mo ago

One year ago I might agree that Kubernetes is an overkill but today? Ask your favorite GPT to generate manifests, get primary app into cluster with telepresence or execute straight from container and switch contexts and clusters like it's 90s again. One reason I dislike Docker Compose and Docker is lack of isolation. Yes sure if you put your arm deep enough you can get it, but on local k8s I can spin cluster per workspace and not worry about conflicting ports between PostgreSQL instances. Before LLMs writing consistent YAMLs was PITA but today on low/development scale it's pretty much free lunch.

iamcreasy 4mo ago
Interesting. I have just started reading about Kubernetes. Is there an reading material that goes over this process you just described?
- johnsmith1840 4mo ago
  Don't. Get a chatgpt subscription and spin up a minikube cluster and launch some stuff and play around. K8s is incredibly deep and complex but with AI it's finally easy to just hello world it.
  - bigstrat2003 4mo ago
    This is absolutely terrible advice. You should never ever use LLMs to work on something you don't understand already, because you have no way to catch the machine when it screws up (and it will screw up). Just like with every other form of automation before LLMs, a smart person only automates things he already knows how to do himself.
    - johnsmith1840 4mo ago
      Yeah no. Getting the first hello world up is more important than anything else. Until you physically see it running learning is slow. I learned k8s through many months of study and pain pre AI. Once I actually got it up learning was FAR easier. This is like using a jupyter notebook to learn python and is always the first thing I point to for someone just starting to learn. Only after should you learn venv, pip install, classes ect. 100% use AI to get started on something you don't understand. I will literally never start to learn about a technical system again without first doing a hello world with AI.

Glyptodon 4mo ago

Even as a solo dev there's generally been a yawning gap between k8s and manual infra that nothing has ever filled that well and it's part of why things like Heroku were so popular for a while.

ghaff 4mo ago
Well, maybe especially as a solo dev. Things like Heroku and other tools that were largely called PaaS at the time were very popular with individuals and small teams but they had limitations for enterprise development--and even ran into barriers once anyone ran into those limitations.
soco 4mo ago
From what I've seen, Azure container apps (ACA) sits exactly there in the middle: a managed Kubernetes, opinionated and full of defaults you may or may not like, but makes the deployment and maintenance a breeze.

mikeocool 4mo ago

I made this decision at a startup (albeit when the eng team was ~30 people, and we had a monolith with ~10 supporting services). I wouldn’t do it again, even for the reasons stated in the article. The uniformity is nice, we were moving from apps running directly ec2 instances provisioned with ansible. Each time we spun up a new service it was a process to get the ec2 instances provisioned just so. But k8s is such a pain in the ass. One thing that I think people new to it don’t realize is that it’s not at all batteries included - to get a basic managed cluster setup, you’re still going to be installing a bunch of additional controllers (ingress, cert-manager, external dns to start). And then you’re on the hook for making sure all those processes stay up (hope the admission webhook controller for a critical resource doesn’t go down!). Then you’ve got to do a major upgrade on not only your cluster, but all of those controllers every ~3 months. And no one is shy about introducing breaking changes. Also you’re introducing a huge amount of complexity with the k8s networking and dns layer that most startups have zero need for (if you’re on EKS, make sure to read about scaling and monitoring CoreDNS). I think there is a real hole in the market for a simple solution that lets you deploy some containers to some instances in a declarative fashion without all of that complexity and does decent LTS versions. I imagine there’s something out there that does this, but k8s has really sucked up all the oxygen.

embedding-shape 4mo ago
> I think there is a real hole in the market for a simple solution that lets you deploy some containers to some instances in a declarative fashion without all of that complexity and does decent LTS versions Hashicorp's Nomad basically is just that, supports various way of running stuff too which is neat. Shame about the license change which basically killed all my interest in it, so seems the hole is indeed still unfilled.
- mikeocool 4mo ago
  Yeah I’ve always meant to check out nomad and never had an opportunity. Though as I recall, it makes heavy use of consul, which I have used in anger, and makes me a little weary (though that experience is likely very out of date).

nitwit005 4mo ago

> First one was uniformity. Every service deploys the same way. My current company makes this claim, but it's not true. They also have serverless apps, and also have some services running directly on EC2. They just think of the Kubernetes deployments as the "standard" way. > Second was shared, hireable knowledge. K8s is basically a lingua franca now. People were demanding experience with Kubernetes, long before it was reasonable to expect it. Everyone added it to their resume, because they had to.

SamuelAdams 4mo ago

This seems to be less about K8’s and more about the infra as code movement. It doesn’t matter if you use K8, CDK, or terraform - you get the same benefits the OP stated across the board. It is nice to be able to have a consistent deployment pattern, with traceability, rollback support, and production approval checks. It’s nice to not have some archaic something stuck in someone’s head. It’s also nice to be able to see how something works by reading the code, which is usually up to date and deployable.

sshine 4mo ago
> less about K8’s and more about the infra as code movement. It doesn’t matter if you use K8, CDK, or terraform - you get the same benefits the OP stated I’d like to gently push back on that. ;-D Terraform, when committed to git, provides organisational memory. But less so uniformity, since all providers are different (and you should expect different things when applying). No tracing besides git. And tfstate is hard to share between developers, unlike kube state. Kubernetes is more the same across providers. And it manages drift after something is applied, which is not a direct argument of OP, but a strong reason over other IAC. And yes, I also enjoy how well deploying works. And how things generally fit together. Liking the networking complexity less so.
- simoncion 4mo ago
  > And tfstate is hard to share between developers... Really? For years and years we put our tfstate files into private S3 buckets at $DAYJOB and it seemed to work just fine. We didn't even take pains to ensure that everyone was on the very same version of the Terraform CLI. What problems did you guys run into?
  - InvertedRhodium 4mo ago
    One person upgrading terraform can break the state format for backwards compatibility. We use direnv with asdf to ensure everyone is in the same version now.
    - 4mo ago

avhception 4mo ago

Well, I totally get the benefits that made those people choose Kubernetes. It's just that those benefits could be had w/o running a massively complicated piece of machinery that is mostly engineered to solve problems I don't have.

shevis 4mo ago

> The CTOs I talked to aren't making a dumb choice. They're solving real problems. Unrelated to the content of the article, this sentence structure is a dead giveaway of LLM writing.

jcattle 4mo ago
This blurb gave me the idea to try and quantize this. Scrape the top HN blogs over the last few years and see how occurences of common phrases change. I'd expect to see a huge increase in "solving real problems" over the last months.
- eamon0989 4mo ago
  I would love to see the results of this if you actually do it!
  - jcattle 4mo ago
    A coding agent should make short work of that. However I'm a bit doubtful if the results would actually be meaningful. I'm thinking that some of the LLM-isms are a bit more complex than just repeated phrases. It's often more that short, punchy writing style with quick setups and punchlines. But would be interesting nonetheless. I really think that some things (like "solving real problems" or "it's not"/"this isn't") would show up.

vasco 4mo ago

It's a bit odd that the author presents no data other than their interviewing and declares that the shift happened recently. It's not true, there's been steady growth of adoption of kubernetes for years. Just reading CNCF surveys from last years before posting would tell them that. Their identified reasons are OK though.

h4kunamata 4mo ago

Taught me that companies follow hype. I worked once at a bankm fully kubernetes, the amount of problems were out of reality from this world. Complexities are being added for no reason at all.

SJC_Hacker 4mo ago
Yeah this was ,ynexperienve as Wellfleet. Nó one réaluadar knew what was going on and we had 5 figure cloud spend bills per month.

crefiz 4mo ago

Another complementary approach is what Vasilios shared today[1] (the ex-Attlasian guy that recently got attraction) [1] https://youtu.be/Iv9hoYTQp_8?si=5YsUxYayFUY-RfKC https://youtu.be/Iv9hoYTQp_8?si=5YsUxYayFUY-RfKC

suralind 4mo ago

So I’m personally a huge fan of k8s and while I agree it may be „complicated”, it’s because deploying applications is complicated. (I want to point out that there is no requirement no set up cert manager, ArgoCD, external secrets, etc. - and many people who’d consider a VPS would happily slap a .env with an unencrypted secret then ssh to update, but when they choose Kubernetes they take the long route of doing proper GitOps and complain that there are so many things to configure :) But I found funny that the OP summarized to use Kubernetes when CTO is no longer the only dev.

vbezhenar 4mo ago
100% agree with you. You can actually treat kubernetes as a glorified docker compose engine. Deploy pods, deploy nginx instead of ingress controller, deploy certbot cronjob instead of cert-manager, and believe it or not, it'll work! On a single server! People often compare Kubernetes with thousands of additional services to a simple VPS, but that's not apples to apples comparison.
solatic 4mo ago
> many people who’d consider a VPS would happily slap a .env with an unencrypted secret then ssh to update I just want to point out that you can totally still do this with Kubernetes. Of course it's not correct, but you can save that unencrypted secret in a .env file right into your container while you're building it - no need to use Kubernetes's support for supplying environment variables from the manifest. And of course, you don't even need a Dockerfile to build that container - you can just exec into a running container, paste it in, and then docker save. Kubernetes doesn't save you from making stupid decisions, it just makes it easier to make better ones.
- suralind 4mo ago
  Perhaps I wasn't clear enough - that was my point as well. You can do that, but when people switch to Kubernetes a lot of them do a proper (or better) job of avoiding that, but compare to previous experience where they'd just ssh to update the env, etc.

reillyse 4mo ago

The reason this is accelerating recently is agents are really good at spinning up k8s clusters. They've made devops work super super simple. Basically all the annoying stuff you know you should do but it's way too much hassle - using let's encrypt to create unique certs for every app in your cluster to enable zero trust, configuring permissions and security profiles for everything etc etc (never mind just standing them up in the first place) - it's all simple now.

phailhaus 4mo ago
The setup may be simple, but what about the maintenance?
- lawn 4mo ago
  Anecdotal of course but I've found that they're excellent at debugging and analysing Kubernetes issues.
- kakwa_ 4mo ago
  And the troubleshooting? Given the number of moving parts, I would be terrified to have to look under the hood of what Talos deployed for me.
  - reillyse 4mo ago
    I’ve found it to be excellent at troubleshooting- recently had a hardware incident and it diagnosed the problem and migrated my cluster to a new machine super fast. I just give the agent access to kubectl and I let it investigate bugs and it does an excellent job - I would say way better than it is at normal coding.
- __turbobrew__ 4mo ago
  Kubernetes MCP

louwrentius 4mo ago

I'm going against the grain but I read: we have a cultural/policy issue and we 'fix' it with tools. I think what you hear is never the whole story, there is much more going on.

stego-tech 4mo ago

OP gets it. Right now, I’m one dinosaur managing a startup’s tech portfolio. Everything lives in my head first, then in my break-glass vault for addressing the bus problem. Our public cloud footprint is a single KMS for backups. We have no VMs, everything is a cloud service. The literal fucking second we have real infrastructure requirements for compute, it’s right to GCE. No ifs, ands, or buts. Here’s our Git Repo, here’s the managed K8s control plane, make it work. If (or when) we need on-prem compute, we add them to the K8s control plane as worker nodes and taint accordingly. It’s just so much more interchangeable, even if the learning curve for non-SDEs can be a little steeper than VMs.

FpUser 4mo ago

I call BS on that. I serve SMB clients and many are happy like a clams with monoliths deployed using those proverbial bash scripts that also does lots of other things. Understanding scripts in the age of AI is trivial for newcomers. I for example fed my own uber script to AI as an experiment and it has produced all encompassing nice documentation with examples and tests.

mianos 4mo ago

This is exactly why I call it 'resume++'. You have to use it to attract talent. People want to use it to expand their employment pool. This is not justification to using it. To use it is a whole different question, and not in any way related to job interviews. I have worked in places that are crazy for not using it and others where using it was even crazier.

siliconc0w 4mo ago

I had a similar experience after a recent job search and started working on a 'kube-lite' that just uses object storage for coordination and normal cloud primitives like auto-scale groups (skiff.pages.dev). I ended up in a different non-SRE role but if you're interested in working on it, please let me know and I'd love to walk you through it.

liampulles 4mo ago

There is a core 20% to Kubernetes which is very nice, mostly being the Deployment and Service management stuff. That along with a very basic GitOps for cluster management (an infra repo for operators using Flux, applying service level yaml from app repos in CI) above a cloud managed Kubernetes cluster, where you still keep your DB and build servers off the cluster, can be quite nice for a small team. Beyond that, there are massive holes of despair to fall down if a novice team starts to engage with extensive operators (starving the control plane), DB operators (distributed persistence) and build operators (spikey, expensive loads). At least, I know that I've had to dig out of those holes. I just hope people don't use k8s in the same way many use microservices: as a way to introduce complexity for complexity's sake.

zbentley 4mo ago
Spot on. I have a lot of trouble convincing cloud folks that for durable state, you probably don’t want kubernetes. It’s not that e.g. the CSI drivers and operators for clustered databases aren’t top notch—they are; the era of “avoid stateful kube services” is long behind us—it’s that the cloud provider managed services for e.g. blob stores or databases are so much more reliable. The S3s and Auroras of the world are expensive for a reason: no matter how good your kube native database operator is, it still doesn’t assume responsibility for a ton of the failure points that managed services do. And that’s true even at modest scale (e.g. upgrades are just harder when you’re running your own DB) and in cost conscious environments (sure, the Elasticache bill is steep, but the salary and velocity cost of fixing memory-leak-caused kube memcached crashes is steeper).
- portly 4mo ago
  Also cluster migrations are required pretty often in my company. Having state on a cluster means migrating that as well, which is a complex and time consuming operation. Having your state in S3 or external database makes migrations a breeze.

chaos_emergent 4mo ago

In addition to all of OP’s points, another reason k8s is getting popular is that LLMs have made them easier to use! It's reasonably well represented in the dataset and there are pretty strong monitoring and observability tools and verification gates to make sure that you've specified your cluster specifications correctly.

zug_zug 4mo ago

Here's my conspiracy theory -- There's a certain type of engineer (maybe 25% of them) who does "hype-driven-development." No matter the technology, they are huge advocates for the technology. The hype may be absolutely real, complete nonsense (e.g. mongodb), or somewhere in between (ai). The vast majority of the time it's hype for a new technology that feels 90% the same from the end-user perspective (react vs vue, docker vs colima, go vs other, whatever vs whatever). These engineers though, only care about something when it's new and trendy enough to be a differentiator. This is because they don't give any hoots about the actual usefulness of anything, they are just trying to differentiate themselves in a market by leveraging vibes rather than raw competence. I think these types of engineer drove kubernetes for companies that don't need it, but tipped the scales enough that it has critical mass. The irony being kubernetes is way too heavy/clumsy an abstraction for most companies. The savings of packing pods onto the same node is usually a tiny fraction of the engineers' salaries who are managing it. The other irony is now that kubernetes isn't the new sexy thing, but a standard tool that AI or a normie can do all the hard work for, the hype driven engineers are off looking for the next thing.

zbentley 4mo ago
The linked article discusses very different reasons for preferring kube. CTOs and hiring managers like it for reasons totally different from the cargo cult/hype-driven engineers.
- zug_zug 4mo ago
  Yeah I read the article and I saw that. And I do think there is a way to use kubernetes with minimal damage, but it requires making firm rules about not focusing on things that aren't needed yet (e.g. istio) and making firm hiring choices about only people who understand that such optimizations are complete wastes of time for a series A startup.
gib444 4mo ago
Tangential but IME those engineers do very very well, as they're good at marketing themselves, but more importantly they often align with the CTO/CEO who wants to present the company as the hottest thing since sliced bread – they signal they're on the same page. Who is going to criticise the engineer who is open to new things / appears innovative / loves MVPs / 110% supportive of the latest bs the CTO is spewing? Yeah

deleted 4mo ago

[deleted]

aaronbrethorst 4mo ago

The knowledge is in the YAML Exactly why I hate CloudFormation, K8S, GitHub Actions, etc. yaml is a terrible format for the knowledge encoded in these artifacts.

rienbdj 4mo ago
There are compile-to-yaml config languages
- aweiland 4mo ago
  Any good ones people would recommend?
  - WorldMaker 4mo ago
    In my experience I tried a few and kept returning to just Typescript with a good YAML serializer library. Most of what you need in YAML programming is "Typed JSON" and Typescript is an excellent "Typed JSON" and "Typed JSON modularization and merger tool". Then just a little bit of glue code to pass your finished "Typed JSON" into a YAML serializer, depending on any "complex" YAML features you might need.
    - rienbdj 4mo ago
      Not a bad solution at all. It does require some discipline to write deterministic templates etc. though.

johnsmith1840 4mo ago

AI, managed control plane, minikube That makes it a no brainer for me for basically any sized project. Small project? -> minikube single node deploy it. Tiny project? -> minimum a docker container I cringe watching anyone build and run code on a raw machine even locally without atleast a container. The endless hours of headaches you avoid is obvious k8s is just the natural extension from this.

ritcgab 4mo ago

Multi-cluster federation is still hard.

metaltyphoon 4mo ago

The grass is not always greener. I would change the Service Fabric crap I have to deal with any day to K8s.

globular-toast 4mo ago

I pushed k8s in the medium sized company I work for much the same reasons. We use flux for gitops which works really well. The problem is we now have as many clusters as we did bare metal hosts before. There's production clusters, dev clusters, ones in other regions etc. The idea was to have "one place, one way to deploy" but it's actually many places. Am I doing it wrong? Should it all be one cluster and just have different nodes for different reasons and RBAC etc?

raesene9 4mo ago
that probably depends on how much security and resource isolation you need. Multi-Tenant security in Kubernetes is not a simple thing, for a wide variety of reasons, and noisy neighbour problems are also potentially a headache.

codemog 4mo ago

It's not uniformity, it's cargo culting and offloading thinking to group norms. Doesn't help engineers are some of the most arrogant people alive and refuse to admit anything is complicated, as they consider it some kind affront to their intelligence. I would not advise asking the majority of CTOs these questions either. Many got to that position by saying what people want to hear, which is the "average" safe answer. They will parrot whatever is "hot" at that time because it's the least risky response. They are not your friend nor a reliable source.

quibono 4mo ago
I think a big part of this whole discussion (and why it's always such a divisive topic) is that there are so many factors to consider that there is no one single golden bullet. An industry standard just makes this issue go away; also it makes the decision making easier since you're not taking risks going for "VM-with-systemd" or "plain docker" or "bare metal" over $STANDARD. > I would not advise asking the majority of CTOs these questions either. Many got to that position by saying what people want to hear, which is the "average" safe answer. Agree; this is the same as asking people why they're not having kids: they either a) don't know or b) don't want to / are not willing to say the truth.
- codemog 4mo ago
  I believe a better phrasing for this is “standards change but your program will always be crappy”. Meaning if you choose all the popular tools and languages of the era because they’re standard and not because they’re the right fit for the project, the standards will eventually evolve and move on like they always do, but your program will sit fossilized and bad.

dzonga 4mo ago

for smaller b2b startups - serverless is still a win.

hanneshdc 4mo ago

I’ve worked for a few startups now and we’ve stayed away from k8s, instead being perfectly happy with ECS services managed via terraform. It gives most of the benefits the author mentions (traceability of changes, clearly written down infrastructure), without the complexity of k8s.

andrewcamel 4mo ago

I’m not sure it will cause a reversal, but I bet the k8s takeover wouldn’t have happened with costs of today. You inherently end up using far more resources than you need - between HA, built in services, overhead per node, etc. I personally will be using more resource efficient approaches in everything I do. Question is just what provides the closest set of benefits without the full k8s weight.

ralferoo 4mo ago

I should add a disclaimer that I've never used kubernetes or docker, the latter mostly because I prefer the mental model and data encapsulation of a VM per service. That said, the app I'm developing in my startup is designed with scalability from the outset. I have a single setup script for each type of node, and can take a fresh ubuntu install and just "wget -O- $URL | sh" as root and it sets up the node from scratch and/or reconfigures it to the latest configuration. That does all the ancillary stuff, setting up sane firewall defaults, blocking SSH from non-whitelisted IPs, setting up NTP, borg backup and zabbix (both require manual work on the respective backing servers currently), setting sane system configs (e.g. systemd logs limited to 100MB), wireguard (for the backends that distribute sqlite databases using litefs), etc. and installing the relevant packages with my software. The actual backend application is built into a debian package automatically, so it's just a case of adding my private repository to apt sources and installing it. Updating a machine is just "ssh root@$MACHINE 'apt-get update ; apt-get install $APP'". I probably could automate that with ansible, but I prefer to upgrade them piecemeal while I'm testing out an upgrade, so I have a couple of bash scripts that do the ssh in a for loop instead with different targets in each. This has the advantage for me of being able to buy any old VPS from a cheap provider and add it to my pool in minutes. I'm sure I could end up with something that's just as easy to update with kubernetes, but it seems like another big learning curve with dependencies that probably change every few months and require me to keep learning new things just to keep it running. I understand my bash scripts, and know they won't just stop working going forwards (modulo exceptional events like having to migrate to systemd scripts, but that kind of change is usually only required on a very few major OS distribution upgrades). I already have enough pain from some of my tech depending on other people's projects (I have a frontend app written in Flutter, and forced SDK upgrades about every 6 months and then resulting issues with toolchains I haven't even chosen to use, like gradle and kotlin, that seem to break everything every release), that I have no great desire to rebuild everything on someone else's deployment framework. When I get to the point of hiring others to help, I'd hope they'd be clued in enough to understand a simple bash script that sets up everything, and logically follow it through.

lanycrost 4mo ago

I'm not asking about kubernetes in interviews too, may be trying to understand do the interviewer understand how containerization, CNI, kube-proxy working under the hood. For me the best indicator of strong engineer is that he is interested enough to go under the hood, and understand the foundations rather the tools.

jessinra98 4mo ago

I’m personally a huge fan of k8s and while I agree it may be „complicated”, it’s because deploying applications is complicated. (I want to point out that there is no requirement no set up cert manager, ArgoCD, external secrets, etc. - and many people who’d consider a VPS would happily slap a .env with an unencrypted secret then ssh to update, but when they choose Kubernetes they take the long route of doing proper GitOps and complain that there are so many things to configure :)

imglorp 4mo ago

> shared, hireable knowledge So it's the same motivation as enterprise java.

TZubiri 4mo ago
But if you just build bash scripts on top of vms, the talent pool is much larger, necessarily. I feel the motivation goes a bit in the opposite direction. Yes a commoditized worker who speaks a common k8s language, but not too common a language, there has to be some selection here, a talent pool of 100K engineers that know linux is too big, but a pool of 5k engineers that know k8s apparently is just right. Gotta filter those resumes somehow!

xylon 4mo ago

I assumed the reason there are so many job offerings for kubernetes engineers is either because these jobs have high churn, or because kubernetes takes more work to maintain. I think linux+nginx+postgres+favourite-lang is still the most sensible way to go. Good engineers don't overcomplicate things.

esbeeb 4mo ago
"linux+nginx+postgres+favourite-lang" - I agree here. Yes, don't over-complicate things.

bionsystem 4mo ago

None of those arguments make sense. Uniformity ? Try deploying openbao inside kube, if kube decides to restart your pods, you're in for unsealing them at 3am, waking up everybody who owns a Shamir key. So bao stays out of the cluster, or pinned to certain nodes, defeating the purpose entirely. Also, with the ultra wide variety of tools at every layer of the stack, uniformity is a joke ; there are no 2 kube cluster deployment that are the same really. Standardized knowledge ? The operating system is standardized knowledge. Any competent SRE should be able to login into a Linux box and figure out what's running there. And if you let your previous ops shadow it all you're just a pretty bad CTO. Tracing who does what ? First of all anybody with admin access can run one time jobs just like anybody with sudo can run one time commands. That's like chapter 01 of the kube doc. Also again at the kube layer itself, below the helm chart, the ops who set that up or updates it can and will change stuff that breaks stuff. Kube isn't necessarily bad and has it's purpose but it's not a product. It's like Linux, a complex piece of tech that requires a lot more knowledge than "just push this helm chart" to work.

bakies 4mo ago
Sounds like you had specific issues with openbap and your cluster provider. Whatever the tech stack it's all presented the same way and easily discoverable. Kubernetes is a cloud operating system- this is exactly the point about standardized knowledge.
- bionsystem 4mo ago
  I give a counter example to a general statement, in logic, it is enough to debunk the original statement. My point is, SOME things wouldn't run predictably inside the cluster (in that case without pinning to nodes, which the article says isn't necessary, and which in general defeats the purpose), so you'd need to run some things outside of it. As per standardized knowledge, I can't see how somebody even proficient with kube, could jump into any app and troubleshoot bad behavior. Apps each have their quirks and subtleties, specific components that behave a certain way. The layers still exists, the kube cluster itself (which again has many component options at every layer of the stack ; hard to know them all), and the app (which will require at least some specialty knowledge). If it's just about pushing helm charts we wouldn't need SRE anymore, just a CI.

kh_hk 4mo ago

I am still unsure if the complexity is an inherent trait of any big organization or a byproduct of using kubernetes

zop_10 4mo ago

[flagged]

jarekdy 3mo ago

[flagged]