6 ms·

I’m baffled to see so many anti-k8s sentiments on HN. Is it because most commenters are developers used to services like heroku, fly.io, render.com etc. Or run

by xiwenc 2y ago

I’m baffled to see so many anti-k8s sentiments on HN. Is it because most commenters are developers used to services like heroku, fly.io, render.com etc. Or run their apps on VM’s?

elktown 2y ago
I think some are just pretty sick and tired of the explosion of needless complexity we've seen in the last decade or so in software, and rightly so. This is an industry-wide problem of deeply misaligned incentives (& some amount of ZIRP gold rush), not specific to this particular case - if this one is even a good example of this to begin with. Honestly, as it stands, I think we'd be seen as pretty useless craftsmen in any other field due to an unhealthy obsession of our tooling and meta-work - consistently throwing any kind of sensible resource usage out of the window in favor of just getting to work with certain tooling. It's some kind of a "Temporarily embarrassed FAANG engineer" situation.
- cwiggs 2y ago
  I agree with this somewhat. The other day I was driving home and I saw a sprinkler head and broke on the side of the road and was spraying water everywhere. It made me think, why aren't sprinkler systems designed with HA in mind? Why aren't there dual water lines with dual sprinkler heads everywhere with an electronic component that detects a break in a line and automatically switches to the backup water line? It's because the downside of having the water spray everywhere, the grass become unhealthy or die is less than how much it would cost to deploy it HA. In the software/tech industry it's common place to just accept that your app can't be down for any amount of time no matter what. No one checked to see how much more it would cost (engineering time & infra costs) to deploy the app so it would be HA, so no one checked to see if it would be worth it. I blame this logic on the low interest rates for a decade. I could be wrong.
  - loire280 2y ago
    This week we had a few minutes of downtime on an internal service because of a node rotation that triggered an alert. The responding engineer started to put together a plan to make the service HA (which would have tripled the cost to serve). I asked how frequently the service went down and how many people would be inconvenienced if it did. They didn't know, but when we checked the metrics it had single-digit minutes of downtime this year and fewer than a dozen daily users. We bumped the threshold on the alert to longer than it takes for a pod to be re-scheduled and resolved the ticket.
    - jack_riminton 2y ago
      This is most sensible thing I’ve read on here in a while. Engineers’ obsession with tinkering and perfection is the slow death of many startups. If you’re doing something important like banking or air traffic control fair enough but a CRUD app for booking hair appointments will survive a bit of downtime
  - fragmede 2y ago
    Why would wanting redundancy be a ZIRP? Is blaming everything on ZIRP like Mercury was in retrograde but for economics dorks?
    - felixgallo 2y ago
      Because the company overhired to the point where people were sitting around dreaming up useless features just to justify their workday.
    - consteval 2y ago
      It depends on the cost of complexity you're adding. Adding another database or whatever is really not that complex so yeah sure, go for it. But a lot of companies are building distributed systems purely because they want this ultra-low downtime. Distributed systems are HARD. You get an entire set of problems you don't get otherwise, and the complexity explodes. Often, in my opinion, this is not justified. Saving a few minutes of downtime in exchange for making your application orders of magnitude more complex is just not worth it. Distributed systems solve distributed problems. They're overkill if you just want better uptime or crisis recovery. You can do that with a monolith and a database and get 99.99% of the way there. That's good enough.
    - addaon 2y ago
      Redundancy, like most engineering choices, is a cost/benefit tradeoff. If the costs are distorted, the result of the tradeoff study will be distorted from the decisions that would be made in "more normal" times.
  - zerkten 2y ago
    You assume that the teams running these systems achieve acceptable uptime and companies aren't making refunds for missed uptime targets when contracts enforce that, or losing customers. There is definitely a vision for HA at many companies, but they are struggling with and without k8s.
- bobobob420 2y ago
  Any software engineer who thinks K8 is complex shouldn’t be a software engineer. It’s really not that hard to manage.
  - LordKeren 2y ago
    I think the key word is “needless” in terms of complexity. There are a lot of k8 projects that probably could benefit from a simpler orchestration system— especially at smaller firms
    - fragmede 2y ago
      do you have a simpler orchestration system you'd recommend?
      - qohen 2y ago
        Nomad, from Hashicorp, comes to mind. https://www.nomadproject.io/ https://www.nomadproject.io/ https://github.com/hashicorp/nomad https://github.com/hashicorp/nomad
        ahoka 2y ago
        How is it more simple?
        Ramiro 2y ago
        Every time I read about Nomad, I wonder the same. I swear I'm not trolling here, I honestly don't get how running Nomad is simpler than Kubernetes. Especially considering that there are substantially more resources and help on Kubernetes than Nomad.
        qohen 2y ago
        Well, for starters, you don't have to have your apps containerized to work with Nomad (though it can handle containers as well as executables). But for some deeper details, I'd suggest checking out the comments in this reddit thread[0] (as well as some of the linked articles therein). E.g. From a comment by /u/Golden_Age_Fallacy: A great use of Nomad is on reduce the burden of on-boarding a team(s) of developers who are unfamiliar with cloud native deployments / systems(even containers!). Nomad jobspecs are very simple and straight forward, as compared to the complexity and pure option overload you get in k8s and helm. From /u/neutralized: It's much easier to use than k8s. Easy to setup, easy to manage, much more shallow learning curve. Nothing super fancy. Just works. I migrated a startup I was at off of a self-managed k8s setup to Nomad a few years ago and they've never looked back. From /u/esity: My team is currently building out a fully automated nomad cluster service offering internally(fortune 10) It's super awesome. Easy. Little headache. Integrates with consul and vault. We are literally planning to replace thousands of vms for K8s with nomad. Containers are faster, more resilient and writing hcl is actually fun once you learn it Now, there is a rather more lengthy comment, by /u/thomasbuchinger, that goes through the pros and cons he experienced in trying Nomad out and his conclusion is that, while he wouldn't discourage anyone from using it, "k3s and a few well-known simple projects give you 80% of Nomands [sic] features. Are as easy to operate, afford you more options in the future and have a ton of documentation/tutorials...available." There are more comments in the thread and again links to a bunch of blogposts/articles/etc., including one from fly.io that seemed pretty detailed, discussing the Googly origins of both k8s and Nomad (fly.io used Nomad but found that it wasn't the best fit for them, which is also discussed in their post -- actually, I'm going to put the link to their post below[1], since I think it is worthwhile). Hope all this helps. [0] https://old.reddit.com/r/devops/comments/11nsxo3/opinions_on_hashicorp_nomad https://old.reddit.com/r/devops/comments/11nsxo3/opinions_on... [1] https://fly.io/blog/carving-the-scheduler-out-of-our-orchestrator/ https://fly.io/blog/carving-the-scheduler-out-of-our-orchest...
- methodical 2y ago
  Fair point but I think the key point here is unnecessary complexity versus necessary complexity. Are zero-downtime deployments and load balancing unnecessary? Perhaps for a personal project, but for any company with a consistent userbase I'd argue these are a non-negotiable, or should be anyways. In a situation where this is the expectation, k8s seems like the simplest answer, or near enough to it.
  - iTokio 2y ago
    They are many ways to do deployments without downtime and load balancing is easy to configure without k8s.
- darby_nine 2y ago
  > It's some kind of a "Temporarily embarrassed FAANG engineer" situation. FAANG engineers made the same mistake, too, even though the analogy implies comparative competency or value.
maayank 2y ago
It’s one of those technologies where there’s merit to use them in some situations but are too often cargo culted.
caniszczyk 2y ago
Hating is a sign of success in some ways :) In some ways, it's nice to see companies move to use mostly open source infrastructure, a lot of it coming from CNCF (https://landscape.cncf.io https://landscape.cncf.io), ASF and other organizations out there (on top of the random things on github).
tryauuum 2y ago
For me it is about VMs. Feel uneasy knowing that any kernel vulnerability will allow a malicious code to escape the container and explore the kubernetes host There are kata-containers I think, they might solve my angst and make me enjoy k8s Overall... There's just nothing cool in kubernetes to me. Containers, load balancers, megabytes of yaml -- I've seen it all. Nothing feels interesting enough to try
- stackskipton 2y ago
  vs the Application getting hacked and running lose on the VM? If you have never dealt with, I have to run these 50 containers plus Nginx/CertBot while figuring out which node is best to run it, yea, I can see you not being thrilled about Kubernetes. For the rest of us though, Kubernetes helps out with that easily.
  - tryauuum 2y ago
    if a 4-core VM with a single application is hacked, that's it if there's a kernel vulnerability in something simple (like dirtycow, which was if I remember correctly about pipes) then the attacker will take over your entire 128 core machine and all the hundreds applications there
moduspol 2y ago
For me personally, I get a little bit salty about it due to imagined, theoretical business needs of being multi-cloud, or being able to deploy on-prem someday if needed. It's tough to explain just how much longer it'll take, how much more expertise is required, how much more fragile it'll be, and how much more money it'll take to build out on Kubernetes instead of your AWS deployment model of choice (VM images on EC2, or Elastic Beanstalk, or ECS / Fargate, or Lambda). I don't want to set up or maintain my own ELK stack, or Prometheus. Or wrestle with CNI plugins. Or Kafka. Or high availability Postgres. Or Argo. Or Helm. Or control plane upgrades. I can get up and running with the AWS equivalent almost immediately, with almost no maintenance, and usually with linear costs starting near zero. I can solve business problems so, so much faster and more efficiently. It's the difference between me being able to blow away expectations and my whole team being quarters behind. That said, when there is a genuine multi-cloud or on-prem requirement, I wouldn't want to do it with anything other than k8s. And it's probably not as bad if you do actually work at a company big enough to have a lot of skilled engineers that understand k8s--that just hasn't been the case anywhere I've worked.
- drawnwren 2y ago
  Genuine question: how are you handling load balancing, log aggregation, failure restart + readiness checks, deployment pipelines, and machine maintenance schedules with these “simple” setups? Because as annoying as getting the prometheus + loki + tempo + promtail stack going on k8s is —- I don’t really believe that writing it from scratch is easier.
  - felixgallo 2y ago
    He named the services. Go read about them.
    - drawnwren 2y ago
      I’m not sure which services you think were named that solve the problems I mentioned, but none were. You’re welcome to go read about them, I do this for a living.
  - 2y ago
archenemybuntu 2y ago
Kubernetes itself is built around mostly solid distributed system principles. It's the ecosystem around it which turns things needlessly complex. Just because you have kubernetes, you don't necessarily need istio, helm, Argo cd, cilium, and whatever half baked stuff is pushed by CNCF yesterday. For example take a look at helm. Its templating is atrocious, and if I am still correct, it doesn't have a way to order resources properly except hooks. Sometimes resource A (deployment) depends on resource B (some CRD). The culture around kubernetes dictates you bring in everything pushed by CNCF. And most of these stuff are half baked MVPs. --- The word devops has created expectations that back end developer should be fighting kubernetes if something goes wrong. --- Containerization is done poorly by many orgs, no care about security and image size. That's a rant for another day. I suspect this isn't a big reason for kubernetes hate here.

by xiwenc 2y ago

I’m baffled to see so many anti-k8s sentiments on HN. Is it because most commenters are developers used to services like heroku, fly.io, render.com etc. Or run their apps on VM’s?

elktown 2y ago

I think some are just pretty sick and tired of the explosion of needless complexity we've seen in the last decade or so in software, and rightly so. This is an industry-wide problem of deeply misaligned incentives (& some amount of ZIRP gold rush), not specific to this particular case - if this one is even a good example of this to begin with. Honestly, as it stands, I think we'd be seen as pretty useless craftsmen in any other field due to an unhealthy obsession of our tooling and meta-work - consistently throwing any kind of sensible resource usage out of the window in favor of just getting to work with certain tooling. It's some kind of a "Temporarily embarrassed FAANG engineer" situation.

cwiggs 2y ago
I agree with this somewhat. The other day I was driving home and I saw a sprinkler head and broke on the side of the road and was spraying water everywhere. It made me think, why aren't sprinkler systems designed with HA in mind? Why aren't there dual water lines with dual sprinkler heads everywhere with an electronic component that detects a break in a line and automatically switches to the backup water line? It's because the downside of having the water spray everywhere, the grass become unhealthy or die is less than how much it would cost to deploy it HA. In the software/tech industry it's common place to just accept that your app can't be down for any amount of time no matter what. No one checked to see how much more it would cost (engineering time & infra costs) to deploy the app so it would be HA, so no one checked to see if it would be worth it. I blame this logic on the low interest rates for a decade. I could be wrong.
- loire280 2y ago
  This week we had a few minutes of downtime on an internal service because of a node rotation that triggered an alert. The responding engineer started to put together a plan to make the service HA (which would have tripled the cost to serve). I asked how frequently the service went down and how many people would be inconvenienced if it did. They didn't know, but when we checked the metrics it had single-digit minutes of downtime this year and fewer than a dozen daily users. We bumped the threshold on the alert to longer than it takes for a pod to be re-scheduled and resolved the ticket.
  - jack_riminton 2y ago
    This is most sensible thing I’ve read on here in a while. Engineers’ obsession with tinkering and perfection is the slow death of many startups. If you’re doing something important like banking or air traffic control fair enough but a CRUD app for booking hair appointments will survive a bit of downtime
- fragmede 2y ago
  Why would wanting redundancy be a ZIRP? Is blaming everything on ZIRP like Mercury was in retrograde but for economics dorks?
  - felixgallo 2y ago
    Because the company overhired to the point where people were sitting around dreaming up useless features just to justify their workday.
  - consteval 2y ago
    It depends on the cost of complexity you're adding. Adding another database or whatever is really not that complex so yeah sure, go for it. But a lot of companies are building distributed systems purely because they want this ultra-low downtime. Distributed systems are HARD. You get an entire set of problems you don't get otherwise, and the complexity explodes. Often, in my opinion, this is not justified. Saving a few minutes of downtime in exchange for making your application orders of magnitude more complex is just not worth it. Distributed systems solve distributed problems. They're overkill if you just want better uptime or crisis recovery. You can do that with a monolith and a database and get 99.99% of the way there. That's good enough.
  - addaon 2y ago
    Redundancy, like most engineering choices, is a cost/benefit tradeoff. If the costs are distorted, the result of the tradeoff study will be distorted from the decisions that would be made in "more normal" times.
- zerkten 2y ago
  You assume that the teams running these systems achieve acceptable uptime and companies aren't making refunds for missed uptime targets when contracts enforce that, or losing customers. There is definitely a vision for HA at many companies, but they are struggling with and without k8s.
bobobob420 2y ago
Any software engineer who thinks K8 is complex shouldn’t be a software engineer. It’s really not that hard to manage.
- LordKeren 2y ago
  I think the key word is “needless” in terms of complexity. There are a lot of k8 projects that probably could benefit from a simpler orchestration system— especially at smaller firms
  - fragmede 2y ago
    do you have a simpler orchestration system you'd recommend?
    - qohen 2y ago
      Nomad, from Hashicorp, comes to mind. https://www.nomadproject.io/ https://www.nomadproject.io/ https://github.com/hashicorp/nomad https://github.com/hashicorp/nomad
      - ahoka 2y ago
        How is it more simple?
        Ramiro 2y ago
        Every time I read about Nomad, I wonder the same. I swear I'm not trolling here, I honestly don't get how running Nomad is simpler than Kubernetes. Especially considering that there are substantially more resources and help on Kubernetes than Nomad.
        qohen 2y ago
        Well, for starters, you don't have to have your apps containerized to work with Nomad (though it can handle containers as well as executables). But for some deeper details, I'd suggest checking out the comments in this reddit thread[0] (as well as some of the linked articles therein). E.g. From a comment by /u/Golden_Age_Fallacy: A great use of Nomad is on reduce the burden of on-boarding a team(s) of developers who are unfamiliar with cloud native deployments / systems(even containers!). Nomad jobspecs are very simple and straight forward, as compared to the complexity and pure option overload you get in k8s and helm. From /u/neutralized: It's much easier to use than k8s. Easy to setup, easy to manage, much more shallow learning curve. Nothing super fancy. Just works. I migrated a startup I was at off of a self-managed k8s setup to Nomad a few years ago and they've never looked back. From /u/esity: My team is currently building out a fully automated nomad cluster service offering internally(fortune 10) It's super awesome. Easy. Little headache. Integrates with consul and vault. We are literally planning to replace thousands of vms for K8s with nomad. Containers are faster, more resilient and writing hcl is actually fun once you learn it Now, there is a rather more lengthy comment, by /u/thomasbuchinger, that goes through the pros and cons he experienced in trying Nomad out and his conclusion is that, while he wouldn't discourage anyone from using it, "k3s and a few well-known simple projects give you 80% of Nomands [sic] features. Are as easy to operate, afford you more options in the future and have a ton of documentation/tutorials...available." There are more comments in the thread and again links to a bunch of blogposts/articles/etc., including one from fly.io that seemed pretty detailed, discussing the Googly origins of both k8s and Nomad (fly.io used Nomad but found that it wasn't the best fit for them, which is also discussed in their post -- actually, I'm going to put the link to their post below[1], since I think it is worthwhile). Hope all this helps. [0] https://old.reddit.com/r/devops/comments/11nsxo3/opinions_on_hashicorp_nomad https://old.reddit.com/r/devops/comments/11nsxo3/opinions_on... [1] https://fly.io/blog/carving-the-scheduler-out-of-our-orchestrator/ https://fly.io/blog/carving-the-scheduler-out-of-our-orchest...
methodical 2y ago
Fair point but I think the key point here is unnecessary complexity versus necessary complexity. Are zero-downtime deployments and load balancing unnecessary? Perhaps for a personal project, but for any company with a consistent userbase I'd argue these are a non-negotiable, or should be anyways. In a situation where this is the expectation, k8s seems like the simplest answer, or near enough to it.
- iTokio 2y ago
  They are many ways to do deployments without downtime and load balancing is easy to configure without k8s.
darby_nine 2y ago
> It's some kind of a "Temporarily embarrassed FAANG engineer" situation. FAANG engineers made the same mistake, too, even though the analogy implies comparative competency or value.

maayank 2y ago

It’s one of those technologies where there’s merit to use them in some situations but are too often cargo culted.

caniszczyk 2y ago

Hating is a sign of success in some ways :) In some ways, it's nice to see companies move to use mostly open source infrastructure, a lot of it coming from CNCF (https://landscape.cncf.io https://landscape.cncf.io), ASF and other organizations out there (on top of the random things on github).

tryauuum 2y ago

For me it is about VMs. Feel uneasy knowing that any kernel vulnerability will allow a malicious code to escape the container and explore the kubernetes host There are kata-containers I think, they might solve my angst and make me enjoy k8s Overall... There's just nothing cool in kubernetes to me. Containers, load balancers, megabytes of yaml -- I've seen it all. Nothing feels interesting enough to try

stackskipton 2y ago
vs the Application getting hacked and running lose on the VM? If you have never dealt with, I have to run these 50 containers plus Nginx/CertBot while figuring out which node is best to run it, yea, I can see you not being thrilled about Kubernetes. For the rest of us though, Kubernetes helps out with that easily.
- tryauuum 2y ago
  if a 4-core VM with a single application is hacked, that's it if there's a kernel vulnerability in something simple (like dirtycow, which was if I remember correctly about pipes) then the attacker will take over your entire 128 core machine and all the hundreds applications there

moduspol 2y ago

For me personally, I get a little bit salty about it due to imagined, theoretical business needs of being multi-cloud, or being able to deploy on-prem someday if needed. It's tough to explain just how much longer it'll take, how much more expertise is required, how much more fragile it'll be, and how much more money it'll take to build out on Kubernetes instead of your AWS deployment model of choice (VM images on EC2, or Elastic Beanstalk, or ECS / Fargate, or Lambda). I don't want to set up or maintain my own ELK stack, or Prometheus. Or wrestle with CNI plugins. Or Kafka. Or high availability Postgres. Or Argo. Or Helm. Or control plane upgrades. I can get up and running with the AWS equivalent almost immediately, with almost no maintenance, and usually with linear costs starting near zero. I can solve business problems so, so much faster and more efficiently. It's the difference between me being able to blow away expectations and my whole team being quarters behind. That said, when there is a genuine multi-cloud or on-prem requirement, I wouldn't want to do it with anything other than k8s. And it's probably not as bad if you do actually work at a company big enough to have a lot of skilled engineers that understand k8s--that just hasn't been the case anywhere I've worked.

drawnwren 2y ago
Genuine question: how are you handling load balancing, log aggregation, failure restart + readiness checks, deployment pipelines, and machine maintenance schedules with these “simple” setups? Because as annoying as getting the prometheus + loki + tempo + promtail stack going on k8s is —- I don’t really believe that writing it from scratch is easier.
- felixgallo 2y ago
  He named the services. Go read about them.
  - drawnwren 2y ago
    I’m not sure which services you think were named that solve the problems I mentioned, but none were. You’re welcome to go read about them, I do this for a living.
- 2y ago

archenemybuntu 2y ago

Kubernetes itself is built around mostly solid distributed system principles. It's the ecosystem around it which turns things needlessly complex. Just because you have kubernetes, you don't necessarily need istio, helm, Argo cd, cilium, and whatever half baked stuff is pushed by CNCF yesterday. For example take a look at helm. Its templating is atrocious, and if I am still correct, it doesn't have a way to order resources properly except hooks. Sometimes resource A (deployment) depends on resource B (some CRD). The culture around kubernetes dictates you bring in everything pushed by CNCF. And most of these stuff are half baked MVPs. --- The word devops has created expectations that back end developer should be fighting kubernetes if something goes wrong. --- Containerization is done poorly by many orgs, no care about security and image size. That's a rant for another day. I suspect this isn't a big reason for kubernetes hate here.