7 ms·

How do you people even keep up with this? I'm going back to cybersecurity after trying DevOps for a year, it's not for me. I miss my sysadmin days, things were

by 3ln00b 11mo ago

How do you people even keep up with this? I'm going back to cybersecurity after trying DevOps for a year, it's not for me. I miss my sysadmin days, things were simple back then and worked. Maybe I'm just getting old and my cognitive abilities are declining. It seems to me that the current tech scene doesn't reward simple.

gryfft 11mo ago
> It seems to me that the current tech scene doesn't reward simple. A deal with the devil was made. The C suite gets to tell a story that k8s practices let you suck every penny out of the compute you already paid for. Modern devs get to do constant busy work adding complexity everywhere, creating job security and opportunities to use fun new toys. "Here's how we're using AI to right size our pods! Never mind the actual costs and reliability compared to traditional infrastructure, we only ever need to talk about the happy path/best case scenarios."
- hexbin010 11mo ago
  Mhm! And Google just sit there laughing at everyone. Mission accomplished
- dangus 11mo ago
  This just seems like sensationalist nonsense spoken by someone who hasn’t done a second of Ops work. Kubernetes is incredibly reliable compared to traditional infrastructure. It eliminates a ton of the configuration management dependency hellscape and inconsistent application deployments that traditional infrastructure entails. Immutable containers provide a major benefit to development velocity and deployment reliability. They are far faster to pull and start than deploying to VMs, which end up needing some kind of annoying deployment pipeline involving building images or having some kind of complex and failure-prone deployment system. Does Kubernetes have its downsides? Yeah, it’s complex overkill for small deployments or monolithic applications. But to be honest, there’s a lot of complexity to configuration management on traditional VMs with a lot of bad, not-so-gracefully aging tooling (cough…Chef Software) And who is really working for a company that has a small deployment? I’d say that most medium-sized tech companies can easily justify the complexity of running a kubernetes cluster. Networking can be complex with Kubernetes, but it’s only as complex as your service architecture. These days there are more solutions than ever that remove a lot of the management burden but leave you with all the benefits of having a cluster, e.g., Talos Linux.
  - vasco 11mo ago
    It was clear they didn't know what they were saying when they think the main reason for kubernetes was to save money. Kubernetes is just easy to complain about.
    - imp0cat 11mo ago
      Exactly, if anything, Kubernetes will require a lot more money.
  - steve1977 11mo ago
    The problem is that some Kubernetes features would have a positive impact on development velocity in theory, however in my experience (25 years of ops and devops), the cost of keeping up often eats up those benefits and often results in a net-negative. This is not always a problem of Kubernetes itself though, but of teams always chasing after the latest shiny thing.
    - mlrtime 11mo ago
      Also a old man from VMS/Sparc days, I'm still doing "devops" and just deployed a realtime streaming webapp tool for our team in a few days to k8s pods. It was incredibly easy and I get so much for free Automatically created for me: - Ingress, TLS, Domain name, Deployment strategy, Dev/Prod environments through helm, Single repo configuration for source code, reproducible dev/prod build+run (Docker)... If a company sets this up correctly developers can create tooling incredibly fast without any tickets from a core infra team. It's all stable and very performant. I'd never go back to the old way of deploying applications after seeing it work well.
      - steve1977 11mo ago
        > just deployed a realtime streaming webapp tool for our team in a few days to k8s pods. How long would you estimate that deployment would have taken with more a „classic“ approach? (e.g. deploying to a Java application server)
- Glamklo 11mo ago
  [dead]
solatic 11mo ago
It's exactly why taking a trip through the ops/infra side is so important for people - you learn why LTS-style engineering is so important. You learn to pick technologies that are stable, reliable, well-supported by a large-enough people who are conservative in their approach, for anything foundational, because the alternative is migration pain again and again.
- selfmodruntime 11mo ago
  I also feel like we as an industry should steer towards a state of "doneness" for OSS solutions. As long as it works, it's fine to keep using technologies that are only sparsely maintained.
  - cyberpunk 11mo ago
    Ingress-Nginx is commonly internet facing though; I think everyone wants at least base image and ssl upgrades on that component…
    - rcxdude 11mo ago
      In which case it's even more important that the updates are not a huge amount of work.
- toredash 11mo ago
  I often find myself trying to tell people that KISS is a good thing. If something is somewhat complex it will be really complex after a few years and a few rotations of personnel.
  - friendzis 11mo ago
    Quite often the tradeoff is not between complexity (to cover a bunch of different cases) and simplicity (do one thing simply), but rather where that complexity lies. Do you have dependency fanout? It probably makes sense to shove all that complexity into the central component and manage it centrally. Otherwise it probably makes sense to make all the components a bit more complex than they could be, but still manageable.
Aeolun 11mo ago
I like devops. It means you get to get ahead of all the issues that you could potentially find in cybersecurity. Sure it's complicated, but at least you'll never be bored. I think the hardest part is that you always feel like you don't have enough time to do everything you need to.
- dangus 11mo ago
  DevOps teams are always running slightly behind and rarely getting ahead of technical debt because they are treated as cost centers by the business (perpetually understaffed) and as “last minute complicated requests that sound simple enough” and “oops our requirements changed” dumping grounds for engineering teams. Plus, the ops side has a lot of challenges that can really be a different beast compared to the application side. The breadth of knowledge needed for the job is staggering and yet you also need depth in terms of knowing how operating systems and networks work.
  - immibis 11mo ago
    > DevOps teams are always running slightly behind and rarely getting ahead of technical debt because they are treated as cost centers by the business This is one of those explanations that sounds reasonable but when you actually experience it you realize the explanation makes no sense. If you're "running behind of technical debt" you'll always feel understaffed no matter how much staffing you have. And adding more staffing will make your tech debt worse. Plus, tech debt doesn't really exist. It's a metaphor for all the little annoyances in your system that add up, but the metaphor makes it sound like it's the problem of management or accounting to solve when it's actually created by developers and solved by developers.
    - dangus 11mo ago
      Hmm, no, you were changing the meaning of my comment. > no matter how much staffing you have That’s not what I said. I said that there tends to be not enough staff. Businesses are more willing to hire software engineers (shipping features = revenue) than hiring DevOps people (keeping the lights on). > tech debt doesn’t really exist. Well that’s news to me. I’m pretty sure it exists. It has an entire Wikipedia article, and that article doesn’t agree with your definition. And yes, more staff would help. Hiring me literally helped my organization fix its lack of monitoring and alerting because nobody had time to address the problem during my team’s day to day responsibilities. Your assertion that it’s not management’s fault is absurd. Management is by definition the bearer if ultimate responsibility. Every problem in any business is something where the buck stops at management. If I ship something with long term problems because management told me to work faster and meet the deadline, that is directly management’s fault. Even me shipping something bad on my own volition is management‘s fault indirectly: they hired the wrong talent (me), or maybe they assigned me to the wrong project where my expertise wasn’t good enough, or they misjudged risks and didn’t leave enough contingency buffer or didn’t make a plan for what to do if we fail. The way businesses view humans are as machine-like resources of labor (Human Resources), they don’t view you as an individual with emotions and thoughts and feelings. When they hire someone they have quantitative measures surrounding that person: how likely are they to perform well, burn out and quit, steal from the company, get run over by a bus, etc. The corporate system actually dictates that management is responsible for the way it arranges and commands its human machines.
makeitdouble 11mo ago
> things were simple back then If you were working in the orgs targeted by k8s, I think it was generally more of a mess. Think about managing a park of 100~200 servers with home made bash scripts and crappy monitoring tools and a modicum of dashboards. Now, k8s has engulfed a lot more than the primary target, but smaller shops go for it because they'r also hoping to hit it big someday I guess. Otherwise, there will be far easier solutions at lower scale.
- dangus 11mo ago
  Even after the bash script era, I don’t think the configuration management landscape gets enough discredit for how bad it is. I never felt like it stopped feeling hacked together and unreliable. E.g., Chef Software, especially after its acquisition, is just a dumpster fire of weird anti-patterns and seemingly incomplete, buggy implementations. Ansible is more of the gold standard but I actually moved to Chef to gain a little more capability. But now I hate both of them. When I just threw this all in the trash in my HomeLab and went to containerization it was a major breath of fresh air and resulted in getting a lot of time back. For organizations, of the best parts about Kubernetes is that it’s so agnostic so that you can drop in replacements with a level of ease that is just about unheard of in the Ops world. If you are a small shop you can just start with something simpler and more manageable like k3s or Talos Linux and basically get all the benefits without the full blown k8s management burden. Would it be simpler to use plain Docker, Docker Swarm, Portainer, something like that? Yeah, but the amount of effort saved versus your ability to adapt in the future seems to favor just choosing Kubernetes as a default option.
  - deleted 11mo ago
    [deleted]
  - reissbaker 11mo ago
    Yup. K8s is a bit of a pain to keep up with, but Chef and even Ansible are much more painful for other reasons once you have more than a handful of nodes to manage. It's also basically a standard API that every cloud provider is forced to implement, meaning it's really easy to onboard new compute from almost anyone. Each K8s cloud provider has its own little quirks, but it's much simpler than the massive sea of difference that each cloud's unique API for VM management was (and the tools to paper over that were generally very leaky abstractions in the pre-K8s world).
merb 11mo ago
ingress-nginx is older than 5-7 years tough. In that time frame you would’ve needed to update your Linux system, which gets hairy most often as well. The sad thing is just that the replacement is just not there and gateway api has a lot of drawbacks that might get fixed in the next release (working with cert manager)
cmckn 11mo ago
The Ingress API has been on ice for like 5 years. The core Kubernetes API doesn't change that much, at least these days. There's an infinite number of (questionable) add-ons you can deploy in your cluster, and I think that's mostly where folks get stuck in the mud.
- sph 11mo ago
  > doesn’t change that much Yet they are retiring a core Ingress that has been around for almost as long as Kubernetes has.
  - deleted 11mo ago
    [deleted]
  - deleted 11mo ago
    [deleted]
  - pestaa 11mo ago
    They are not retiring the API. Nginx Ingress is one of the many projects that implements this API, and you are free to migrate to another implementation.
- cesnja 11mo ago
  But the Gateway API has only been generally available for two years now. And the last time I checked, most managed K8S solutions recommend the Ingress API while Gateway support is still experimental.
  - deleted 11mo ago
    [deleted]
  - p_l 11mo ago
    We also now have multiple full featured Ingress implementations that work better than the old nginx-ingress
- yrro 11mo ago
  And here's me still using OpenShift routes... :)
nunez 11mo ago
/r/kubernetes had this announcement up about five mins after it dropped at Kubecon. It's a huge deal. So many tutorials and products used ingress-nginx for basic ingress, so them throwing in the towel (but not really) is big news. That said, (a) the Gateway API supercedes Ingress and provides much more functionality without much more complexity, and (b) NGINX and HAproxy have Gateway controllers. To generally answer your question, I use HN, /r/devops and /r/kubernetes to stay current. I'm also working on a weekly blog series wherein I'll be doing an overview and quick start guide for every CNCF project in their portfolio. There's hundreds (thousands?) of projects in the collection, so it will keep me busy until I retire, probably :)
- locknitpicker 11mo ago
  > /r/kubernetes had this announcement up about five mins after it dropped at Kubecon. It's a huge deal. So many tutorials and products used ingress-nginx for basic ingress, so them throwing in the towel (but not really) is big news. I was one of those whose first reaction was surprise, because ingress was the most critical and hardest aspect of a kubernetes rollout to implement and get up and running on a vanilla deployment. It's what cloud providers offer out of the box as a major selling point to draw in customers. But then I browsed through the Gateway API docs, and it is a world of difference. It turns a hard problem that requires so many tutorials and products to help anyone get something running into a trivially solvable problem. The improvements on their security model is undoubtedly better and alone clearly justifies getting rid of ingress. Change might be inconvenient, but you need change to get rid of pain points.
  - nunez 11mo ago
    Gateway works at Layer 4 (TCPRoute, UDPRoute). Massive improvement over Services + service mesh hackery!
Tractor8626 11mo ago
Cybersecurity is easier? Isn't it all about constantly updating and patching obsolete vulnerable stuff - most annoying part of ops?
steve1977 11mo ago
In my experience, many teams keep up with this by spending a lot of time keeping up with this and less time developing the actual product. Which, you probably guessed it, results in products much shittier than what we had 10 or 20 years ago. But hey, it keeps a lot of people busy, which means it also keeps a lot of managers and consultants and trainers busy.
jitl 11mo ago
i prefer current era where i never have to ssh to debug a node. if a node is misbehaving or even needs a patch i destroy it. one command, works every time.
- secondcoming 11mo ago
  How can you not be interested in what took down your node???
  - jitl 11mo ago
    oh, i am interested, but i can’t remember the last time i needed ssh to figure out the issue, or needing to fix a node besides by destroying it. last time it was a silly app deciding to use a host volume on root partition to cache stuff using all the disk space. remediate in the moment by destroying node, fix it forever by moving the app to a node type with instance attached NVMe device and putting the volume there + container that nukes the data volume if it runs out of space.
  - otterley 11mo ago
    That’s what telemetry services are for. If you have all the logs and metrics from the host, then you can research and construct the story from those. You don’t necessarily need the host to be alive anymore.
nonameiguess 11mo ago
Honestly, a lot of the Hacker News discourse every single time anything having to do with Kubernetes comes up reads like uninformed annoyed griping from people who have barely or not used it. Kubernetes itself has been around since 2014. ingress-nginx was the original example of how to implement an Ingress controller. Ingress itself is not going away, which seems to a misconception of a lot of replies to your comment. A lot of tutorials use this because a lot of tutorials simply copied the Kubernetes upstream documentation's own tutorials, which used toy examples of how to do things, including ingress-nginx itself, which was meant to be a toy example of how to implement an Ingress controller. Nonetheless, it was around a full decade before they finally decided to retire it. It's not like this is something they introduced, advertised as the ideal fit for all production use cases, and then promptly changed their minds. It's been over a decade. Part of the problem here is the Kubernetes devs not really following their own advice, as annotations are supposed to be notes that don't implement functionality, but ingress-nginx allowed you to inject arbitrary configuration with them, which ended up being a terrible idea in the main use Kubernetes is really meant for, which is you're an organization running a multi-tenant platform offering application layer services to other organizations, which it is great for, but Hacker News with its "everything is either a week one startup or a solo indy dev" is blind to for whatever reason. Nonetheless, they still kept it alive for over a decade. Hacker News also has the exact wrong idea about who does and should use Kubernetes. It's not FAANGs, which operate at a scale way too big for it and do this kind of thing using in-house tech they develop themselves. Even Google doesn't use it. It's more for the Home Depots and BMWs of the world, organizations which are large-scale but not primarily software companies, running thousands if not millions of applications in different physical locations run by different local teams, but not necessarily serving planet-scale web users. They can deal with changing providers once every ten years. I would invite everyone who thinks this is unmanageable complexity to try dipping their toes into the legal and accounting worlds that Fortune 500s have to deal with. They can handle some complexity.
Daviey 11mo ago
Not just that, but technologies which took me many months or even years to become and expert at, the latest generation of engineers seem to be able to pick up in weeks. It's scary how fast the world is moving.
szszrk 11mo ago
It's a much less of a deal than it seems. Yeah, it is a popular project that has been around for a while, but this is just another day at work. Things evolve, there are migration paths no matter if you want to stay with ingresses or move on... Kubernetes is promoting Gateway API for a while now. It's in GA for 2 years already (while Ingress was in GA quite late, 2020/K8s 1.19?). Sun-setting ingress-nginx was not exactly a secret. The whole Ingress in k8s is marked in docs as "frozen" for a while as well. There are no radical steps yet, but it's clear that Gateway API is something to get interested in. Meanwhile Nginx Gateway Fabric [1] (which implements gateway API) is there, still uses nginx under the hood and remains opensource. They even have a "migration tool" to convert objects [3]. There are still a few months of support and time to move on to a different controller. Kubernetes still continues support for ingress so if you want to switch and keep using Ingress, there are other controllers [2]. [1] https://gateway-api.sigs.k8s.io/implementations/#nginx-gateway-fabric https://gateway-api.sigs.k8s.io/implementations/#nginx-gatew... [2] https://gateway-api.sigs.k8s.io/implementations/#gateway-controller-implementation-status https://gateway-api.sigs.k8s.io/implementations/#gateway-con... [3] https://docs.nginx.com/nginx-gateway-fabric/install/ingress-to-gateway/ https://docs.nginx.com/nginx-gateway-fabric/install/ingress-...
- KaiserPro 11mo ago
  > this is just another day at work. But the point is this, it worked, it does work and will, if given developer time continue to work. I now need to schedual in time to test the changes, then adjust the metrics and alerting that we have. For no gain. It just feels like kuberenetes is carbon fibre programming.
  - szszrk 11mo ago
    I honestly don't get it. As a person who managed k8s in teams of 1-3 people. It's not that much effort. Things get sunset all the time.
  - herzzolf 11mo ago
    > if given developer time continue to work. well that's the root of the problem, no? there's no one who wants to maintain the complex lua written to make nginx cloud native. they were looking for maintainers for quite some time with no one stepping up and I'm not surprised, their issue tracker always was full of very entitled people, so you would be doing a stressful/thankless job... for what exactly?
pjmlp 11mo ago
We don't, I focus mainly on backend, DevOps happens because in many small teams someone has to have multiple roles, and I end up taking DevOps responsibilities as well. One thing that I push for nowadays, after a few scars is managed platforms.
wvh 11mo ago
I feel the same, especially the feeling old and jaded part, but I disagree that things were easier. Systems such as Kubernetes are not worse than trying to administer a zillion servers and networks by hand in the late '90s (or with tools like Puppet and Ansible a bit later), let alone HA shenanigans; neither are they a magical solution, more of a side-step and necessary evolution of scale. There is a wild-grow of 80% solved problems in the Kubernetes space though, and especially the DevOps landscape seems to be plagued by half-solutions at the moment. I think part of the complexity arises from everything being interconnected services instead of simple stand-alone software binaries. Things talking with other things, not necessarily from the same maker or ecosystem. I don't understand decisions such as these though, retiring de facto standards such as Ingress NGINX. I can't name a single of our customers at $WORKPLACE that's running something else.
brookritz 11mo ago
I once installed some kubernetes based software by following the instructions and watching many unicode/ascii-art animations on the commandline. I've also learned that the 8 in k8s stands for 8 letters: 'ubernete'. I've decided that D4s is not for me.
mmcnl 11mo ago
Things weren't simpler. The complexity was simply not visible because different teams/department were all doing a small part of what now a single team is doing with Kubernetes. Yes, for that single team it is more complex. But now it's 1 team that does it all, instead of 5 separate teams responsible for development, storage, networking, disaster recovery, etc. Kubernetes is a gift.
zzyzxd 11mo ago
If your infrastructure can justify the complexity of Kubernetes, keeping up with Kubernetes native software is extremely easy comparing to anything else I have dealt with. I had some horror story managing nginx instances on 3 servers with ansible. To me that's much harder than working with ingress controllers in Kubernetes. Replacing an ingress controller in Kubernetes is also a well documented practice, with minimum or even zero downtime if you want to. Generally, if your engineering team can reasonably keep things simple, it's good. However, business needs to grow and infrastructure needs to scale out. Sometimes trying too hard to be simple is, in my experience, how things become unmanageably complex. I find well-engineered complexity to be much more pleasant to work with.

by 3ln00b 11mo ago

gryfft 11mo ago

> It seems to me that the current tech scene doesn't reward simple. A deal with the devil was made. The C suite gets to tell a story that k8s practices let you suck every penny out of the compute you already paid for. Modern devs get to do constant busy work adding complexity everywhere, creating job security and opportunities to use fun new toys. "Here's how we're using AI to right size our pods! Never mind the actual costs and reliability compared to traditional infrastructure, we only ever need to talk about the happy path/best case scenarios."

hexbin010 11mo ago
Mhm! And Google just sit there laughing at everyone. Mission accomplished
dangus 11mo ago
This just seems like sensationalist nonsense spoken by someone who hasn’t done a second of Ops work. Kubernetes is incredibly reliable compared to traditional infrastructure. It eliminates a ton of the configuration management dependency hellscape and inconsistent application deployments that traditional infrastructure entails. Immutable containers provide a major benefit to development velocity and deployment reliability. They are far faster to pull and start than deploying to VMs, which end up needing some kind of annoying deployment pipeline involving building images or having some kind of complex and failure-prone deployment system. Does Kubernetes have its downsides? Yeah, it’s complex overkill for small deployments or monolithic applications. But to be honest, there’s a lot of complexity to configuration management on traditional VMs with a lot of bad, not-so-gracefully aging tooling (cough…Chef Software) And who is really working for a company that has a small deployment? I’d say that most medium-sized tech companies can easily justify the complexity of running a kubernetes cluster. Networking can be complex with Kubernetes, but it’s only as complex as your service architecture. These days there are more solutions than ever that remove a lot of the management burden but leave you with all the benefits of having a cluster, e.g., Talos Linux.
- vasco 11mo ago
  It was clear they didn't know what they were saying when they think the main reason for kubernetes was to save money. Kubernetes is just easy to complain about.
  - imp0cat 11mo ago
    Exactly, if anything, Kubernetes will require a lot more money.
- steve1977 11mo ago
  The problem is that some Kubernetes features would have a positive impact on development velocity in theory, however in my experience (25 years of ops and devops), the cost of keeping up often eats up those benefits and often results in a net-negative. This is not always a problem of Kubernetes itself though, but of teams always chasing after the latest shiny thing.
  - mlrtime 11mo ago
    Also a old man from VMS/Sparc days, I'm still doing "devops" and just deployed a realtime streaming webapp tool for our team in a few days to k8s pods. It was incredibly easy and I get so much for free Automatically created for me: - Ingress, TLS, Domain name, Deployment strategy, Dev/Prod environments through helm, Single repo configuration for source code, reproducible dev/prod build+run (Docker)... If a company sets this up correctly developers can create tooling incredibly fast without any tickets from a core infra team. It's all stable and very performant. I'd never go back to the old way of deploying applications after seeing it work well.
    - steve1977 11mo ago
      > just deployed a realtime streaming webapp tool for our team in a few days to k8s pods. How long would you estimate that deployment would have taken with more a „classic“ approach? (e.g. deploying to a Java application server)
Glamklo 11mo ago
[dead]

solatic 11mo ago

It's exactly why taking a trip through the ops/infra side is so important for people - you learn why LTS-style engineering is so important. You learn to pick technologies that are stable, reliable, well-supported by a large-enough people who are conservative in their approach, for anything foundational, because the alternative is migration pain again and again.

selfmodruntime 11mo ago
I also feel like we as an industry should steer towards a state of "doneness" for OSS solutions. As long as it works, it's fine to keep using technologies that are only sparsely maintained.
- cyberpunk 11mo ago
  Ingress-Nginx is commonly internet facing though; I think everyone wants at least base image and ssl upgrades on that component…
  - rcxdude 11mo ago
    In which case it's even more important that the updates are not a huge amount of work.
toredash 11mo ago
I often find myself trying to tell people that KISS is a good thing. If something is somewhat complex it will be really complex after a few years and a few rotations of personnel.
- friendzis 11mo ago
  Quite often the tradeoff is not between complexity (to cover a bunch of different cases) and simplicity (do one thing simply), but rather where that complexity lies. Do you have dependency fanout? It probably makes sense to shove all that complexity into the central component and manage it centrally. Otherwise it probably makes sense to make all the components a bit more complex than they could be, but still manageable.

Aeolun 11mo ago

I like devops. It means you get to get ahead of all the issues that you could potentially find in cybersecurity. Sure it's complicated, but at least you'll never be bored. I think the hardest part is that you always feel like you don't have enough time to do everything you need to.

dangus 11mo ago
DevOps teams are always running slightly behind and rarely getting ahead of technical debt because they are treated as cost centers by the business (perpetually understaffed) and as “last minute complicated requests that sound simple enough” and “oops our requirements changed” dumping grounds for engineering teams. Plus, the ops side has a lot of challenges that can really be a different beast compared to the application side. The breadth of knowledge needed for the job is staggering and yet you also need depth in terms of knowing how operating systems and networks work.
- immibis 11mo ago
  > DevOps teams are always running slightly behind and rarely getting ahead of technical debt because they are treated as cost centers by the business This is one of those explanations that sounds reasonable but when you actually experience it you realize the explanation makes no sense. If you're "running behind of technical debt" you'll always feel understaffed no matter how much staffing you have. And adding more staffing will make your tech debt worse. Plus, tech debt doesn't really exist. It's a metaphor for all the little annoyances in your system that add up, but the metaphor makes it sound like it's the problem of management or accounting to solve when it's actually created by developers and solved by developers.
  - dangus 11mo ago
    Hmm, no, you were changing the meaning of my comment. > no matter how much staffing you have That’s not what I said. I said that there tends to be not enough staff. Businesses are more willing to hire software engineers (shipping features = revenue) than hiring DevOps people (keeping the lights on). > tech debt doesn’t really exist. Well that’s news to me. I’m pretty sure it exists. It has an entire Wikipedia article, and that article doesn’t agree with your definition. And yes, more staff would help. Hiring me literally helped my organization fix its lack of monitoring and alerting because nobody had time to address the problem during my team’s day to day responsibilities. Your assertion that it’s not management’s fault is absurd. Management is by definition the bearer if ultimate responsibility. Every problem in any business is something where the buck stops at management. If I ship something with long term problems because management told me to work faster and meet the deadline, that is directly management’s fault. Even me shipping something bad on my own volition is management‘s fault indirectly: they hired the wrong talent (me), or maybe they assigned me to the wrong project where my expertise wasn’t good enough, or they misjudged risks and didn’t leave enough contingency buffer or didn’t make a plan for what to do if we fail. The way businesses view humans are as machine-like resources of labor (Human Resources), they don’t view you as an individual with emotions and thoughts and feelings. When they hire someone they have quantitative measures surrounding that person: how likely are they to perform well, burn out and quit, steal from the company, get run over by a bus, etc. The corporate system actually dictates that management is responsible for the way it arranges and commands its human machines.

makeitdouble 11mo ago

> things were simple back then If you were working in the orgs targeted by k8s, I think it was generally more of a mess. Think about managing a park of 100~200 servers with home made bash scripts and crappy monitoring tools and a modicum of dashboards. Now, k8s has engulfed a lot more than the primary target, but smaller shops go for it because they'r also hoping to hit it big someday I guess. Otherwise, there will be far easier solutions at lower scale.

dangus 11mo ago
Even after the bash script era, I don’t think the configuration management landscape gets enough discredit for how bad it is. I never felt like it stopped feeling hacked together and unreliable. E.g., Chef Software, especially after its acquisition, is just a dumpster fire of weird anti-patterns and seemingly incomplete, buggy implementations. Ansible is more of the gold standard but I actually moved to Chef to gain a little more capability. But now I hate both of them. When I just threw this all in the trash in my HomeLab and went to containerization it was a major breath of fresh air and resulted in getting a lot of time back. For organizations, of the best parts about Kubernetes is that it’s so agnostic so that you can drop in replacements with a level of ease that is just about unheard of in the Ops world. If you are a small shop you can just start with something simpler and more manageable like k3s or Talos Linux and basically get all the benefits without the full blown k8s management burden. Would it be simpler to use plain Docker, Docker Swarm, Portainer, something like that? Yeah, but the amount of effort saved versus your ability to adapt in the future seems to favor just choosing Kubernetes as a default option.
- deleted 11mo ago
  [deleted]
- reissbaker 11mo ago
  Yup. K8s is a bit of a pain to keep up with, but Chef and even Ansible are much more painful for other reasons once you have more than a handful of nodes to manage. It's also basically a standard API that every cloud provider is forced to implement, meaning it's really easy to onboard new compute from almost anyone. Each K8s cloud provider has its own little quirks, but it's much simpler than the massive sea of difference that each cloud's unique API for VM management was (and the tools to paper over that were generally very leaky abstractions in the pre-K8s world).

merb 11mo ago

ingress-nginx is older than 5-7 years tough. In that time frame you would’ve needed to update your Linux system, which gets hairy most often as well. The sad thing is just that the replacement is just not there and gateway api has a lot of drawbacks that might get fixed in the next release (working with cert manager)

cmckn 11mo ago

The Ingress API has been on ice for like 5 years. The core Kubernetes API doesn't change that much, at least these days. There's an infinite number of (questionable) add-ons you can deploy in your cluster, and I think that's mostly where folks get stuck in the mud.

sph 11mo ago
> doesn’t change that much Yet they are retiring a core Ingress that has been around for almost as long as Kubernetes has.
- deleted 11mo ago
  [deleted]
- deleted 11mo ago
  [deleted]
- pestaa 11mo ago
  They are not retiring the API. Nginx Ingress is one of the many projects that implements this API, and you are free to migrate to another implementation.
cesnja 11mo ago
But the Gateway API has only been generally available for two years now. And the last time I checked, most managed K8S solutions recommend the Ingress API while Gateway support is still experimental.
- deleted 11mo ago
  [deleted]
- p_l 11mo ago
  We also now have multiple full featured Ingress implementations that work better than the old nginx-ingress
yrro 11mo ago
And here's me still using OpenShift routes... :)

nunez 11mo ago

/r/kubernetes had this announcement up about five mins after it dropped at Kubecon. It's a huge deal. So many tutorials and products used ingress-nginx for basic ingress, so them throwing in the towel (but not really) is big news. That said, (a) the Gateway API supercedes Ingress and provides much more functionality without much more complexity, and (b) NGINX and HAproxy have Gateway controllers. To generally answer your question, I use HN, /r/devops and /r/kubernetes to stay current. I'm also working on a weekly blog series wherein I'll be doing an overview and quick start guide for every CNCF project in their portfolio. There's hundreds (thousands?) of projects in the collection, so it will keep me busy until I retire, probably :)

locknitpicker 11mo ago
> /r/kubernetes had this announcement up about five mins after it dropped at Kubecon. It's a huge deal. So many tutorials and products used ingress-nginx for basic ingress, so them throwing in the towel (but not really) is big news. I was one of those whose first reaction was surprise, because ingress was the most critical and hardest aspect of a kubernetes rollout to implement and get up and running on a vanilla deployment. It's what cloud providers offer out of the box as a major selling point to draw in customers. But then I browsed through the Gateway API docs, and it is a world of difference. It turns a hard problem that requires so many tutorials and products to help anyone get something running into a trivially solvable problem. The improvements on their security model is undoubtedly better and alone clearly justifies getting rid of ingress. Change might be inconvenient, but you need change to get rid of pain points.
- nunez 11mo ago
  Gateway works at Layer 4 (TCPRoute, UDPRoute). Massive improvement over Services + service mesh hackery!

Tractor8626 11mo ago

Cybersecurity is easier? Isn't it all about constantly updating and patching obsolete vulnerable stuff - most annoying part of ops?

steve1977 11mo ago

In my experience, many teams keep up with this by spending a lot of time keeping up with this and less time developing the actual product. Which, you probably guessed it, results in products much shittier than what we had 10 or 20 years ago. But hey, it keeps a lot of people busy, which means it also keeps a lot of managers and consultants and trainers busy.

jitl 11mo ago

i prefer current era where i never have to ssh to debug a node. if a node is misbehaving or even needs a patch i destroy it. one command, works every time.

secondcoming 11mo ago
How can you not be interested in what took down your node???
- jitl 11mo ago
  oh, i am interested, but i can’t remember the last time i needed ssh to figure out the issue, or needing to fix a node besides by destroying it. last time it was a silly app deciding to use a host volume on root partition to cache stuff using all the disk space. remediate in the moment by destroying node, fix it forever by moving the app to a node type with instance attached NVMe device and putting the volume there + container that nukes the data volume if it runs out of space.
- otterley 11mo ago
  That’s what telemetry services are for. If you have all the logs and metrics from the host, then you can research and construct the story from those. You don’t necessarily need the host to be alive anymore.

nonameiguess 11mo ago

Honestly, a lot of the Hacker News discourse every single time anything having to do with Kubernetes comes up reads like uninformed annoyed griping from people who have barely or not used it. Kubernetes itself has been around since 2014. ingress-nginx was the original example of how to implement an Ingress controller. Ingress itself is not going away, which seems to a misconception of a lot of replies to your comment. A lot of tutorials use this because a lot of tutorials simply copied the Kubernetes upstream documentation's own tutorials, which used toy examples of how to do things, including ingress-nginx itself, which was meant to be a toy example of how to implement an Ingress controller. Nonetheless, it was around a full decade before they finally decided to retire it. It's not like this is something they introduced, advertised as the ideal fit for all production use cases, and then promptly changed their minds. It's been over a decade. Part of the problem here is the Kubernetes devs not really following their own advice, as annotations are supposed to be notes that don't implement functionality, but ingress-nginx allowed you to inject arbitrary configuration with them, which ended up being a terrible idea in the main use Kubernetes is really meant for, which is you're an organization running a multi-tenant platform offering application layer services to other organizations, which it is great for, but Hacker News with its "everything is either a week one startup or a solo indy dev" is blind to for whatever reason. Nonetheless, they still kept it alive for over a decade. Hacker News also has the exact wrong idea about who does and should use Kubernetes. It's not FAANGs, which operate at a scale way too big for it and do this kind of thing using in-house tech they develop themselves. Even Google doesn't use it. It's more for the Home Depots and BMWs of the world, organizations which are large-scale but not primarily software companies, running thousands if not millions of applications in different physical locations run by different local teams, but not necessarily serving planet-scale web users. They can deal with changing providers once every ten years. I would invite everyone who thinks this is unmanageable complexity to try dipping their toes into the legal and accounting worlds that Fortune 500s have to deal with. They can handle some complexity.

Daviey 11mo ago

Not just that, but technologies which took me many months or even years to become and expert at, the latest generation of engineers seem to be able to pick up in weeks. It's scary how fast the world is moving.

szszrk 11mo ago

It's a much less of a deal than it seems. Yeah, it is a popular project that has been around for a while, but this is just another day at work. Things evolve, there are migration paths no matter if you want to stay with ingresses or move on... Kubernetes is promoting Gateway API for a while now. It's in GA for 2 years already (while Ingress was in GA quite late, 2020/K8s 1.19?). Sun-setting ingress-nginx was not exactly a secret. The whole Ingress in k8s is marked in docs as "frozen" for a while as well. There are no radical steps yet, but it's clear that Gateway API is something to get interested in. Meanwhile Nginx Gateway Fabric [1] (which implements gateway API) is there, still uses nginx under the hood and remains opensource. They even have a "migration tool" to convert objects [3]. There are still a few months of support and time to move on to a different controller. Kubernetes still continues support for ingress so if you want to switch and keep using Ingress, there are other controllers [2]. [1] https://gateway-api.sigs.k8s.io/implementations/#nginx-gateway-fabric https://gateway-api.sigs.k8s.io/implementations/#nginx-gatew... [2] https://gateway-api.sigs.k8s.io/implementations/#gateway-controller-implementation-status https://gateway-api.sigs.k8s.io/implementations/#gateway-con... [3] https://docs.nginx.com/nginx-gateway-fabric/install/ingress-to-gateway/ https://docs.nginx.com/nginx-gateway-fabric/install/ingress-...

KaiserPro 11mo ago
> this is just another day at work. But the point is this, it worked, it does work and will, if given developer time continue to work. I now need to schedual in time to test the changes, then adjust the metrics and alerting that we have. For no gain. It just feels like kuberenetes is carbon fibre programming.
- szszrk 11mo ago
  I honestly don't get it. As a person who managed k8s in teams of 1-3 people. It's not that much effort. Things get sunset all the time.
- herzzolf 11mo ago
  > if given developer time continue to work. well that's the root of the problem, no? there's no one who wants to maintain the complex lua written to make nginx cloud native. they were looking for maintainers for quite some time with no one stepping up and I'm not surprised, their issue tracker always was full of very entitled people, so you would be doing a stressful/thankless job... for what exactly?

pjmlp 11mo ago

We don't, I focus mainly on backend, DevOps happens because in many small teams someone has to have multiple roles, and I end up taking DevOps responsibilities as well. One thing that I push for nowadays, after a few scars is managed platforms.

wvh 11mo ago

I feel the same, especially the feeling old and jaded part, but I disagree that things were easier. Systems such as Kubernetes are not worse than trying to administer a zillion servers and networks by hand in the late '90s (or with tools like Puppet and Ansible a bit later), let alone HA shenanigans; neither are they a magical solution, more of a side-step and necessary evolution of scale. There is a wild-grow of 80% solved problems in the Kubernetes space though, and especially the DevOps landscape seems to be plagued by half-solutions at the moment. I think part of the complexity arises from everything being interconnected services instead of simple stand-alone software binaries. Things talking with other things, not necessarily from the same maker or ecosystem. I don't understand decisions such as these though, retiring de facto standards such as Ingress NGINX. I can't name a single of our customers at $WORKPLACE that's running something else.

brookritz 11mo ago

I once installed some kubernetes based software by following the instructions and watching many unicode/ascii-art animations on the commandline. I've also learned that the 8 in k8s stands for 8 letters: 'ubernete'. I've decided that D4s is not for me.

mmcnl 11mo ago

Things weren't simpler. The complexity was simply not visible because different teams/department were all doing a small part of what now a single team is doing with Kubernetes. Yes, for that single team it is more complex. But now it's 1 team that does it all, instead of 5 separate teams responsible for development, storage, networking, disaster recovery, etc. Kubernetes is a gift.

zzyzxd 11mo ago

If your infrastructure can justify the complexity of Kubernetes, keeping up with Kubernetes native software is extremely easy comparing to anything else I have dealt with. I had some horror story managing nginx instances on 3 servers with ansible. To me that's much harder than working with ingress controllers in Kubernetes. Replacing an ingress controller in Kubernetes is also a well documented practice, with minimum or even zero downtime if you want to. Generally, if your engineering team can reasonably keep things simple, it's good. However, business needs to grow and infrastructure needs to scale out. Sometimes trying too hard to be simple is, in my experience, how things become unmanageably complex. I find well-engineered complexity to be much more pleasant to work with.