7 ms·

A couple years ago I came into a company that had recently switched almost everything to Kubernetes. The company expected the devs to learn how to own and opera

A couple years ago I came into a company that had recently switched almost everything to Kubernetes. The company expected the devs to learn how to own and operate the cluster. Most of the devs there couldn't care less about anything except how to deploy their code to it (probably rightfully so). But I was shocked to find the k8s api was exposed to the internet and unauthenticated. I brought the problem up with the higher-ups and they were pretty lethargic about it. Moral of the story I guess is if you're gonna choose to operate infrastructure like Kubernetes make sure you have experienced people trained to do so and don't expect your devs to wear another hat.

andhapp 3y ago
DevOps is the new way, I think. Developers create the code and also know how to run and maintain infra :-)
- sharts 3y ago
  I don’t think that’s true anymore. Most companies that seriously deploy k8s tend to have dedicated teams for it. There’s no way an app developer is going to be able to do their work while also knowing the ins and outs of cluster mgmt.
  - bonestamp2 3y ago
    > Most companies that seriously deploy k8s tend to have dedicated teams for it. Ya, and everywhere I've worked, that team is called "devops". The application developers are a different team. Maybe that's not the norm, but I thought it was.
    - funcDropShadow 3y ago
      > Ya, and everywhere I've worked, that team is called "devops". But then that team is only doing operations and not development. Unless they are committers to k8s.
      - bonestamp2 3y ago
        Yes, in my experience they've always done development for the operational tasks.
  - politelemon 3y ago
    That's the problem, many companies don't realize they need to be serious, they just have a few people who want k8s in their stack.
  - jml78 3y ago
    They don’t need to know k8s administration but they need to understand how their services are going to function and scale in k8s.
    - funcDropShadow 3y ago
      But somebody has to know k8s administration.
      - jml78 3y ago
        Yeah, that is me bro. I started back in 2016 building k8s clusters from scratch via terraform and bash scripting. It was ugly awful but worked thru Covid with zero downtime when we switched to hosted solutions from AWS and Google cloud.
  - hosh 3y ago
    There are app developers like that. There is a class of problems where you need to be able to do both application development and system work to be able to do well. What a lot of people miss about larger k8s deployment is that you can create an internal platform team, like a micro-heroku within the company. The company would have to hire people who can create _products_ and also knowing how to work with infra in order to create a platform. It would be this rare combination of product, dev, ops, and focusing on supporting the rest of the engineering team and giving up on the glory of launching something for end users.
    - marcus_holmes 3y ago
      This is one of those cycles that IT does regularly every other decade. In the early days, computers were huge things that were maintained by a team of specialists (and who actually soldered things together). Getting anything changed involved a series of paper forms and waiting for a specialist to do it for you. Then came the microcomputer revolution and computers became small things that sat on desks and were maintained by the office geek. Changing anything meant talking to him (it was always him) and it would be done by lunchtime, and you owed him a coffee. Then companies learned that this meant that no two offices had the same configuration, application suite, or even operating system. IT was centralised, and every office issued with the same stuff. Getting anything changed meant filling in forms and waiting for System Administrators to get around to it. Then Agile and the Cloud happened and while the rest of the company had to carry on filling in forms, the IT Department turned into Engineering and DevOps was born. Every developer learned how to deploy their stuff to the cloud by themselves. Then organisations worked out that that meant that applications were deployed badly, using a wide variety of tools and technologies, and essential security practices got skipped because no-one was responsible for them. So DevOps was turned back into SysAdmin and the forms reinvented as JIRA tickets. And so we go on. There are micro-cycles within these ones, like the one about terminals (are you typing on the actual computer, or on a terminal that talks to the actual computer which is somewhere else? The answer depends on which decade you're in). The ideal solution is somewhere in between (enough central control for it not to get into a mess, but enough decentralisation to prevent bureaucracy and allow for agility in the actual business). But that is a very hard balance to find and keep.
  - metalforever 3y ago
    Everywhere I've been recently has devs doing the ops. This includes big places you have probably heard of.
- mac-chaffee 3y ago
  DevOps still must draw the line somewhere. DevOps people typically don't physically plug cables into switches, for instance. With the ever-growing complexity of infra, I see that line having to shift more and more, with either PaaS products or platform engineering teams filling the space by providing something like Kubernetes as a service to the DevOps folks.
- ramraj07 3y ago
  Or do what actually productive people do and use managed services like Elastic Beanstalk and RDS.
  - pc86 3y ago
    If that makes you productive just imagine how productive you'd be if you pushed your code to a VM and were done.
- starttoaster 3y ago
  It usually doesn't work that way, in my experience. My title is "Senior DevOps Engineer" and I do my fair share of writing code as well as maintaining infra. But honestly, hiring for people who do what I do seems to be very hit and miss. Most people excel at one or the other, and so they only pick up tickets that fit their skillset. They might dive into a ticket outside their comfort zone here or there, but not commonly. It would seem to me that many DevOps engineering teams are just one team who has a manager that hired and manages some Devs and some Ops folk, and just wrangles each of them into doing tasks that fit the goal of the team and their individual talents. And from an outsider's perspective it would seem like one team of jacks of all trades, but internally it's very much known who is going to take on certain kinds of tasks and the siloes are all very much in place.
  - diarrhea 3y ago
    That situation sounds reasonable though. I don’t believe in tearing down all silos. Expertise and preferences exist, it’s human nature. I always thought the DevOps mantra of having one person be good at both was a fruitless endeavour. Have Dev and Ops collaborate closely, yes, but let them be separate people (not teams!). It’s fine. You know how to dev and ops, and I don’t doubt it. I do too. But if we focused that same energy and expertise, we’d be double the devs/ops. Nevertheless, people like us are excellent glue for organisations. At the same time, teams and organisations of only us would be quite pointless. I feel similarly about fullstack. You’re a front end dev that has written a controller and used an ORM before. Often not too inspiring. Backend devs are still needed, whereas full stack devs form excellent glue.
    - jahewson 3y ago
      > Have Dev and Ops collaborate closely That’s what DevOps is… or, was. I remember when it was new; that was the whole thing: dev and ops should work together. A break from the norm of the time that dev throws it over the wall to ops and never thinks about it again. What a tragedy that it’s been cargo-culted into its current meaning of “devs do ops” (poorly).
    - starttoaster 3y ago
- yjftsjthsd-h 3y ago
  You're replying to an anecdote about devs not knowing - or caring at least - how to run and maintain infra
lakomen 3y ago
That's what I've been saying all along. For me, when I use something I need to understand it. I'm usually a quick learner (3rd iteration is where I start making sense of a complex topic). I beat my head against k8s for 2 weeks, every day, I wanted to setup a 3 node control pane on my own, and it was just too hard. I gave up. Also, I don't have many resources at my disposal, k8s requires a separate storage cluster. All-together I would've required 9 servers, 3 control pane, 3 app or pods, 3 storage. Way too expensive. And then think further, all this has to be updated sometime. I don't have the brain and time resources to do that. So, if I want k8s, I need to pay for expensive cloud. Or, I can remain with my 1 server with backups and build systems on demand. Yes, I hear the booksmarties scream "but but failsafety". 1 incubator server, specialized systems when something takes off. Some companies can afford to pay for expensive cloud, more power to them. I can't. I'm busy solving problems at hand, not dealing with additional problems of a system I don't fully understand. Yes, k8s is a nice comfortable way, if you can afford it. But it's expensive and wasteful of resources. And we do have a climate crisis.
- mjburgess 3y ago
  It's better to pay the cost of abstraction when you've good evidence of its necessity
- ncrmro 3y ago
  K3s/k9s can be ran on a single server and gives you the declarative benefits and the entire helm ecosystem. Everyone assumes you have to run multi node.
- thinkmassive 3y ago
  > All-together I would've required 9 servers, 3 control pane, 3 app or pods, 3 storage. … Yes, k8s is a nice comfortable way, if you can afford it. But it’s expensive and wasteful of resources. And we do have a climate crisis. You can run each of those roles on the same machine, achieving high availability from 3 total. For learning purposes you can run this all in VMs on a single machine. There are plenty of good reasons not to try kubernetes when it doesn’t fit your use case, but the overhead it imposes is mostly in time and expertise. Claiming it’s more costly in compute resources, and especially contributing climate crisis, is reaching.
ChicagoDave 3y ago
The bigger issue is that container deployment and management is a big reseller business and they actually don’t care about whether it’s appropriate or not or if the eventual maintenance and DevOps are in place.
gz5 3y ago
and initiatives to make devs/ops/systems use vpn, bastion, static permitted IPs, etc. are often DOA due to their impacts on automation and speed. a foss alternative is kubeztl - it takes kubectl off the networks - no public IP at all - but w/o users needing agents: https://github.com/openziti-test-kitchen/kubeztl/tree/main https://github.com/openziti-test-kitchen/kubeztl/tree/main disclosure: i am a maintainer and the software overlay in the middle (helps enforce outbound-only, pre-authorized connects only) needs to be managed (self-hosted foss or hosted saas), so there are still trade-offs.
aprdm 3y ago
Yes, same with cloud and giving devs full access to AWS/Azure/You-name-it. Disaster waiting to happen
ern 3y ago
I increasingly feel like the development community maintains an equilibrium of "Accidental Complexity" and "Essential Complexity". As better methods for managing Essential Complexity come to the fore, Accidental Complexity gets jacked up to keep the workload high, so the geeks stay stimulated and the consultancies collect the $$$.
- steve1977 3y ago
  I‘m not even sure it’s an equilibrium. I’m in tech for around 25 years now, I feel developer productivity and output is not the same but lower today.
  - renegade-otter 3y ago
    Between mind-blowing complexity and never-ending distractions, I find doing actual work near impossible. After getting laid off, I just... didn't look for work. I instead started writing my own product for pull request assignments and direct Slack notifications for those (my personal frustration). I needed to get away from it all and decompress. It's been refreshing writing code.
- travisgriggs 3y ago
  I have been feeling this for a while now. It’s like the army of eager labor, the ranks of managers who derive value from larger work forces, and sick amount of VC money, has really eclipsed emphases on various axes of efficiency seeking that used to dominate the industry (often in overrated/oversold ways admittedly). Remember when “tech stack X enabled you to do y per coney more with z fewer developers?
- renegade-otter 3y ago
  In VC-backed startups, "growth" does not refer to user growth or profits. Most of the "daily active user" metrics are bullshit. I've worked for a company once that gave away employees free funding money to make purchases. Yes, to get the number of orders up before a board meeting. It was blatant, unethical, but not really illegal. So what is growth? The number of developers you have doing stuff. You hire more and more, and every few months you parade some Patagonia vests through your offices to show them. "Observe, more devs! Growth!" The problems is, the devs all have to do something. I stopped feeling productive years ago. It's all just never-ending grind, dealing with complexity that has no f--ing business being there. And the younger engineers have no idea. They think this is normal.
uberduper 3y ago
I don't recall Kubernetes ever allowing unauthenticated access to the api. I could be wrong tho since I never considered trying to run it sans auth.
- parasubvert 3y ago
  Kubernetes (and many popular distros) allowing anonymous access, even when authentication is enabled. The system:anonymous user is generally not authorized to view resources, but might be inadvertently. To test this, if you hit the K8s API server unauthenticated, do you get a 401 error or a 403 error? If the latter, you're at risk.
  - uberduper 3y ago
    I didn't word my statement well. Yes, you can enable unauthenticated requests and you can grant the anonymous role access to the api. I didn't recall `--anonymous-auth` to be true by default.
    - peddling-brink 3y ago
      It is on certain clouds. But that isn't overtly dangerous until someone binds system:anonymous to cluster admin.
    - raesene9 3y ago
      Most of the major cloud distros (AKS being the notable exception) do have --anonymous-auth enabled, although it's generally just /version and a couple of other endpoints exposed. Makes it easy to find out what clusters are running what versions of k8s which is interesting, but not a major security issue.
- raesene9 3y ago
  It did in the old days of the "Insecure API port" which listened on 8080/TCP, but that's been gone a while now. Whilst it usually was just bound to localhost, I did encounter a distribution that, by default, bound it to the container network, meaning that anyone with access to one pod, got to be cluster admin! Generally speaking though I don't think any distributions do that now.

andhapp 3y ago

DevOps is the new way, I think. Developers create the code and also know how to run and maintain infra :-)

sharts 3y ago
I don’t think that’s true anymore. Most companies that seriously deploy k8s tend to have dedicated teams for it. There’s no way an app developer is going to be able to do their work while also knowing the ins and outs of cluster mgmt.
- bonestamp2 3y ago
  > Most companies that seriously deploy k8s tend to have dedicated teams for it. Ya, and everywhere I've worked, that team is called "devops". The application developers are a different team. Maybe that's not the norm, but I thought it was.
  - funcDropShadow 3y ago
    > Ya, and everywhere I've worked, that team is called "devops". But then that team is only doing operations and not development. Unless they are committers to k8s.
    - bonestamp2 3y ago
      Yes, in my experience they've always done development for the operational tasks.
- politelemon 3y ago
  That's the problem, many companies don't realize they need to be serious, they just have a few people who want k8s in their stack.
- jml78 3y ago
  They don’t need to know k8s administration but they need to understand how their services are going to function and scale in k8s.
  - funcDropShadow 3y ago
    But somebody has to know k8s administration.
    - jml78 3y ago
      Yeah, that is me bro. I started back in 2016 building k8s clusters from scratch via terraform and bash scripting. It was ugly awful but worked thru Covid with zero downtime when we switched to hosted solutions from AWS and Google cloud.
- hosh 3y ago
  There are app developers like that. There is a class of problems where you need to be able to do both application development and system work to be able to do well. What a lot of people miss about larger k8s deployment is that you can create an internal platform team, like a micro-heroku within the company. The company would have to hire people who can create _products_ and also knowing how to work with infra in order to create a platform. It would be this rare combination of product, dev, ops, and focusing on supporting the rest of the engineering team and giving up on the glory of launching something for end users.
  - marcus_holmes 3y ago
    This is one of those cycles that IT does regularly every other decade. In the early days, computers were huge things that were maintained by a team of specialists (and who actually soldered things together). Getting anything changed involved a series of paper forms and waiting for a specialist to do it for you. Then came the microcomputer revolution and computers became small things that sat on desks and were maintained by the office geek. Changing anything meant talking to him (it was always him) and it would be done by lunchtime, and you owed him a coffee. Then companies learned that this meant that no two offices had the same configuration, application suite, or even operating system. IT was centralised, and every office issued with the same stuff. Getting anything changed meant filling in forms and waiting for System Administrators to get around to it. Then Agile and the Cloud happened and while the rest of the company had to carry on filling in forms, the IT Department turned into Engineering and DevOps was born. Every developer learned how to deploy their stuff to the cloud by themselves. Then organisations worked out that that meant that applications were deployed badly, using a wide variety of tools and technologies, and essential security practices got skipped because no-one was responsible for them. So DevOps was turned back into SysAdmin and the forms reinvented as JIRA tickets. And so we go on. There are micro-cycles within these ones, like the one about terminals (are you typing on the actual computer, or on a terminal that talks to the actual computer which is somewhere else? The answer depends on which decade you're in). The ideal solution is somewhere in between (enough central control for it not to get into a mess, but enough decentralisation to prevent bureaucracy and allow for agility in the actual business). But that is a very hard balance to find and keep.
- metalforever 3y ago
  Everywhere I've been recently has devs doing the ops. This includes big places you have probably heard of.
mac-chaffee 3y ago
DevOps still must draw the line somewhere. DevOps people typically don't physically plug cables into switches, for instance. With the ever-growing complexity of infra, I see that line having to shift more and more, with either PaaS products or platform engineering teams filling the space by providing something like Kubernetes as a service to the DevOps folks.
ramraj07 3y ago
Or do what actually productive people do and use managed services like Elastic Beanstalk and RDS.
- pc86 3y ago
  If that makes you productive just imagine how productive you'd be if you pushed your code to a VM and were done.
starttoaster 3y ago
It usually doesn't work that way, in my experience. My title is "Senior DevOps Engineer" and I do my fair share of writing code as well as maintaining infra. But honestly, hiring for people who do what I do seems to be very hit and miss. Most people excel at one or the other, and so they only pick up tickets that fit their skillset. They might dive into a ticket outside their comfort zone here or there, but not commonly. It would seem to me that many DevOps engineering teams are just one team who has a manager that hired and manages some Devs and some Ops folk, and just wrangles each of them into doing tasks that fit the goal of the team and their individual talents. And from an outsider's perspective it would seem like one team of jacks of all trades, but internally it's very much known who is going to take on certain kinds of tasks and the siloes are all very much in place.
- diarrhea 3y ago
  That situation sounds reasonable though. I don’t believe in tearing down all silos. Expertise and preferences exist, it’s human nature. I always thought the DevOps mantra of having one person be good at both was a fruitless endeavour. Have Dev and Ops collaborate closely, yes, but let them be separate people (not teams!). It’s fine. You know how to dev and ops, and I don’t doubt it. I do too. But if we focused that same energy and expertise, we’d be double the devs/ops. Nevertheless, people like us are excellent glue for organisations. At the same time, teams and organisations of only us would be quite pointless. I feel similarly about fullstack. You’re a front end dev that has written a controller and used an ORM before. Often not too inspiring. Backend devs are still needed, whereas full stack devs form excellent glue.
  - jahewson 3y ago
    > Have Dev and Ops collaborate closely That’s what DevOps is… or, was. I remember when it was new; that was the whole thing: dev and ops should work together. A break from the norm of the time that dev throws it over the wall to ops and never thinks about it again. What a tragedy that it’s been cargo-culted into its current meaning of “devs do ops” (poorly).
  - starttoaster 3y ago
yjftsjthsd-h 3y ago
You're replying to an anecdote about devs not knowing - or caring at least - how to run and maintain infra

lakomen 3y ago

That's what I've been saying all along. For me, when I use something I need to understand it. I'm usually a quick learner (3rd iteration is where I start making sense of a complex topic). I beat my head against k8s for 2 weeks, every day, I wanted to setup a 3 node control pane on my own, and it was just too hard. I gave up. Also, I don't have many resources at my disposal, k8s requires a separate storage cluster. All-together I would've required 9 servers, 3 control pane, 3 app or pods, 3 storage. Way too expensive. And then think further, all this has to be updated sometime. I don't have the brain and time resources to do that. So, if I want k8s, I need to pay for expensive cloud. Or, I can remain with my 1 server with backups and build systems on demand. Yes, I hear the booksmarties scream "but but failsafety". 1 incubator server, specialized systems when something takes off. Some companies can afford to pay for expensive cloud, more power to them. I can't. I'm busy solving problems at hand, not dealing with additional problems of a system I don't fully understand. Yes, k8s is a nice comfortable way, if you can afford it. But it's expensive and wasteful of resources. And we do have a climate crisis.

mjburgess 3y ago
It's better to pay the cost of abstraction when you've good evidence of its necessity
ncrmro 3y ago
K3s/k9s can be ran on a single server and gives you the declarative benefits and the entire helm ecosystem. Everyone assumes you have to run multi node.
thinkmassive 3y ago
> All-together I would've required 9 servers, 3 control pane, 3 app or pods, 3 storage. … Yes, k8s is a nice comfortable way, if you can afford it. But it’s expensive and wasteful of resources. And we do have a climate crisis. You can run each of those roles on the same machine, achieving high availability from 3 total. For learning purposes you can run this all in VMs on a single machine. There are plenty of good reasons not to try kubernetes when it doesn’t fit your use case, but the overhead it imposes is mostly in time and expertise. Claiming it’s more costly in compute resources, and especially contributing climate crisis, is reaching.

ChicagoDave 3y ago

The bigger issue is that container deployment and management is a big reseller business and they actually don’t care about whether it’s appropriate or not or if the eventual maintenance and DevOps are in place.

gz5 3y ago

and initiatives to make devs/ops/systems use vpn, bastion, static permitted IPs, etc. are often DOA due to their impacts on automation and speed. a foss alternative is kubeztl - it takes kubectl off the networks - no public IP at all - but w/o users needing agents: https://github.com/openziti-test-kitchen/kubeztl/tree/main https://github.com/openziti-test-kitchen/kubeztl/tree/main disclosure: i am a maintainer and the software overlay in the middle (helps enforce outbound-only, pre-authorized connects only) needs to be managed (self-hosted foss or hosted saas), so there are still trade-offs.

aprdm 3y ago

Yes, same with cloud and giving devs full access to AWS/Azure/You-name-it. Disaster waiting to happen

ern 3y ago

I increasingly feel like the development community maintains an equilibrium of "Accidental Complexity" and "Essential Complexity". As better methods for managing Essential Complexity come to the fore, Accidental Complexity gets jacked up to keep the workload high, so the geeks stay stimulated and the consultancies collect the $$$.

steve1977 3y ago
I‘m not even sure it’s an equilibrium. I’m in tech for around 25 years now, I feel developer productivity and output is not the same but lower today.
- renegade-otter 3y ago
  Between mind-blowing complexity and never-ending distractions, I find doing actual work near impossible. After getting laid off, I just... didn't look for work. I instead started writing my own product for pull request assignments and direct Slack notifications for those (my personal frustration). I needed to get away from it all and decompress. It's been refreshing writing code.
travisgriggs 3y ago
I have been feeling this for a while now. It’s like the army of eager labor, the ranks of managers who derive value from larger work forces, and sick amount of VC money, has really eclipsed emphases on various axes of efficiency seeking that used to dominate the industry (often in overrated/oversold ways admittedly). Remember when “tech stack X enabled you to do y per coney more with z fewer developers?
renegade-otter 3y ago
In VC-backed startups, "growth" does not refer to user growth or profits. Most of the "daily active user" metrics are bullshit. I've worked for a company once that gave away employees free funding money to make purchases. Yes, to get the number of orders up before a board meeting. It was blatant, unethical, but not really illegal. So what is growth? The number of developers you have doing stuff. You hire more and more, and every few months you parade some Patagonia vests through your offices to show them. "Observe, more devs! Growth!" The problems is, the devs all have to do something. I stopped feeling productive years ago. It's all just never-ending grind, dealing with complexity that has no f--ing business being there. And the younger engineers have no idea. They think this is normal.

uberduper 3y ago

I don't recall Kubernetes ever allowing unauthenticated access to the api. I could be wrong tho since I never considered trying to run it sans auth.

parasubvert 3y ago
Kubernetes (and many popular distros) allowing anonymous access, even when authentication is enabled. The system:anonymous user is generally not authorized to view resources, but might be inadvertently. To test this, if you hit the K8s API server unauthenticated, do you get a 401 error or a 403 error? If the latter, you're at risk.
- uberduper 3y ago
  I didn't word my statement well. Yes, you can enable unauthenticated requests and you can grant the anonymous role access to the api. I didn't recall `--anonymous-auth` to be true by default.
  - peddling-brink 3y ago
    It is on certain clouds. But that isn't overtly dangerous until someone binds system:anonymous to cluster admin.
  - raesene9 3y ago
    Most of the major cloud distros (AKS being the notable exception) do have --anonymous-auth enabled, although it's generally just /version and a couple of other endpoints exposed. Makes it easy to find out what clusters are running what versions of k8s which is interesting, but not a major security issue.
raesene9 3y ago
It did in the old days of the "Insecure API port" which listened on 8080/TCP, but that's been gone a while now. Whilst it usually was just bound to localhost, I did encounter a distribution that, by default, bound it to the container network, meaning that anyone with access to one pod, got to be cluster admin! Generally speaking though I don't think any distributions do that now.