4 ms·

My question when looking at Kubernetes for small teams is always the same. Why? In the blog, there are multiple days of downtime, a complete cluster rebuild, a

My question when looking at Kubernetes for small teams is always the same. Why? In the blog, there are multiple days of downtime, a complete cluster rebuild, a description of how individual experts have to be crowned as the technology is too complex to jump in and out of in any real production environment, handling versioning of helm and k8s, a description of managing the underlying scripts to rebuild for disaster (I'm assuming there's a data persistence/backup step here that goes unmentioned!), and on, and on and on. When you're already using cloud primitives, why not use your existing expertise there, their serverless offerings, and learn the IaC tooling of choice for that provider? Yes it will be more expensive on your cloud bell. But when you measure the TCO, is it really?

teaearlgraycold 3y ago
We're starting to use k8s as a small team because the simpler offerings with GPUs available don't meet our needs. It's clear they're either built for someone else or are less reliable than an EKS cluster would be.
- nostrebored 3y ago
  I'd encourage you to look at the problem space and evaluate if ECS or an external abstraction layer (like Ray) meets your needs. I've seen both work in completely separate domains (e.g. inference on real time video streams vs. model building) -- but obviously ymmv, tech is a big domain and pretending I understand exactly what you're doing would be silly. Sometimes there is a real answer to the why!
  - teaearlgraycold 3y ago
    Ah, well today I learned about ECS. I guess we’ll migrate to that once I need to add complexity to our EKS setup. I’m new to this stuff, so it’s hard to dig through all of the possible different solutions. I looked into Ray a bit but it seemed a little too complicated vs. just running a CUDA accelerated docker container. Most of the streamlined solutions in this space are not made for full stack web developers deploying a service that happens to need a GPU. They’re for ML devs who are trying to own the production side of their part of the product.
belval 3y ago
Especially considering that the author seems to be using some Azure specific features anyway: > While being vendor-agnostic is a great idea, for us, it came with a high opportunity cost. After a while, we decided to go all-in on AKS-related Azure products, like the container registry, security scanning, auth, etc. For us, this resulted in an improved developer experience, simplified security ( centralized access management with Azure Entra Id), and more, which led to faster time-to-market and reduced costs (volume benefits).
- deleted 3y ago
  [deleted]
liveoneggs 3y ago
k8s is half-baked at best but people enjoy copy-paste yaml recipes, which half-baked products lend themselves to, so it is loved
- auspiv 3y ago
  I work for a US subsidiary of a very large oil company. We are migrating from Azure to AWS for many things (it is deemed "OneCloud"). A very large number of our new EC2 instances, and even our EKS instances, were provisioned within the last 6 months as T2 instances. Some, if we were lucky, were T3. T3 was released 10 years ago. Copy + paste indeed.
  - secondcoming 3y ago
    Think of the cost savings though!
deleted 3y ago
[deleted]
menschmanfred 3y ago
Our setup works very very well. And in smaller setups you would have a shared cluster or fully managed like gke etc.
cortesoft 3y ago
Do people try to push it that strongly for small teams? Lots of us work on bigger teams and enjoy more of the benefits. However, I also still use Kubernetes for my personal projects, because I really appreciate the level of abstraction it supplies. Everyone always points out that you can do all the things k8s does in other ways, but what I like about it defines a common way to do everything. I don't care that there are 50 ways to do it, I just like having one way. What this allows is for tools to seamlessly work together. It is trivial to have all sorts of cool functionality with minimal configuration.
- karolist 3y ago
  This. It's the npm install 100 packages and do everything with JS vs Rails arguments all over again.
- politelemon 3y ago
  > Do people try to push it that strongly for small teams? Yes. You have to understand that a lot of people without the benefit of experience will often base their technology choices on blog posts. K8S has a lot of mindshare and blog attention, so it gets seen as the only way to run a container in a production environment, while all the important aspects of it are ignored.
  - cortesoft 3y ago
    I get that, but I just get frustrated in the same way I get frustrated with all the "you don't need it" responses to any topic... what about all of us that DO work for bigger companies and DO need to use this stuff? Where can we gather to talk about it without being constantly told we don't need the features?
    - ehutch79 3y ago
      They don’t read those blogs. And if they do, the decision makers have enough experience to know that “your dog blog doesn’t need k8s” doesn’t apply to their 100000 mau app
dilyevsky 3y ago
I would think it's more dependent on technology requirements more than the size of the team. If all you need is some variation of LAMP stack, then you'd probably be better off with a paas like render, fly or the like.
- nostrebored 3y ago
  Totally a valid point! I think size of team matters as the impact of k8s ownership as a fraction of your development velocity changes immensely as you're able to afford a platform team who can build tooling to remove the cognitive load of deploying to and managing k8s. At an ~400 engineer company I worked at, k8s bugs that actually impacted our team were in the single digits over a year, but a large part of that was the platform team that managed the ecosystem around k8s deployments.
datadeft 3y ago
Same exact question I ask every single time. We just decided against k8s, again, in 2024. We are going to go with AWS ECS and Azure Container Apps (the infra has to exist in both clouds). ECS and Container Apps provides all the benefits of k8s without the cons. What we want is a to be able to execute container (Docker) images with autoscaling and control which group of instances can talk to each other. What we do not want to do: - learn all of the error modes of k8s - learn all the network modes of k8s - learn the tooling of k8s (and the pitfalls) - learn how to embed yaml into yaml the right way (I have seen some of the tools are doing this) - do upgrades of k8s and figuring out what has changed the way that is backward incompatible - learn how to manage certificates for k8s the right way - learn how to debug DNS issues in a distributed system (https://github.com/kubernetes/kubernetes/issues/110550 https://github.com/kubernetes/kubernetes/issues/110550 and many more) I could go on and on but many people and companies figured out the hard way that k8s complexity is not justified.
ManBeardPc 3y ago
My experience with Kubernetes has been mostly bad. I always see an explosion of complexity and there is something that needs fixing all the time. The knowledge required comes on top of the existing stack. Maybe I'm biased and just have the wrong kind of projects, but so far everything I encountered could be built with a simple tech stack on virtual or native hardware. A reverse proxy/webserver, some frontend library/framework, a backend, database, maybe some queues/logs/caching solutions on any server Linux distribution. Maintenance is minimal, dirt cheap, no vendor lock-in and easy to teach. Is everyone building the next Amazon/Netflix/Goole and needs to scale to infinity? I feel there is such a huge amount of software and companies that will never require or benefit from Kubernetes.
- mhitza 3y ago
  Company CTOs in my experience get sold very easily the idea of infinite scalability. In practice not many companies reach that point, but many that go down this road have to build on top of dozens of layers of compute/networking abstractions that only few experts on the team can manage, if any, competently. I think the cost of self-managed Linux VMs and monoliths is smaller than the cloud vendors made it seem. Containers are nice when you have to deal with a language like Python and it's packaging ecosystem, but when Go/Rust/.Net/etc binaries are placed in containers as well... I think sight of what we're trying to solve in real life has been kind of lost.
  - ManBeardPc 3y ago
    Monoliths are so much easier for smaller teams. No additional tooling needed, no service discovery, instead of networks calls you have function calls, can share resources, etc. Much less overhead as well, so you may not even need to scale. The amount of requests a single Go/Rust server can handle on a dedicated machine is insanely high with modern hardware.
marcinzm 3y ago
> But when you measure the TCO, is it really? I'm in the ML space and every small company I try to avoid EKS. Then I hate my life. Sagemaker, for example, is a giant abstracted away mess with random holes (ie: these types of jobs don't work on this GPU type, etc.) compared to just running things on EKS. The same goes to trying to deploy a more complex third party application. I could just deploy their Helm chart or I could spend a lot of time deploying it somehow in our environment.
- nostrebored 3y ago
  I had to sell Sagemaker and I agree. It is the wrong abstraction layer without the right escape hatches. I am super pro Ray for handling these types of workloads now. Huge shout out to anyone here working on maintaining that project.
bionsystem 3y ago
I see it all the time at different layers of the stack. At some point some knowledge is lost due to people turnover and the solution is to change the technology, instead of paying somebody full time to re-understand it. Why not rewrite XXX part in YYY language as nobody understands XXX anymore ? Linux VMs require a good sysadmin with a taste in digging into existing scripts and playbooks. With Kubernetes we can start from scratch and we only needs a kubernetes expert ! (or so they say). Right now I'm working a lot with an oversized maven configuration that nobody understands ; I'm paid only to dig into it and maybe refactor some parts. It's made way too complicated for the task and does a lot of non-standard stuff to work around problems it created itself. But when I arrived people were blaming jenkins and wanted to move to gitlab because jenkins was becoming too complicated to work around maven (also !). Next thing you know somebody could try kubernetes or moving to the cloud or switching from RedHat to NixOS or whatever, and the problem would still be maven.

teaearlgraycold 3y ago

We're starting to use k8s as a small team because the simpler offerings with GPUs available don't meet our needs. It's clear they're either built for someone else or are less reliable than an EKS cluster would be.

nostrebored 3y ago
I'd encourage you to look at the problem space and evaluate if ECS or an external abstraction layer (like Ray) meets your needs. I've seen both work in completely separate domains (e.g. inference on real time video streams vs. model building) -- but obviously ymmv, tech is a big domain and pretending I understand exactly what you're doing would be silly. Sometimes there is a real answer to the why!
- teaearlgraycold 3y ago
  Ah, well today I learned about ECS. I guess we’ll migrate to that once I need to add complexity to our EKS setup. I’m new to this stuff, so it’s hard to dig through all of the possible different solutions. I looked into Ray a bit but it seemed a little too complicated vs. just running a CUDA accelerated docker container. Most of the streamlined solutions in this space are not made for full stack web developers deploying a service that happens to need a GPU. They’re for ML devs who are trying to own the production side of their part of the product.

belval 3y ago

Especially considering that the author seems to be using some Azure specific features anyway: > While being vendor-agnostic is a great idea, for us, it came with a high opportunity cost. After a while, we decided to go all-in on AKS-related Azure products, like the container registry, security scanning, auth, etc. For us, this resulted in an improved developer experience, simplified security ( centralized access management with Azure Entra Id), and more, which led to faster time-to-market and reduced costs (volume benefits).

deleted 3y ago
[deleted]

liveoneggs 3y ago

k8s is half-baked at best but people enjoy copy-paste yaml recipes, which half-baked products lend themselves to, so it is loved

auspiv 3y ago
I work for a US subsidiary of a very large oil company. We are migrating from Azure to AWS for many things (it is deemed "OneCloud"). A very large number of our new EC2 instances, and even our EKS instances, were provisioned within the last 6 months as T2 instances. Some, if we were lucky, were T3. T3 was released 10 years ago. Copy + paste indeed.
- secondcoming 3y ago
  Think of the cost savings though!

deleted 3y ago

[deleted]

menschmanfred 3y ago

Our setup works very very well. And in smaller setups you would have a shared cluster or fully managed like gke etc.

cortesoft 3y ago

Do people try to push it that strongly for small teams? Lots of us work on bigger teams and enjoy more of the benefits. However, I also still use Kubernetes for my personal projects, because I really appreciate the level of abstraction it supplies. Everyone always points out that you can do all the things k8s does in other ways, but what I like about it defines a common way to do everything. I don't care that there are 50 ways to do it, I just like having one way. What this allows is for tools to seamlessly work together. It is trivial to have all sorts of cool functionality with minimal configuration.

karolist 3y ago
This. It's the npm install 100 packages and do everything with JS vs Rails arguments all over again.
politelemon 3y ago
> Do people try to push it that strongly for small teams? Yes. You have to understand that a lot of people without the benefit of experience will often base their technology choices on blog posts. K8S has a lot of mindshare and blog attention, so it gets seen as the only way to run a container in a production environment, while all the important aspects of it are ignored.
- cortesoft 3y ago
  I get that, but I just get frustrated in the same way I get frustrated with all the "you don't need it" responses to any topic... what about all of us that DO work for bigger companies and DO need to use this stuff? Where can we gather to talk about it without being constantly told we don't need the features?
  - ehutch79 3y ago
    They don’t read those blogs. And if they do, the decision makers have enough experience to know that “your dog blog doesn’t need k8s” doesn’t apply to their 100000 mau app

dilyevsky 3y ago

I would think it's more dependent on technology requirements more than the size of the team. If all you need is some variation of LAMP stack, then you'd probably be better off with a paas like render, fly or the like.

nostrebored 3y ago
Totally a valid point! I think size of team matters as the impact of k8s ownership as a fraction of your development velocity changes immensely as you're able to afford a platform team who can build tooling to remove the cognitive load of deploying to and managing k8s. At an ~400 engineer company I worked at, k8s bugs that actually impacted our team were in the single digits over a year, but a large part of that was the platform team that managed the ecosystem around k8s deployments.

datadeft 3y ago

Same exact question I ask every single time. We just decided against k8s, again, in 2024. We are going to go with AWS ECS and Azure Container Apps (the infra has to exist in both clouds). ECS and Container Apps provides all the benefits of k8s without the cons. What we want is a to be able to execute container (Docker) images with autoscaling and control which group of instances can talk to each other. What we do not want to do: - learn all of the error modes of k8s - learn all the network modes of k8s - learn the tooling of k8s (and the pitfalls) - learn how to embed yaml into yaml the right way (I have seen some of the tools are doing this) - do upgrades of k8s and figuring out what has changed the way that is backward incompatible - learn how to manage certificates for k8s the right way - learn how to debug DNS issues in a distributed system (https://github.com/kubernetes/kubernetes/issues/110550 https://github.com/kubernetes/kubernetes/issues/110550 and many more) I could go on and on but many people and companies figured out the hard way that k8s complexity is not justified.

ManBeardPc 3y ago

My experience with Kubernetes has been mostly bad. I always see an explosion of complexity and there is something that needs fixing all the time. The knowledge required comes on top of the existing stack. Maybe I'm biased and just have the wrong kind of projects, but so far everything I encountered could be built with a simple tech stack on virtual or native hardware. A reverse proxy/webserver, some frontend library/framework, a backend, database, maybe some queues/logs/caching solutions on any server Linux distribution. Maintenance is minimal, dirt cheap, no vendor lock-in and easy to teach. Is everyone building the next Amazon/Netflix/Goole and needs to scale to infinity? I feel there is such a huge amount of software and companies that will never require or benefit from Kubernetes.

mhitza 3y ago
Company CTOs in my experience get sold very easily the idea of infinite scalability. In practice not many companies reach that point, but many that go down this road have to build on top of dozens of layers of compute/networking abstractions that only few experts on the team can manage, if any, competently. I think the cost of self-managed Linux VMs and monoliths is smaller than the cloud vendors made it seem. Containers are nice when you have to deal with a language like Python and it's packaging ecosystem, but when Go/Rust/.Net/etc binaries are placed in containers as well... I think sight of what we're trying to solve in real life has been kind of lost.
- ManBeardPc 3y ago
  Monoliths are so much easier for smaller teams. No additional tooling needed, no service discovery, instead of networks calls you have function calls, can share resources, etc. Much less overhead as well, so you may not even need to scale. The amount of requests a single Go/Rust server can handle on a dedicated machine is insanely high with modern hardware.

marcinzm 3y ago

> But when you measure the TCO, is it really? I'm in the ML space and every small company I try to avoid EKS. Then I hate my life. Sagemaker, for example, is a giant abstracted away mess with random holes (ie: these types of jobs don't work on this GPU type, etc.) compared to just running things on EKS. The same goes to trying to deploy a more complex third party application. I could just deploy their Helm chart or I could spend a lot of time deploying it somehow in our environment.

nostrebored 3y ago
I had to sell Sagemaker and I agree. It is the wrong abstraction layer without the right escape hatches. I am super pro Ray for handling these types of workloads now. Huge shout out to anyone here working on maintaining that project.

bionsystem 3y ago

I see it all the time at different layers of the stack. At some point some knowledge is lost due to people turnover and the solution is to change the technology, instead of paying somebody full time to re-understand it. Why not rewrite XXX part in YYY language as nobody understands XXX anymore ? Linux VMs require a good sysadmin with a taste in digging into existing scripts and playbooks. With Kubernetes we can start from scratch and we only needs a kubernetes expert ! (or so they say). Right now I'm working a lot with an oversized maven configuration that nobody understands ; I'm paid only to dig into it and maybe refactor some parts. It's made way too complicated for the task and does a lot of non-standard stuff to work around problems it created itself. But when I arrived people were blaming jenkins and wanted to move to gitlab because jenkins was becoming too complicated to work around maven (also !). Next thing you know somebody could try kubernetes or moving to the cloud or switching from RedHat to NixOS or whatever, and the problem would still be maven.