13 ms·

These kinds of posts always focus on the complexity of running k8s, the large amount of concepts it has, the lack of a need to scale, and that there is a "wide

by jorams 7y ago

These kinds of posts always focus on the complexity of running k8s, the large amount of concepts it has, the lack of a need to scale, and that there is a "wide variety of tools" that can replace it, but the advice never seems to become more concrete. We are running a relatively small system on k8s. The cluster contains just a few nodes, a couple of which are serving web traffic and a variable number of others that are running background workers. The number of background workers is scaled up based on the amount of work to be done, then scaled down once no longer necessary. Some cronjobs trigger every once in a while. It runs on GKE. All of this could run on anything that runs containers, and the scaling could probably be replaced by a single beefy server. In fact, we can run all of this on a single developer machine if there is no load. The following k8s concepts are currently visible to us developers: Pod, Deployment, Job, CronJob, Service, Ingress, ConfigMap, Secret. The hardest one to understand is Ingress, because it is mapped to a GCE load balancer. All the rest is predictable and easy to grasp. I know k8s is a monster to run, but none of us have to deal with that part at all. Running on GKE gives us the following things, in addition to just running it all, without any effort on our part: centralized logging, centralized monitoring with alerts, rolling deployments with easy rollbacks, automatic VM scaling, automatic VM upgrades. How would we replace GKE in this equation? what would we have to give up? What new tools and concepts would we need to learn? How much of those would be vendor-specific? If anyone has a solution that is actually simpler and just as easy to set up, I'm very much interested.

envoked 7y ago
I'm in the same camp. I think a lot of these anti-k8s articles are written by software developers who haven't really been exposed to the world of SRE and mostly think in terms of web servers. A few years ago I joined a startup where everything (including the db) was running on one, not-backed-up, non-reproducible, VM. In the process of "productionizing" I ran into a lot of open questions: How do we handle deploys with potentially updated system dependencies? Where should we store secrets (not the repo)? How do we manage/deploy cronjobs? How do internal services communicate? All things a dedicated SRE team managed in my previous role. GKE offered a solution to each of those problems while allowing me to still focus on application development. There's definitely been some growing pains (prematurely trying to run our infra on ephemeral nodes) but for the most part, it's provided a solid foundation without much effort.
- mslm 7y ago
  Exactly, all these articles seem to come from operational novices, who think in terms of 1-2 click solutions. K8s is not a 1-2 click solution, and clearly isn't designed to be; it's solving particular tough operational problems that if you don't know exist in the first place you won't really be able to evaluate these kinds of things properly. If a group literally doesn't have the need to answer questions like the ones you posed, then OK, don't bother with these tools. But that's all that needs to be said - no need for a new article every week on it.
  - collyw 7y ago
    > it's solving particular tough operational problems that if you don't know exist in the first place They probably don't exist for the majority of people using it. We are using k8s for when we need to scale, but at the moment we have a handful of customers and it isn't changing quickly any time soon.
    - ownagefool 7y ago
      They basically exist for everyone. As soon as you go down the road of actually doing infrastructure-as-code, using (not running) k8s is probably as good as any other solution, and arguably better than most when you grow into anything complex. Most of the complaints are false equivalence: i.e. running k8s is harder than just using AWS, which I already know. Of course it is. You don't manage AWS. How big do you think their code base is? If you don't know k8s already, and you're a start-up looking for a niche, maybe now isn't the time to learn k8s, at least not from the business point of view (personal growth, another issue). But when you do know k8s, it makes a lot of sense to just rent a cluster and put your app there, because when you want to build better tests, it's easy, when you want to do zero trust, it's easy, when you want to integrate with vault, it's easy, when you want to encrypt, it's easy, when you want to add a mesh for tracing, metrics and maybe auth, it's easy. What's not easy is inheriting a similarly done product that's entirely bespoke.
      - collyw 7y ago
LoSboccacc 7y ago
same we use kubernet not for scale but for the environment repeatability. we can spin any branch at any time on any provider and be sure it's exactly as we have it in production down to networking and routing, to built that out of plain devops tools would require a lot more in depth knowledge and it won't port exactly one provider to another
VvR-Ox 7y ago
> These kinds of posts always focus on the complexity of running k8s... Yes but that is also the worst already that you could criticize about k8s. Complexity is dangerous because if things are growing beyond a certain threshold X you will have side effects that nobody can predict, a very steep learning curve and therefor many people screwing up something in their (first) setups as well as maintainability nightmares. Probably some day someone will prove me wrong but right now one of my biggest goals to improve security, reliability and people being able to contribute is reducing complexity. After all this is what many of us do when they refactor systems. I am sticking with the UNIX philosophy at this point and in the foreseeable future I will not have a big dev-team at my disposal as companies like Alphabet have to maintain and safe-guard all of this complexity.
- hedora 7y ago
  From a developer perspective, k8s seems ripe for disruption. It does a bunch of junk that is trivial to accomplish on one machine - open network ports, keep services running, log stuff, run in a jail with dropped privileges, and set proper file permissions on secret files. The mechanisms for all of this, and for resource management, are transparent to unix developers, but in kubernetes they are not. Instead, you have to understand a architectural spaghetti torrent to write and execute “hello world”. It used to be similar with RDBMS systems. It took months and a highly paid consultant to get a working SQL install. Then, you’d hire a team to manage the database, not because the hardware was expensive, but because you’d dropped $100k’s (in 90’s dollars) on the installation process. Then mysql came along, and it didn’t have durability or transactions, but it let you be up and running in a few hours, and have a dynamic web page a few hours after that. If it died, you only lost a few hours or minutes of transactions, assuming somone in your organization spent an afternoon learning cron and mysqldump. I imagine someone will get sufficiently fed up with k8s to do the same. There is clearly demand. I wish them luck.
  - p_l 7y ago
    It seems to be. When you start implementing something, you'll soon find that most of that complexity is inherent, not accidental. Been there, done that, didn't even get the T-Shirt. Today it's much easier to package nicer API on top of the rather generic k8s one. There are ways to deploy it easier (in fact, I'd wager that a lot of complexity in deploying k8s is accidental due to deploy tools themselves, not k8s itself. Just look at OpenShift deploy scripts...)
Glyptodon 7y ago
You're asking exactly what I've been wondering. The answer in this thread this so far has been "maybe dokku." I can totally buy that K8s is overkill, but the alternatives for small scale operators seem to require a lot more work internally than just using a K8s service.
geggam 7y ago
Running k8s on GKE is the only rational choice for anyone who isnt a large enterprise with a team of Google SREs to support k8s. Period.
rclayton 7y ago
1000%. If you take a little bit of time to learn k8s and run it on a hosted environment (e.g. EKS), it’s a fantastic solution. We are much happier with it than ECS, Elastic Beanstalk, etc.
AlchemistCamp 7y ago
> If anyone has a solution that is actually simpler and just as easy to set up, I'm very much interested. Sure. @levelsio runs Nomad List (~100k MRR) all on a single Linode VPS. He uses their automated backups service, but it's a simple setup. No k8s, no Docker, just some PHP running on a Linux server. As I understand it, he was learning to code as he built his businesses. https://twitter.com/levelsio/status/1177562806192730113 https://twitter.com/levelsio/status/1177562806192730113
- icedchai 7y ago
  Many startups are not as resource efficient. The ones I am familiar with spend 50%+ of their MRR on cloud costs.
  - p_l 7y ago
    My first production k8s use was actually due to resource efficiencies. We looked at the >50 applications we had to serve, the technical debt that would lead us down the road of building customized distribution to avoid package requirements incompabilties, and various other things, and decided to go with containers - just to avoid the conflicts involved. Thanks to k8s, we generally keep to 1/5th of the original cost, thanks to bin packing of servers, and sleep sounder thanks to automatic restarts of failed pods, ability to easily allocate computing resources per container, globall configure load balancing (we had to scratch use of cloud-provider's load balancer because our number of paths was too big for URL mapping API). Everything can be moved to pretty much every k8s hosting that runs 1.15, biggest difference would be hooking the load balancer to the external network and possibly storage.
  - AlchemistCamp 7y ago
    That's very surprising to me! As an employee at multiple startups in multiple countries, I've never seen cloud costs anywhere near payroll.
    - icedchai 7y ago
      Well, I said 50% of monthly revenue, not total costs.
halbritt 7y ago
I'm curious how, with your experience, that you came to the conclusion that k8s is a monster to run?
hrgiger 7y ago
This, kubernetes is container orchestration framework. If you know your needs then you have opportunity to make a better decision and planning. I had similar experience with ingress also I would like to add installing Kubernetes on bare metal is a pain in the neck, even with kubespray. Here is a comparison with other frameworks, from 2018: https://arxiv.org/pdf/2002.02806.pdf https://arxiv.org/pdf/2002.02806.pdf
honkycat 7y ago
Agreed. I've been saying for years that if you go with Docker-Compose, or Amazon ECS, or something lower level, you are just going to end up rebuilding a shittier version of Kubernetes. I think the real alternative is Heroku or running on VMs, but then you do not get service discovery, or a cloud agnostic API for querying running services, or automatic restarts, or rolling updates, or encrypted secrets, or automatic log aggregation, or plug-and-play monitoring, or VM scaling, or an EXCELLENT decoupled solution for deploying my applications ( keel.sh ), or liveness and readiness probes... But nobody needs those things right?
- NicoJuicy 7y ago
  Why would docker be shittier? It's easier and flexible. Eg. Sidecar pattern resolves most things (eg. logging)
- sk5t 7y ago
  You do in fact get a lot of this stuff with ECS and Fargate - rolling updates, automatic restart, log aggregation, auto scaling, some discovery bits, healthchecks, Secrets Manager or Parameter Store if you want, etc.
  - throwaway894345 7y ago
    This. We’ve been chugging along happily on Fargate for a while now. We looked into EKS, but there is a ton of stuff you have to do manually that is built into Fargate or at least Fargate makes it trivial to integrate with other AWS services (logging, monitoring, IAM integration, cert manager, load balancers, secrets management, etc). I’d like to use Kubernetes, but Fargate just works out of the box. Fortunately, AWS seems to be making EKS better all the time.
    - Logishort 7y ago
      And what if Amazon feels like changing Fargate or charging differently or deprecating it or... Whats your strategy for that?
rawoke083600 7y ago
"Pod, Deployment, Job, CronJob, Service, Ingress, ConfigMap, Secre" Wow as a new developer coming onboard your company, I will walk out the door after seeing that, and the fact that you admit its a small serivce.
- nrb 7y ago
  It's an afternoon worth of research to understand the basic concepts. Then, with the powerful and intuitive tooling you can spin up your own cluster on your computer in minutes and practice deploying containers that: - are automatically assigned to an appropriate machine(node) based on explicit resource limits you define, enabling reliable performance - horizontally scale (even automatically if you want!) - can be deployed with a rolling update strategy to preserve uptime during deployments - can rollback with swiftness and ease - have liveness checks that restart unhealthy apps(pods) automatically and prevent bad deploys from being widely released - abstracts away your infrastructure, allowing these exact same configs to power a cluster on-prem, in the cloud on bare metal or vms, with a hosted k8s service, or some combination of all of them All of that functionality is unlocked with just a few lines of config or kubectl command, and there are tools that abstract this stuff to simplify it even more or automate more of it. You definitely want some experienced people around to avoid some of the footguns and shortcuts but over the last several years I think k8s has easily proven itself as a substantial net-positive for many shops.
  - scarface74 7y ago
    So why should I do all of that instead of throwing a little money at AWS, run ECS and actually spend my time creating my product? Heck, if my needs are simple enough why should I even use ECS instead of just putting my web app on some VM's in an auto-scaling group behind a load balancer and used managed services?
    - geggam 7y ago
      Hate to tell you this but that second idea is much easier to manage from an operations standpoint and significantly more reliable. Also easier to debug and monitor... but you run your business to make developers happy right ?

by jorams 7y ago

envoked 7y ago

I'm in the same camp. I think a lot of these anti-k8s articles are written by software developers who haven't really been exposed to the world of SRE and mostly think in terms of web servers. A few years ago I joined a startup where everything (including the db) was running on one, not-backed-up, non-reproducible, VM. In the process of "productionizing" I ran into a lot of open questions: How do we handle deploys with potentially updated system dependencies? Where should we store secrets (not the repo)? How do we manage/deploy cronjobs? How do internal services communicate? All things a dedicated SRE team managed in my previous role. GKE offered a solution to each of those problems while allowing me to still focus on application development. There's definitely been some growing pains (prematurely trying to run our infra on ephemeral nodes) but for the most part, it's provided a solid foundation without much effort.

mslm 7y ago
Exactly, all these articles seem to come from operational novices, who think in terms of 1-2 click solutions. K8s is not a 1-2 click solution, and clearly isn't designed to be; it's solving particular tough operational problems that if you don't know exist in the first place you won't really be able to evaluate these kinds of things properly. If a group literally doesn't have the need to answer questions like the ones you posed, then OK, don't bother with these tools. But that's all that needs to be said - no need for a new article every week on it.
- collyw 7y ago
  > it's solving particular tough operational problems that if you don't know exist in the first place They probably don't exist for the majority of people using it. We are using k8s for when we need to scale, but at the moment we have a handful of customers and it isn't changing quickly any time soon.
  - ownagefool 7y ago
    They basically exist for everyone. As soon as you go down the road of actually doing infrastructure-as-code, using (not running) k8s is probably as good as any other solution, and arguably better than most when you grow into anything complex. Most of the complaints are false equivalence: i.e. running k8s is harder than just using AWS, which I already know. Of course it is. You don't manage AWS. How big do you think their code base is? If you don't know k8s already, and you're a start-up looking for a niche, maybe now isn't the time to learn k8s, at least not from the business point of view (personal growth, another issue). But when you do know k8s, it makes a lot of sense to just rent a cluster and put your app there, because when you want to build better tests, it's easy, when you want to do zero trust, it's easy, when you want to integrate with vault, it's easy, when you want to encrypt, it's easy, when you want to add a mesh for tracing, metrics and maybe auth, it's easy. What's not easy is inheriting a similarly done product that's entirely bespoke.
    - collyw 7y ago

LoSboccacc 7y ago

same we use kubernet not for scale but for the environment repeatability. we can spin any branch at any time on any provider and be sure it's exactly as we have it in production down to networking and routing, to built that out of plain devops tools would require a lot more in depth knowledge and it won't port exactly one provider to another

VvR-Ox 7y ago

> These kinds of posts always focus on the complexity of running k8s... Yes but that is also the worst already that you could criticize about k8s. Complexity is dangerous because if things are growing beyond a certain threshold X you will have side effects that nobody can predict, a very steep learning curve and therefor many people screwing up something in their (first) setups as well as maintainability nightmares. Probably some day someone will prove me wrong but right now one of my biggest goals to improve security, reliability and people being able to contribute is reducing complexity. After all this is what many of us do when they refactor systems. I am sticking with the UNIX philosophy at this point and in the foreseeable future I will not have a big dev-team at my disposal as companies like Alphabet have to maintain and safe-guard all of this complexity.

hedora 7y ago
From a developer perspective, k8s seems ripe for disruption. It does a bunch of junk that is trivial to accomplish on one machine - open network ports, keep services running, log stuff, run in a jail with dropped privileges, and set proper file permissions on secret files. The mechanisms for all of this, and for resource management, are transparent to unix developers, but in kubernetes they are not. Instead, you have to understand a architectural spaghetti torrent to write and execute “hello world”. It used to be similar with RDBMS systems. It took months and a highly paid consultant to get a working SQL install. Then, you’d hire a team to manage the database, not because the hardware was expensive, but because you’d dropped $100k’s (in 90’s dollars) on the installation process. Then mysql came along, and it didn’t have durability or transactions, but it let you be up and running in a few hours, and have a dynamic web page a few hours after that. If it died, you only lost a few hours or minutes of transactions, assuming somone in your organization spent an afternoon learning cron and mysqldump. I imagine someone will get sufficiently fed up with k8s to do the same. There is clearly demand. I wish them luck.
- p_l 7y ago
  It seems to be. When you start implementing something, you'll soon find that most of that complexity is inherent, not accidental. Been there, done that, didn't even get the T-Shirt. Today it's much easier to package nicer API on top of the rather generic k8s one. There are ways to deploy it easier (in fact, I'd wager that a lot of complexity in deploying k8s is accidental due to deploy tools themselves, not k8s itself. Just look at OpenShift deploy scripts...)

Glyptodon 7y ago

You're asking exactly what I've been wondering. The answer in this thread this so far has been "maybe dokku." I can totally buy that K8s is overkill, but the alternatives for small scale operators seem to require a lot more work internally than just using a K8s service.

geggam 7y ago

Running k8s on GKE is the only rational choice for anyone who isnt a large enterprise with a team of Google SREs to support k8s. Period.

rclayton 7y ago

1000%. If you take a little bit of time to learn k8s and run it on a hosted environment (e.g. EKS), it’s a fantastic solution. We are much happier with it than ECS, Elastic Beanstalk, etc.

AlchemistCamp 7y ago

> If anyone has a solution that is actually simpler and just as easy to set up, I'm very much interested. Sure. @levelsio runs Nomad List (~100k MRR) all on a single Linode VPS. He uses their automated backups service, but it's a simple setup. No k8s, no Docker, just some PHP running on a Linux server. As I understand it, he was learning to code as he built his businesses. https://twitter.com/levelsio/status/1177562806192730113 https://twitter.com/levelsio/status/1177562806192730113

icedchai 7y ago
Many startups are not as resource efficient. The ones I am familiar with spend 50%+ of their MRR on cloud costs.
- p_l 7y ago
  My first production k8s use was actually due to resource efficiencies. We looked at the >50 applications we had to serve, the technical debt that would lead us down the road of building customized distribution to avoid package requirements incompabilties, and various other things, and decided to go with containers - just to avoid the conflicts involved. Thanks to k8s, we generally keep to 1/5th of the original cost, thanks to bin packing of servers, and sleep sounder thanks to automatic restarts of failed pods, ability to easily allocate computing resources per container, globall configure load balancing (we had to scratch use of cloud-provider's load balancer because our number of paths was too big for URL mapping API). Everything can be moved to pretty much every k8s hosting that runs 1.15, biggest difference would be hooking the load balancer to the external network and possibly storage.
- AlchemistCamp 7y ago
  That's very surprising to me! As an employee at multiple startups in multiple countries, I've never seen cloud costs anywhere near payroll.
  - icedchai 7y ago
    Well, I said 50% of monthly revenue, not total costs.

halbritt 7y ago

I'm curious how, with your experience, that you came to the conclusion that k8s is a monster to run?

hrgiger 7y ago

This, kubernetes is container orchestration framework. If you know your needs then you have opportunity to make a better decision and planning. I had similar experience with ingress also I would like to add installing Kubernetes on bare metal is a pain in the neck, even with kubespray. Here is a comparison with other frameworks, from 2018: https://arxiv.org/pdf/2002.02806.pdf https://arxiv.org/pdf/2002.02806.pdf

honkycat 7y ago

Agreed. I've been saying for years that if you go with Docker-Compose, or Amazon ECS, or something lower level, you are just going to end up rebuilding a shittier version of Kubernetes. I think the real alternative is Heroku or running on VMs, but then you do not get service discovery, or a cloud agnostic API for querying running services, or automatic restarts, or rolling updates, or encrypted secrets, or automatic log aggregation, or plug-and-play monitoring, or VM scaling, or an EXCELLENT decoupled solution for deploying my applications ( keel.sh ), or liveness and readiness probes... But nobody needs those things right?

NicoJuicy 7y ago
Why would docker be shittier? It's easier and flexible. Eg. Sidecar pattern resolves most things (eg. logging)
sk5t 7y ago
You do in fact get a lot of this stuff with ECS and Fargate - rolling updates, automatic restart, log aggregation, auto scaling, some discovery bits, healthchecks, Secrets Manager or Parameter Store if you want, etc.
- throwaway894345 7y ago
  This. We’ve been chugging along happily on Fargate for a while now. We looked into EKS, but there is a ton of stuff you have to do manually that is built into Fargate or at least Fargate makes it trivial to integrate with other AWS services (logging, monitoring, IAM integration, cert manager, load balancers, secrets management, etc). I’d like to use Kubernetes, but Fargate just works out of the box. Fortunately, AWS seems to be making EKS better all the time.
  - Logishort 7y ago
    And what if Amazon feels like changing Fargate or charging differently or deprecating it or... Whats your strategy for that?

rawoke083600 7y ago

"Pod, Deployment, Job, CronJob, Service, Ingress, ConfigMap, Secre" Wow as a new developer coming onboard your company, I will walk out the door after seeing that, and the fact that you admit its a small serivce.

nrb 7y ago
It's an afternoon worth of research to understand the basic concepts. Then, with the powerful and intuitive tooling you can spin up your own cluster on your computer in minutes and practice deploying containers that: - are automatically assigned to an appropriate machine(node) based on explicit resource limits you define, enabling reliable performance - horizontally scale (even automatically if you want!) - can be deployed with a rolling update strategy to preserve uptime during deployments - can rollback with swiftness and ease - have liveness checks that restart unhealthy apps(pods) automatically and prevent bad deploys from being widely released - abstracts away your infrastructure, allowing these exact same configs to power a cluster on-prem, in the cloud on bare metal or vms, with a hosted k8s service, or some combination of all of them All of that functionality is unlocked with just a few lines of config or kubectl command, and there are tools that abstract this stuff to simplify it even more or automate more of it. You definitely want some experienced people around to avoid some of the footguns and shortcuts but over the last several years I think k8s has easily proven itself as a substantial net-positive for many shops.
- scarface74 7y ago
  So why should I do all of that instead of throwing a little money at AWS, run ECS and actually spend my time creating my product? Heck, if my needs are simple enough why should I even use ECS instead of just putting my web app on some VM's in an auto-scaling group behind a load balancer and used managed services?
  - geggam 7y ago
    Hate to tell you this but that second idea is much easier to manage from an operations standpoint and significantly more reliable. Also easier to debug and monitor... but you run your business to make developers happy right ?