7 ms·

All of this glosses over the biggest issue with Kubernetes: it's still ridiculously complex, and troubleshooting issues that arise (and they will arise), can le

All of this glosses over the biggest issue with Kubernetes: it's still ridiculously complex, and troubleshooting issues that arise (and they will arise), can leave you struggling for days pouring over docs, code, GitHub issues, stackoverflow... All of the positives you listed rely on super complex abstractions that can easily blow up without a clear answer as to "why". Compared to something like scp and restarting services, I would personally not pay the Kubernetes tax unless I absolutely had to.

dustinmoris 7y ago
Maybe, but the point with containers and kubernets is to treat it like cattle, not pets. If something blows up or dies, then with Kubernetes it's often faster to just tear down the entire namespace and bring it up again. If the entire cluster is dead, then just spin up a new cluster and run your yaml files on it and kill your old cluster. Treat it like cattle, when it doesn't serve your purpose anymore then shoot it. This is one of the biggest advantages of Kubes, but often overlooked because traditional Ops people keep treating infrastructure like a pet. Only thing you should treat like a pet is your persistence layer, which is presumably outside Kubes, somehting like DynamoDb, Firestore, CosmosDb, SQL server, whatever.
- reacharavindh 7y ago
  This is not good engineering. If somebody told me this at a business, I’d not trust them anymore with my infrastructure. So, you say that problems happen, and you consciously don’t want to know/solve them. A recurring problem in you view is solved with constantly building new K8s clusters and your whole infrastructure in it every time!?! Simple example - A microservice that leaks memory.... let it keep restarting as it crashes?! I remember at one of my first jobs, at a healthcare system for a hospital in India, their Java app was so poorly written that it kept leaking memory and bloated beyond GC could help and will crash every morning at around 11 AM and then again at around 3 PM. The end users - Doctors, nurses, pharmacists knew about this behavior and had breaks during that time. Absolutely bullshit engineering! It’s a shame on those that wrote that shitty code, and shame on whoever reckless to suggest a ever rebuilding K8s clusters.
  - dustinmoris 7y ago
    Nothing is this black and white. I tried to emphasise just a simple philosophy that life gets a lot easier if you make things easily replaceable. That was the message I tried to convey, but of course if there is a deep problem with something it needs proper investigation + fixing, but that is an actual code/application problem.
  - carlio 7y ago
    This is something that has been bothering me for the last couple of years. I consistently work with developers who no longer care about performance issues, assuming that k8s and the ops team will take care of it by adding more CPU or RAM or just restarting. What happened to writing reliable code that performed well?
    - lllr_finger 7y ago
      I've witnessed the same thing, however there is nothing mutually exclusive about having performant code running in Kubernetes. There's a trade-off between performance and productivity, and maintaining a sense of pragmatism is a good skill to have (that's directed towards those that use scaling up/out as a reason for being lax about performance).
    - novok 7y ago
      Business incentives. It's a classic incentive tension between more time on nicer code that does the same thing or building more features. Code expands to it's performance budget and all. At least on backend you can quantify the cost fairly easily. If you bring it up to your business people they will notice easy win and then push the devs to make more efficient code. If it's a small $$ difference although, the devs are probably prioritizing correctly.
  - kemitche 7y ago
    > let it keep restarting as it crashes?! Yes, "let it keep restarting while it crashes and while I investigate the issue" is MUCH preferred to "everything's down and my boss is on my ass to fix the memory issue." The bug exists either way, but in one world my site is still up while I fix the bug and prioritize it against other work and in another world my site is hard-down.
- irishsultan 7y ago
  So what if there is a recurring issue? Are you never going to debug it?
- high_5 7y ago
  The problem is that your "cattle" has mad cow disease and no amount of restarting will help.
- nickik 7y ago
  When cattle are sick, you need to heal them. Not shoot them in the head and bring in new cattle. If you your software behaves badly you need to FIX THE SOFTWARE. Just doing the old 'just restart everything' is typical windows admin behavoir and a recipy for making bad unstable systems. Kubernetes absolutly does do strang things, crahes on strange things, does strange things and not tell you about it. I like the system, but to pretend its this unbelievable great thing is an exaturation.
  - dustinmoris 7y ago
    You fundamentally misunderstood what I said. I agree with you, treat your software like a pet. I am saying though, treat your infrastructure like cattle. Infrastructure problems <> Software problems. So yeah, if you have severe bugs in your app, go ahead and fix it, but that has nothing to do with Kubes or not Kubes anymore.
vorpalhex 7y ago
Kubernetes can be very complex.. or it can be very simple. Just like a production server can be very simple, or extremely complex, or a linux distro, or an app.. Kubernetes by itself is a very minimal layer. If you install every extension you can into it, then yes, you'll hit all kinds of weird problems, but that's not a Kubernetes problem.
wpietri 7y ago
Exactly. A year or so ago I thought, hey, maybe I should redo my personal infrastructure using Kubernetes. Long story short, it was way too much of a pain in the ass. As background, I've done time as a professional sysadmin. My current infrastructure is all Chef-based, with maybe a dozen custom cookbooks. But Chef felt kinda heavy and clunky, and the many VMs I had definitely seemed heavy compared with containerization. I thought switching to Kubernetes would be pretty straightforward. Surprise! It was not. I moved the least complex thing I run, my home lighting daemon to it; it's stateless and nothing connects to it, but it was still a struggle to get it up and running. Then I tried adding more stateful services and got bogged down in bugs, mysteries, and Kubernetes complexity. I set it aside, thinking I'd come back to it later when I had more time. That time never quite arrived, and a month or so ago my home lights stopped working. Why? I couldn't tell. A bunch of internal Kubernetes certificates expired, so none of the commands worked. Eventually, I just copy-pasted stuff out of Stack Overflow and randomly rebooted things, and eventually it started working again. I'll happily look at it again when I have to do serious volume and can afford somebody to focus full-time on Kubernetes. But for anything small or casual, I'll be looking elsewhere.
- busterarm 7y ago
  At work we're building an entire service platform on top of managed kubernetes services, agnostic to cloud provider. We had already had bad experiences running K8s ourselves. Going into it we knew how much of a PITA it would be but we vastly underestimated how much, IMO. Would not do again -- I would quit first.
  - pnako 7y ago
    Fire and Motion https://www.joelonsoftware.com/2002/01/06/fire-and-motion/ https://www.joelonsoftware.com/2002/01/06/fire-and-motion/ Written 18 years ago, so obviously not about Kubernagus, but it does explain the same phenomenon. Replace Microsoft with cloud providers and that's more or less the same argument.
- cnf 7y ago
  I run my home automation and infrastructure on kubernetes, and for me that is one of the smoothest ways of doing it. I find it quite easy to deal with, and much prefer it to the “classic” way of doing it.
Townley 7y ago
Agree that the k8s tax, as described, is a huge issue. But I think the biggest issue is immaturity of the ecosystem, with complexity coming in second. You can at least throw an expensive developer at the complexity issue. But when it comes to reliable installations (even helm charts for major software is a coin flip in terms of whether they’ll work), fast moving versioning that reminds me of the JavaScript Wild West (the recent RBAC on by default implementation comes to mind, even if its a good thing), and unresolved problems around provider-agnostic volumes and load balancing... those are headaches that persist long after you’ve learned the difference between a replicaSet and a deployment.
- Rapzid 7y ago
  To further this point about the ecosystem, and this is AWS specific. You need, or have needed, to install a handful of extra services/controllers onto your EKS cluster to get it to integrate the way most would expect with AWS. Autoscalling? Install and configure the autoscaler. IAM roles? Install Kube2IAM. DNS/ALB/etc etc? etc etc etc. After a slog you get everything going. Suddenly a service is throwing errors because it doesn't have IAM permissions. You look into it and it's not getting the role from the kube2iam proxy. Kube2iam is throwing some strange error about a nil or interface cast. Let's pretend you know Go like I do. The error message still tells you nothing specific about what the issue may be. Google leads you to github and you locate an issue with the same symptoms. It's been open for over a year and nobody seems to have any clue what's going on. Good times :) Stay safe everyone, and run a staging cluster!
Summershard 7y ago
True. I'm currently in the middle of writing a paper on extending Kubernetes Scheduler through Scheduler Extender[1]. The process has been really painful. [1]: https://github.com/kubernetes/community/blob/master/contributors/design-proposals/scheduling/scheduler_extender.md https://github.com/kubernetes/community/blob/master/contribu...
- shaklee3 7y ago
  You're saying a feature that's in alpha, released 2 months ago is painful? You should at least wait until a feature is beta until expecting it to be easier to use.
  - Summershard 7y ago
    Scheduler Extender was initially released over 4 years ago[1]. What you are referring to is Scheduling Framework[2], which indeed is a new feature (and will replace/contain Scheduler Extender). [1]: https://github.com/kubernetes/kubernetes/pull/13580 https://github.com/kubernetes/kubernetes/pull/13580 [2]: https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/20180409-scheduling-framework.md https://github.com/kubernetes/enhancements/blob/master/keps/... EDIT: formatting
    - shaklee3 7y ago
      Sorry about that. Can you explain the problems using the extender?
markbnj 7y ago
> All of this glosses over the biggest issue with Kubernetes: it's still ridiculously complex, and troubleshooting issues that arise (and they will arise), can leave you struggling for days pouring over docs, code, GitHub issues, stackoverflow. It's probably good at this point to distinguish between on-prem and managed installations of k8s. In almost four years of running production workloads on Google's GKE we've had... I don't know perhaps 3-4 real head-scratchers where we had to spend a couple of days digging into things. Notably none of these issues have ever left any of our clusters or workloads inoperable. It isn't hyperbole to say that in general the system just works, 24x7x365.
- mpmpmpmp 7y ago
  Agreed. We moved from ECS to GKE specifically because we didn't have the resources to handle what was supposed to be a "managed" container service with ECS. Had agent issues constantly where we couldn't deploy. It did take a little bit to learn k8s no doubt. But now it requires changes so little I usually have to think for a minute to remember how something works because it's been so long since I needed to touch it.
atombender 7y ago
Kubernetes, once you know it, is significantly easier than cobbling together an environment from "classical" solutions that combine Puppet/Chef/Ansible, homegrown shell scripts, static VMs, and SSH. Sure, you can bring up a single VM with those technologies and be up and running quickly. But a real production environment will need automatic scaling (both of processes and nodes), CPU/memory limits, rolling app/infra upgrades, distributed log collection and monitoring, resilience to node failure, load balancing, stateful services (e.g. a database; anything that stores its state on disk and can't use a distributed file system), etc., and you end up building a very, very poor man's Kubernetes dealing with all of the above. With Kubernetes, all of the work has been done, and you only need to deal with high-level primitives. "Nodes" become an abstraction. You just specify what should run, and the cluster takes care of it. I've been there, many times. I ran stuff the "classical" Unix way -- successfully, but painfully -- for about 15 years and I'm not going back there. There are alternatives, of course. Terraform and CloudFormation and things like that. There's Nomad. You can even cobble together something with Docker. But those solutions all require a lot more custom glue from the ops team than Kubernetes.
- candiddevmike 7y ago
  The majority of what you posted reiterates the post I responded to , and it doesn't address the complexity of those features or their implementation. Additionally, I challenge your assertion that "real production environments" need automatic scaling.
  - atombender 7y ago
    You missed my point. I was contrasting Kubernetes with the alternative: Critics often highlight Kubernetes' complexity, forgetting/ignoring that replicating its functionality is also complex and often not composable or transferable to new projects/clusters. It's hard to design a good, flexible Puppet (or whatever) configuration that grows with a company, can be maintained across teams, handles redundancy, and all of those other things. Not all environments need automatic scaling, but they need redundancy, and from a Kubernetes perspective those are two sides of the same coin. A classical setup that automatically allows a new node to start up to take over from a dysfunctional/dead one isn't trivial. Much of Kubernetes' operational complexity also melts away if you choose a managed cloud such as Digital Ocean, Azure, or Google Cloud Platform. I can speak from experience, as I've both set up Kubernetes from scratch on AWS (fun challenge, wouldn't want to do it often) and I am also administering several clusters on Google Cloud. The latter requires almost no classical "system administration". Most of the concerns are "hoisted" up to the Kubernetes layer. If something is wrong, it's almost never related to a node or hardware; it's all pod orchestration and application configuration, with some occasional bits relating to DNS, load balancing, and persistent disks. And if I start a new project I can just boot up a cluster (literally a single command) and have my operational platform ready to serve apps, much like the "one click deploy" promise of, say, Heroku or Zeit, except I have almost complete control of the platform. In my opinion, Kubernetes beats everything else even on a single node.
darau1 7y ago
You could use this argument for literally anything, though. I spent days poring over docs, googling, SOF, github issues, making comments, the whole works when I learned any new software/technology. The argument doesn't hold water, IMO.
kemitche 7y ago
> can leave you struggling for days pouring over docs, code, GitHub issues, stackoverflow I've had that when running code straight on a VM, when running on Docker, and when running on k8s. I can't think of a way to deploy code right now that lets you completely avoid issues with systems that you require but are possibly unfamiliar with, except maybe "serverless" functions. \ And of those three, I much preferred the k8s failure states simply because k8s made running _my code_ much easier.
- jacurtis 7y ago
  > I can't think of a way to deploy code right now that lets you completely avoid issues with systems that you require but are possibly unfamiliar with, except maybe "serverless" functions. This is basically the same comment I was going to write, so I'll just jump onto it. But whenever I hear people complain about how complex XXX solution is for deployment, I always think, "ok, I agree that it sucks, but what's the alternative?" Deploying something right now with all of its ancillary services is a chore, no matter how you do it. K8s is a pain in the ass to set up, I agree. But it seems to maintain itself the best once it is running. And longterm maintainability cannot be overlooked when considering deployment solutions. When I look out in the sea of deployment services and options that exist right now, each option has its own tradeoffs. Another service might elimite or minimize anothers' tradeoffs, but it then introduces its' own tradeoffs. You are trading one evil for another. And this makes it nearly impossible to say "X solution is the best deployment solution in 2020". Do you value scalability? Speed? Cost? Ease of learning? There are different solutions to optimize each of these schools of thought, but it ultimately comes down to what you value most, and another developer isn't going to value things in the same way, so for them, another solution is better. The only drop-dead simple, fast, scalable, deployment solution I have seen right now is static site hosting on tools like Netlify or AWS Amplify (among others). But these only work for static generated sites, which were already pretty easy to deploy, and they are not an option for most sites outside of marketing sites, landing pages, and blogs. They aren't going to work for service based sites, nor will they likely replace something being deployed with K8s right now. So they are almost moot in this argument, but I bring it up, because they are arguably, the only "best deployment solution" right now if you are building a site that can meet its' narrow criteria.
segmondy 7y ago
You can make an argument that Linux is ridiculously complex and troubleshooting issues that arise can leave you struggling for days pouring over docs, code, etc and that msdos is a much simpler system and be sort of right.
jopython 7y ago
You could try k3s which is much easier to deploy.

dustinmoris 7y ago

Maybe, but the point with containers and kubernets is to treat it like cattle, not pets. If something blows up or dies, then with Kubernetes it's often faster to just tear down the entire namespace and bring it up again. If the entire cluster is dead, then just spin up a new cluster and run your yaml files on it and kill your old cluster. Treat it like cattle, when it doesn't serve your purpose anymore then shoot it. This is one of the biggest advantages of Kubes, but often overlooked because traditional Ops people keep treating infrastructure like a pet. Only thing you should treat like a pet is your persistence layer, which is presumably outside Kubes, somehting like DynamoDb, Firestore, CosmosDb, SQL server, whatever.

reacharavindh 7y ago
This is not good engineering. If somebody told me this at a business, I’d not trust them anymore with my infrastructure. So, you say that problems happen, and you consciously don’t want to know/solve them. A recurring problem in you view is solved with constantly building new K8s clusters and your whole infrastructure in it every time!?! Simple example - A microservice that leaks memory.... let it keep restarting as it crashes?! I remember at one of my first jobs, at a healthcare system for a hospital in India, their Java app was so poorly written that it kept leaking memory and bloated beyond GC could help and will crash every morning at around 11 AM and then again at around 3 PM. The end users - Doctors, nurses, pharmacists knew about this behavior and had breaks during that time. Absolutely bullshit engineering! It’s a shame on those that wrote that shitty code, and shame on whoever reckless to suggest a ever rebuilding K8s clusters.
- dustinmoris 7y ago
  Nothing is this black and white. I tried to emphasise just a simple philosophy that life gets a lot easier if you make things easily replaceable. That was the message I tried to convey, but of course if there is a deep problem with something it needs proper investigation + fixing, but that is an actual code/application problem.
- carlio 7y ago
  This is something that has been bothering me for the last couple of years. I consistently work with developers who no longer care about performance issues, assuming that k8s and the ops team will take care of it by adding more CPU or RAM or just restarting. What happened to writing reliable code that performed well?
  - lllr_finger 7y ago
    I've witnessed the same thing, however there is nothing mutually exclusive about having performant code running in Kubernetes. There's a trade-off between performance and productivity, and maintaining a sense of pragmatism is a good skill to have (that's directed towards those that use scaling up/out as a reason for being lax about performance).
  - novok 7y ago
    Business incentives. It's a classic incentive tension between more time on nicer code that does the same thing or building more features. Code expands to it's performance budget and all. At least on backend you can quantify the cost fairly easily. If you bring it up to your business people they will notice easy win and then push the devs to make more efficient code. If it's a small $$ difference although, the devs are probably prioritizing correctly.
- kemitche 7y ago
  > let it keep restarting as it crashes?! Yes, "let it keep restarting while it crashes and while I investigate the issue" is MUCH preferred to "everything's down and my boss is on my ass to fix the memory issue." The bug exists either way, but in one world my site is still up while I fix the bug and prioritize it against other work and in another world my site is hard-down.
irishsultan 7y ago
So what if there is a recurring issue? Are you never going to debug it?
high_5 7y ago
The problem is that your "cattle" has mad cow disease and no amount of restarting will help.
nickik 7y ago
When cattle are sick, you need to heal them. Not shoot them in the head and bring in new cattle. If you your software behaves badly you need to FIX THE SOFTWARE. Just doing the old 'just restart everything' is typical windows admin behavoir and a recipy for making bad unstable systems. Kubernetes absolutly does do strang things, crahes on strange things, does strange things and not tell you about it. I like the system, but to pretend its this unbelievable great thing is an exaturation.
- dustinmoris 7y ago
  You fundamentally misunderstood what I said. I agree with you, treat your software like a pet. I am saying though, treat your infrastructure like cattle. Infrastructure problems <> Software problems. So yeah, if you have severe bugs in your app, go ahead and fix it, but that has nothing to do with Kubes or not Kubes anymore.

vorpalhex 7y ago

Kubernetes can be very complex.. or it can be very simple. Just like a production server can be very simple, or extremely complex, or a linux distro, or an app.. Kubernetes by itself is a very minimal layer. If you install every extension you can into it, then yes, you'll hit all kinds of weird problems, but that's not a Kubernetes problem.

wpietri 7y ago

Exactly. A year or so ago I thought, hey, maybe I should redo my personal infrastructure using Kubernetes. Long story short, it was way too much of a pain in the ass. As background, I've done time as a professional sysadmin. My current infrastructure is all Chef-based, with maybe a dozen custom cookbooks. But Chef felt kinda heavy and clunky, and the many VMs I had definitely seemed heavy compared with containerization. I thought switching to Kubernetes would be pretty straightforward. Surprise! It was not. I moved the least complex thing I run, my home lighting daemon to it; it's stateless and nothing connects to it, but it was still a struggle to get it up and running. Then I tried adding more stateful services and got bogged down in bugs, mysteries, and Kubernetes complexity. I set it aside, thinking I'd come back to it later when I had more time. That time never quite arrived, and a month or so ago my home lights stopped working. Why? I couldn't tell. A bunch of internal Kubernetes certificates expired, so none of the commands worked. Eventually, I just copy-pasted stuff out of Stack Overflow and randomly rebooted things, and eventually it started working again. I'll happily look at it again when I have to do serious volume and can afford somebody to focus full-time on Kubernetes. But for anything small or casual, I'll be looking elsewhere.

busterarm 7y ago
At work we're building an entire service platform on top of managed kubernetes services, agnostic to cloud provider. We had already had bad experiences running K8s ourselves. Going into it we knew how much of a PITA it would be but we vastly underestimated how much, IMO. Would not do again -- I would quit first.
- pnako 7y ago
  Fire and Motion https://www.joelonsoftware.com/2002/01/06/fire-and-motion/ https://www.joelonsoftware.com/2002/01/06/fire-and-motion/ Written 18 years ago, so obviously not about Kubernagus, but it does explain the same phenomenon. Replace Microsoft with cloud providers and that's more or less the same argument.
cnf 7y ago
I run my home automation and infrastructure on kubernetes, and for me that is one of the smoothest ways of doing it. I find it quite easy to deal with, and much prefer it to the “classic” way of doing it.

Townley 7y ago

Agree that the k8s tax, as described, is a huge issue. But I think the biggest issue is immaturity of the ecosystem, with complexity coming in second. You can at least throw an expensive developer at the complexity issue. But when it comes to reliable installations (even helm charts for major software is a coin flip in terms of whether they’ll work), fast moving versioning that reminds me of the JavaScript Wild West (the recent RBAC on by default implementation comes to mind, even if its a good thing), and unresolved problems around provider-agnostic volumes and load balancing... those are headaches that persist long after you’ve learned the difference between a replicaSet and a deployment.

Rapzid 7y ago
To further this point about the ecosystem, and this is AWS specific. You need, or have needed, to install a handful of extra services/controllers onto your EKS cluster to get it to integrate the way most would expect with AWS. Autoscalling? Install and configure the autoscaler. IAM roles? Install Kube2IAM. DNS/ALB/etc etc? etc etc etc. After a slog you get everything going. Suddenly a service is throwing errors because it doesn't have IAM permissions. You look into it and it's not getting the role from the kube2iam proxy. Kube2iam is throwing some strange error about a nil or interface cast. Let's pretend you know Go like I do. The error message still tells you nothing specific about what the issue may be. Google leads you to github and you locate an issue with the same symptoms. It's been open for over a year and nobody seems to have any clue what's going on. Good times :) Stay safe everyone, and run a staging cluster!

Summershard 7y ago

True. I'm currently in the middle of writing a paper on extending Kubernetes Scheduler through Scheduler Extender[1]. The process has been really painful. [1]: https://github.com/kubernetes/community/blob/master/contributors/design-proposals/scheduling/scheduler_extender.md https://github.com/kubernetes/community/blob/master/contribu...

shaklee3 7y ago
You're saying a feature that's in alpha, released 2 months ago is painful? You should at least wait until a feature is beta until expecting it to be easier to use.
- Summershard 7y ago
  Scheduler Extender was initially released over 4 years ago[1]. What you are referring to is Scheduling Framework[2], which indeed is a new feature (and will replace/contain Scheduler Extender). [1]: https://github.com/kubernetes/kubernetes/pull/13580 https://github.com/kubernetes/kubernetes/pull/13580 [2]: https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/20180409-scheduling-framework.md https://github.com/kubernetes/enhancements/blob/master/keps/... EDIT: formatting
  - shaklee3 7y ago
    Sorry about that. Can you explain the problems using the extender?

markbnj 7y ago

> All of this glosses over the biggest issue with Kubernetes: it's still ridiculously complex, and troubleshooting issues that arise (and they will arise), can leave you struggling for days pouring over docs, code, GitHub issues, stackoverflow. It's probably good at this point to distinguish between on-prem and managed installations of k8s. In almost four years of running production workloads on Google's GKE we've had... I don't know perhaps 3-4 real head-scratchers where we had to spend a couple of days digging into things. Notably none of these issues have ever left any of our clusters or workloads inoperable. It isn't hyperbole to say that in general the system just works, 24x7x365.

mpmpmpmp 7y ago
Agreed. We moved from ECS to GKE specifically because we didn't have the resources to handle what was supposed to be a "managed" container service with ECS. Had agent issues constantly where we couldn't deploy. It did take a little bit to learn k8s no doubt. But now it requires changes so little I usually have to think for a minute to remember how something works because it's been so long since I needed to touch it.

atombender 7y ago

Kubernetes, once you know it, is significantly easier than cobbling together an environment from "classical" solutions that combine Puppet/Chef/Ansible, homegrown shell scripts, static VMs, and SSH. Sure, you can bring up a single VM with those technologies and be up and running quickly. But a real production environment will need automatic scaling (both of processes and nodes), CPU/memory limits, rolling app/infra upgrades, distributed log collection and monitoring, resilience to node failure, load balancing, stateful services (e.g. a database; anything that stores its state on disk and can't use a distributed file system), etc., and you end up building a very, very poor man's Kubernetes dealing with all of the above. With Kubernetes, all of the work has been done, and you only need to deal with high-level primitives. "Nodes" become an abstraction. You just specify what should run, and the cluster takes care of it. I've been there, many times. I ran stuff the "classical" Unix way -- successfully, but painfully -- for about 15 years and I'm not going back there. There are alternatives, of course. Terraform and CloudFormation and things like that. There's Nomad. You can even cobble together something with Docker. But those solutions all require a lot more custom glue from the ops team than Kubernetes.

candiddevmike 7y ago
The majority of what you posted reiterates the post I responded to , and it doesn't address the complexity of those features or their implementation. Additionally, I challenge your assertion that "real production environments" need automatic scaling.
- atombender 7y ago
  You missed my point. I was contrasting Kubernetes with the alternative: Critics often highlight Kubernetes' complexity, forgetting/ignoring that replicating its functionality is also complex and often not composable or transferable to new projects/clusters. It's hard to design a good, flexible Puppet (or whatever) configuration that grows with a company, can be maintained across teams, handles redundancy, and all of those other things. Not all environments need automatic scaling, but they need redundancy, and from a Kubernetes perspective those are two sides of the same coin. A classical setup that automatically allows a new node to start up to take over from a dysfunctional/dead one isn't trivial. Much of Kubernetes' operational complexity also melts away if you choose a managed cloud such as Digital Ocean, Azure, or Google Cloud Platform. I can speak from experience, as I've both set up Kubernetes from scratch on AWS (fun challenge, wouldn't want to do it often) and I am also administering several clusters on Google Cloud. The latter requires almost no classical "system administration". Most of the concerns are "hoisted" up to the Kubernetes layer. If something is wrong, it's almost never related to a node or hardware; it's all pod orchestration and application configuration, with some occasional bits relating to DNS, load balancing, and persistent disks. And if I start a new project I can just boot up a cluster (literally a single command) and have my operational platform ready to serve apps, much like the "one click deploy" promise of, say, Heroku or Zeit, except I have almost complete control of the platform. In my opinion, Kubernetes beats everything else even on a single node.

darau1 7y ago

You could use this argument for literally anything, though. I spent days poring over docs, googling, SOF, github issues, making comments, the whole works when I learned any new software/technology. The argument doesn't hold water, IMO.

kemitche 7y ago

> can leave you struggling for days pouring over docs, code, GitHub issues, stackoverflow I've had that when running code straight on a VM, when running on Docker, and when running on k8s. I can't think of a way to deploy code right now that lets you completely avoid issues with systems that you require but are possibly unfamiliar with, except maybe "serverless" functions. \ And of those three, I much preferred the k8s failure states simply because k8s made running _my code_ much easier.

jacurtis 7y ago
> I can't think of a way to deploy code right now that lets you completely avoid issues with systems that you require but are possibly unfamiliar with, except maybe "serverless" functions. This is basically the same comment I was going to write, so I'll just jump onto it. But whenever I hear people complain about how complex XXX solution is for deployment, I always think, "ok, I agree that it sucks, but what's the alternative?" Deploying something right now with all of its ancillary services is a chore, no matter how you do it. K8s is a pain in the ass to set up, I agree. But it seems to maintain itself the best once it is running. And longterm maintainability cannot be overlooked when considering deployment solutions. When I look out in the sea of deployment services and options that exist right now, each option has its own tradeoffs. Another service might elimite or minimize anothers' tradeoffs, but it then introduces its' own tradeoffs. You are trading one evil for another. And this makes it nearly impossible to say "X solution is the best deployment solution in 2020". Do you value scalability? Speed? Cost? Ease of learning? There are different solutions to optimize each of these schools of thought, but it ultimately comes down to what you value most, and another developer isn't going to value things in the same way, so for them, another solution is better. The only drop-dead simple, fast, scalable, deployment solution I have seen right now is static site hosting on tools like Netlify or AWS Amplify (among others). But these only work for static generated sites, which were already pretty easy to deploy, and they are not an option for most sites outside of marketing sites, landing pages, and blogs. They aren't going to work for service based sites, nor will they likely replace something being deployed with K8s right now. So they are almost moot in this argument, but I bring it up, because they are arguably, the only "best deployment solution" right now if you are building a site that can meet its' narrow criteria.

segmondy 7y ago

You can make an argument that Linux is ridiculously complex and troubleshooting issues that arise can leave you struggling for days pouring over docs, code, etc and that msdos is a much simpler system and be sort of right.

jopython 7y ago

You could try k3s which is much easier to deploy.