6 ms·

The argument that Kubernetes adds complexity is, in my opinion, bogus. Kubernetes is a "define once, forget about it" type of infrastructure. You define the sta

The argument that Kubernetes adds complexity is, in my opinion, bogus. Kubernetes is a "define once, forget about it" type of infrastructure. You define the state you want your infrastructure to and Kubernetes takes care of maintaining that state. Tools like Ansible and Puppet, as great as they are, do not guarantee your infrastructure will end up in the state you defined and you easily end up with broken services. The only complexity in kubernetes is the fact that it forces you to think and carefully design your infra in a way people aren't used to, yet. More upfront, careful thinking isn't complexity. It can only benefit you in the long run. There is, however, a learning curve to Kubernetes, but it isn't this sharp. It does require you to sit down and read the doc for 8 hours, but that a small price to pay. A few month back I wrote a blog post[1] that, through walking through the few different infrastructures my company experimented with over the years, surfaces many reasons one would want to use [a managed] Kubernetes. (For a shorter read, you can probably start at reading at [2]) [1]: https://boxunix.com/post/bare_metal_to_kube https://boxunix.com/post/bare_metal_to_kube [2]: https://boxunix.com/post/bare_metal_to_kube/#_hardware_infrastructure https://boxunix.com/post/bare_metal_to_kube/#_hardware_infra...

JMTQp8lwXL 8y ago
This has been my experience. It isn't that complex, just a steep learning curve.
- TooCleverByHalf 8y ago
  If complexity isn't the reason for the steep learning curve what is?
  - JMTQp8lwXL 8y ago
    The documentation around it isn't that great. Once you know the key actors -- which are a dozen or so key types of resources, and how they play together, you understand how to create the configuration for your environment. The YAML files are quite intimidating at first, if you look at examples out there. I'm probably not a great teacher, but I think I could boil it down to a slide deck of no more than 25, and an hour's worth of time, and clearly teach how to interface with Kubernetes for a basic web app w/ horizontal scaling, Let's Encrypt SSL certificates, etc.
    - NicoJuicy 8y ago
      I'd be interested
  - ecnahc515 8y ago
    My personal take is that it's because Kubernetes has a lot of compositional features and abstractions, rather than a bunch of one off ones that don't play well with other features. A good example is label selectors with Services, Pods, and Deployments. You create a deployment, which is basically a scaling group for pods, and pods are the unit of deployment running containers. A service exposes your pods ports to other components through a cluster local virtual IP, or a load balancer. You cannot simply say "this service is for this app". You must instead say, this service will map it's service port, to the following pod port, for all pods that match the following label selector. There are 5 different concepts (pods, deployments, services, labels and selectors) you need to learn before you can actually make your app accessible. But what's nice is that these concepts are used elsewhere in Kubernetes. Labels and selectors are used to query over any group of objects, and allow grouping different things in arbitrary ways. Pods are concepts that get reused by anything that deploys a container. There's Deployments, Statefulsets, Daemonsets, Jobs, CronJobs, these all create pods. So it's nice that pods are their own decoupled concept, but now you gotta understand how they're used, and relate to the other Resource types. Part of what's great about this all is also that now that people are building custom operators/controllers, you can understand how to use them fairly easily because they build on all these existing concepts.
    - jacques_chester 8y ago
      I find that in Kubernetes I do a great deal of tedious boilerplate work. Sometimes I just want to run some damn software and let the platform do the boring stuff. Disclosure: I work for Pivotal, we have a hand in a few such things.
      - JMTQp8lwXL 8y ago
        I've tried that, but then been burned by PaaS switching things up in unexpected ways. Best to make the investment and just own it. I absolutely share the sentiment, though. Good news is, it only took me a week to really pick up k8s.
        jacques_chester 8y ago
edvald 8y ago
I agree to some extent. One of the key value propositions of Kubernetes is indeed in its abstractions, which are mostly quite good. And I really like its declarative approach. However, I've been dealing with its ins and outs for a while now, partly because I'm developing tooling around it, and I've found there are already a lot of idiosyncrasies in there. Some APIs have very frustrating flaws, and most annoying of all is how tightly some core functionality is coupled to kubectl. A remarkable amount of logic is embedded in the CLI, as opposed to the API layer, and if you really get in the weeds you're likely to start pulling your hair. Which is to say, even once you get through the learning curve, you may still find yourself struggling with it as you use it. Sometimes when you declare your intent, you find that it just doesn't work, and it can take a while to figure out why. Before you know it, you're using various operators that are meant to mask some of the deficiencies in the core orchestrator, and by extension you're outside of the core abstractions. And don't get me started on Istio... (but that's a tangent). Anyhow. If you're not trying to do anything unusual, it's a good orchestrator. Not perfect, and for sure heavy, but a good choice for a lot of use cases.
- jordanbeiber 8y ago
  Pretty much my experience as well. I’ve actually started to wrap calls to kubectl to not have to replicate code and logic taking place in the CLI. And istio... I can’t even... envoy is awesome though!
arp242 8y ago
This is not my experience with k8s at all. Sure, everything will work fine when it works (although I don't think it's as easy as you make it out to be), but when something unexpected happens it's a massive pain to debug. "8 hours" initial investment is already large for a small/simple scenario; and it's not the full costs either, because fixing problems will be much harder/time-consuming. I wrote a post about this just the other day[1]: > What does it mean for a framework, library, or tool to be “easy”? There are many possible definitions one could use, but my definition is usually that it’s easy to debug. I often see people advertise a particular program, framework, library, file format, or something else as easy because “look with how little effort I can do task X, this is so easy!” That’s great, but an incomplete picture. [..] > Abstractions which make something easier to write often come at the cost of make things harder to understand. Sometimes this is a good trade-off, but often it’s not. In general I will happily spend a little but more effort writing something now if that makes things easier to understand and debug later on, as it’s often a net time-saver. [1]: https://arp242.net/weblog/easy.html https://arp242.net/weblog/easy.html
- ravedave5 8y ago
  We spent weeks and weeks at work working through k8 performance issues, we got nowhere and fell back to a crappy work around.
  - techntoke 8y ago
    Sounds like it wasn't K8S performance issues then, but performance issues with your environment.
    - Carpetsmoker 8y ago
      I don't think it matters if it was a k8s issue or not, what matters is that k8s made it much harder to get to the bottom of it. We actually had similar issues when we deployed k8s. In the end it turned out to be a misconfiguration, but took weeks to figure out, and only because our entire dev team looked at it (and not the k8s guru who implemented it all).
ukoki 8y ago
The internet is both full of people complaining that Kubernetes is too complicated and full of people arguing that it makes things easier. The people saying it's simple are looking up from the world of IaaS and non-declarative orchestration software (Ansible, Puppet etc) and saying "hey this is much easier". The people saying it's complicated are looking down from the world of PaaS and FaaS (Heroku, AWS Lambda, Google App Engine) and saying "why they hell would I want to manage this?"
ken 8y ago
It's pretty common for new technologies to advertise themselves as "adopt, and forget about it", but in my experience it's unheard of that any actually deliver on this promise. Any technology you adopt today is a technology you're going to have to troubleshoot tomorrow. (I don't think the 15,000 Kubernetes questions on StackOverflow are all from initial setup.) I can't remember the last [application / service / file format / website / language / anything related to computer software] that was so simple and reliable that I wasn't searching the internet for answers (and banging my head against the wall because of) the very next month. It was probably something on my C=64. As Kernighan said back in the 1970's, "Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it?" I've never used Kubernetes, but I've read some articles about it and watched some videos, and despite the nonstop bragging about its simplicity (red flag #1), I'm not sure I can figure out how to deploy with it. I'm fairly certain I wouldn't have any hope of fixing it when it breaks next month. Hearing testimonials only from people who say "it doesn't break!" is red flag #2. No technology works perfectly for everyone, so I want to hear from the people who had to troubleshoot it, not the people who think it's all sunshine and rainbows. And those people are not kind, and make it sound like the cost is way more than just "8 hours reading the docs" -- in fact, the docs are often called out as part of the problem.
- oso2k 8y ago
  Disclaimer: I work for Red Hat as an OpenShift Consulting Architect. If you want a gentle, free introduction to OpenShift (our Kubernetes distribution), I recommend trying out Katacoda portal, Learn OpenShift [0]. Katacoda [1] also has vanilla Kubernetes lessons as well. [0] https://learn.openshift.com/ https://learn.openshift.com/ [1] https://www.katacoda.com/ https://www.katacoda.com/
- zebraflask 8y ago
  I have yet to see a K instance that doesn't somehow manage to break.
  - oso2k 8y ago
    In my experience, the clusters that break are clusters that are architected poorly and/or go unmaintained. There's also some of us that believe clusters need to be treated more like cattle rather than pets.
fukbezos 8y ago
Could you recommend some reading on hosting/scaling a 3rd party hosted windows app using private cloud with containers?
reitanqild 8y ago
> There is, however, a learning curve to Kubernetes, but it isn't this sharp. It does require you to sit down and read the doc for 8 hours, but that a small price to pay. Last time I sat down with k8s docs it was a broken mess. It was impossible for me to follow any of the examples I tried through.
- jordanbeiber 8y ago
  And then you start digging through GH issues... that is a proper rabbit hole.
jrs95 8y ago
I would not recommend most people to run their own k8s cluster. Even Amazon's EKS is enough of a pain in the ass I don't think it's close to a "define once, forget about it" type of experience. However, this is pretty much my experience using GKE on Google Cloud. You can get a cluster up and running and start deploying stuff to it in a matter of minutes.
ttiurani 8y ago
I agree with this: after you know k8s, it is pretty darn great. Two more points: 1. K8s of 2019 is significantly better than K8s of 2017. With stable Deployments, StatefulSets and CronJobs and binaryData in ConfigMaps etc., I haven't missed a single feature in my latest setup: I just typed away helm'd manifests and things just worked. This is in stark contrast to the dance around missing features with initContainer and crond container kludges you had to do in 2017, when I built my first kubernetes setup. 2. People tend to conflate maintaining a k8s cluster to using it. Setting up your own k8s cluster with HA masters, is likely still a royal pain (disclaimer: my attempt at that are from 2017 so could be wrong, see point 1). But for a small company, whipping up a cluster from GKE is a breeze, and for big corps, company-wide cluster(s) setup by an ops team (OpenShift has been ok) is the way to go. The end result is that as a developer you just push containers and apply manifests.
- neop1x 8y ago
  I am scared of Helm. I would expect most people install helm chart like a package in Ubuntu, without knowing what it is going to do, what images are going to run and why. What if something malfunctions? What if helm upgrade won't success? Are you going to call a Helm support line?
  - ttiurani 8y ago
    I was referring to using helm to configure my own charts to work in different test environments. Wrt public helm charts, you want to understand the chart install. Most of them are really dead simple if you know k8s.
westurner 8y ago
> Tools like Ansible and Puppet, as great as they are, do not guarantee your infrastructure will end up in the state you defined and you easily end up with broken services. False dilemma. Ansible and Puppet are great tools for configuring kubernetes, kubernetes worker nodes, and building container images. Kubernetes does not solve for host OS maintenance; though there are a number of host OS projects which remove most of what they consider to be unnecessary services, there's still need to upgrade kubernetes nodes and move pods out of the way first (which can be done with e.g. Puppet or Ansible). As well, it may not be appropriate for monitoring to depend upon kubernetes; there again you have nodes to manage with an SCM tool.
eeeeeeeeeeeee 8y ago
I have never heard any infrastructure explained as “define once, forget about it.” Ever. Everything requires maintenance, monitoring, upgrades, security, etc. And things just randomly break. I like Kubernetes but it is not without problems. And when your entire business is now leaning on Kubernetes, it can really hurt when it has problems and you can’t figure it out quickly. And if your expertise is not running infrastructure, a lot of people, as this post explains, are better off on AWS or Google Cloud using managed services. Or you might have a problem with the underlying storage platform that Kubernetes uses, so you now have to be a storage expert. I think it’s telling how many job offers I receive from companies already running Kubernetes and need help with it.

JMTQp8lwXL 8y ago

This has been my experience. It isn't that complex, just a steep learning curve.

TooCleverByHalf 8y ago
If complexity isn't the reason for the steep learning curve what is?
- JMTQp8lwXL 8y ago
  The documentation around it isn't that great. Once you know the key actors -- which are a dozen or so key types of resources, and how they play together, you understand how to create the configuration for your environment. The YAML files are quite intimidating at first, if you look at examples out there. I'm probably not a great teacher, but I think I could boil it down to a slide deck of no more than 25, and an hour's worth of time, and clearly teach how to interface with Kubernetes for a basic web app w/ horizontal scaling, Let's Encrypt SSL certificates, etc.
  - NicoJuicy 8y ago
    I'd be interested
- ecnahc515 8y ago
  My personal take is that it's because Kubernetes has a lot of compositional features and abstractions, rather than a bunch of one off ones that don't play well with other features. A good example is label selectors with Services, Pods, and Deployments. You create a deployment, which is basically a scaling group for pods, and pods are the unit of deployment running containers. A service exposes your pods ports to other components through a cluster local virtual IP, or a load balancer. You cannot simply say "this service is for this app". You must instead say, this service will map it's service port, to the following pod port, for all pods that match the following label selector. There are 5 different concepts (pods, deployments, services, labels and selectors) you need to learn before you can actually make your app accessible. But what's nice is that these concepts are used elsewhere in Kubernetes. Labels and selectors are used to query over any group of objects, and allow grouping different things in arbitrary ways. Pods are concepts that get reused by anything that deploys a container. There's Deployments, Statefulsets, Daemonsets, Jobs, CronJobs, these all create pods. So it's nice that pods are their own decoupled concept, but now you gotta understand how they're used, and relate to the other Resource types. Part of what's great about this all is also that now that people are building custom operators/controllers, you can understand how to use them fairly easily because they build on all these existing concepts.
  - jacques_chester 8y ago
    I find that in Kubernetes I do a great deal of tedious boilerplate work. Sometimes I just want to run some damn software and let the platform do the boring stuff. Disclosure: I work for Pivotal, we have a hand in a few such things.
    - JMTQp8lwXL 8y ago
      I've tried that, but then been burned by PaaS switching things up in unexpected ways. Best to make the investment and just own it. I absolutely share the sentiment, though. Good news is, it only took me a week to really pick up k8s.
      - jacques_chester 8y ago

edvald 8y ago

I agree to some extent. One of the key value propositions of Kubernetes is indeed in its abstractions, which are mostly quite good. And I really like its declarative approach. However, I've been dealing with its ins and outs for a while now, partly because I'm developing tooling around it, and I've found there are already a lot of idiosyncrasies in there. Some APIs have very frustrating flaws, and most annoying of all is how tightly some core functionality is coupled to kubectl. A remarkable amount of logic is embedded in the CLI, as opposed to the API layer, and if you really get in the weeds you're likely to start pulling your hair. Which is to say, even once you get through the learning curve, you may still find yourself struggling with it as you use it. Sometimes when you declare your intent, you find that it just doesn't work, and it can take a while to figure out why. Before you know it, you're using various operators that are meant to mask some of the deficiencies in the core orchestrator, and by extension you're outside of the core abstractions. And don't get me started on Istio... (but that's a tangent). Anyhow. If you're not trying to do anything unusual, it's a good orchestrator. Not perfect, and for sure heavy, but a good choice for a lot of use cases.

jordanbeiber 8y ago
Pretty much my experience as well. I’ve actually started to wrap calls to kubectl to not have to replicate code and logic taking place in the CLI. And istio... I can’t even... envoy is awesome though!

arp242 8y ago

This is not my experience with k8s at all. Sure, everything will work fine when it works (although I don't think it's as easy as you make it out to be), but when something unexpected happens it's a massive pain to debug. "8 hours" initial investment is already large for a small/simple scenario; and it's not the full costs either, because fixing problems will be much harder/time-consuming. I wrote a post about this just the other day[1]: > What does it mean for a framework, library, or tool to be “easy”? There are many possible definitions one could use, but my definition is usually that it’s easy to debug. I often see people advertise a particular program, framework, library, file format, or something else as easy because “look with how little effort I can do task X, this is so easy!” That’s great, but an incomplete picture. [..] > Abstractions which make something easier to write often come at the cost of make things harder to understand. Sometimes this is a good trade-off, but often it’s not. In general I will happily spend a little but more effort writing something now if that makes things easier to understand and debug later on, as it’s often a net time-saver. [1]: https://arp242.net/weblog/easy.html https://arp242.net/weblog/easy.html

ravedave5 8y ago
We spent weeks and weeks at work working through k8 performance issues, we got nowhere and fell back to a crappy work around.
- techntoke 8y ago
  Sounds like it wasn't K8S performance issues then, but performance issues with your environment.
  - Carpetsmoker 8y ago
    I don't think it matters if it was a k8s issue or not, what matters is that k8s made it much harder to get to the bottom of it. We actually had similar issues when we deployed k8s. In the end it turned out to be a misconfiguration, but took weeks to figure out, and only because our entire dev team looked at it (and not the k8s guru who implemented it all).

ukoki 8y ago

The internet is both full of people complaining that Kubernetes is too complicated and full of people arguing that it makes things easier. The people saying it's simple are looking up from the world of IaaS and non-declarative orchestration software (Ansible, Puppet etc) and saying "hey this is much easier". The people saying it's complicated are looking down from the world of PaaS and FaaS (Heroku, AWS Lambda, Google App Engine) and saying "why they hell would I want to manage this?"

ken 8y ago

It's pretty common for new technologies to advertise themselves as "adopt, and forget about it", but in my experience it's unheard of that any actually deliver on this promise. Any technology you adopt today is a technology you're going to have to troubleshoot tomorrow. (I don't think the 15,000 Kubernetes questions on StackOverflow are all from initial setup.) I can't remember the last [application / service / file format / website / language / anything related to computer software] that was so simple and reliable that I wasn't searching the internet for answers (and banging my head against the wall because of) the very next month. It was probably something on my C=64. As Kernighan said back in the 1970's, "Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it?" I've never used Kubernetes, but I've read some articles about it and watched some videos, and despite the nonstop bragging about its simplicity (red flag #1), I'm not sure I can figure out how to deploy with it. I'm fairly certain I wouldn't have any hope of fixing it when it breaks next month. Hearing testimonials only from people who say "it doesn't break!" is red flag #2. No technology works perfectly for everyone, so I want to hear from the people who had to troubleshoot it, not the people who think it's all sunshine and rainbows. And those people are not kind, and make it sound like the cost is way more than just "8 hours reading the docs" -- in fact, the docs are often called out as part of the problem.

oso2k 8y ago
Disclaimer: I work for Red Hat as an OpenShift Consulting Architect. If you want a gentle, free introduction to OpenShift (our Kubernetes distribution), I recommend trying out Katacoda portal, Learn OpenShift [0]. Katacoda [1] also has vanilla Kubernetes lessons as well. [0] https://learn.openshift.com/ https://learn.openshift.com/ [1] https://www.katacoda.com/ https://www.katacoda.com/
zebraflask 8y ago
I have yet to see a K instance that doesn't somehow manage to break.
- oso2k 8y ago
  In my experience, the clusters that break are clusters that are architected poorly and/or go unmaintained. There's also some of us that believe clusters need to be treated more like cattle rather than pets.

fukbezos 8y ago

Could you recommend some reading on hosting/scaling a 3rd party hosted windows app using private cloud with containers?

reitanqild 8y ago

> There is, however, a learning curve to Kubernetes, but it isn't this sharp. It does require you to sit down and read the doc for 8 hours, but that a small price to pay. Last time I sat down with k8s docs it was a broken mess. It was impossible for me to follow any of the examples I tried through.

jordanbeiber 8y ago
And then you start digging through GH issues... that is a proper rabbit hole.

jrs95 8y ago

I would not recommend most people to run their own k8s cluster. Even Amazon's EKS is enough of a pain in the ass I don't think it's close to a "define once, forget about it" type of experience. However, this is pretty much my experience using GKE on Google Cloud. You can get a cluster up and running and start deploying stuff to it in a matter of minutes.

ttiurani 8y ago

I agree with this: after you know k8s, it is pretty darn great. Two more points: 1. K8s of 2019 is significantly better than K8s of 2017. With stable Deployments, StatefulSets and CronJobs and binaryData in ConfigMaps etc., I haven't missed a single feature in my latest setup: I just typed away helm'd manifests and things just worked. This is in stark contrast to the dance around missing features with initContainer and crond container kludges you had to do in 2017, when I built my first kubernetes setup. 2. People tend to conflate maintaining a k8s cluster to using it. Setting up your own k8s cluster with HA masters, is likely still a royal pain (disclaimer: my attempt at that are from 2017 so could be wrong, see point 1). But for a small company, whipping up a cluster from GKE is a breeze, and for big corps, company-wide cluster(s) setup by an ops team (OpenShift has been ok) is the way to go. The end result is that as a developer you just push containers and apply manifests.

neop1x 8y ago
I am scared of Helm. I would expect most people install helm chart like a package in Ubuntu, without knowing what it is going to do, what images are going to run and why. What if something malfunctions? What if helm upgrade won't success? Are you going to call a Helm support line?
- ttiurani 8y ago
  I was referring to using helm to configure my own charts to work in different test environments. Wrt public helm charts, you want to understand the chart install. Most of them are really dead simple if you know k8s.

westurner 8y ago

> Tools like Ansible and Puppet, as great as they are, do not guarantee your infrastructure will end up in the state you defined and you easily end up with broken services. False dilemma. Ansible and Puppet are great tools for configuring kubernetes, kubernetes worker nodes, and building container images. Kubernetes does not solve for host OS maintenance; though there are a number of host OS projects which remove most of what they consider to be unnecessary services, there's still need to upgrade kubernetes nodes and move pods out of the way first (which can be done with e.g. Puppet or Ansible). As well, it may not be appropriate for monitoring to depend upon kubernetes; there again you have nodes to manage with an SCM tool.

eeeeeeeeeeeee 8y ago

I have never heard any infrastructure explained as “define once, forget about it.” Ever. Everything requires maintenance, monitoring, upgrades, security, etc. And things just randomly break. I like Kubernetes but it is not without problems. And when your entire business is now leaning on Kubernetes, it can really hurt when it has problems and you can’t figure it out quickly. And if your expertise is not running infrastructure, a lot of people, as this post explains, are better off on AWS or Google Cloud using managed services. Or you might have a problem with the underlying storage platform that Kubernetes uses, so you now have to be a storage expert. I think it’s telling how many job offers I receive from companies already running Kubernetes and need help with it.