6 ms·

We're moving away from Kubernetes because managing even a simple implementation is a full-time job. Obviously Kubernetes is meant for enterprises, but I've seen

We're moving away from Kubernetes because managing even a simple implementation is a full-time job. Obviously Kubernetes is meant for enterprises, but I've seen seed round/series A startups with a small engineering team using it. We were at a crossroad. Either we hire an SRE that has extensive kubernetes experience for $130K/year or we move to a managed platform until we actually need those capabilities. We're happy with the move and the pressure relief is tremendous. Now we can focus on features and not whether or not deploying is using ENV vars from our local machines and causing the system to crash.

organsnyder 7y ago
In my experience, Kubernetes is a lot of work to install and maintain. However, I've found managed offerings—especially ones like Azure that actually abstract away the complexity (Amazon exposes all the nitty-gritty, in my experience)—make it easy to use for even smaller shops. Of course, not every workload benefits from the features k8s provides.
- bproven 7y ago
  I agree on EKS,IMO is it an unfinished bare bones "managed" service. Upgrades of control plane and nodes is a royal PITA - very manual and error prone. In GKE it is literally 2 gcloud commands. I think AWS hamstrings kubernetes still bc it is holding onto ECS. Hopefully they see the light soon, beef it up and make it equal to AKS/GKE. At least make masters/control plane free...ugh
- merb 7y ago
  did you ever looked at k3s? what is more difficult about k3s than managing ansible? what about kubespray? it's basically k8s managed via ansible? what makes it so difficult?
  - bpicolo 7y ago
    When folk want managed k8s, they effectively wanted something akin to a "serverless" compute pool that they point the k8s API at and worry about nothing else - not underlying hosts, networking configuration, storage, etc. The ideal is how close it gets to that. K3s still leaves hardware management to you.
    - merb 7y ago
      ansible, etc, vm's, etc as well?!
  - organsnyder 7y ago
    For work projects, I don't want to care about hardware—or "machines" of any sort, for that matter. Give me a ready-to-go platform with networking, storage, node management, etc. already configured. I don't even want to have to run scripts. At home, I'm happy to play around with such things—and just might (I just purchased an old Dell server to take care of some home tasks and provide a sandbox environment). But at work I want to be as far away from the bare metal (physical machines or VMs) as possible.
manigandham 7y ago
Why not just used managed Kubernetes? Every cloud vendor has it, even Tier 2 operators like DigitalOcean. Running Kubernetes is vastly different for using it to run your apps. You really shouldn’t do the former unless you have to be on-Prem or have some other need.
- 013a 7y ago
  There's a ton at play here. "Managed Kubernetes" really runs the spectrum between "one step above just installing it yourself on a bunch of VMs" and "I spend 1% of my time managing anything below the product." Each cloud provider exists somewhere different on this spectrum, with none of them being in quite the same location, and some of them have multiple different products which exist at different points. For example: AWS is among the most bare-bones. EKS is just a managed control plane; coming from GKE, you might click "create an cluster" then be very confused how there are no options for, say, instance size, or how many... because you have to do that all yourself. There are tools like eksctl or Rancher which can help with this, but ultimately, you're managing those instances. You're doing capacity planning (you think kube would be a great pick to integrate with spot fleets because of its ability to schedule and move workloads to a new instance when one goes down? have fun setting it up, hope you like ops work.). You're doing auto-scaling (and that ASG? its not going to know about your pod resource requests, so you either need some very smart manual coordination between the two, or you need to set up cluster-autoscaler). You're setting up cluster metrics (definitely need metrics-server. not heapster, that was last year, metrics-server is this year. but how to visualize? do i host grafana in the cluster? then i need to worry about authn. cloudwatch really isn't made for these kinds of things... maybe I'll just give datadog a few thousand bucks.) Crap, 1.16 is out already? They only support 9 months of releases with security updates?! I feel like I just upgraded my nodes! Oh well, time to lose a day replicating this update across all of my environments. I'd go on, but you get the point. There is nothing "managed" about EKS. DigitalOcean is pretty similar to this (it does provision instances, but the tooling beyond that is barebones). Google Cloud/GKE is "more managed" in a few sense; the cloud dashboard provides some great management capabilities out-of-the-box, such that you may not need to reach for something like Datadog, and the autoscaler works really well without a lot of tinkering. There are still underlying instances, so you're worrying about ingress protection, OS hardening, OS upgrades, etc... but its not as bad as AWS. Not by a long shot. The holy grail (for some companies) is really something like Azure AKS + Azure Container Instances. No instances to manage. Click a button for a kubernetes cluster. Schedule workloads. Get functional metrics, logging, tracing, dashboards out of the box. Don't worry about OS upgrades, hardening, autoscaling, upgrading the cluster, etc; we'll do it all for you, or at least make it one click to configure. That's the ideal situation. I haven't used AKS/ACI so I can't comment on whether Azure gets us there, but the idea is sound; even if its more expensive. This sounds like an anti-Kube post, right? Wrong (its a Tide ad). The beautiful thing about Kubernetes is that it can span this spectrum. The same exact API surface can scale from a fully-managed abstract platform where you just say "take this git repo and run it" (see: Gitlab Auto-DevOps), all the way to powering millions of workloads across dozens of federated clusters at Fortune 500 companies. But, to the OPs point: We're close to solving that right end of the spectrum, and a lot further away from the left end. We're getting there, but we're not there yet. There isn't enough abstracted management of these compute resources... yet. But there's enough money and desire for there to be that I know we'll get there.
kissgyorgy 7y ago
I don't understand. You had a problem; you needed to run a distributed system with containers which Kubernetes is perfect for. Did your problem disappeared or did your requirement changed or did you change to a different container orchestrator tool?
- sergiotapia 7y ago
  I'm saying we didn't have a problem that Kubernetes solved. A lot of seed-round/Series A startups also are quite simple, but hopping onto Kubernetes because it's "enterprise".
  - vitalus 7y ago
    Adopting any piece of technology just because it's a fad or otherwise trendy is rarely a good justification. It sounds like the primary reason you're deciding to move away from it is because you don't face any of the problems that it's there to solve, rather than it being an operational burden.
    - EdwardDiego 7y ago
      To be fair, k8s imposes a large complexity load. So what it gives you has got to be worth the time your developers/devops/sysops will spend learning to work with it. We're migrating to k8s, and it's fantastic for us, but we have a large complex system that we're moving into the cloud, which really benefits from k8s features - loving the horizontal pod autoscalers, especially when I can expose Kafka topic lag to them (via Prometheus) as a metric to scale certain apps on.
  - manigandham 7y ago
    It's really not "enterprise". It's because you have a resilient platform that provides automatic deployments, rolling upgrades, failover, auto-repair, logging, monitoring, storage/stateful data management, and more out of the box with a simple declarative YAML configuration. Kubernetes can run a simple container with 1 line if that's all you need, or you can scale up to several different services with a few more files, all the way a massive deployment with thousands of containers. How you use it is up to you, but you do need to read and understand the basics. However there's absolutely no need to run Kubernetes yourself unless you have a serious reason to. If you must, I highly recommend using something like Rancher [1] that can install and manage the cluster for you, even on cloud providers. 1. https://rancher.com/ https://rancher.com/
movedx 7y ago
> Kubernetes is meant for enterprises Actually it's meant for virtually no one, because very few people have the problems it solves. Like AutoScaling Groups, Serverless, and Cloud its self, it's a tool that solves a pain point in a specific domain. About 0.001% of the business world have that problem. > We're happy with the move and the pressure relief is tremendous. I'll bet it was. I've convinced businesses to go in the completely opposite direction to K8s. I've told them to developed a monolith before they start optimising into microservices. The startups that listened to me are still around and in round C. The others closed up shop ages ago (minus one of them) because they never got to market in time. K8s is a tool. Docker is a tool. Cloud is a tool. Businesses have to utilise tools as efficiently as possible to get their solutions out the door if they're to survive. Using K8s from the ground up is a death sentence.
- atombender 7y ago
  The "Kubernetes is only for Google-scale companies" meme needs to die. Kubernetes is a useful tool even if you have a single node. As a case in point: I just set up a small new app on GKE. Because I'm experienced with Kubernetes, within a few minutes I had my app (a React front-end written in TypeScript and a backend written in Go) running and receiving HTTPS traffic. I entered just a handful of shell commands to get from an empty cluster to a functional one. It's a cluster with a single node, no resilience. But this is genuinely useful, and a perfectly appropriate use case for Kubernetes. The alternative is the old way — VMs, perhaps with a sprinkling of Terraform and/or Salt/Ansible/Chef/Puppet, dealing with Linux, installing OS packages, controlling everything over SSH — or some high-level PaaS like AppEngine or Heroku. While it's an example that shows that Kubernetes "scales down", I'm now also free to scale up. While today it's a small app with essentially zero traffic, when/if the need should arise, I can just expand the nodepool, add a horizontal pod autoscaler, let the cluster span multiple regions/zones, and so on, and I'll have something that can handle whatever is thrown at it. My company has a ton of apps on multiple Kubernetes clusters, none of them very big. From my perspective, it's a huge win in operational simplicity over the "old" way. Docker itself is a benefit, but the true benefits come when you can treat your entire cluster as a virtualized resource and just throw containers and load balancers and persistent disks at it.
ngrilly 7y ago
Moving away to what?
- k__ 7y ago
  Maybe a serverless stack?
pjmlp 7y ago
I keep using managed platforms, Kubernetes is the new NoSQL.
rmetzler 7y ago
Don’t do Kubernetes or any other container orchestration when you don’t have CI/CD in place yet. I really think this is meant to work in tandem and changing Kubernetes YAML on a developer laptop can blow up fast.

organsnyder 7y ago

In my experience, Kubernetes is a lot of work to install and maintain. However, I've found managed offerings—especially ones like Azure that actually abstract away the complexity (Amazon exposes all the nitty-gritty, in my experience)—make it easy to use for even smaller shops. Of course, not every workload benefits from the features k8s provides.

bproven 7y ago
I agree on EKS,IMO is it an unfinished bare bones "managed" service. Upgrades of control plane and nodes is a royal PITA - very manual and error prone. In GKE it is literally 2 gcloud commands. I think AWS hamstrings kubernetes still bc it is holding onto ECS. Hopefully they see the light soon, beef it up and make it equal to AKS/GKE. At least make masters/control plane free...ugh
merb 7y ago
did you ever looked at k3s? what is more difficult about k3s than managing ansible? what about kubespray? it's basically k8s managed via ansible? what makes it so difficult?
- bpicolo 7y ago
  When folk want managed k8s, they effectively wanted something akin to a "serverless" compute pool that they point the k8s API at and worry about nothing else - not underlying hosts, networking configuration, storage, etc. The ideal is how close it gets to that. K3s still leaves hardware management to you.
  - merb 7y ago
    ansible, etc, vm's, etc as well?!
- organsnyder 7y ago
  For work projects, I don't want to care about hardware—or "machines" of any sort, for that matter. Give me a ready-to-go platform with networking, storage, node management, etc. already configured. I don't even want to have to run scripts. At home, I'm happy to play around with such things—and just might (I just purchased an old Dell server to take care of some home tasks and provide a sandbox environment). But at work I want to be as far away from the bare metal (physical machines or VMs) as possible.

manigandham 7y ago

Why not just used managed Kubernetes? Every cloud vendor has it, even Tier 2 operators like DigitalOcean. Running Kubernetes is vastly different for using it to run your apps. You really shouldn’t do the former unless you have to be on-Prem or have some other need.

013a 7y ago
There's a ton at play here. "Managed Kubernetes" really runs the spectrum between "one step above just installing it yourself on a bunch of VMs" and "I spend 1% of my time managing anything below the product." Each cloud provider exists somewhere different on this spectrum, with none of them being in quite the same location, and some of them have multiple different products which exist at different points. For example: AWS is among the most bare-bones. EKS is just a managed control plane; coming from GKE, you might click "create an cluster" then be very confused how there are no options for, say, instance size, or how many... because you have to do that all yourself. There are tools like eksctl or Rancher which can help with this, but ultimately, you're managing those instances. You're doing capacity planning (you think kube would be a great pick to integrate with spot fleets because of its ability to schedule and move workloads to a new instance when one goes down? have fun setting it up, hope you like ops work.). You're doing auto-scaling (and that ASG? its not going to know about your pod resource requests, so you either need some very smart manual coordination between the two, or you need to set up cluster-autoscaler). You're setting up cluster metrics (definitely need metrics-server. not heapster, that was last year, metrics-server is this year. but how to visualize? do i host grafana in the cluster? then i need to worry about authn. cloudwatch really isn't made for these kinds of things... maybe I'll just give datadog a few thousand bucks.) Crap, 1.16 is out already? They only support 9 months of releases with security updates?! I feel like I just upgraded my nodes! Oh well, time to lose a day replicating this update across all of my environments. I'd go on, but you get the point. There is nothing "managed" about EKS. DigitalOcean is pretty similar to this (it does provision instances, but the tooling beyond that is barebones). Google Cloud/GKE is "more managed" in a few sense; the cloud dashboard provides some great management capabilities out-of-the-box, such that you may not need to reach for something like Datadog, and the autoscaler works really well without a lot of tinkering. There are still underlying instances, so you're worrying about ingress protection, OS hardening, OS upgrades, etc... but its not as bad as AWS. Not by a long shot. The holy grail (for some companies) is really something like Azure AKS + Azure Container Instances. No instances to manage. Click a button for a kubernetes cluster. Schedule workloads. Get functional metrics, logging, tracing, dashboards out of the box. Don't worry about OS upgrades, hardening, autoscaling, upgrading the cluster, etc; we'll do it all for you, or at least make it one click to configure. That's the ideal situation. I haven't used AKS/ACI so I can't comment on whether Azure gets us there, but the idea is sound; even if its more expensive. This sounds like an anti-Kube post, right? Wrong (its a Tide ad). The beautiful thing about Kubernetes is that it can span this spectrum. The same exact API surface can scale from a fully-managed abstract platform where you just say "take this git repo and run it" (see: Gitlab Auto-DevOps), all the way to powering millions of workloads across dozens of federated clusters at Fortune 500 companies. But, to the OPs point: We're close to solving that right end of the spectrum, and a lot further away from the left end. We're getting there, but we're not there yet. There isn't enough abstracted management of these compute resources... yet. But there's enough money and desire for there to be that I know we'll get there.

kissgyorgy 7y ago

I don't understand. You had a problem; you needed to run a distributed system with containers which Kubernetes is perfect for. Did your problem disappeared or did your requirement changed or did you change to a different container orchestrator tool?

sergiotapia 7y ago
I'm saying we didn't have a problem that Kubernetes solved. A lot of seed-round/Series A startups also are quite simple, but hopping onto Kubernetes because it's "enterprise".
- vitalus 7y ago
  Adopting any piece of technology just because it's a fad or otherwise trendy is rarely a good justification. It sounds like the primary reason you're deciding to move away from it is because you don't face any of the problems that it's there to solve, rather than it being an operational burden.
  - EdwardDiego 7y ago
    To be fair, k8s imposes a large complexity load. So what it gives you has got to be worth the time your developers/devops/sysops will spend learning to work with it. We're migrating to k8s, and it's fantastic for us, but we have a large complex system that we're moving into the cloud, which really benefits from k8s features - loving the horizontal pod autoscalers, especially when I can expose Kafka topic lag to them (via Prometheus) as a metric to scale certain apps on.
- manigandham 7y ago
  It's really not "enterprise". It's because you have a resilient platform that provides automatic deployments, rolling upgrades, failover, auto-repair, logging, monitoring, storage/stateful data management, and more out of the box with a simple declarative YAML configuration. Kubernetes can run a simple container with 1 line if that's all you need, or you can scale up to several different services with a few more files, all the way a massive deployment with thousands of containers. How you use it is up to you, but you do need to read and understand the basics. However there's absolutely no need to run Kubernetes yourself unless you have a serious reason to. If you must, I highly recommend using something like Rancher [1] that can install and manage the cluster for you, even on cloud providers. 1. https://rancher.com/ https://rancher.com/

movedx 7y ago

> Kubernetes is meant for enterprises Actually it's meant for virtually no one, because very few people have the problems it solves. Like AutoScaling Groups, Serverless, and Cloud its self, it's a tool that solves a pain point in a specific domain. About 0.001% of the business world have that problem. > We're happy with the move and the pressure relief is tremendous. I'll bet it was. I've convinced businesses to go in the completely opposite direction to K8s. I've told them to developed a monolith before they start optimising into microservices. The startups that listened to me are still around and in round C. The others closed up shop ages ago (minus one of them) because they never got to market in time. K8s is a tool. Docker is a tool. Cloud is a tool. Businesses have to utilise tools as efficiently as possible to get their solutions out the door if they're to survive. Using K8s from the ground up is a death sentence.

atombender 7y ago
The "Kubernetes is only for Google-scale companies" meme needs to die. Kubernetes is a useful tool even if you have a single node. As a case in point: I just set up a small new app on GKE. Because I'm experienced with Kubernetes, within a few minutes I had my app (a React front-end written in TypeScript and a backend written in Go) running and receiving HTTPS traffic. I entered just a handful of shell commands to get from an empty cluster to a functional one. It's a cluster with a single node, no resilience. But this is genuinely useful, and a perfectly appropriate use case for Kubernetes. The alternative is the old way — VMs, perhaps with a sprinkling of Terraform and/or Salt/Ansible/Chef/Puppet, dealing with Linux, installing OS packages, controlling everything over SSH — or some high-level PaaS like AppEngine or Heroku. While it's an example that shows that Kubernetes "scales down", I'm now also free to scale up. While today it's a small app with essentially zero traffic, when/if the need should arise, I can just expand the nodepool, add a horizontal pod autoscaler, let the cluster span multiple regions/zones, and so on, and I'll have something that can handle whatever is thrown at it. My company has a ton of apps on multiple Kubernetes clusters, none of them very big. From my perspective, it's a huge win in operational simplicity over the "old" way. Docker itself is a benefit, but the true benefits come when you can treat your entire cluster as a virtualized resource and just throw containers and load balancers and persistent disks at it.

ngrilly 7y ago

Moving away to what?

k__ 7y ago
Maybe a serverless stack?

pjmlp 7y ago

I keep using managed platforms, Kubernetes is the new NoSQL.

rmetzler 7y ago

Don’t do Kubernetes or any other container orchestration when you don’t have CI/CD in place yet. I really think this is meant to work in tandem and changing Kubernetes YAML on a developer laptop can blow up fast.