4 ms·

I work for a mid size company with 30-40 engineers managing 20-30 very diverse apps in terms of scale requirements and architectural complexity. It took our dev

I work for a mid size company with 30-40 engineers managing 20-30 very diverse apps in terms of scale requirements and architectural complexity. It took our devops team (4-5 people) probably 18 months to learn and fully migrate all our apps to Kubernetes. The upfront cost was massive, but nowadays the app teams own their deployments, configurations, SLA's, monitors, and cost metrics. Introducing Kubernetes into our org allowed us to do this; we would have never gotten here with our legacy deployment and orchestration Frankenstein. The change has been so positive, that I adopted Kubernetes for my solo projects and I am having a blast. I understand Coinbase's position, and they need to stick to what works for them. I just wanted to bring up a positive POV for a technology I am becoming a fan of.

Gene_Parmesan 6y ago
Do you have any intuition as to what percentage of the benefits your company has seen come from kubernetes specifically, and how much just from the exercise of spending 18 months working on a modern, coherent, efficient infrastructure?
- DevKoala 6y ago
  Kubernetes is almost a standard, all our engineers are eager to learn it. It is extensible, some app teams have started building operators to tackle problems in a more efficient way. It allowed us to standardize metrics and monitoring, we didn’t have a clear story for this before. It is cloud agnostic and with great compatibility; I was part of a migration from GKE to EKS, and it was painless.
ec109685 6y ago
One other advantage of Kubernetes that is overlooked in the article is the benefit with a heterogeneous cluster of instant auto scaling. For example you have 10 apps on a k8s cluster that each use the same resources, you can give 20% buffer for the cluster, which would let any single app use 300% of their allocation instantly. With VMs, you’ll be stuck waiting for VMs to spin up or have to give each app their own large buffer to handle bursts of traffic.
- dijit 6y ago
  Isn't that the benefit of cloud? Maybe I'm spoiled by GCP and it's not the same in other providers; but I can have a brand new debian VM configured and operational in less than 10s (less the time that I run my own startup script). Debian machines spawn with incredible speed; not much slower (if at all slower) than a new container.
  - p_l 6y ago
    Spawning a bare bones VM is easy and fast, the problem is getting the applications there, which is where k8s shines.
    - dijit 6y ago
      Maybe I'm spoiled then, it takes less than 2s to spin up my application on a new instance. :\ There are also many options for creating Debian based image files with all you need baked in. I guess it depends very much on the application you have.
      - p_l 6y ago
        A lot depends on whether "bake application into AMI, use ASG to scale and zero-downtime deploy" is applicable or cost effective. I think I've been in total of one company where it was somewhat true, and even there we didn't do it.
- amerine 6y ago
  It amazing to me how many people associate this with kube. You’ve had this for years in numerous ways, the only difference is some yaml and marketing.
  - jayd16 6y ago
    What other solution is cloud agnostic as well as has an easy local dev story?
thinkharderdev 6y ago
Exactly. A lot of the comments here are along the lines of "we setup a k8s cluster and now managing it is a huge burden" which is not surprising. The power of k8s is to allow separation of concerns in your technology organization. You can have a dedicated team to build and maintain the underlying cluster and then app development teams are consumers who deploy their application on the common infrastructure. Kubernetes provides a nice abstraction layer so those two teams/orgs can interact through a well-defined API. As a dev team, we can manage our own infrastructure and pipeline through declarative configurations and let someone else manage the underlying compute and network infrastructure. As long as you don't fall into the anti-pattern of "every team builds it's own k8s cluster" then you should be able to derive some nice economies of scale.
yCombLinks 6y ago
You're comparing 18 months of migrating to K8s to doing nothing, rather than 18 months of other possible solutions.
vandalatyou 6y ago
I'm a fan of cloud native kubernetes. But: I've also been put in the terrible position of supporting a platform sold to run on kubernetes in an air gapped, on prem, bare metal environment. But: I've also been put into the terrible position of fighting vendors armed with agile, k8s and microservices selling that combined mayhem as a replacement for an openmp,openacc based massively parallel, on prem, bare metal HPC system with a strongly conservative development and operational tradition. I understand loving efficiency and time savings but not everything simply works better with k8s.
- folmar 6y ago
  > I've also been put in the terrible position of supporting a platform sold to run on kubernetes in an air gapped, on prem, bare metal environment. Would you share your experience?
  - vandalatyou 6y ago
    Sorry, can't share details as I enjoy being employed.

Gene_Parmesan 6y ago

Do you have any intuition as to what percentage of the benefits your company has seen come from kubernetes specifically, and how much just from the exercise of spending 18 months working on a modern, coherent, efficient infrastructure?

DevKoala 6y ago
Kubernetes is almost a standard, all our engineers are eager to learn it. It is extensible, some app teams have started building operators to tackle problems in a more efficient way. It allowed us to standardize metrics and monitoring, we didn’t have a clear story for this before. It is cloud agnostic and with great compatibility; I was part of a migration from GKE to EKS, and it was painless.

ec109685 6y ago

One other advantage of Kubernetes that is overlooked in the article is the benefit with a heterogeneous cluster of instant auto scaling. For example you have 10 apps on a k8s cluster that each use the same resources, you can give 20% buffer for the cluster, which would let any single app use 300% of their allocation instantly. With VMs, you’ll be stuck waiting for VMs to spin up or have to give each app their own large buffer to handle bursts of traffic.

dijit 6y ago
Isn't that the benefit of cloud? Maybe I'm spoiled by GCP and it's not the same in other providers; but I can have a brand new debian VM configured and operational in less than 10s (less the time that I run my own startup script). Debian machines spawn with incredible speed; not much slower (if at all slower) than a new container.
- p_l 6y ago
  Spawning a bare bones VM is easy and fast, the problem is getting the applications there, which is where k8s shines.
  - dijit 6y ago
    Maybe I'm spoiled then, it takes less than 2s to spin up my application on a new instance. :\ There are also many options for creating Debian based image files with all you need baked in. I guess it depends very much on the application you have.
    - p_l 6y ago
      A lot depends on whether "bake application into AMI, use ASG to scale and zero-downtime deploy" is applicable or cost effective. I think I've been in total of one company where it was somewhat true, and even there we didn't do it.
amerine 6y ago
It amazing to me how many people associate this with kube. You’ve had this for years in numerous ways, the only difference is some yaml and marketing.
- jayd16 6y ago
  What other solution is cloud agnostic as well as has an easy local dev story?

thinkharderdev 6y ago

Exactly. A lot of the comments here are along the lines of "we setup a k8s cluster and now managing it is a huge burden" which is not surprising. The power of k8s is to allow separation of concerns in your technology organization. You can have a dedicated team to build and maintain the underlying cluster and then app development teams are consumers who deploy their application on the common infrastructure. Kubernetes provides a nice abstraction layer so those two teams/orgs can interact through a well-defined API. As a dev team, we can manage our own infrastructure and pipeline through declarative configurations and let someone else manage the underlying compute and network infrastructure. As long as you don't fall into the anti-pattern of "every team builds it's own k8s cluster" then you should be able to derive some nice economies of scale.

yCombLinks 6y ago

You're comparing 18 months of migrating to K8s to doing nothing, rather than 18 months of other possible solutions.

vandalatyou 6y ago

I'm a fan of cloud native kubernetes. But: I've also been put in the terrible position of supporting a platform sold to run on kubernetes in an air gapped, on prem, bare metal environment. But: I've also been put into the terrible position of fighting vendors armed with agile, k8s and microservices selling that combined mayhem as a replacement for an openmp,openacc based massively parallel, on prem, bare metal HPC system with a strongly conservative development and operational tradition. I understand loving efficiency and time savings but not everything simply works better with k8s.

folmar 6y ago
> I've also been put in the terrible position of supporting a platform sold to run on kubernetes in an air gapped, on prem, bare metal environment. Would you share your experience?
- vandalatyou 6y ago
  Sorry, can't share details as I enjoy being employed.