7 ms·

A story about a Kubernetes migration

92 points by marcelo_lebre 8y ago

zerogvt 8y ago
It seems that k8s has won the deployment race by and large. I see a lot of success stories around (I'm hearing nice things from the DevOps teams in my organization as well). Yet I'm curious to hear a few cases where things did not pan out quite right. Note: The 5-15s DNS problem seems a pretty serious one. Weird that it didn't get more publicity (and a proper fix).
- zimbatm 8y ago
  There are a lot of things that can go wrong with K8s but there is always a way to fix them. For example a common mistake it to forget to allocate limits on pods, which then brings the worker node to capacity. I think the failure scenario is soft, it's just going to cost more engineering time to figure out how to upgrade the cluster to the new version, find out why this network overlay isn't performing as expected or debug this external resource that isn't being allocated properly, configure RBAC properly, play with various resource deployment strategies, tune how pods are being moved during a node auto-scaling event... The nice thing is that at the end it gives a unified API for all of the things, it forces some consistency in the infrastructure. My personal rule of thumb is that unless the client specifically need auto-scaling or have more than 100 services to run, have a 5 people devops team, just use Terraform. For a small number of servers a better strategy is to have a base image with Docker and monitoring, and use Terraform to deploy the infrastructure. CI can then use docker-compose to deploy the containers onto the hosts directly. This approach is much more stable and doesn't require to learn as many things as K8s. This can be run by a 1 man DevOps team without a sweat.
- geggam 8y ago
  Hard to fix something broken by design. Using TCP as the main message bus then using layers upon layers of NAT needs to be revisited, routing is the solution.
geekuillaume 8y ago
I'm working with Kubernetes recently and the learning curve is quite hard. I hope the team will improve kubectl to make it more user-friendly (error messages are hard to understand for beginners). A lot of cloud providers now have a way to easily deploy and manage a k8s cluster on their servers but I cannot find a tools that help with the deployment of a basic service, something like dokku but on Kubernetes. http://dokku.viewdocs.io/dokku/ http://dokku.viewdocs.io/dokku/
- halbritt 8y ago
  Have you looked at helm?
TeeWEE 8y ago
For me kubernetes is also a breeze. There is some learning curve because we started with Helm, Tiller, Grafana, Prometheus right from the start. But the kubectl command is easy to work with, and the k8s yaml files are really a breeze of fresh air compared to Ansible playbooks. We're not on production yet, but moving soon.
- y4mi 8y ago
  Uuuh, seriously? I always preferred ansibles to kubernetes yaml I'm using both daily and can work with either though
  - SmirkingRevenge 8y ago
    Can't speak for the OP but I dislike the direction they seem to be heading, incrementally (and perhaps accidentally) - yaml as a Turing complete programming language.
    - geerlingguy 8y ago
      > yaml as a Turing complete programming language. If someone is authoring Ansible playbooks this way, this is definitely not a best practice. Code should go into modules, plugins, filters, etc. Playbooks should be YAML, with extremely minimal use of any coding constructs.
- mgoetzke 8y ago
  I have not worked with Kubernetes yet, but I do have experience with ansible and I was under the impression that Kubernetes is working on a higher abstraction level than ansible. Do kubernetes files really concern themselves with little details such as how a database or application is configured ? I assumed that kubernetes is more about having 'images' of pre-installed machines (e.g via ansible) and having kubernetes just 'clone' them into production and interconnect them.
  - ryukafalz 8y ago
    You are correct, Kubernetes does operate at a higher level of abstraction. By the time you're deploying to Kubernetes, you'll already have images that can be used to run your applications. However, those images typically will be unconfigured aside from sane defaults. The final configuration (connecting an application to a database, etc) is indeed handled through Kubernetes.
mosselman 8y ago
I am setting up a swarm deployment of one of my apps as an experiment and I must say the learning curve is hardly there. I tried kubernetes, but I found that most resources that try to explain how it works are focussing too much on github-size deployments. I just want 2 instances of my app, a database and traefik with lets encrypt. Does anyone know of a proper resource for the 'just a tad more than dokku' size?
- sandGorgon 8y ago
  Same here. I think if you have less than 100 servers, then k8s is a real overkill. Swarm is much easier to reason about and run. It's a godsend for startups without dedicated devops.
  - mosselman 8y ago
    Swarm is working out really nicely so far.
  - hinkley 8y ago
    > dedicated devops. I have dreamed a dream, but now that dream is gone from me.
- shawabawa3 8y ago
  Setting up a kubernetes cluster itself is probably the biggest hurdle. Also, bear in mind if it's just for a single service the resource overhead of kubernetes may be significant, possibly even more than 50%. I'd strongly recommend using a hosted k8s - either GKE, EKS, or I believe digital ocean have just released one. If you want to use an existing VPS just to test it out, see the docs here https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/ https://kubernetes.io/docs/setup/independent/create-cluster-... Once you have the cluster running, kompose[1] might be a nice tool if you're used to using docker-compose, however I'd say just use it as a guideline - you'll probably want to rewrite most of what it generates at one point or another [1] https://github.com/kubernetes/kompose https://github.com/kubernetes/kompose
  - 8y ago
tekkk 8y ago
I hope that the original title of the story was intended sarcasm: "Unbabel migrated to Kubernetes and you won’t believe what happened next!" But so they managed to consolidate their infrastructure around Kubernetes and Google Cloud which made the management of their servers easier and faster? I wonder how much actual money they saved but I guess it will pay off for them in the long run. I've been dabbling with Kubernetes for some time now but God forbid it can be a bit complicated. Time required to become well-versed with Kubernetes is a hefty investment which is not for all organizations. Lots of small things that can drive up your blood-pressure when figuring them out. Were it simpler I would be much more inclined to be using it but now it's only in the "learning for funsies" -category. I feel people who've developed k8s have been more of the theoretical sort and not the regular-joe-dummy-kind like me.
- falcolas 8y ago
  Saying that Kubernetes is a bit complicated seems like saying that water can be a bit wet. Even their documentation can't keep up. And with a release cycle of 3 months, and a deprecation cycle of 6 months, you need a team dedicated to keeping up with K8s state-of-the-art; so much of that knowledge you picked up a year ago is at best stale, and at worst wrong. Sure, it makes setting up and keeping a set of containers up simple. But that's never really been that hard. To paraphrase an article from a few weeks ago: "We made microservices to address the problems with monoliths." "We made containers to address the problems with microservices." "We made Kubernetes to address the problems with containers."
  - eeZah7Ux 8y ago
    "Now we have a distributed monolith that requires 2x less developers to build and 5x more system engineers to deploy"
    - falcolas 8y ago
      Well, to be fair, all of those developers probably found themselves filling a systems engineer role "because the product developers are best equipped to handle the running and support of their own applications".
zedpm 8y ago
It sounds like they really wanted to switch to K8s and rationalized it. The cons of their existing solution are minor and easily addressed with correct use of Ansible, and the massive complexity of K8s is understated. As an example, they suggest that there's a heavy cognitive load associated with having devs run some Ansible playbooks, and then argue that to avoid that, they just have to introduce an entirely new toolchain via workshops and tutorials. Right.
- halbritt 8y ago
  Regardless of your skepticism, the benefits are real. Scaling applications in k8s, updating, and keeping configs consistent are a great deal easier for me than using Ansible or any other config management tool. In the end, it's a singular platform that one can build tooling against that allows an organization to abstract away the infrastructure. My team has done that (on top of k8s). As such, a developer can spin up a new environment with the click of a button, deploy whatever code they like, scale the environment, etc. with very little to no training. Those capabilities were a tremendous accelerator for my organization. Sure, you can build something similar with Ansible on AWS, but then you're married to AWS, you have to worry about sizing, and the cost of idle instances. In my experience, it's just a great deal more overhead.
  - falcolas 8y ago
    With ECS running on Fargate, idle instances don't exist. Throw in service autoscaling, and you have a simple scaling solution with no K8s cluster management required.
    - oppositelock 8y ago
      Or, you use EKS, no k8s cluster management required either. I'm running a production service on EKS, also tried it on GKE. Both take away most of the cluster management pain.
      - halbritt 8y ago
beat 8y ago
If you find Kubernetes a headache at first, consider looking at OpenShift. It's Red Hat's wrapper for Kubernetes, and does make some things easier.
- Carpetsmoker 8y ago
  I'm not sure if we've got enough wrappers yet. I think we want several more!
OJFord 8y ago
A story about a migration to Kubernetes. (As it is, I expected it to be about migrating k8s version. (Still much better than OP though...))
Fishkins 8y ago
Since they mention it a couple times in the article, how do other folks handle auth for their k8s dashboards? I'm trying to figure out the best approach that right now.
- colek42 8y ago
  You could build an authorization proxy that creates a token with the Kube API server and sets the Authorization header. This probably exists, but a project I worked on: https://github.com/boxboat/okta-nginx https://github.com/boxboat/okta-nginx might be a good starting point.
- tlynchpin 8y ago
  Me too. I haven't tried any of this but here's a suggestion: https://akomljen.com/protect-kubernetes-external-endpoints-with-oauth2-proxy/ https://akomljen.com/protect-kubernetes-external-endpoints-w...
user5994461 8y ago
Facepalm. This entire blog post reads like they didn't figure out how to deploy to more than one server with ansible, while ansible is made just for that.
sulam 8y ago
"The amount of instances multiplying was also clogging up our DevOps team." Let's pause a moment and appreciate that if you have a DevOps team, you're not doing DevOps.

92 points by marcelo_lebre 8y ago

It seems that k8s has won the deployment race by and large. I see a lot of success stories around (I'm hearing nice things from the DevOps teams in my organization as well). Yet I'm curious to hear a few cases where things did not pan out quite right. Note: The 5-15s DNS problem seems a pretty serious one. Weird that it didn't get more publicity (and a proper fix).

zimbatm 8y ago
There are a lot of things that can go wrong with K8s but there is always a way to fix them. For example a common mistake it to forget to allocate limits on pods, which then brings the worker node to capacity. I think the failure scenario is soft, it's just going to cost more engineering time to figure out how to upgrade the cluster to the new version, find out why this network overlay isn't performing as expected or debug this external resource that isn't being allocated properly, configure RBAC properly, play with various resource deployment strategies, tune how pods are being moved during a node auto-scaling event... The nice thing is that at the end it gives a unified API for all of the things, it forces some consistency in the infrastructure. My personal rule of thumb is that unless the client specifically need auto-scaling or have more than 100 services to run, have a 5 people devops team, just use Terraform. For a small number of servers a better strategy is to have a base image with Docker and monitoring, and use Terraform to deploy the infrastructure. CI can then use docker-compose to deploy the containers onto the hosts directly. This approach is much more stable and doesn't require to learn as many things as K8s. This can be run by a 1 man DevOps team without a sweat.
geggam 8y ago
Hard to fix something broken by design. Using TCP as the main message bus then using layers upon layers of NAT needs to be revisited, routing is the solution.

geekuillaume 8y ago

I'm working with Kubernetes recently and the learning curve is quite hard. I hope the team will improve kubectl to make it more user-friendly (error messages are hard to understand for beginners). A lot of cloud providers now have a way to easily deploy and manage a k8s cluster on their servers but I cannot find a tools that help with the deployment of a basic service, something like dokku but on Kubernetes. http://dokku.viewdocs.io/dokku/ http://dokku.viewdocs.io/dokku/

halbritt 8y ago
Have you looked at helm?

TeeWEE 8y ago

For me kubernetes is also a breeze. There is some learning curve because we started with Helm, Tiller, Grafana, Prometheus right from the start. But the kubectl command is easy to work with, and the k8s yaml files are really a breeze of fresh air compared to Ansible playbooks. We're not on production yet, but moving soon.

y4mi 8y ago
Uuuh, seriously? I always preferred ansibles to kubernetes yaml I'm using both daily and can work with either though
- SmirkingRevenge 8y ago
  Can't speak for the OP but I dislike the direction they seem to be heading, incrementally (and perhaps accidentally) - yaml as a Turing complete programming language.
  - geerlingguy 8y ago
    > yaml as a Turing complete programming language. If someone is authoring Ansible playbooks this way, this is definitely not a best practice. Code should go into modules, plugins, filters, etc. Playbooks should be YAML, with extremely minimal use of any coding constructs.
mgoetzke 8y ago
I have not worked with Kubernetes yet, but I do have experience with ansible and I was under the impression that Kubernetes is working on a higher abstraction level than ansible. Do kubernetes files really concern themselves with little details such as how a database or application is configured ? I assumed that kubernetes is more about having 'images' of pre-installed machines (e.g via ansible) and having kubernetes just 'clone' them into production and interconnect them.
- ryukafalz 8y ago
  You are correct, Kubernetes does operate at a higher level of abstraction. By the time you're deploying to Kubernetes, you'll already have images that can be used to run your applications. However, those images typically will be unconfigured aside from sane defaults. The final configuration (connecting an application to a database, etc) is indeed handled through Kubernetes.

mosselman 8y ago

I am setting up a swarm deployment of one of my apps as an experiment and I must say the learning curve is hardly there. I tried kubernetes, but I found that most resources that try to explain how it works are focussing too much on github-size deployments. I just want 2 instances of my app, a database and traefik with lets encrypt. Does anyone know of a proper resource for the 'just a tad more than dokku' size?

sandGorgon 8y ago
Same here. I think if you have less than 100 servers, then k8s is a real overkill. Swarm is much easier to reason about and run. It's a godsend for startups without dedicated devops.
- mosselman 8y ago
  Swarm is working out really nicely so far.
- hinkley 8y ago
  > dedicated devops. I have dreamed a dream, but now that dream is gone from me.
shawabawa3 8y ago
Setting up a kubernetes cluster itself is probably the biggest hurdle. Also, bear in mind if it's just for a single service the resource overhead of kubernetes may be significant, possibly even more than 50%. I'd strongly recommend using a hosted k8s - either GKE, EKS, or I believe digital ocean have just released one. If you want to use an existing VPS just to test it out, see the docs here https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/ https://kubernetes.io/docs/setup/independent/create-cluster-... Once you have the cluster running, kompose[1] might be a nice tool if you're used to using docker-compose, however I'd say just use it as a guideline - you'll probably want to rewrite most of what it generates at one point or another [1] https://github.com/kubernetes/kompose https://github.com/kubernetes/kompose
- 8y ago

tekkk 8y ago

I hope that the original title of the story was intended sarcasm: "Unbabel migrated to Kubernetes and you won’t believe what happened next!" But so they managed to consolidate their infrastructure around Kubernetes and Google Cloud which made the management of their servers easier and faster? I wonder how much actual money they saved but I guess it will pay off for them in the long run. I've been dabbling with Kubernetes for some time now but God forbid it can be a bit complicated. Time required to become well-versed with Kubernetes is a hefty investment which is not for all organizations. Lots of small things that can drive up your blood-pressure when figuring them out. Were it simpler I would be much more inclined to be using it but now it's only in the "learning for funsies" -category. I feel people who've developed k8s have been more of the theoretical sort and not the regular-joe-dummy-kind like me.

falcolas 8y ago
Saying that Kubernetes is a bit complicated seems like saying that water can be a bit wet. Even their documentation can't keep up. And with a release cycle of 3 months, and a deprecation cycle of 6 months, you need a team dedicated to keeping up with K8s state-of-the-art; so much of that knowledge you picked up a year ago is at best stale, and at worst wrong. Sure, it makes setting up and keeping a set of containers up simple. But that's never really been that hard. To paraphrase an article from a few weeks ago: "We made microservices to address the problems with monoliths." "We made containers to address the problems with microservices." "We made Kubernetes to address the problems with containers."
- eeZah7Ux 8y ago
  "Now we have a distributed monolith that requires 2x less developers to build and 5x more system engineers to deploy"
  - falcolas 8y ago
    Well, to be fair, all of those developers probably found themselves filling a systems engineer role "because the product developers are best equipped to handle the running and support of their own applications".

zedpm 8y ago

It sounds like they really wanted to switch to K8s and rationalized it. The cons of their existing solution are minor and easily addressed with correct use of Ansible, and the massive complexity of K8s is understated. As an example, they suggest that there's a heavy cognitive load associated with having devs run some Ansible playbooks, and then argue that to avoid that, they just have to introduce an entirely new toolchain via workshops and tutorials. Right.

halbritt 8y ago
Regardless of your skepticism, the benefits are real. Scaling applications in k8s, updating, and keeping configs consistent are a great deal easier for me than using Ansible or any other config management tool. In the end, it's a singular platform that one can build tooling against that allows an organization to abstract away the infrastructure. My team has done that (on top of k8s). As such, a developer can spin up a new environment with the click of a button, deploy whatever code they like, scale the environment, etc. with very little to no training. Those capabilities were a tremendous accelerator for my organization. Sure, you can build something similar with Ansible on AWS, but then you're married to AWS, you have to worry about sizing, and the cost of idle instances. In my experience, it's just a great deal more overhead.
- falcolas 8y ago
  With ECS running on Fargate, idle instances don't exist. Throw in service autoscaling, and you have a simple scaling solution with no K8s cluster management required.
  - oppositelock 8y ago
    Or, you use EKS, no k8s cluster management required either. I'm running a production service on EKS, also tried it on GKE. Both take away most of the cluster management pain.
    - halbritt 8y ago

beat 8y ago

If you find Kubernetes a headache at first, consider looking at OpenShift. It's Red Hat's wrapper for Kubernetes, and does make some things easier.

Carpetsmoker 8y ago
I'm not sure if we've got enough wrappers yet. I think we want several more!

OJFord 8y ago

A story about a migration to Kubernetes. (As it is, I expected it to be about migrating k8s version. (Still much better than OP though...))

Fishkins 8y ago

Since they mention it a couple times in the article, how do other folks handle auth for their k8s dashboards? I'm trying to figure out the best approach that right now.

colek42 8y ago
You could build an authorization proxy that creates a token with the Kube API server and sets the Authorization header. This probably exists, but a project I worked on: https://github.com/boxboat/okta-nginx https://github.com/boxboat/okta-nginx might be a good starting point.
tlynchpin 8y ago
Me too. I haven't tried any of this but here's a suggestion: https://akomljen.com/protect-kubernetes-external-endpoints-with-oauth2-proxy/ https://akomljen.com/protect-kubernetes-external-endpoints-w...

user5994461 8y ago

Facepalm. This entire blog post reads like they didn't figure out how to deploy to more than one server with ansible, while ansible is made just for that.

sulam 8y ago

"The amount of instances multiplying was also clogging up our DevOps team." Let's pause a moment and appreciate that if you have a DevOps team, you're not doing DevOps.