25 ms·

What we learned after a year on Kubernetes

213 points by ecliptik 6y ago

t3rabytes 6y ago
> Increased billing from cross-AZ traffic Yep, we (Basecamp) have been bit by this with EKS too. By default the ALB Ingress Controller will put all cluster nodes into the target group and a request can hit kube-proxy in any AZ before being directed to the right place, causing a lot of inter-AZ traffic churn. It's mildly annoying. (with alb-ingress-controller you can change target-type to IP to have traffic go direct to pods that will actually back a given service instead of via kube-proxy on a random node, but the big fault there is that if alb-ingress-controller gets hung up on something and you deploy, you can send traffic to nowhere because the target group still has old IPs for pods that don't exist).
- GauntletWizard 6y ago
  It's so worth it, though. You may not realize it, but your Healthchecks don't work unless you are using target-type IP. In the standard mode, AWS healthchecks each kubernetes node, but not your pods themselves. It will keep sending traffic to pods that are unhealthy (such as in state: terminating) if it already has connections open through kube-proxy to them, and stop sending traffic to healthy pods if the connection to them was through a node it considers "unhealthy". I wrote at tool[1] to handle this at Houseparty before alb-ingress-controller supported this mode, and it gained us a whole nine as measured by the ALB. The same failure mode actually exists with instances, though they typically rotate much more slowly. Monitor your ALB-ingress-controller and treat it as critical infrastructure, because it is. [1]https://github.com/GauntletWizard/targetgroupcontroller https://github.com/GauntletWizard/targetgroupcontroller
  - t3rabytes 6y ago
    > It's so worth it, though. It was until it caused a production outage, then it became very not worth it.
    - GauntletWizard 6y ago
      ALB Ingress Controller didn't cause a hard outage, lack of monitoring on critical infrastructure caused a hard outage. The ALB Ingress Controller was already critical infrastructure beforehand, because the NodePort method still relies on it adding and removing instances from the targetgroup - It just takes longer for that to become an issue. The fact that it's hours instead of days hurts, but also makes it much easier to diagnose.
  - rsanders 6y ago
    Can you expand on this: "AWS healthchecks each kubernetes node, but not your pods themselves". Are you talking about a keepalive connection to an unhealthy pod which is reused for multiple requests? So the failure modes are, if I understand you correctly, a) the ALB keeps sending requests through an established keep-alive HTTP connection which terminates in an unhealthy pod, but which it sees as healthy because the node is healthy and can route traffic to another, healthy pod, and b) the health of an established HTTP keepalive connection is perceived to be that of the node rather than the destination pod, so nodes which become unhealthy can cause the ALB to unnecessarily terminate a keepalive connection. We had to switch to using target-type=instance because of issues with pods not being deregistered. I'd prefer to use target-type IP but it seemed like preventing 500s on rollouts required a bit of testing and tuning with a very specific approach. e.g. introducing a longish delay on pod termination with a lifecycle hook and using the pod readiness gate support recently added to alb-ingress-controller.
    - GauntletWizard 6y ago
      You've got it exactly right. Your problem of pods not being deregistered is a real problem, but also with a quick fix: The default "Deregistration delay" for ALBs is 300 seconds but for kubernetes pods the TerminationGracePeriod defaults to 60 seconds. This means that your load balancer keeps trying that pod for 4 whole minutes after it's been hard-shutdown. Here's the annotation that I used to fix that: alb.ingress.kubernetes.io/target-group-attributes: deregistration_delay.timeout_seconds=30,slow_start.duration_seconds=30
- ckdarby 6y ago
  What you're looking for is called Service Topology it is in Alpha. https://kubernetes.io/docs/concepts/services-networking/service-topology/ https://kubernetes.io/docs/concepts/services-networking/serv...
  - DuskStar 6y ago
    I genuinely cannot wait for that to make it to EKS.
    - ckdarby 6y ago
      Yep, we were getting an insane amount of AZ chatter with Apache Pulsar because of all the components: Request(us-west-1a) -> Pulsar Proxy(us-west-1b) -> Broker(us-west-1c) -> Bookie (us-west-1a) Though for anyone using Pulsar on EKS who need multi-az for disaster recovery for the time being could deploy in a single AZ and then use the geo-replication feature in Pulsar to copy over to another AZ.
- qubit23 6y ago
  Regarding the alb-ingress-controller blackholing traffic with an IP target type ingress, check out support for pod readiness gates: https://kubernetes-sigs.github.io/aws-alb-ingress-controller/guide/ingress/pod-conditions/ https://kubernetes-sigs.github.io/aws-alb-ingress-controller.... Assuming you're using a rolling deployment, old pods won't be stopped until the new ones are added to the ALB target group and are in a healthy state. This has eliminated the small number of ELB 500s we'd see on deploys.
ishcheklein 6y ago
We are using k8s as well for a project that we also provide on-prem install too. We decided to use kustomize for now vs doing helm for now. Curious, have someone had experience with both to compare. What are the benefits of using helm?
- dpc_pw 6y ago
  `helm` is an absolute garbage. I'll reserve judgment about `kustomize` until I have more practical experience, but so far it looks to me that it's going to be another YAML disaster. The problem with k8s ecosystem is that developers conflated k8s using yaml for a semi-human-readable serialization of k8s resources, as reason to employ YAML to absolutely everything. What I'd like is to generate serialized yaml files ... you know ... with code. Using some typed programming language, to be able to build any abstractions required for the job, and get some checks, compiler errors, and even ability to have asserts, unit-tests and so on. Instead, I have to dig YAML with a pickaxe in the YAML-mine without any technology to assist me.
  - justicezyx 6y ago
    Insider joke: K8s was born to solve the same problem Borg solved or simply design in such a way to not have that problem. One of them is the config files hell. Borgcfg files ranks #3 in terms of line count across all of Google. (Borgcfg language is similar to Jsonnet, helm) In terms of the amount of power wielded buy a single CLI, `borgcfg`, this little program that has less than 50k lines of nontesting code, stands at a peak dwarfs almost anything else inside Google. I declared in 2015 that k8s will eventually enjoy the same borgcfg problem. And I was advocating code driving application management approach as you suggested, not the configuration as code (behind the infrastructure as code facade). Well, it seems I am still on the track to be on the correct camp...
    - esprehn 6y ago
      Relevant thread (with you) from the nearby past: https://news.ycombinator.com/item?id=20849454 https://news.ycombinator.com/item?id=20849454 The yaml monster in k8s is quite unpleasant. It's always been surprising to me how we somehow ended up with executable BUILD files but totally static deploy files.
deleted 6y ago
[deleted]
nuker 6y ago
It reads like a press release, no gotchas, no complaints... https://k8s.af/ https://k8s.af/
- marinj 6y ago
  There are some complains that we can share, but honestly they are not really that fun to read. We are still early in our transition and one major complaint I have is about helm and lack of flexibility of it and operator being very flexible but a huge time sink. I think a lot of complaints we would usually have are reduced due to the fact that we use managed K8s with GKE. All other complaints I can think of are related to our application and some architecture decisions we made early on, but that won't be interesting to anyone but people at GitLab.
  - nuker 6y ago
    > complains that we can share, but honestly they are not really that fun to read. "You are mistaken counselor" (c) The Descendants
zelphirkalt 6y ago
I do have only very minimal xp in k8s, as I have tried to use it only once (and never again). To me it seem like mOst of the comments are about the unusable and unmaintainable yaml configuration files. This seems like exactly the issue I had a few years ago, when I was cursing, because I could not find a spec or docs for the yaml stuff in all the pages and pages of seemingly important documentation and thus could not figure out how to transform a docker only deployment into k8s deployment. I also remember having to set up helm, according to whatever tutorial or official guide I was following. It was all a distant nightmare, that I do not wish to repeat.
- regularfry 6y ago
  There's definitely a need for an opinionated layer over the top, with much simpler configuration. Probably more than one, with different use cases. Nobody should have to wrangle that much yaml.
  - cpursley 6y ago
    https://convox.com https://convox.com It's basically heroku on top of k8s and you bring your own cloud.
- louiskottmann 6y ago
  You can start off with an example you would find online, then use "kubectl explain" to read about the options used and other ones you want to use.
  - deleted 6y ago
    [deleted]
- sippeangelo 6y ago
  This. I'm currently trying to figure this mess out and all I want is a reference for the different objects and what parameters they take, or what the hell an "apiVersion" is and which ones are available. Instead their "documentation" is a bunch of unhelpful guides that never show examples of what I need to know. How this has reached widespread adoption is beyond me.
pjmlp 6y ago
I learned not to use it and wait until the k8s fad goes away.
- bsaul 6y ago
  i'm curious to hear about your alternative deployment strategy (no snark intended)
  - TheDong 6y ago
    For many simple applications, an aws autoscaling group and application load balancer is as good as k8s and far simpler. If you're running microservices (to the point where you have so many small services that having an asg per service), than the above strategy does waste resources compared to k8s I guess... But you'd need a lot of microservices to justify the significant overhead k8s has.
    - aequitas 6y ago
      But now you have to manage those VM's, the OS lifecycle, different distributions, application deployment, configuration and dependencies. And we're back at containers. Or you have to go towards lambda. For K8s you need to manage the vm's/os as well but that part is mostly trivial as it's highly focussed on immutable/disposable infrastructure.
  - evil-olive 6y ago
    We use Nomad from Hashicorp at $dayjob and I'm quite happy with it. https://www.nomadproject.io/ https://www.nomadproject.io/ https://www.hashicorp.com/resources/introduction-nomad https://www.hashicorp.com/resources/introduction-nomad https://blog.cloudflare.com/how-we-use-hashicorp-nomad/ https://blog.cloudflare.com/how-we-use-hashicorp-nomad/
  - pjmlp 6y ago
    VMs as usual.
    - 6y ago
stepbeek 6y ago
Was anyone else surprised by the timeline of this post? I was under the impression that Gitlab had been fully on K8S for years now, given the hard push of their K8S offering 18 months ago.
- jarv 6y ago
  It has been possible to run GitLab the application on K8s for some time now for the majority of self-managed deployments. There were some limitations however, for running GitLab on K8s at the very large scale of GitLab.com that we have been working through in the last year. (disclaimer: blog post author)
  - stepbeek 6y ago
    > August 2019 when we migrated the GitLab Container Registry to Kubernetes, the first service to move. Though this was a critical and high traffic service, it was a good choice for the first migration because it is a stateless application with only a few external dependencies I was under the impression that porting a large stateless service was something that wasn't too bad to do with K8S? What limitations prevented this from going ahead earlier?
    - jarv 6y ago
      Yeah, this is why we decided to migrate it first about a year ago, and we probably could have done it sooner than that. Other services were not yet ready to migrate until recently due to some blocking issues which was a factor in the decision. disclaimer: blog post author
Keycap 6y ago
I'm not sure if i get it but normally internal traffic across AZs should be included? It was also not clear to me after reading their issue page.
ChicagoDave 6y ago
My question. Why not convert the rails code/APIs to blob/FaaS and skip all the DevOps complexity? Are containers providing enough long-term cost-effectiveness to a complete cloud-native application architecture? Or is it just mapping VMs to something “like” VMs so your topology remains mostly the same?
- auspex 6y ago
  Serverless has its own complexity. Deploying a smaller, lightweight purpose built image, with no orchestration lock-in is the main benefit.
  - ChicagoDave 6y ago
    Those are fair infrastructure reasons, but not necessarily cost-effective. There are always competing perspectives. What does the business want. What do the developers want. What does the infrastructure team want. When power is centralized in one area, decisions can be made that overlook those competing purposes. I see containers being adopted without openly discussing the options and long-term consequences.
deleted 6y ago
[deleted]
solatic 6y ago
> We found that with an application that increases its memory utilization over time, low requests (which reserves memory for each pod) and a generous hard limit on utilization was a recipe for node saturation and a high rate of evictions. To adjust for this we eventually decided to use higher requests and lower limit which took pressure off of the nodes and allowed pods to be recycled without putting too much pressure on the node. Kubernetes works at its most efficient when you scale by adding more Pods, rather than scaling within a Pod. Requests and limits should be equal to each other, and a HorizontalPodAutoscaler (or KEDA Scales object) should be used to add more Pods as utilization increases. Using Cluster Autoscaler then makes sure that your cluster has sufficient Nodes to schedule the Pods. Then you just need to set up alerting to tell you if the autoscaling reaches minimums (indicating that you should consider lowering the minimum, to improve efficiency) or maximums (indicating that you should consider increasing the maximums), and you can leave things well enough alone, unless the speed at which scaling occurs is insufficiently slow for your use case.

213 points by ecliptik 6y ago

> Increased billing from cross-AZ traffic Yep, we (Basecamp) have been bit by this with EKS too. By default the ALB Ingress Controller will put all cluster nodes into the target group and a request can hit kube-proxy in any AZ before being directed to the right place, causing a lot of inter-AZ traffic churn. It's mildly annoying. (with alb-ingress-controller you can change target-type to IP to have traffic go direct to pods that will actually back a given service instead of via kube-proxy on a random node, but the big fault there is that if alb-ingress-controller gets hung up on something and you deploy, you can send traffic to nowhere because the target group still has old IPs for pods that don't exist).

GauntletWizard 6y ago
It's so worth it, though. You may not realize it, but your Healthchecks don't work unless you are using target-type IP. In the standard mode, AWS healthchecks each kubernetes node, but not your pods themselves. It will keep sending traffic to pods that are unhealthy (such as in state: terminating) if it already has connections open through kube-proxy to them, and stop sending traffic to healthy pods if the connection to them was through a node it considers "unhealthy". I wrote at tool[1] to handle this at Houseparty before alb-ingress-controller supported this mode, and it gained us a whole nine as measured by the ALB. The same failure mode actually exists with instances, though they typically rotate much more slowly. Monitor your ALB-ingress-controller and treat it as critical infrastructure, because it is. [1]https://github.com/GauntletWizard/targetgroupcontroller https://github.com/GauntletWizard/targetgroupcontroller
- t3rabytes 6y ago
  > It's so worth it, though. It was until it caused a production outage, then it became very not worth it.
  - GauntletWizard 6y ago
    ALB Ingress Controller didn't cause a hard outage, lack of monitoring on critical infrastructure caused a hard outage. The ALB Ingress Controller was already critical infrastructure beforehand, because the NodePort method still relies on it adding and removing instances from the targetgroup - It just takes longer for that to become an issue. The fact that it's hours instead of days hurts, but also makes it much easier to diagnose.
- rsanders 6y ago
  Can you expand on this: "AWS healthchecks each kubernetes node, but not your pods themselves". Are you talking about a keepalive connection to an unhealthy pod which is reused for multiple requests? So the failure modes are, if I understand you correctly, a) the ALB keeps sending requests through an established keep-alive HTTP connection which terminates in an unhealthy pod, but which it sees as healthy because the node is healthy and can route traffic to another, healthy pod, and b) the health of an established HTTP keepalive connection is perceived to be that of the node rather than the destination pod, so nodes which become unhealthy can cause the ALB to unnecessarily terminate a keepalive connection. We had to switch to using target-type=instance because of issues with pods not being deregistered. I'd prefer to use target-type IP but it seemed like preventing 500s on rollouts required a bit of testing and tuning with a very specific approach. e.g. introducing a longish delay on pod termination with a lifecycle hook and using the pod readiness gate support recently added to alb-ingress-controller.
  - GauntletWizard 6y ago
    You've got it exactly right. Your problem of pods not being deregistered is a real problem, but also with a quick fix: The default "Deregistration delay" for ALBs is 300 seconds but for kubernetes pods the TerminationGracePeriod defaults to 60 seconds. This means that your load balancer keeps trying that pod for 4 whole minutes after it's been hard-shutdown. Here's the annotation that I used to fix that: alb.ingress.kubernetes.io/target-group-attributes: deregistration_delay.timeout_seconds=30,slow_start.duration_seconds=30
ckdarby 6y ago
What you're looking for is called Service Topology it is in Alpha. https://kubernetes.io/docs/concepts/services-networking/service-topology/ https://kubernetes.io/docs/concepts/services-networking/serv...
- DuskStar 6y ago
  I genuinely cannot wait for that to make it to EKS.
  - ckdarby 6y ago
    Yep, we were getting an insane amount of AZ chatter with Apache Pulsar because of all the components: Request(us-west-1a) -> Pulsar Proxy(us-west-1b) -> Broker(us-west-1c) -> Bookie (us-west-1a) Though for anyone using Pulsar on EKS who need multi-az for disaster recovery for the time being could deploy in a single AZ and then use the geo-replication feature in Pulsar to copy over to another AZ.
qubit23 6y ago
Regarding the alb-ingress-controller blackholing traffic with an IP target type ingress, check out support for pod readiness gates: https://kubernetes-sigs.github.io/aws-alb-ingress-controller/guide/ingress/pod-conditions/ https://kubernetes-sigs.github.io/aws-alb-ingress-controller.... Assuming you're using a rolling deployment, old pods won't be stopped until the new ones are added to the ALB target group and are in a healthy state. This has eliminated the small number of ELB 500s we'd see on deploys.

ishcheklein 6y ago

We are using k8s as well for a project that we also provide on-prem install too. We decided to use kustomize for now vs doing helm for now. Curious, have someone had experience with both to compare. What are the benefits of using helm?

dpc_pw 6y ago
`helm` is an absolute garbage. I'll reserve judgment about `kustomize` until I have more practical experience, but so far it looks to me that it's going to be another YAML disaster. The problem with k8s ecosystem is that developers conflated k8s using yaml for a semi-human-readable serialization of k8s resources, as reason to employ YAML to absolutely everything. What I'd like is to generate serialized yaml files ... you know ... with code. Using some typed programming language, to be able to build any abstractions required for the job, and get some checks, compiler errors, and even ability to have asserts, unit-tests and so on. Instead, I have to dig YAML with a pickaxe in the YAML-mine without any technology to assist me.
- justicezyx 6y ago
  Insider joke: K8s was born to solve the same problem Borg solved or simply design in such a way to not have that problem. One of them is the config files hell. Borgcfg files ranks #3 in terms of line count across all of Google. (Borgcfg language is similar to Jsonnet, helm) In terms of the amount of power wielded buy a single CLI, `borgcfg`, this little program that has less than 50k lines of nontesting code, stands at a peak dwarfs almost anything else inside Google. I declared in 2015 that k8s will eventually enjoy the same borgcfg problem. And I was advocating code driving application management approach as you suggested, not the configuration as code (behind the infrastructure as code facade). Well, it seems I am still on the track to be on the correct camp...
  - esprehn 6y ago
    Relevant thread (with you) from the nearby past: https://news.ycombinator.com/item?id=20849454 https://news.ycombinator.com/item?id=20849454 The yaml monster in k8s is quite unpleasant. It's always been surprising to me how we somehow ended up with executable BUILD files but totally static deploy files.

deleted 6y ago

[deleted]

nuker 6y ago

It reads like a press release, no gotchas, no complaints... https://k8s.af/ https://k8s.af/

marinj 6y ago
There are some complains that we can share, but honestly they are not really that fun to read. We are still early in our transition and one major complaint I have is about helm and lack of flexibility of it and operator being very flexible but a huge time sink. I think a lot of complaints we would usually have are reduced due to the fact that we use managed K8s with GKE. All other complaints I can think of are related to our application and some architecture decisions we made early on, but that won't be interesting to anyone but people at GitLab.
- nuker 6y ago
  > complains that we can share, but honestly they are not really that fun to read. "You are mistaken counselor" (c) The Descendants

zelphirkalt 6y ago

I do have only very minimal xp in k8s, as I have tried to use it only once (and never again). To me it seem like mOst of the comments are about the unusable and unmaintainable yaml configuration files. This seems like exactly the issue I had a few years ago, when I was cursing, because I could not find a spec or docs for the yaml stuff in all the pages and pages of seemingly important documentation and thus could not figure out how to transform a docker only deployment into k8s deployment. I also remember having to set up helm, according to whatever tutorial or official guide I was following. It was all a distant nightmare, that I do not wish to repeat.

regularfry 6y ago
There's definitely a need for an opinionated layer over the top, with much simpler configuration. Probably more than one, with different use cases. Nobody should have to wrangle that much yaml.
- cpursley 6y ago
  https://convox.com https://convox.com It's basically heroku on top of k8s and you bring your own cloud.
louiskottmann 6y ago
You can start off with an example you would find online, then use "kubectl explain" to read about the options used and other ones you want to use.
- deleted 6y ago
  [deleted]
sippeangelo 6y ago
This. I'm currently trying to figure this mess out and all I want is a reference for the different objects and what parameters they take, or what the hell an "apiVersion" is and which ones are available. Instead their "documentation" is a bunch of unhelpful guides that never show examples of what I need to know. How this has reached widespread adoption is beyond me.

pjmlp 6y ago

I learned not to use it and wait until the k8s fad goes away.

bsaul 6y ago
i'm curious to hear about your alternative deployment strategy (no snark intended)
- TheDong 6y ago
  For many simple applications, an aws autoscaling group and application load balancer is as good as k8s and far simpler. If you're running microservices (to the point where you have so many small services that having an asg per service), than the above strategy does waste resources compared to k8s I guess... But you'd need a lot of microservices to justify the significant overhead k8s has.
  - aequitas 6y ago
    But now you have to manage those VM's, the OS lifecycle, different distributions, application deployment, configuration and dependencies. And we're back at containers. Or you have to go towards lambda. For K8s you need to manage the vm's/os as well but that part is mostly trivial as it's highly focussed on immutable/disposable infrastructure.
- evil-olive 6y ago
  We use Nomad from Hashicorp at $dayjob and I'm quite happy with it. https://www.nomadproject.io/ https://www.nomadproject.io/ https://www.hashicorp.com/resources/introduction-nomad https://www.hashicorp.com/resources/introduction-nomad https://blog.cloudflare.com/how-we-use-hashicorp-nomad/ https://blog.cloudflare.com/how-we-use-hashicorp-nomad/
- pjmlp 6y ago
  VMs as usual.
  - 6y ago

stepbeek 6y ago

Was anyone else surprised by the timeline of this post? I was under the impression that Gitlab had been fully on K8S for years now, given the hard push of their K8S offering 18 months ago.

jarv 6y ago
It has been possible to run GitLab the application on K8s for some time now for the majority of self-managed deployments. There were some limitations however, for running GitLab on K8s at the very large scale of GitLab.com that we have been working through in the last year. (disclaimer: blog post author)
- stepbeek 6y ago
  > August 2019 when we migrated the GitLab Container Registry to Kubernetes, the first service to move. Though this was a critical and high traffic service, it was a good choice for the first migration because it is a stateless application with only a few external dependencies I was under the impression that porting a large stateless service was something that wasn't too bad to do with K8S? What limitations prevented this from going ahead earlier?
  - jarv 6y ago
    Yeah, this is why we decided to migrate it first about a year ago, and we probably could have done it sooner than that. Other services were not yet ready to migrate until recently due to some blocking issues which was a factor in the decision. disclaimer: blog post author

Keycap 6y ago

I'm not sure if i get it but normally internal traffic across AZs should be included? It was also not clear to me after reading their issue page.

ChicagoDave 6y ago

My question. Why not convert the rails code/APIs to blob/FaaS and skip all the DevOps complexity? Are containers providing enough long-term cost-effectiveness to a complete cloud-native application architecture? Or is it just mapping VMs to something “like” VMs so your topology remains mostly the same?

auspex 6y ago
Serverless has its own complexity. Deploying a smaller, lightweight purpose built image, with no orchestration lock-in is the main benefit.
- ChicagoDave 6y ago
  Those are fair infrastructure reasons, but not necessarily cost-effective. There are always competing perspectives. What does the business want. What do the developers want. What does the infrastructure team want. When power is centralized in one area, decisions can be made that overlook those competing purposes. I see containers being adopted without openly discussing the options and long-term consequences.

deleted 6y ago

[deleted]

solatic 6y ago

> We found that with an application that increases its memory utilization over time, low requests (which reserves memory for each pod) and a generous hard limit on utilization was a recipe for node saturation and a high rate of evictions. To adjust for this we eventually decided to use higher requests and lower limit which took pressure off of the nodes and allowed pods to be recycled without putting too much pressure on the node. Kubernetes works at its most efficient when you scale by adding more Pods, rather than scaling within a Pod. Requests and limits should be equal to each other, and a HorizontalPodAutoscaler (or KEDA Scales object) should be used to add more Pods as utilization increases. Using Cluster Autoscaler then makes sure that your cluster has sufficient Nodes to schedule the Pods. Then you just need to set up alerting to tell you if the autoscaling reaches minimums (indicating that you should consider lowering the minimum, to improve efficiency) or maximums (indicating that you should consider increasing the maximums), and you can leave things well enough alone, unless the speed at which scaling occurs is insufficiently slow for your use case.