4 ms·

> Increased billing from cross-AZ traffic Yep, we (Basecamp) have been bit by this with EKS too. By default the ALB Ingress Controller will put all cluster nod

> Increased billing from cross-AZ traffic Yep, we (Basecamp) have been bit by this with EKS too. By default the ALB Ingress Controller will put all cluster nodes into the target group and a request can hit kube-proxy in any AZ before being directed to the right place, causing a lot of inter-AZ traffic churn. It's mildly annoying. (with alb-ingress-controller you can change target-type to IP to have traffic go direct to pods that will actually back a given service instead of via kube-proxy on a random node, but the big fault there is that if alb-ingress-controller gets hung up on something and you deploy, you can send traffic to nowhere because the target group still has old IPs for pods that don't exist).

GauntletWizard 6y ago
It's so worth it, though. You may not realize it, but your Healthchecks don't work unless you are using target-type IP. In the standard mode, AWS healthchecks each kubernetes node, but not your pods themselves. It will keep sending traffic to pods that are unhealthy (such as in state: terminating) if it already has connections open through kube-proxy to them, and stop sending traffic to healthy pods if the connection to them was through a node it considers "unhealthy". I wrote at tool[1] to handle this at Houseparty before alb-ingress-controller supported this mode, and it gained us a whole nine as measured by the ALB. The same failure mode actually exists with instances, though they typically rotate much more slowly. Monitor your ALB-ingress-controller and treat it as critical infrastructure, because it is. [1]https://github.com/GauntletWizard/targetgroupcontroller https://github.com/GauntletWizard/targetgroupcontroller
- t3rabytes 6y ago
  > It's so worth it, though. It was until it caused a production outage, then it became very not worth it.
  - GauntletWizard 6y ago
    ALB Ingress Controller didn't cause a hard outage, lack of monitoring on critical infrastructure caused a hard outage. The ALB Ingress Controller was already critical infrastructure beforehand, because the NodePort method still relies on it adding and removing instances from the targetgroup - It just takes longer for that to become an issue. The fact that it's hours instead of days hurts, but also makes it much easier to diagnose.
- rsanders 6y ago
  Can you expand on this: "AWS healthchecks each kubernetes node, but not your pods themselves". Are you talking about a keepalive connection to an unhealthy pod which is reused for multiple requests? So the failure modes are, if I understand you correctly, a) the ALB keeps sending requests through an established keep-alive HTTP connection which terminates in an unhealthy pod, but which it sees as healthy because the node is healthy and can route traffic to another, healthy pod, and b) the health of an established HTTP keepalive connection is perceived to be that of the node rather than the destination pod, so nodes which become unhealthy can cause the ALB to unnecessarily terminate a keepalive connection. We had to switch to using target-type=instance because of issues with pods not being deregistered. I'd prefer to use target-type IP but it seemed like preventing 500s on rollouts required a bit of testing and tuning with a very specific approach. e.g. introducing a longish delay on pod termination with a lifecycle hook and using the pod readiness gate support recently added to alb-ingress-controller.
  - GauntletWizard 6y ago
    You've got it exactly right. Your problem of pods not being deregistered is a real problem, but also with a quick fix: The default "Deregistration delay" for ALBs is 300 seconds but for kubernetes pods the TerminationGracePeriod defaults to 60 seconds. This means that your load balancer keeps trying that pod for 4 whole minutes after it's been hard-shutdown. Here's the annotation that I used to fix that: alb.ingress.kubernetes.io/target-group-attributes: deregistration_delay.timeout_seconds=30,slow_start.duration_seconds=30
ckdarby 6y ago
What you're looking for is called Service Topology it is in Alpha. https://kubernetes.io/docs/concepts/services-networking/service-topology/ https://kubernetes.io/docs/concepts/services-networking/serv...
- DuskStar 6y ago
  I genuinely cannot wait for that to make it to EKS.
  - ckdarby 6y ago
    Yep, we were getting an insane amount of AZ chatter with Apache Pulsar because of all the components: Request(us-west-1a) -> Pulsar Proxy(us-west-1b) -> Broker(us-west-1c) -> Bookie (us-west-1a) Though for anyone using Pulsar on EKS who need multi-az for disaster recovery for the time being could deploy in a single AZ and then use the geo-replication feature in Pulsar to copy over to another AZ.
qubit23 6y ago
Regarding the alb-ingress-controller blackholing traffic with an IP target type ingress, check out support for pod readiness gates: https://kubernetes-sigs.github.io/aws-alb-ingress-controller/guide/ingress/pod-conditions/ https://kubernetes-sigs.github.io/aws-alb-ingress-controller.... Assuming you're using a rolling deployment, old pods won't be stopped until the new ones are added to the ALB target group and are in a healthy state. This has eliminated the small number of ELB 500s we'd see on deploys.

GauntletWizard 6y ago

It's so worth it, though. You may not realize it, but your Healthchecks don't work unless you are using target-type IP. In the standard mode, AWS healthchecks each kubernetes node, but not your pods themselves. It will keep sending traffic to pods that are unhealthy (such as in state: terminating) if it already has connections open through kube-proxy to them, and stop sending traffic to healthy pods if the connection to them was through a node it considers "unhealthy". I wrote at tool[1] to handle this at Houseparty before alb-ingress-controller supported this mode, and it gained us a whole nine as measured by the ALB. The same failure mode actually exists with instances, though they typically rotate much more slowly. Monitor your ALB-ingress-controller and treat it as critical infrastructure, because it is. [1]https://github.com/GauntletWizard/targetgroupcontroller https://github.com/GauntletWizard/targetgroupcontroller

t3rabytes 6y ago
> It's so worth it, though. It was until it caused a production outage, then it became very not worth it.
- GauntletWizard 6y ago
  ALB Ingress Controller didn't cause a hard outage, lack of monitoring on critical infrastructure caused a hard outage. The ALB Ingress Controller was already critical infrastructure beforehand, because the NodePort method still relies on it adding and removing instances from the targetgroup - It just takes longer for that to become an issue. The fact that it's hours instead of days hurts, but also makes it much easier to diagnose.
rsanders 6y ago
Can you expand on this: "AWS healthchecks each kubernetes node, but not your pods themselves". Are you talking about a keepalive connection to an unhealthy pod which is reused for multiple requests? So the failure modes are, if I understand you correctly, a) the ALB keeps sending requests through an established keep-alive HTTP connection which terminates in an unhealthy pod, but which it sees as healthy because the node is healthy and can route traffic to another, healthy pod, and b) the health of an established HTTP keepalive connection is perceived to be that of the node rather than the destination pod, so nodes which become unhealthy can cause the ALB to unnecessarily terminate a keepalive connection. We had to switch to using target-type=instance because of issues with pods not being deregistered. I'd prefer to use target-type IP but it seemed like preventing 500s on rollouts required a bit of testing and tuning with a very specific approach. e.g. introducing a longish delay on pod termination with a lifecycle hook and using the pod readiness gate support recently added to alb-ingress-controller.
- GauntletWizard 6y ago
  You've got it exactly right. Your problem of pods not being deregistered is a real problem, but also with a quick fix: The default "Deregistration delay" for ALBs is 300 seconds but for kubernetes pods the TerminationGracePeriod defaults to 60 seconds. This means that your load balancer keeps trying that pod for 4 whole minutes after it's been hard-shutdown. Here's the annotation that I used to fix that: alb.ingress.kubernetes.io/target-group-attributes: deregistration_delay.timeout_seconds=30,slow_start.duration_seconds=30

ckdarby 6y ago

What you're looking for is called Service Topology it is in Alpha. https://kubernetes.io/docs/concepts/services-networking/service-topology/ https://kubernetes.io/docs/concepts/services-networking/serv...

DuskStar 6y ago
I genuinely cannot wait for that to make it to EKS.
- ckdarby 6y ago
  Yep, we were getting an insane amount of AZ chatter with Apache Pulsar because of all the components: Request(us-west-1a) -> Pulsar Proxy(us-west-1b) -> Broker(us-west-1c) -> Bookie (us-west-1a) Though for anyone using Pulsar on EKS who need multi-az for disaster recovery for the time being could deploy in a single AZ and then use the geo-replication feature in Pulsar to copy over to another AZ.

qubit23 6y ago

Regarding the alb-ingress-controller blackholing traffic with an IP target type ingress, check out support for pod readiness gates: https://kubernetes-sigs.github.io/aws-alb-ingress-controller/guide/ingress/pod-conditions/ https://kubernetes-sigs.github.io/aws-alb-ingress-controller.... Assuming you're using a rolling deployment, old pods won't be stopped until the new ones are added to the ALB target group and are in a healthy state. This has eliminated the small number of ELB 500s we'd see on deploys.