4 ms·

It's so worth it, though. You may not realize it, but your Healthchecks don't work unless you are using target-type IP. In the standard mode, AWS healthchecks e

It's so worth it, though. You may not realize it, but your Healthchecks don't work unless you are using target-type IP. In the standard mode, AWS healthchecks each kubernetes node, but not your pods themselves. It will keep sending traffic to pods that are unhealthy (such as in state: terminating) if it already has connections open through kube-proxy to them, and stop sending traffic to healthy pods if the connection to them was through a node it considers "unhealthy". I wrote at tool[1] to handle this at Houseparty before alb-ingress-controller supported this mode, and it gained us a whole nine as measured by the ALB. The same failure mode actually exists with instances, though they typically rotate much more slowly. Monitor your ALB-ingress-controller and treat it as critical infrastructure, because it is. [1]https://github.com/GauntletWizard/targetgroupcontroller https://github.com/GauntletWizard/targetgroupcontroller

t3rabytes 6y ago
> It's so worth it, though. It was until it caused a production outage, then it became very not worth it.
- GauntletWizard 6y ago
  ALB Ingress Controller didn't cause a hard outage, lack of monitoring on critical infrastructure caused a hard outage. The ALB Ingress Controller was already critical infrastructure beforehand, because the NodePort method still relies on it adding and removing instances from the targetgroup - It just takes longer for that to become an issue. The fact that it's hours instead of days hurts, but also makes it much easier to diagnose.
rsanders 6y ago
Can you expand on this: "AWS healthchecks each kubernetes node, but not your pods themselves". Are you talking about a keepalive connection to an unhealthy pod which is reused for multiple requests? So the failure modes are, if I understand you correctly, a) the ALB keeps sending requests through an established keep-alive HTTP connection which terminates in an unhealthy pod, but which it sees as healthy because the node is healthy and can route traffic to another, healthy pod, and b) the health of an established HTTP keepalive connection is perceived to be that of the node rather than the destination pod, so nodes which become unhealthy can cause the ALB to unnecessarily terminate a keepalive connection. We had to switch to using target-type=instance because of issues with pods not being deregistered. I'd prefer to use target-type IP but it seemed like preventing 500s on rollouts required a bit of testing and tuning with a very specific approach. e.g. introducing a longish delay on pod termination with a lifecycle hook and using the pod readiness gate support recently added to alb-ingress-controller.
- GauntletWizard 6y ago
  You've got it exactly right. Your problem of pods not being deregistered is a real problem, but also with a quick fix: The default "Deregistration delay" for ALBs is 300 seconds but for kubernetes pods the TerminationGracePeriod defaults to 60 seconds. This means that your load balancer keeps trying that pod for 4 whole minutes after it's been hard-shutdown. Here's the annotation that I used to fix that: alb.ingress.kubernetes.io/target-group-attributes: deregistration_delay.timeout_seconds=30,slow_start.duration_seconds=30

t3rabytes 6y ago

> It's so worth it, though. It was until it caused a production outage, then it became very not worth it.

GauntletWizard 6y ago
ALB Ingress Controller didn't cause a hard outage, lack of monitoring on critical infrastructure caused a hard outage. The ALB Ingress Controller was already critical infrastructure beforehand, because the NodePort method still relies on it adding and removing instances from the targetgroup - It just takes longer for that to become an issue. The fact that it's hours instead of days hurts, but also makes it much easier to diagnose.

rsanders 6y ago

Can you expand on this: "AWS healthchecks each kubernetes node, but not your pods themselves". Are you talking about a keepalive connection to an unhealthy pod which is reused for multiple requests? So the failure modes are, if I understand you correctly, a) the ALB keeps sending requests through an established keep-alive HTTP connection which terminates in an unhealthy pod, but which it sees as healthy because the node is healthy and can route traffic to another, healthy pod, and b) the health of an established HTTP keepalive connection is perceived to be that of the node rather than the destination pod, so nodes which become unhealthy can cause the ALB to unnecessarily terminate a keepalive connection. We had to switch to using target-type=instance because of issues with pods not being deregistered. I'd prefer to use target-type IP but it seemed like preventing 500s on rollouts required a bit of testing and tuning with a very specific approach. e.g. introducing a longish delay on pod termination with a lifecycle hook and using the pod readiness gate support recently added to alb-ingress-controller.

GauntletWizard 6y ago
You've got it exactly right. Your problem of pods not being deregistered is a real problem, but also with a quick fix: The default "Deregistration delay" for ALBs is 300 seconds but for kubernetes pods the TerminationGracePeriod defaults to 60 seconds. This means that your load balancer keeps trying that pod for 4 whole minutes after it's been hard-shutdown. Here's the annotation that I used to fix that: alb.ingress.kubernetes.io/target-group-attributes: deregistration_delay.timeout_seconds=30,slow_start.duration_seconds=30