3 ms·

Check out https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#service-upstream https://kubernetes.github.io/ingress-nginx/us

Check out https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#service-upstream https://kubernetes.github.io/ingress-nginx/user-guide/nginx-... if you're running nginx. Consider setting it to true instead of the default (false) --- By default the NGINX ingress controller uses a list of all endpoints (Pod IP/port) in the NGINX upstream configuration. The nginx.ingress.kubernetes.io/service-upstream annotation disables that behavior and instead uses a single upstream in NGINX, the service's Cluster IP and port. This can be desirable for things like zero-downtime deployments .

nhoughto 5y ago
ah good tip, I don't care about session affinity or custom balancing algos so that works. I'd imagine running in GKE or AWS you would also avoid the DNAT / conntrack overhead as pods by default use a routable VPC IP instead of a magic CNI IP. Would have to test that though. Quote from related issue: The NGINX ingress controller does not uses Services to route traffic to the pods. Instead it uses the Endpoints API in order to bypass kube-proxy to allow NGINX features like session affinity and custom load balancing algorithms. It also removes some overhead, such as conntrack entries for iptables DNAT.
- sascha_sl 5y ago
  How does this remove conntrack overhead? It should add more, because the node with the ingress controller now has to hold an extra <Cluster IP Cluster Port, Pod IP, Pod Port> mapping, regardless of what CNI is used (flannel in gateway mode also eliminates this overhead, by the way - you just need to make sure there is nothing like the default EC2 source destination check in place).
nopurpose 5y ago
You just moved problem from ingress controller to kube-proxy (or it's replacement) on the same node. Race condition is still present as before.
sascha_sl 5y ago
This is not a good idea, you're doubling the load on conntrack. Maybe ingress-nginx should fix their config generation instead. Had a cluster with at least one change per second to the ingresses, and nginx would regularly just die (and orphan existing requests after 20 seconds).

nhoughto 5y ago

ah good tip, I don't care about session affinity or custom balancing algos so that works. I'd imagine running in GKE or AWS you would also avoid the DNAT / conntrack overhead as pods by default use a routable VPC IP instead of a magic CNI IP. Would have to test that though. Quote from related issue: The NGINX ingress controller does not uses Services to route traffic to the pods. Instead it uses the Endpoints API in order to bypass kube-proxy to allow NGINX features like session affinity and custom load balancing algorithms. It also removes some overhead, such as conntrack entries for iptables DNAT.

sascha_sl 5y ago
How does this remove conntrack overhead? It should add more, because the node with the ingress controller now has to hold an extra <Cluster IP Cluster Port, Pod IP, Pod Port> mapping, regardless of what CNI is used (flannel in gateway mode also eliminates this overhead, by the way - you just need to make sure there is nothing like the default EC2 source destination check in place).

nopurpose 5y ago

You just moved problem from ingress controller to kube-proxy (or it's replacement) on the same node. Race condition is still present as before.

sascha_sl 5y ago

This is not a good idea, you're doubling the load on conntrack. Maybe ingress-nginx should fix their config generation instead. Had a cluster with at least one change per second to the ingresses, and nginx would regularly just die (and orphan existing requests after 20 seconds).