3 ms·

Likely a dumb question (apologies in advance): can that in any way replace the need to spinning a native LoadBallancer on AWS?

iooi 8y ago
I don't think this a dumb question, ELBs on AWS are stupid expensive for what they do.
- skywhopper 8y ago
  I agree it's a reasonable question. And ELBs can be expensive at small scales. They can handle a huge amount of traffic with an exceptional level of reliability, but at small scales there are probably much better options available.
sisk 8y ago
No need for apology: there’s a lot of _stuff_ when it comes to kubernetes networking. Using the IPVS mode instead of iptables won’t change anything with regards to external load balancers. The proxy mode of kube-proxy sets the mechanism by which cluster virtual IPs (i.e., the “Cluster IP” assigned to a service) are mapped to zero or more “real” pod IPs (typically assigned by an overlay network like Calico or Flannel). So, if I have a service that has been assigned a cluster IP of 10.10.10.10, kube-proxy uses iptables, IPVS, or a userspace proxy to intercept packets destined for that service and redirect them to one of the pods that fulfills that service’s label selector (maybe, for example, one of 10.20.0.1 or 10.20.0.2). Similarly with a LoadBalancer-type service, in addition to the above, your service will be assigned a nodePort for each port you've defined. kube-proxy will then add the proper rules—using the proxy-mode you've selected whether it's IPVS or not—to each node. This means that any traffic that hits any node on the selected nodePort will be redirected to one of the pods that fulfills the service label-selector. That's why if you look at the kubernetes-provisioned _real_ load balancer, you'll see it simply sends traffic to the selected nodePort on any of your nodes. IPVS, iptables, or the user space proxy (depending on the proxy-mode you've configured) is then responsible for sending the traffic to one of the right pods and ports from there. In other words, kube-proxy, by way of IPVS or iptables rules, makes sure that things destined for "services" (a not-real, totally virtual thing) end up at one of the right pods. It's a wholly inter-cluster routing concern. It's a lot but hopefully that makes sense.

iooi 8y ago

I don't think this a dumb question, ELBs on AWS are stupid expensive for what they do.

skywhopper 8y ago
I agree it's a reasonable question. And ELBs can be expensive at small scales. They can handle a huge amount of traffic with an exceptional level of reliability, but at small scales there are probably much better options available.

sisk 8y ago

No need for apology: there’s a lot of _stuff_ when it comes to kubernetes networking. Using the IPVS mode instead of iptables won’t change anything with regards to external load balancers. The proxy mode of kube-proxy sets the mechanism by which cluster virtual IPs (i.e., the “Cluster IP” assigned to a service) are mapped to zero or more “real” pod IPs (typically assigned by an overlay network like Calico or Flannel). So, if I have a service that has been assigned a cluster IP of 10.10.10.10, kube-proxy uses iptables, IPVS, or a userspace proxy to intercept packets destined for that service and redirect them to one of the pods that fulfills that service’s label selector (maybe, for example, one of 10.20.0.1 or 10.20.0.2). Similarly with a LoadBalancer-type service, in addition to the above, your service will be assigned a nodePort for each port you've defined. kube-proxy will then add the proper rules—using the proxy-mode you've selected whether it's IPVS or not—to each node. This means that any traffic that hits any node on the selected nodePort will be redirected to one of the pods that fulfills the service label-selector. That's why if you look at the kubernetes-provisioned _real_ load balancer, you'll see it simply sends traffic to the selected nodePort on any of your nodes. IPVS, iptables, or the user space proxy (depending on the proxy-mode you've configured) is then responsible for sending the traffic to one of the right pods and ports from there. In other words, kube-proxy, by way of IPVS or iptables rules, makes sure that things destined for "services" (a not-real, totally virtual thing) end up at one of the right pods. It's a wholly inter-cluster routing concern. It's a lot but hopefully that makes sense.