4 ms·

> Generally on k8s, you'd use a LoadBalancer Are you sure? All encountered k8s installations (including ones rolled from scratch by me) use Ingress controllers

> Generally on k8s, you'd use a LoadBalancer Are you sure? All encountered k8s installations (including ones rolled from scratch by me) use Ingress controllers as the option for getting traffic into the cluster. Community NGINX Ingress controller is the de facto standard. One need to use LoadBalancer service type because of its managed origin (metallb is a different beast and I wouldn't recommend it, if you need to load balance traffic to your ingresses on on-premises infra, it's better to do it outside of k8s). Anyway, you lose all flexibility and observability of Ingress solutions with LoadBalancer service types if you use them directly as traffic routers to your backend.

q3k 7y ago
They're complementary - LoadBalancer Services are L3, Ingresses are L4+. More often than not in-cluster Ingress providers (eg. nginx-ingress-controller) will in fact use LoadBalancers to actually route L3 traffic into the cluster in the first place. Without being able to create LoadBalancer services there's no easy way to get any traffic into your cluster other than using NodePorts, and these have tons of shortcomings.
- pepemon 7y ago
  NodePort is a bad practice for a such case, you are right. You get additional unnecessary routing on CNI level between the cluster nodes just to get traffic into the Ingress controller pod, for example. The solution here is to use hostNetwork mode with the pool of workers solely dedicated to scheduling Ingress controllers. Traffic directly hits NGINX/Envoy/HAProxy/Traefik/whatever and gets into the cluster without additional intermediates. You need a load balancing solution for this pool of Ingress controllers, that's right, but as I said before, this setup gives you the flexibility to cook load balancing as you desire. BTW, community NGINX Ingress controller is able to ingress L3 (TCP) traffic into the cluster.
  - q3k 7y ago
    > You need a load balancing solution for this pool of Ingress controllers, that's right, but as I said before, this setup gives you the flexibility to cook load balancing as you desire. Sure, but this means that you cannot use LoadBalancers, which is painful. It means every payload has to be configured both at k8s level and then externally. That somewhat defeats the use of k8s as a self-service platform internally in an organization (other dev/ops teams need to go through a centralized ops channel to get traffic ingressing into the cluster if for some reason they can't use an Ingress). > BTW, community NGINX Ingress controller is able to ingress L3 (TCP) traffic into the cluster. Yes, but it's configurable via a single ConfigMap (which limits self-service if you're running a multi-tenant orga-wide cluster, unless you bring your own automation), and still you only have one namespace of ports, ie. a single external address for all ports - unless you complicate your 'external' LB system even further. With all these caveats, I really don't understand why not just run metallb.
nicolast 7y ago
Not all services exposed to the outside are HTTP(S).
- pepemon 7y ago
  Yes, Ingress resources implicitly assume that you want to get HTTP(S) traffic into the cluster, but as for example, ingress-nginx is able to expose gRPC via additional annotations and generic L3 via related ConfigMap.
DelightOne 7y ago
You lead me down a rabbit hole, thank you! Do you know whether ingress controllers like nginx-ingress-controller honor the readiness status of pods from a service and so only sends traffic to ready pods from the concerned deployment?
- llarsson 7y ago
  They send traffic to the Service that exposes your Pods, so yes, it only gets routed to Endpoints that work according to readiness tests.
  - DelightOne 7y ago
    hmm makes sense, thank you!
  - rcconf 7y ago
    I remember seeing documentation on this a while ago, but I can't seem to find any documentation from nginx-ingress that talks about how nginx-ingress uses readinessprobe or livenessprobe. I was looking into this because if I have a pod with 3 containers, if 1 container wasn't running, nginx-ingress stopped serving traffic. I actually want to continue serving traffic if a certain container is still running, not all 3 running necessarily Do you know or have any documentation on how nginx-ingress actually does its readiness/liveness checks?
    - xnxn 7y ago
      ingress-nginx doesn't do readiness/liveness checks itself; the kubelet does[1]. If any of the containers in a Pod aren't ready, the endpoints controller[2] removes the Endpoint object corresponding to that Pod. ingress-nginx watches these Endpoints objects[3] to determine which Pods it should send traffic to. Edit: As to your use case, I think you should remove the readiness probe from that one container you don't care about. [1]: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/ https://kubernetes.io/docs/tasks/configure-pod-container/con... [2]: https://kubernetes.io/docs/concepts/overview/components/#kube-controller-manager https://kubernetes.io/docs/concepts/overview/components/#kub... [3]: https://kubernetes.github.io/ingress-nginx/user-guide/miscellaneous/#why-endpoints-and-not-services https://kubernetes.github.io/ingress-nginx/user-guide/miscel...

q3k 7y ago

They're complementary - LoadBalancer Services are L3, Ingresses are L4+. More often than not in-cluster Ingress providers (eg. nginx-ingress-controller) will in fact use LoadBalancers to actually route L3 traffic into the cluster in the first place. Without being able to create LoadBalancer services there's no easy way to get any traffic into your cluster other than using NodePorts, and these have tons of shortcomings.

pepemon 7y ago
NodePort is a bad practice for a such case, you are right. You get additional unnecessary routing on CNI level between the cluster nodes just to get traffic into the Ingress controller pod, for example. The solution here is to use hostNetwork mode with the pool of workers solely dedicated to scheduling Ingress controllers. Traffic directly hits NGINX/Envoy/HAProxy/Traefik/whatever and gets into the cluster without additional intermediates. You need a load balancing solution for this pool of Ingress controllers, that's right, but as I said before, this setup gives you the flexibility to cook load balancing as you desire. BTW, community NGINX Ingress controller is able to ingress L3 (TCP) traffic into the cluster.
- q3k 7y ago
  > You need a load balancing solution for this pool of Ingress controllers, that's right, but as I said before, this setup gives you the flexibility to cook load balancing as you desire. Sure, but this means that you cannot use LoadBalancers, which is painful. It means every payload has to be configured both at k8s level and then externally. That somewhat defeats the use of k8s as a self-service platform internally in an organization (other dev/ops teams need to go through a centralized ops channel to get traffic ingressing into the cluster if for some reason they can't use an Ingress). > BTW, community NGINX Ingress controller is able to ingress L3 (TCP) traffic into the cluster. Yes, but it's configurable via a single ConfigMap (which limits self-service if you're running a multi-tenant orga-wide cluster, unless you bring your own automation), and still you only have one namespace of ports, ie. a single external address for all ports - unless you complicate your 'external' LB system even further. With all these caveats, I really don't understand why not just run metallb.

nicolast 7y ago

Not all services exposed to the outside are HTTP(S).

pepemon 7y ago
Yes, Ingress resources implicitly assume that you want to get HTTP(S) traffic into the cluster, but as for example, ingress-nginx is able to expose gRPC via additional annotations and generic L3 via related ConfigMap.

DelightOne 7y ago

You lead me down a rabbit hole, thank you! Do you know whether ingress controllers like nginx-ingress-controller honor the readiness status of pods from a service and so only sends traffic to ready pods from the concerned deployment?

llarsson 7y ago
They send traffic to the Service that exposes your Pods, so yes, it only gets routed to Endpoints that work according to readiness tests.
- DelightOne 7y ago
  hmm makes sense, thank you!
- rcconf 7y ago
  I remember seeing documentation on this a while ago, but I can't seem to find any documentation from nginx-ingress that talks about how nginx-ingress uses readinessprobe or livenessprobe. I was looking into this because if I have a pod with 3 containers, if 1 container wasn't running, nginx-ingress stopped serving traffic. I actually want to continue serving traffic if a certain container is still running, not all 3 running necessarily Do you know or have any documentation on how nginx-ingress actually does its readiness/liveness checks?
  - xnxn 7y ago
    ingress-nginx doesn't do readiness/liveness checks itself; the kubelet does[1]. If any of the containers in a Pod aren't ready, the endpoints controller[2] removes the Endpoint object corresponding to that Pod. ingress-nginx watches these Endpoints objects[3] to determine which Pods it should send traffic to. Edit: As to your use case, I think you should remove the readiness probe from that one container you don't care about. [1]: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/ https://kubernetes.io/docs/tasks/configure-pod-container/con... [2]: https://kubernetes.io/docs/concepts/overview/components/#kube-controller-manager https://kubernetes.io/docs/concepts/overview/components/#kub... [3]: https://kubernetes.github.io/ingress-nginx/user-guide/miscellaneous/#why-endpoints-and-not-services https://kubernetes.github.io/ingress-nginx/user-guide/miscel...