3 ms·

Fortunately there's a bunch of ingress controllers you can use: Traefik, Voyager, HAProxy, and probably several others. And it's surprisingly trivial to write y

Fortunately there's a bunch of ingress controllers you can use: Traefik, Voyager, HAProxy, and probably several others. And it's surprisingly trivial to write your own. So ingresses is not currently a weak point in relation to vendor lock-in happens. And Kubernetes already supports plenty of non-Google tech; as an open source project, Kubernetes is refreshingly non-Google-focused (there are a bunch of players, notably Red Hat and Microsoft, ensuring this). As an aside, the current ingresses (including the Nginx one and Google's own GLB one) all have annoying deficiencies (subpar support for TLS and per-route timeout settings among the biggest). Ingress support is, and has been for a long time, Kubernetes' weakest point. For example, GLBs max out at 20 TLS certs (ridiculous if you're hosting many customers on a SaaS solution) and default to a timeout of 30 seconds (you can't control this using ingress annotations; you have to manually go in and edit the backends via API or UI) (which doesnt work for big streaming requests and WebSockets). These are also very trivial problems compared to the complex ones that are being solved by big new features in Kubernetes proper, so it's a bit surprising that ingress implementations are lagging to this extent.

mephitix 9y ago
>> As an aside, the current ingresses (including the Nginx one and Google's own GLB one) all have annoying deficiencies Agree, this is along the lines of what I meant. For example GCE ingress allows you to reference a global static IP while this is not possible with solely nginx ingress due to limitations with the TCP load balancer. There are separate ingress annotations for GKE/nginx/haproxy, etc. If I want to use the global-static-ip GCE annotation, this will make it a little harder to move to Azure.
- bonesss 9y ago
  Totally fair points, and vendor lock-in is something to be highly vigilant of. For some context, though: the ingress functionality is rather recent in Kubernetes and is still fleshing out in terms of demands in the market and across cloud providers. Another year or two will do a lot for equality and saner networking solutions for small-scale and bespoke K8s deployments. I got a little shock that I had to set up an ingress controller on each of my nodes (on our VMWare cluster), but going through the nuts and bolts of it I can see why it's hanging a bit behind the rest of the product. I think the devs are being smart letting the ecosystem mature a little instead of pushing out something half-baked and prematurely limiting.
- manigandham 9y ago
  You can always layer these. Have a simple global ingress pointing to a service or daemon set of nginx/haproxy.
thockingoog 9y ago
The problem with Ingress "lagging" is that it was designed to be a lowest-common-denominator API - it only absorbs logic that exists in the majority of realistic implementations. For better or worse, cloud LBs are vastly more limited in feature-set than Nginx or Envoy, so Ingress is too. This is a big topic for debate, and will be on the agenda at KubeCon in O(days).
- lobster_johnson 9y ago
  This is understandable, of course. In hindsight, I suspect the current concept of an abstract, one-size-fits-all ingress was, and is, going in the wrong direction. With CRDs, we could have each ingress controller provide its own, native ingress object ("nginx-ingress") that had the exact features it supported (with schema validation). The ingress controller would then create or delete cloud-specific CRDs ("google-loadbalancer") based on what flavour of cloud you're running under, which Kubernetes could pick up and use. Or something like that. But as you say, some of the friction exists because cloud LBs are limited in the first place. The arbitrary cert limit on GCP is particularly egregious. We run a SaaS solution with about 100 vendor domains, which means we've been forced to use the Nginx ingress controller and terminate TLS there, instead of at the GLB level where it arguably belongs. (We could run 10 GLBs, but that would require splitting our ingresses into 10 separate ingresses, with the duplication and potential for copy/paste errors that would ensue.) But thirdly, it's also true that several of the ingress implementations are just a bit sloppy. Traefik, Voyager and haproxy-ingress all have issues with using TLS certs (all of them have open issues about serving both HTTP and HTTPS at the same time, I believe). A lot of today's ergonomics could be solved by polishing up these projects.
  - thockingoog 9y ago
    First, Ingress had a purpose to serve, and it has served that purpose - it is relatively easy to handle low-complexity apps with generic Ingress. But low-complexity apps don't stay that way. What you're describing is very much the way my brain has gone. In my experience, most users end up using at least one non-portable annotation on Ingress. The logical conclusion then, is that people care about features MORE than portability in this facet of the API. This is not surprising to me, given how religious the debate tends to be...

mephitix 9y ago

>> As an aside, the current ingresses (including the Nginx one and Google's own GLB one) all have annoying deficiencies Agree, this is along the lines of what I meant. For example GCE ingress allows you to reference a global static IP while this is not possible with solely nginx ingress due to limitations with the TCP load balancer. There are separate ingress annotations for GKE/nginx/haproxy, etc. If I want to use the global-static-ip GCE annotation, this will make it a little harder to move to Azure.

bonesss 9y ago
Totally fair points, and vendor lock-in is something to be highly vigilant of. For some context, though: the ingress functionality is rather recent in Kubernetes and is still fleshing out in terms of demands in the market and across cloud providers. Another year or two will do a lot for equality and saner networking solutions for small-scale and bespoke K8s deployments. I got a little shock that I had to set up an ingress controller on each of my nodes (on our VMWare cluster), but going through the nuts and bolts of it I can see why it's hanging a bit behind the rest of the product. I think the devs are being smart letting the ecosystem mature a little instead of pushing out something half-baked and prematurely limiting.
manigandham 9y ago
You can always layer these. Have a simple global ingress pointing to a service or daemon set of nginx/haproxy.

thockingoog 9y ago

The problem with Ingress "lagging" is that it was designed to be a lowest-common-denominator API - it only absorbs logic that exists in the majority of realistic implementations. For better or worse, cloud LBs are vastly more limited in feature-set than Nginx or Envoy, so Ingress is too. This is a big topic for debate, and will be on the agenda at KubeCon in O(days).

lobster_johnson 9y ago
This is understandable, of course. In hindsight, I suspect the current concept of an abstract, one-size-fits-all ingress was, and is, going in the wrong direction. With CRDs, we could have each ingress controller provide its own, native ingress object ("nginx-ingress") that had the exact features it supported (with schema validation). The ingress controller would then create or delete cloud-specific CRDs ("google-loadbalancer") based on what flavour of cloud you're running under, which Kubernetes could pick up and use. Or something like that. But as you say, some of the friction exists because cloud LBs are limited in the first place. The arbitrary cert limit on GCP is particularly egregious. We run a SaaS solution with about 100 vendor domains, which means we've been forced to use the Nginx ingress controller and terminate TLS there, instead of at the GLB level where it arguably belongs. (We could run 10 GLBs, but that would require splitting our ingresses into 10 separate ingresses, with the duplication and potential for copy/paste errors that would ensue.) But thirdly, it's also true that several of the ingress implementations are just a bit sloppy. Traefik, Voyager and haproxy-ingress all have issues with using TLS certs (all of them have open issues about serving both HTTP and HTTPS at the same time, I believe). A lot of today's ergonomics could be solved by polishing up these projects.
- thockingoog 9y ago
  First, Ingress had a purpose to serve, and it has served that purpose - it is relatively easy to handle low-complexity apps with generic Ingress. But low-complexity apps don't stay that way. What you're describing is very much the way my brain has gone. In my experience, most users end up using at least one non-portable annotation on Ingress. The logical conclusion then, is that people care about features MORE than portability in this facet of the API. This is not surprising to me, given how religious the debate tends to be...