4 ms·

Ingress going GA seems big. I just read through the release notes but didnt see as much about Ingress Controllers. I always found it strange that Ingress Contro

Ingress going GA seems big. I just read through the release notes but didnt see as much about Ingress Controllers. I always found it strange that Ingress Controllers are external components provided by the CSPs or 3rd parties (if you are going multi-cloud support as I am). Looking at NGINX, Kong, Traefik, and the loose change of community-contributed extensions to NGINX, Kong, Traefik, I always wonder why it isn't just part of the core. Anyone know if that is the eventual plan? If so, what happens to the small companies building solutions, do their products just become 2nd class citizens?

starefossen 6y ago
Ingress going GA refers to the Ingress API resource in the Kubernetes API becoming stable. There are no discussion (or Kubernetes Enhancement Proposals (KEPs) regarding providing a default ingress controller. There are two areas that Kubernetes has taken a hands of approach and it is ingress traffic (and external load balancing) and storage. Those are provided by external vendors through APIs.
candiddevmike 6y ago
This is basically why ingress isn't seen as a sexy way to integrate with Kube. Companies like F5 etc know they don't want to be a "swappable" ingress provider, so they create stickier solutions like custom operators. Additionally, most companies graduate from ingress to service mesh, or skip ingress entirely, so service mesh is the key business to capture and where most companies are targeting their efforts.
- cmckn 6y ago
  I've used Ingress for some personal projects, and have also felt the pull towards service mesh. I've tried to set up Istio and Linkerd a couple times each, and bailed every time. These products need to be deeply integrated into the cluster (e.g. Istio installs its own ingress controller), so I've had to tear down and re-build my cluster in order to properly set up the mesh. I wish they were more approachable. Or I wish I were smarter!
  - deleted 6y ago
    [deleted]
csunbird 6y ago
On an unrelated note, did anyone here try Skipper as their ingress controller? We are torn between Skipper and HAProxy. Our use case is transferring large size files and NGINX is rumored to have bad performance in that regard.
- rdli 6y ago
  Have you tried Ambassador? Built on Envoy Proxy. We have submitted as well to the CNCF as an official project too. (Disclosure: I work on Ambassador).
  - csunbird 6y ago
    We will certainly give them a try!
jrockway 6y ago
I think the problem is that HTTP reverse proxies do so much, it's hard to specify what Kubernetes should do and what you should get an external solution to do. (Personally I am appalled at how org-wide SSO works right now, and almost started my own company to sell an open-source solution to the problem. But... I don't know how to start a company and then there was a pandemic. Someone else will sell you a not-as-good solution for mega-bucks.) In my opinion, people are always going to want their special thing; compatibility with their legacy nginx.conf, support for HTTP/3, whatever. Kubernetes can never satisfy people with those requirements. For that reason, I think people need to agree on a service discovery protocol, a route discovery protocol, and then hook those into their frontend proxy of choice. For example, I never really liked Ingress (lots of unspecified landmines, like what order rules are evaluated in when matching a request to a route), and so just run Envoy inside Kubernetes. Envoy is then fed a static route table and a valid TLS certificate, and handles all traffic in an explicit and easy to debug way. I wrote something to glue Envoy's service discovery to Kubernetes so my static config file contains less boilerplate (https://github.com/jrockway/ekglue https://github.com/jrockway/ekglue), and it's served me well. (I need to write something to add route table entries to DNS, though. I currently do that by editing my DNS records in a web interface, which is tedious and uninteresting.) With that in mind, what people should really be focusing on is standardizing xDS (Envoy's config language basically), and hooking it into their container orchestration framework and web server of choice. Then you can run Nginx on Openshift or Apache on ECS and everyone is using the same tools to do the very boring gruntwork of figuring out what your backends are called, and what syntax you use to say "send all requests that match example.com/foobar to a backend my container orchestration framework calls foo-bar-v2". I am sure it will sort itself out eventually -- xDS has a lot of momentum in things that aren't Envoy (notably gRPC), so this could all be a solved problem outside of Kubernetes soon enough.
- dastbe 6y ago
  (I work at AWS, this is a personal opinion :) ) IMO, I'd like to see a broad standardization on xDS/UDP to all services where direct dataplane integration would be beneficial. Why can't I direct Envoy to autoscaling so they can tell me what endpoints are available and what certs to use? or to RDS to direct me to nearby replicas and assist with authentication/authorization? I think the typical solution would be to embed more magic into the infrastructure, but that breaks down as you split things across the cloud and on-prem. While some customers only want magic and will change whatever they have to to get it, other customers don't/won't buy into that. Having a general-purpose method for a control plane to configure an arbitrary dataplane would be a big benefit for them.
harpratap 6y ago
The path going forward is to split L7 layer proxies and L3/L4 layer - https://kubernetes-sigs.github.io/service-apis/spec/ https://kubernetes-sigs.github.io/service-apis/spec/ "Ingress" is too broad of a term and it does too much and too little depending on your use case

starefossen 6y ago

Ingress going GA refers to the Ingress API resource in the Kubernetes API becoming stable. There are no discussion (or Kubernetes Enhancement Proposals (KEPs) regarding providing a default ingress controller. There are two areas that Kubernetes has taken a hands of approach and it is ingress traffic (and external load balancing) and storage. Those are provided by external vendors through APIs.

candiddevmike 6y ago

This is basically why ingress isn't seen as a sexy way to integrate with Kube. Companies like F5 etc know they don't want to be a "swappable" ingress provider, so they create stickier solutions like custom operators. Additionally, most companies graduate from ingress to service mesh, or skip ingress entirely, so service mesh is the key business to capture and where most companies are targeting their efforts.

cmckn 6y ago
I've used Ingress for some personal projects, and have also felt the pull towards service mesh. I've tried to set up Istio and Linkerd a couple times each, and bailed every time. These products need to be deeply integrated into the cluster (e.g. Istio installs its own ingress controller), so I've had to tear down and re-build my cluster in order to properly set up the mesh. I wish they were more approachable. Or I wish I were smarter!
- deleted 6y ago
  [deleted]

csunbird 6y ago

On an unrelated note, did anyone here try Skipper as their ingress controller? We are torn between Skipper and HAProxy. Our use case is transferring large size files and NGINX is rumored to have bad performance in that regard.

rdli 6y ago
Have you tried Ambassador? Built on Envoy Proxy. We have submitted as well to the CNCF as an official project too. (Disclosure: I work on Ambassador).
- csunbird 6y ago
  We will certainly give them a try!

jrockway 6y ago

I think the problem is that HTTP reverse proxies do so much, it's hard to specify what Kubernetes should do and what you should get an external solution to do. (Personally I am appalled at how org-wide SSO works right now, and almost started my own company to sell an open-source solution to the problem. But... I don't know how to start a company and then there was a pandemic. Someone else will sell you a not-as-good solution for mega-bucks.) In my opinion, people are always going to want their special thing; compatibility with their legacy nginx.conf, support for HTTP/3, whatever. Kubernetes can never satisfy people with those requirements. For that reason, I think people need to agree on a service discovery protocol, a route discovery protocol, and then hook those into their frontend proxy of choice. For example, I never really liked Ingress (lots of unspecified landmines, like what order rules are evaluated in when matching a request to a route), and so just run Envoy inside Kubernetes. Envoy is then fed a static route table and a valid TLS certificate, and handles all traffic in an explicit and easy to debug way. I wrote something to glue Envoy's service discovery to Kubernetes so my static config file contains less boilerplate (https://github.com/jrockway/ekglue https://github.com/jrockway/ekglue), and it's served me well. (I need to write something to add route table entries to DNS, though. I currently do that by editing my DNS records in a web interface, which is tedious and uninteresting.) With that in mind, what people should really be focusing on is standardizing xDS (Envoy's config language basically), and hooking it into their container orchestration framework and web server of choice. Then you can run Nginx on Openshift or Apache on ECS and everyone is using the same tools to do the very boring gruntwork of figuring out what your backends are called, and what syntax you use to say "send all requests that match example.com/foobar to a backend my container orchestration framework calls foo-bar-v2". I am sure it will sort itself out eventually -- xDS has a lot of momentum in things that aren't Envoy (notably gRPC), so this could all be a solved problem outside of Kubernetes soon enough.

dastbe 6y ago
(I work at AWS, this is a personal opinion :) ) IMO, I'd like to see a broad standardization on xDS/UDP to all services where direct dataplane integration would be beneficial. Why can't I direct Envoy to autoscaling so they can tell me what endpoints are available and what certs to use? or to RDS to direct me to nearby replicas and assist with authentication/authorization? I think the typical solution would be to embed more magic into the infrastructure, but that breaks down as you split things across the cloud and on-prem. While some customers only want magic and will change whatever they have to to get it, other customers don't/won't buy into that. Having a general-purpose method for a control plane to configure an arbitrary dataplane would be a big benefit for them.

harpratap 6y ago

The path going forward is to split L7 layer proxies and L3/L4 layer - https://kubernetes-sigs.github.io/service-apis/spec/ https://kubernetes-sigs.github.io/service-apis/spec/ "Ingress" is too broad of a term and it does too much and too little depending on your use case