3 ms·

Could you provide some more detail about your statement that "nginx-ingress will lose source-ip info"? I'm currently working towards a k8s-on-metal system usin

Could you provide some more detail about your statement that "nginx-ingress will lose source-ip info"? I'm currently working towards a k8s-on-metal system using nginx for ingress and was led to believe that I could use either X-Forwarded-For or proxy_protocol to preserve the originating IP address. Is the traffic bounced through an SNAT rule on it's way to the ingress controller or similar?

sandGorgon 10y ago
you should google for proxy-protocol. so here's the thing - nginx cannot INJECT proxy protocol. I'm not an expert in nginx.. but atleast the versions used in k8s couldnt. It can read/pass-through proxy protocol headers. So if you have haproxy in front of nginx.. it will work fine (or ELB/GLB). Which is why there's another beta ingress - the "keepalived-vip" repo in k8s ingress. However, not a lot of people seem to be using haproxy primarily because haproxy cannot have a zero-downtime deploy. A couple of weeks back on k8s slack, I was discussing using Github multibinder to solve that one problem and move ahead with haproxy ingress. protip: please join the #sig-onpremise channel on slack. There's not a lot of momentum for bare metal deploys otherwise. https://githubengineering.com/glb-part-2-haproxy-zero-downtime-zero-delay-reloads-with-multibinder/ https://githubengineering.com/glb-part-2-haproxy-zero-downti...
- predakanga 10y ago
  I'm familiar with proxy-protocol, and while I haven't injected it with nginx, it seems that it does support it: https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_protocol https://nginx.org/en/docs/stream/ngx_stream_proxy_module.htm... For reference, the feature was added in nginx 1.9.2, and the ingress-nginx controller appears to use nginx 1.11.9: https://github.com/kubernetes/ingress/blob/0.9.1/images/nginx-slim/build.sh https://github.com/kubernetes/ingress/blob/0.9.1/images/ngin... Regardless, would I be right in assuming that your issues are mostly related to the ingress of TCP streams? My usecase is almost exclusively HTTP, so I might be missing the worst of it. Thanks for the pointer to the #sig-onpremise channel, I'll be sure to check it out. Been meaning to look into the SIGs since reading https://coreos.com/blog/self-hosted-kubernetes.html https://coreos.com/blog/self-hosted-kubernetes.html
  - sandGorgon 10y ago
    please double check - nginx's support for proxy protocol is to be able to read it. it cannot support injecting it. if nginx is the first thing your traffic hits, it will not inject proxy protocol headers. Again, I will not stake my reputation on this... but I remember going deep into this about 2 months back. if you are using nginx ingress, then you need to move all of your existing config into their own config file format. Since setting up full TLS internally in kubernetes is buggy, I didnt want to terminate my traffic on the ingress. Also I really did not want to move my (fairly complex) nginx configuration to the ingress. Which is why I'm a big believer in haproxy ingress - it was designed to interface with nginx. However, you are already one of the "enlightened" ones. You actually do know what an ingress is. A significant amount of k8s slack traffic is "how do I setup a loadbalancer/how to make the cluster available to the outside world"
    - justinsb 10y ago
      nginx can inject proxy_protocol - I know because I've enabled it by mistake! The kubernetes nginx ingress controller doesn't support it though. I think that is because most people use HTTP headers (X-Forwarded-For) instead. But if you do want it, you should open an issue explaining the use case so we can check our assumptions!
      - sandGorgon 10y ago
        @justinsb - ur the expert here and i will not dispute this :D I still believe that there is NO better tool to use kubernetes than kops. I now use the 100.64 subnet that you guys figured out ("carrier grade NAT" seriously?) in docker swarm. but genuine question - can u show me how ? Because I honestly went crawling into the nginx codebase and could only find client side handling of proxy protocol. I found NO place where it showed how it could be injected and chain reverse proxies together (that is what it was invented for in the first place). P.S. just a quick google reveals this link which sort of confirms my suspicion - http://blog.haproxy.com/haproxy/proxy-protocol/ http://blog.haproxy.com/haproxy/proxy-protocol/
        10y ago
- zimbatm 10y ago
  What's wrong with using the X-Forwarded-For/X-Forwarded-Proto headers as usual? Or do you mean for non-HTTP protocol?
  - monktastic1 10y ago
    Yeah, looks like he's talking about L4 proxying. Check out http://www.haproxy.org/download/1.8/doc/proxy-protocol.txt http://www.haproxy.org/download/1.8/doc/proxy-protocol.txt.

sandGorgon 10y ago

you should google for proxy-protocol. so here's the thing - nginx cannot INJECT proxy protocol. I'm not an expert in nginx.. but atleast the versions used in k8s couldnt. It can read/pass-through proxy protocol headers. So if you have haproxy in front of nginx.. it will work fine (or ELB/GLB). Which is why there's another beta ingress - the "keepalived-vip" repo in k8s ingress. However, not a lot of people seem to be using haproxy primarily because haproxy cannot have a zero-downtime deploy. A couple of weeks back on k8s slack, I was discussing using Github multibinder to solve that one problem and move ahead with haproxy ingress. protip: please join the #sig-onpremise channel on slack. There's not a lot of momentum for bare metal deploys otherwise. https://githubengineering.com/glb-part-2-haproxy-zero-downtime-zero-delay-reloads-with-multibinder/ https://githubengineering.com/glb-part-2-haproxy-zero-downti...

predakanga 10y ago
I'm familiar with proxy-protocol, and while I haven't injected it with nginx, it seems that it does support it: https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_protocol https://nginx.org/en/docs/stream/ngx_stream_proxy_module.htm... For reference, the feature was added in nginx 1.9.2, and the ingress-nginx controller appears to use nginx 1.11.9: https://github.com/kubernetes/ingress/blob/0.9.1/images/nginx-slim/build.sh https://github.com/kubernetes/ingress/blob/0.9.1/images/ngin... Regardless, would I be right in assuming that your issues are mostly related to the ingress of TCP streams? My usecase is almost exclusively HTTP, so I might be missing the worst of it. Thanks for the pointer to the #sig-onpremise channel, I'll be sure to check it out. Been meaning to look into the SIGs since reading https://coreos.com/blog/self-hosted-kubernetes.html https://coreos.com/blog/self-hosted-kubernetes.html
- sandGorgon 10y ago
  please double check - nginx's support for proxy protocol is to be able to read it. it cannot support injecting it. if nginx is the first thing your traffic hits, it will not inject proxy protocol headers. Again, I will not stake my reputation on this... but I remember going deep into this about 2 months back. if you are using nginx ingress, then you need to move all of your existing config into their own config file format. Since setting up full TLS internally in kubernetes is buggy, I didnt want to terminate my traffic on the ingress. Also I really did not want to move my (fairly complex) nginx configuration to the ingress. Which is why I'm a big believer in haproxy ingress - it was designed to interface with nginx. However, you are already one of the "enlightened" ones. You actually do know what an ingress is. A significant amount of k8s slack traffic is "how do I setup a loadbalancer/how to make the cluster available to the outside world"
  - justinsb 10y ago
    nginx can inject proxy_protocol - I know because I've enabled it by mistake! The kubernetes nginx ingress controller doesn't support it though. I think that is because most people use HTTP headers (X-Forwarded-For) instead. But if you do want it, you should open an issue explaining the use case so we can check our assumptions!
    - sandGorgon 10y ago
      @justinsb - ur the expert here and i will not dispute this :D I still believe that there is NO better tool to use kubernetes than kops. I now use the 100.64 subnet that you guys figured out ("carrier grade NAT" seriously?) in docker swarm. but genuine question - can u show me how ? Because I honestly went crawling into the nginx codebase and could only find client side handling of proxy protocol. I found NO place where it showed how it could be injected and chain reverse proxies together (that is what it was invented for in the first place). P.S. just a quick google reveals this link which sort of confirms my suspicion - http://blog.haproxy.com/haproxy/proxy-protocol/ http://blog.haproxy.com/haproxy/proxy-protocol/
      - 10y ago
zimbatm 10y ago
What's wrong with using the X-Forwarded-For/X-Forwarded-Proto headers as usual? Or do you mean for non-HTTP protocol?
- monktastic1 10y ago
  Yeah, looks like he's talking about L4 proxying. Check out http://www.haproxy.org/download/1.8/doc/proxy-protocol.txt http://www.haproxy.org/download/1.8/doc/proxy-protocol.txt.