6 ms·
I'm familiar with proxy-protocol, and while I haven't injected it with nginx, it seems that it does support it: https://nginx.org/en/docs/stream/ngx_stream_prox
by predakanga 10y ago
I'm familiar with proxy-protocol, and while I haven't injected it with nginx, it seems that it does support it: https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_protocol https://nginx.org/en/docs/stream/ngx_stream_proxy_module.htm...
For reference, the feature was added in nginx 1.9.2, and the ingress-nginx controller appears to use nginx 1.11.9: https://github.com/kubernetes/ingress/blob/0.9.1/images/nginx-slim/build.sh https://github.com/kubernetes/ingress/blob/0.9.1/images/ngin...
Regardless, would I be right in assuming that your issues are mostly related to the ingress of TCP streams? My usecase is almost exclusively HTTP, so I might be missing the worst of it.
Thanks for the pointer to the #sig-onpremise channel, I'll be sure to check it out. Been meaning to look into the SIGs since reading https://coreos.com/blog/self-hosted-kubernetes.html https://coreos.com/blog/self-hosted-kubernetes.html
- sandGorgon 10y agoplease double check - nginx's support for proxy protocol is to be able to read it. it cannot support injecting it. if nginx is the first thing your traffic hits, it will not inject proxy protocol headers. Again, I will not stake my reputation on this... but I remember going deep into this about 2 months back. if you are using nginx ingress, then you need to move all of your existing config into their own config file format. Since setting up full TLS internally in kubernetes is buggy, I didnt want to terminate my traffic on the ingress. Also I really did not want to move my (fairly complex) nginx configuration to the ingress. Which is why I'm a big believer in haproxy ingress - it was designed to interface with nginx. However, you are already one of the "enlightened" ones. You actually do know what an ingress is. A significant amount of k8s slack traffic is "how do I setup a loadbalancer/how to make the cluster available to the outside world"
- justinsb 10y agonginx can inject proxy_protocol - I know because I've enabled it by mistake! The kubernetes nginx ingress controller doesn't support it though. I think that is because most people use HTTP headers (X-Forwarded-For) instead. But if you do want it, you should open an issue explaining the use case so we can check our assumptions!
- sandGorgon 10y ago@justinsb - ur the expert here and i will not dispute this :D I still believe that there is NO better tool to use kubernetes than kops. I now use the 100.64 subnet that you guys figured out ("carrier grade NAT" seriously?) in docker swarm. but genuine question - can u show me how ? Because I honestly went crawling into the nginx codebase and could only find client side handling of proxy protocol. I found NO place where it showed how it could be injected and chain reverse proxies together (that is what it was invented for in the first place). P.S. just a quick google reveals this link which sort of confirms my suspicion - http://blog.haproxy.com/haproxy/proxy-protocol/ http://blog.haproxy.com/haproxy/proxy-protocol/
- justinsb 10y agoIt's the same proxy_protocol keyword for both, which is why this is so confusing. In the listen line it means "remove proxy protocol from the inbound connection", as a top level directive "proxy_protocol on" on the server it means "add proxy protocol to the outbound connection" This commit should show the difference: https://github.com/kubernetes/ingress/commit/6fa461c2a7891b4f20fffd1c9f7fe2fce5085cd3 https://github.com/kubernetes/ingress/commit/6fa461c2a7891b4... This is the nginx function AFAICT: https://github.com/nginx/nginx/blob/b580770f3afaeec48a15cb8c9d2f9cf72b14e4da/src/stream/ngx_stream_proxy_module.c#L909 https://github.com/nginx/nginx/blob/b580770f3afaeec48a15cb8c... Looking at that though, maybe it only works with SSL passthrough... but that is the typical use-case for using proxy protocol instead of X-Forwarded-For