3 ms·
It is all ridiculously complicated. Envoy's documentation makes it pretty clear what concessions are made: https://www.envoyproxy.io/docs/envoy/latest/configur
by jrockway 5y ago
It is all ridiculously complicated. Envoy's documentation makes it pretty clear what concessions are made:
https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/headers#x-forwarded-for https://www.envoyproxy.io/docs/envoy/latest/configuration/ht... I link the documentation not for the informative aspect, but just for you to gawk at the length and how worried the author is about it all ;)
I personally prefer that the load balancer speak "proxy protocol" to the front proxy, and then the front proxy simply make a decision as to what the external IP is and relay that to upstream applications with a proprietary header (x-envoy-external-address). Upstream applications that really want to be careful about the external address then verify the signature of envoy's TLS certificate when the connection comes in. (I use some mTLS for my homelab stuff, but don't have any software that cares to check this. I use Envoy's rate limiter rather than application specific rate limiting, and use Envoy's logs to get the IP address and the x-b3-trace-id header to correlate access logs with application logs.)
Proxy protocol is regrettably problematic from time to time, however. By default, Kubernetes tries to be "smart" about routing. If you have a LoadBalancer with internal IP 10.123.234.56 and external IP 54.43.32.21, traffic originating in the cluster with a destination address 54.43.32.21 does not transit the cloud provider's load balancer, it just gets rewritten to the internal IP address 10.123.234.56. This means that it doesn't use the proxy protocol, and the front proxy rejects the connection. This is generally not a problem, but will be annoying if you host an internal container registry in-cluster, because containers will be named for the external container registry and will thus resolve to the external IP address. The connection will skip your cloud provider's load balancer which adds the proxy header, and your front proxy will reject the connection, causing failing pulls. Most people will never hit this, because their cloud provider just provides a container registry (when I set up all my container stuff, my cloud provider of choice didn't offer this; they do now). But if you ever think internal applications will intend to use the public Internet to connect to other internal applications, watch out. This can come up if you do OIDC things internally, for example. (Apps will want to grab the .well-known configuration keys and JWKS key material from the external address that it will eventually redirect clients to. You will be confused when the configuration fails to load.)
In conclusion, what a mess.
- adam-p 5y agoThat Envoy doc is... impressive. I'll find a place to link to it in the post.