3 ms·

A big inconvenience is that for HTTP2 Routes or edge / re-encrypt Routes with a custom TLS certificate, the TLS certificate and keys must be inline in the Route

A big inconvenience is that for HTTP2 Routes or edge / re-encrypt Routes with a custom TLS certificate, the TLS certificate and keys must be inline in the Route resource instead of referencing a secret like Ingress resources do. I think this is a big oversight where Routes mix secrets and Ingress configuration together. It makes GitOps annoying because I don't want to treat the whole Route resource as a secret that needs to be encrypted or stored in vault. Do I also then treat Route resources as sensitive and deny some users access on the account they could contain private keys? I also have to worry about keeping the route updated before certificates expire instead of having it taken care of by cert-manager. So we use Traefik's IngressRoute.

m3adow 5y ago
That I can relate to. For us, there's only one dev team which uses HTTP2 (financial industry, so HTTP2 is still seen as "new = beta") and encountered that problem. I have no idea how they solved it though.
- nullify88 5y ago
  FWIW, although I've known for a while that OpenShift coverts ingress resources to routes, I just found out that the Ingress Controller sets up a watch on the secretref which keeps the inline TLS in the Route in sync. That could be enough for some people.
smarterclayton 5y ago
For context, the reason Routes were designed to inline cert info (about 6-12mo before ingress) was that ingress controllers (which run on nodes) having the ability to read all secrets (which means the ingress controller has to be considered to be as powerful as any ServiceAccount on the cluster) was too scary. The alternative chosen was to prevent ingress controllers from seeing any secret but the ones the user decides to expose via the route. Later on, the pattern of referencing secrets in extensions became more common, and things like the NodeAuthorizer (which allows nodes to only read the secrets associated with the pods scheduled onto them) demonstrated a possible different pattern we could have chosen to implement (although nothing that can be efficiently implemented without changing kube itself today). Agree routes should have added a ref - that was feedback informing Gateway API, and once that hits GA and provides the best of both routes and ingress, we would probably suggest using that instead. Routes is mostly frozen for all but critical new features now though so we can ensure Gateway has everything we need to replace it while still providing the necessary forward compat. I would treat routes as sensitive. Note that within a namespace there is minimal cross user security (not part of the kube / openshift threat model), so giving namespace read access to a specific set of routes and only infrastructure users access to all routes, OR using a wildcard cert on the routers and keeping all key material out of the user’s space. On 4.x versions you could also create multiple ingress controllers and assign them to different namespaces, preventing leak between them.

m3adow 5y ago

That I can relate to. For us, there's only one dev team which uses HTTP2 (financial industry, so HTTP2 is still seen as "new = beta") and encountered that problem. I have no idea how they solved it though.

nullify88 5y ago
FWIW, although I've known for a while that OpenShift coverts ingress resources to routes, I just found out that the Ingress Controller sets up a watch on the secretref which keeps the inline TLS in the Route in sync. That could be enough for some people.

smarterclayton 5y ago

For context, the reason Routes were designed to inline cert info (about 6-12mo before ingress) was that ingress controllers (which run on nodes) having the ability to read all secrets (which means the ingress controller has to be considered to be as powerful as any ServiceAccount on the cluster) was too scary. The alternative chosen was to prevent ingress controllers from seeing any secret but the ones the user decides to expose via the route. Later on, the pattern of referencing secrets in extensions became more common, and things like the NodeAuthorizer (which allows nodes to only read the secrets associated with the pods scheduled onto them) demonstrated a possible different pattern we could have chosen to implement (although nothing that can be efficiently implemented without changing kube itself today). Agree routes should have added a ref - that was feedback informing Gateway API, and once that hits GA and provides the best of both routes and ingress, we would probably suggest using that instead. Routes is mostly frozen for all but critical new features now though so we can ensure Gateway has everything we need to replace it while still providing the necessary forward compat. I would treat routes as sensitive. Note that within a namespace there is minimal cross user security (not part of the kube / openshift threat model), so giving namespace read access to a specific set of routes and only infrastructure users access to all routes, OR using a wildcard cert on the routers and keeping all key material out of the user’s space. On 4.x versions you could also create multiple ingress controllers and assign them to different namespaces, preventing leak between them.