3 ms·

A lot of extensions ask for “read all secrets” which is effectively root on the cluster - especially ingress controllers - because service account tokens for co

A lot of extensions ask for “read all secrets” which is effectively root on the cluster - especially ingress controllers - because service account tokens for controllers are in secrets. Once we turn off service account tokens (they will he generated for pods on demand on each host) then that attack gets much smaller, although at that point is an extension that reads secrets is still as privileged as any pod on the cluster. Being able to create a pod in a namespace makes you also able to read any secret in that namespace.

jrockway 7y ago
I can believe that. Ingress controllers do a ton of crazy things for no good reason (including, as you mention, access secrets in the namespace of the service they are exposing; the secrets contain TLS certificates). I wrote my own thing as an alternative to Ingress. I have Envoy and the public TLS certificates in a namespace, then send it a list of services over xDS from another service (in its own namespace, with read/list access to services and endpoints): https://github.com/jrockway/ekglue https://github.com/jrockway/ekglue This is limited in the sense that you have to edit the route table and reload Envoy to add new routes, but at least you don't have to configure the backends. (And Envoy doesn't have to poll DNS every 5 seconds for every upstream.) I don't manage the route table because what I've learned from Ingress is that route tables don't compose nearly as well as people think. They are ordered, and that creates all kind of craziness. (ingress-nginx, for example, orders by the creation timestamp on resources. So if you delete and reapply your Ingress rules, the controller behaves differently! That seems like a disaster waiting to happen, so I just manage it as one chunk. I think that is what we did at Google; it was always Quite The Procedure to create a new publicly-routable DNS name.)

jrockway 7y ago

I can believe that. Ingress controllers do a ton of crazy things for no good reason (including, as you mention, access secrets in the namespace of the service they are exposing; the secrets contain TLS certificates). I wrote my own thing as an alternative to Ingress. I have Envoy and the public TLS certificates in a namespace, then send it a list of services over xDS from another service (in its own namespace, with read/list access to services and endpoints): https://github.com/jrockway/ekglue https://github.com/jrockway/ekglue This is limited in the sense that you have to edit the route table and reload Envoy to add new routes, but at least you don't have to configure the backends. (And Envoy doesn't have to poll DNS every 5 seconds for every upstream.) I don't manage the route table because what I've learned from Ingress is that route tables don't compose nearly as well as people think. They are ordered, and that creates all kind of craziness. (ingress-nginx, for example, orders by the creation timestamp on resources. So if you delete and reapply your Ingress rules, the controller behaves differently! That seems like a disaster waiting to happen, so I just manage it as one chunk. I think that is what we did at Google; it was always Quite The Procedure to create a new publicly-routable DNS name.)