3 ms·

Sure but a lot of that scope is regularly added on to kubernetes clusters (metrics, logging, etc) so unless you keep your cluster super lean you will end up wit

Sure but a lot of that scope is regularly added on to kubernetes clusters (metrics, logging, etc) so unless you keep your cluster super lean you will end up with a large attach surface anyway.

jrockway 7y ago
Logging agents don't need to talk to the control plane, so shouldn't be able to compromise it. In the case of a per-machine log collector ("DaemonSet"), it just reads files in a particular location that your container runtime writes. In the case of per-pod log collectors ("sidecars"), no special privileges are needed. (Your application just opens a network connection inside the pod network; the 127.0.0.1 address that you write log data to doesn't even exist outside the pod.) Prometheus does talk to the control plane, but via a service account that only has read and list permissions, so shouldn't be able to compromise it. The truly paranoid could have static configurations (scrape foo.bar.svc.cluster.local:1234, not "foreach pod, scape $POD_ID:1234"), but if reading/listing pods can compromise the control plane, that would probably be a P0 security bug in Kubernetes. Obviously, anything running in your cluster has the potential to exploit a kernel bug to read files on the node that it shouldn't have access to, including other service accounts. The guidance there should be to be careful about what permissions you grant any application.
- smarterclayton 7y ago
  A lot of extensions ask for “read all secrets” which is effectively root on the cluster - especially ingress controllers - because service account tokens for controllers are in secrets. Once we turn off service account tokens (they will he generated for pods on demand on each host) then that attack gets much smaller, although at that point is an extension that reads secrets is still as privileged as any pod on the cluster. Being able to create a pod in a namespace makes you also able to read any secret in that namespace.
  - jrockway 6y ago
    I can believe that. Ingress controllers do a ton of crazy things for no good reason (including, as you mention, access secrets in the namespace of the service they are exposing; the secrets contain TLS certificates). I wrote my own thing as an alternative to Ingress. I have Envoy and the public TLS certificates in a namespace, then send it a list of services over xDS from another service (in its own namespace, with read/list access to services and endpoints): https://github.com/jrockway/ekglue https://github.com/jrockway/ekglue This is limited in the sense that you have to edit the route table and reload Envoy to add new routes, but at least you don't have to configure the backends. (And Envoy doesn't have to poll DNS every 5 seconds for every upstream.) I don't manage the route table because what I've learned from Ingress is that route tables don't compose nearly as well as people think. They are ordered, and that creates all kind of craziness. (ingress-nginx, for example, orders by the creation timestamp on resources. So if you delete and reapply your Ingress rules, the controller behaves differently! That seems like a disaster waiting to happen, so I just manage it as one chunk. I think that is what we did at Google; it was always Quite The Procedure to create a new publicly-routable DNS name.)

Sure but a lot of that scope is regularly added on to kubernetes clusters (metrics, logging, etc) so unless you keep your cluster super lean you will end up with a large attach surface anyway.

jrockway 7y ago

Logging agents don't need to talk to the control plane, so shouldn't be able to compromise it. In the case of a per-machine log collector ("DaemonSet"), it just reads files in a particular location that your container runtime writes. In the case of per-pod log collectors ("sidecars"), no special privileges are needed. (Your application just opens a network connection inside the pod network; the 127.0.0.1 address that you write log data to doesn't even exist outside the pod.) Prometheus does talk to the control plane, but via a service account that only has read and list permissions, so shouldn't be able to compromise it. The truly paranoid could have static configurations (scrape foo.bar.svc.cluster.local:1234, not "foreach pod, scape $POD_ID:1234"), but if reading/listing pods can compromise the control plane, that would probably be a P0 security bug in Kubernetes. Obviously, anything running in your cluster has the potential to exploit a kernel bug to read files on the node that it shouldn't have access to, including other service accounts. The guidance there should be to be careful about what permissions you grant any application.

smarterclayton 7y ago
A lot of extensions ask for “read all secrets” which is effectively root on the cluster - especially ingress controllers - because service account tokens for controllers are in secrets. Once we turn off service account tokens (they will he generated for pods on demand on each host) then that attack gets much smaller, although at that point is an extension that reads secrets is still as privileged as any pod on the cluster. Being able to create a pod in a namespace makes you also able to read any secret in that namespace.
- jrockway 6y ago
  I can believe that. Ingress controllers do a ton of crazy things for no good reason (including, as you mention, access secrets in the namespace of the service they are exposing; the secrets contain TLS certificates). I wrote my own thing as an alternative to Ingress. I have Envoy and the public TLS certificates in a namespace, then send it a list of services over xDS from another service (in its own namespace, with read/list access to services and endpoints): https://github.com/jrockway/ekglue https://github.com/jrockway/ekglue This is limited in the sense that you have to edit the route table and reload Envoy to add new routes, but at least you don't have to configure the backends. (And Envoy doesn't have to poll DNS every 5 seconds for every upstream.) I don't manage the route table because what I've learned from Ingress is that route tables don't compose nearly as well as people think. They are ordered, and that creates all kind of craziness. (ingress-nginx, for example, orders by the creation timestamp on resources. So if you delete and reapply your Ingress rules, the controller behaves differently! That seems like a disaster waiting to happen, so I just manage it as one chunk. I think that is what we did at Google; it was always Quite The Procedure to create a new publicly-routable DNS name.)