3 ms·

Openshift sells itself as a more secure Kubernetes ... anyone knows how much of this it mitigates?

by chvid 7y ago

meddlepal 7y ago
OpenShift has way more scope (read: surface area) than Kubernetes... I'm skeptical.
- pat2man 7y ago
  Sure but a lot of that scope is regularly added on to kubernetes clusters (metrics, logging, etc) so unless you keep your cluster super lean you will end up with a large attach surface anyway.
  - jrockway 7y ago
    Logging agents don't need to talk to the control plane, so shouldn't be able to compromise it. In the case of a per-machine log collector ("DaemonSet"), it just reads files in a particular location that your container runtime writes. In the case of per-pod log collectors ("sidecars"), no special privileges are needed. (Your application just opens a network connection inside the pod network; the 127.0.0.1 address that you write log data to doesn't even exist outside the pod.) Prometheus does talk to the control plane, but via a service account that only has read and list permissions, so shouldn't be able to compromise it. The truly paranoid could have static configurations (scrape foo.bar.svc.cluster.local:1234, not "foreach pod, scape $POD_ID:1234"), but if reading/listing pods can compromise the control plane, that would probably be a P0 security bug in Kubernetes. Obviously, anything running in your cluster has the potential to exploit a kernel bug to read files on the node that it shouldn't have access to, including other service accounts. The guidance there should be to be careful about what permissions you grant any application.
    - smarterclayton 7y ago
      A lot of extensions ask for “read all secrets” which is effectively root on the cluster - especially ingress controllers - because service account tokens for controllers are in secrets. Once we turn off service account tokens (they will he generated for pods on demand on each host) then that attack gets much smaller, although at that point is an extension that reads secrets is still as privileged as any pod on the cluster. Being able to create a pod in a namespace makes you also able to read any secret in that namespace.
      - jrockway 7y ago
        I can believe that. Ingress controllers do a ton of crazy things for no good reason (including, as you mention, access secrets in the namespace of the service they are exposing; the secrets contain TLS certificates). I wrote my own thing as an alternative to Ingress. I have Envoy and the public TLS certificates in a namespace, then send it a list of services over xDS from another service (in its own namespace, with read/list access to services and endpoints): https://github.com/jrockway/ekglue https://github.com/jrockway/ekglue This is limited in the sense that you have to edit the route table and reload Envoy to add new routes, but at least you don't have to configure the backends. (And Envoy doesn't have to poll DNS every 5 seconds for every upstream.) I don't manage the route table because what I've learned from Ingress is that route tables don't compose nearly as well as people think. They are ordered, and that creates all kind of craziness. (ingress-nginx, for example, orders by the creation timestamp on resources. So if you delete and reapply your Ingress rules, the controller behaves differently! That seems like a disaster waiting to happen, so I just manage it as one chunk. I think that is what we did at Google; it was always Quite The Procedure to create a new publicly-routable DNS name.)
chrisfosterelli 7y ago
I haven't administered an OpenShift cluster but I've used one and you notice the difference in policy approach. OpenShift seems to have a lot of default policies that are more aggressive than Kubernetes. A lot of container images from Docker Hub just don't run at all on OpenShift without modification due to this. The biggest thing is that they don't allow containers to run as root, but they also have other features like blocking hostPath mounts and offering SELinux integration. Nearly all of this can be turned off (if you have permissions to do that in the RBAC) but they generally take the "secure by default" approach.
- pat2man 7y ago
  OpenShift also has some features that help with known attacks (easy cluster upgrades, image vulnerability scanning, etc).
- freedomben 7y ago
  Correct. OpenShift by default will start your container with a "random" UID (it's not technically random but to a user it will feel that way), so not only is root disallowed but you also can't anticipate being any specific user. This can feel annoying at times but has important implications for running on the host. you can read more here if interested: https://cookbook.openshift.org/users-and-role-based-access-control/why-do-my-applications-run-as-a-random-user-id.html https://cookbook.openshift.org/users-and-role-based-access-c... OpenShift also moves to CRI-O and rootless containers in later version, which further protects hosts.
cpitman 7y ago
Let me take a stab at this, as a Red Hat architect that spends a lot of time on OpenShift(OCP) implementations: OpenShift helps mitigate the following, by providing secure defaults. It is totally possible for someone (a cluster level admin) to un-configure these security controls, so keep that in mind. These are partially mitigated by Red Hat providing trusted image content and having a well oiled machine for patching/CVE response: * Compromised images in registry (use Red Hat base images, and in-cluster source or docker builds to lock down registry access. Registry not exposed by default.) * Application vulnerability/Application exploit (again, Red Hat base images for runtime, regularly patched by Red Hat. Images have to run as non-root user. Doesn't solve self-created app vulnerabilities) These are mitigated by OCP's locked down default RBAC: * Writable host-path (host-path locked down by default. Not sure why host-path is special here though, seems like any PV could be used for persistence) * Privileged container (also locked down, admin must enable for a user/service account) * host-path mount (specifically for getting access to the file system, OCP locks down this via RBAC by default. Layer on SELinux to prevent an escalation from allowing file-system access) * list kubernetes secrets (RBAC to limit scope, and additionally a node can only access secrets for pods scheduled to it, in case node is exploited) * access container service-account (default service-account has pretty limited permissions) * access Kubernetes dashboard (containers have no special access to the console, which requires auth) These are just miscellaneous config/security controls: * Exposed dashboard (OCP console has authentication by default, no default passwords) * clear container-logs (built in log aggregation would require an additional exploit to defeat) * cluster internal networking (depends on the sdn plugin, but most installs limit networking to containers in the same namespace by default) * access tiller endpoint (No Helm 2, support for Helm 3 without tiller coming soon. Driven by tiller security/RBAC concerns)

Openshift sells itself as a more secure Kubernetes ... anyone knows how much of this it mitigates?

by chvid 7y ago

Openshift sells itself as a more secure Kubernetes ... anyone knows how much of this it mitigates?

meddlepal 7y ago

OpenShift has way more scope (read: surface area) than Kubernetes... I'm skeptical.

pat2man 7y ago
Sure but a lot of that scope is regularly added on to kubernetes clusters (metrics, logging, etc) so unless you keep your cluster super lean you will end up with a large attach surface anyway.
- jrockway 7y ago
  Logging agents don't need to talk to the control plane, so shouldn't be able to compromise it. In the case of a per-machine log collector ("DaemonSet"), it just reads files in a particular location that your container runtime writes. In the case of per-pod log collectors ("sidecars"), no special privileges are needed. (Your application just opens a network connection inside the pod network; the 127.0.0.1 address that you write log data to doesn't even exist outside the pod.) Prometheus does talk to the control plane, but via a service account that only has read and list permissions, so shouldn't be able to compromise it. The truly paranoid could have static configurations (scrape foo.bar.svc.cluster.local:1234, not "foreach pod, scape $POD_ID:1234"), but if reading/listing pods can compromise the control plane, that would probably be a P0 security bug in Kubernetes. Obviously, anything running in your cluster has the potential to exploit a kernel bug to read files on the node that it shouldn't have access to, including other service accounts. The guidance there should be to be careful about what permissions you grant any application.
  - smarterclayton 7y ago
    A lot of extensions ask for “read all secrets” which is effectively root on the cluster - especially ingress controllers - because service account tokens for controllers are in secrets. Once we turn off service account tokens (they will he generated for pods on demand on each host) then that attack gets much smaller, although at that point is an extension that reads secrets is still as privileged as any pod on the cluster. Being able to create a pod in a namespace makes you also able to read any secret in that namespace.
    - jrockway 7y ago
      I can believe that. Ingress controllers do a ton of crazy things for no good reason (including, as you mention, access secrets in the namespace of the service they are exposing; the secrets contain TLS certificates). I wrote my own thing as an alternative to Ingress. I have Envoy and the public TLS certificates in a namespace, then send it a list of services over xDS from another service (in its own namespace, with read/list access to services and endpoints): https://github.com/jrockway/ekglue https://github.com/jrockway/ekglue This is limited in the sense that you have to edit the route table and reload Envoy to add new routes, but at least you don't have to configure the backends. (And Envoy doesn't have to poll DNS every 5 seconds for every upstream.) I don't manage the route table because what I've learned from Ingress is that route tables don't compose nearly as well as people think. They are ordered, and that creates all kind of craziness. (ingress-nginx, for example, orders by the creation timestamp on resources. So if you delete and reapply your Ingress rules, the controller behaves differently! That seems like a disaster waiting to happen, so I just manage it as one chunk. I think that is what we did at Google; it was always Quite The Procedure to create a new publicly-routable DNS name.)

chrisfosterelli 7y ago

I haven't administered an OpenShift cluster but I've used one and you notice the difference in policy approach. OpenShift seems to have a lot of default policies that are more aggressive than Kubernetes. A lot of container images from Docker Hub just don't run at all on OpenShift without modification due to this. The biggest thing is that they don't allow containers to run as root, but they also have other features like blocking hostPath mounts and offering SELinux integration. Nearly all of this can be turned off (if you have permissions to do that in the RBAC) but they generally take the "secure by default" approach.

pat2man 7y ago
OpenShift also has some features that help with known attacks (easy cluster upgrades, image vulnerability scanning, etc).
freedomben 7y ago
Correct. OpenShift by default will start your container with a "random" UID (it's not technically random but to a user it will feel that way), so not only is root disallowed but you also can't anticipate being any specific user. This can feel annoying at times but has important implications for running on the host. you can read more here if interested: https://cookbook.openshift.org/users-and-role-based-access-control/why-do-my-applications-run-as-a-random-user-id.html https://cookbook.openshift.org/users-and-role-based-access-c... OpenShift also moves to CRI-O and rootless containers in later version, which further protects hosts.

cpitman 7y ago

Let me take a stab at this, as a Red Hat architect that spends a lot of time on OpenShift(OCP) implementations: OpenShift helps mitigate the following, by providing secure defaults. It is totally possible for someone (a cluster level admin) to un-configure these security controls, so keep that in mind. These are partially mitigated by Red Hat providing trusted image content and having a well oiled machine for patching/CVE response: * Compromised images in registry (use Red Hat base images, and in-cluster source or docker builds to lock down registry access. Registry not exposed by default.) * Application vulnerability/Application exploit (again, Red Hat base images for runtime, regularly patched by Red Hat. Images have to run as non-root user. Doesn't solve self-created app vulnerabilities) These are mitigated by OCP's locked down default RBAC: * Writable host-path (host-path locked down by default. Not sure why host-path is special here though, seems like any PV could be used for persistence) * Privileged container (also locked down, admin must enable for a user/service account) * host-path mount (specifically for getting access to the file system, OCP locks down this via RBAC by default. Layer on SELinux to prevent an escalation from allowing file-system access) * list kubernetes secrets (RBAC to limit scope, and additionally a node can only access secrets for pods scheduled to it, in case node is exploited) * access container service-account (default service-account has pretty limited permissions) * access Kubernetes dashboard (containers have no special access to the console, which requires auth) These are just miscellaneous config/security controls: * Exposed dashboard (OCP console has authentication by default, no default passwords) * clear container-logs (built in log aggregation would require an additional exploit to defeat) * cluster internal networking (depends on the sdn plugin, but most installs limit networking to containers in the same namespace by default) * access tiller endpoint (No Helm 2, support for Helm 3 without tiller coming soon. Driven by tiller security/RBAC concerns)