5 ms·

If someone can access your etcd, they can write new pods to run into it which will then be distributed to nodes to run. Those new pods could reference existing

by TheDong 10y ago

If someone can access your etcd, they can write new pods to run into it which will then be distributed to nodes to run. Those new pods could reference existing secrets, be in any namespace (e.g. kube-system), and are exactly the same as root on all k8s nodes. Because of that, having access to etcd is equivalent to having admin cluster access. The secrets are no more secret if they're encrypted in etcd or not since anyone with access to etcd can launch pods that will decrypt arbitrary secrets. This is true iwth vault as well. Once secrets are in vault, access to etcd will still result in all secrets being available to the attacker. The threat model is no better. There are plans to encrypt them in etcd for the sake of them not appearing in backups and because people like you constantly bitch about it, but it doesn't mater and saying they shouldn't be called secrets because of this is FUD.

otterley 10y ago
First, I'm offended by your characterization of my argument as "bitching," as though our concerns are unjustified. At any rate, you bring up two points: First, that etcd is insecure, which is a problem in itself that needs to be addressed. Communications between K8S and etcd and between applications and etcd need to be secured, and K8S itself needs role-based ACLs in the core. Second, for the Vault use case, the best practice is not to place Vault tokens in etcd that grant direct access to any secrets. Instead, the best practice (as Kelsey's vault-controller project does) is to pass a "wrapper token," which is a single-use token. The application consumes the wrapper token (hopefully quickly) and exchanges it for a longer-lived token, which can then be used to access the secret data. Once the wrapper token has been used, it is subsequently worthless to someone who has direct etcd access. And Vault's auditing capabilities can help you detect misuses of wrapper tokens.
- soamv 10y ago
  The comment you're replying to doesn't say that etcd is insecure; just that access to etcd is equivalent in power to Kubernetes API access. It's hard to have a productive discussion in such a confrontational tone, especially when you don't really have all the facts. If you have questions about how to secure Kubernetes communication with etcd, feel free to ask!
  - otterley 10y ago
    What facts do you think I don't possess? Does etcd have an ACL capability yet? Does it possess auditing capability? AFAIK etcd only supports TLS client authentication, which is arguably a fraction of what it needs in order to be considered a secure store in any meaningful sense. Consul is miles ahead in this regard. By all means, correct me if I'm mistaken.
    - TheDong 10y ago
      What etcd does and doesn't support doesn't matter here. Pods do not access etcd. Nothing that is not fully trusted has access. You don't need ACLs when the only users of etcd are 100% trusted (aka just kubernetes core components). The kubernetes API provides secrets to pods and can do its own validation (and there's experimental support for that). It can provide its own auditing. That's where it actually matters, not etcd.
      - otterley 10y ago
        The fact that you don't allow anything to connect to etcd besides the K8S components doesn't really make etcd itself inherently secure. It also implies that access to etcd is an all-or-nothing proposition, which doesn't work well in complex environments. People will need to get into etcd in order to perform debugging, maintenance, etc., and at big organizations, they will need differing levels of access depending on their respective roles. And your assumption that only K8S will access etcd isn't necessarily shared by others. In the model implementation, perhaps that is so; but in the real world, implementors may desire to share etcd with other applications. This too necessitates additional security options for etcd itself.
        mugsie 10y ago
        If someone was actually interested in security, they would not allow tenant (or user / etc) defined components access the base infrastructure's etcd, even if it had ACLs. Basic separation of concerns would call for a separate instance of etcd. In a complex environment, spinning up a new etcd should not be a big deal. Now, having ACLs, is not a bad thing, but they should only be used to lock down the base infrastructure access even more, and still have a separate instance for other applications.

by TheDong 10y ago

otterley 10y ago

First, I'm offended by your characterization of my argument as "bitching," as though our concerns are unjustified. At any rate, you bring up two points: First, that etcd is insecure, which is a problem in itself that needs to be addressed. Communications between K8S and etcd and between applications and etcd need to be secured, and K8S itself needs role-based ACLs in the core. Second, for the Vault use case, the best practice is not to place Vault tokens in etcd that grant direct access to any secrets. Instead, the best practice (as Kelsey's vault-controller project does) is to pass a "wrapper token," which is a single-use token. The application consumes the wrapper token (hopefully quickly) and exchanges it for a longer-lived token, which can then be used to access the secret data. Once the wrapper token has been used, it is subsequently worthless to someone who has direct etcd access. And Vault's auditing capabilities can help you detect misuses of wrapper tokens.

soamv 10y ago
The comment you're replying to doesn't say that etcd is insecure; just that access to etcd is equivalent in power to Kubernetes API access. It's hard to have a productive discussion in such a confrontational tone, especially when you don't really have all the facts. If you have questions about how to secure Kubernetes communication with etcd, feel free to ask!
- otterley 10y ago
  What facts do you think I don't possess? Does etcd have an ACL capability yet? Does it possess auditing capability? AFAIK etcd only supports TLS client authentication, which is arguably a fraction of what it needs in order to be considered a secure store in any meaningful sense. Consul is miles ahead in this regard. By all means, correct me if I'm mistaken.
  - TheDong 10y ago
    What etcd does and doesn't support doesn't matter here. Pods do not access etcd. Nothing that is not fully trusted has access. You don't need ACLs when the only users of etcd are 100% trusted (aka just kubernetes core components). The kubernetes API provides secrets to pods and can do its own validation (and there's experimental support for that). It can provide its own auditing. That's where it actually matters, not etcd.
    - otterley 10y ago
      The fact that you don't allow anything to connect to etcd besides the K8S components doesn't really make etcd itself inherently secure. It also implies that access to etcd is an all-or-nothing proposition, which doesn't work well in complex environments. People will need to get into etcd in order to perform debugging, maintenance, etc., and at big organizations, they will need differing levels of access depending on their respective roles. And your assumption that only K8S will access etcd isn't necessarily shared by others. In the model implementation, perhaps that is so; but in the real world, implementors may desire to share etcd with other applications. This too necessitates additional security options for etcd itself.
      - mugsie 10y ago
        If someone was actually interested in security, they would not allow tenant (or user / etc) defined components access the base infrastructure's etcd, even if it had ACLs. Basic separation of concerns would call for a separate instance of etcd. In a complex environment, spinning up a new etcd should not be a big deal. Now, having ACLs, is not a bad thing, but they should only be used to lock down the base infrastructure access even more, and still have a separate instance for other applications.