6 ms·
Kelsey, I (and others) would kindly appreciate it if you'd stop calling anything in Kubernetes as it exists today a "secret" and writing new code to further enc
by otterley 10y ago
Kelsey, I (and others) would kindly appreciate it if you'd stop calling anything in Kubernetes as it exists today a "secret" and writing new code to further encourage its use. People should be aware in no uncertain terms that the "secrets" store in etcd is totally unencrypted and insecure.
If you'd like to find a place to help, I'd suggest focusing your efforts on connecting Kubernetes to Hashicorp Vault, which is truly secure, and deprecating the old unencrypted etcd-backed implementation.
- coleca 10y agoDidn't he already do this? https://github.com/kelseyhightower/vault-controller https://github.com/kelseyhightower/vault-controller
- otterley 10y ago"This is a prototype. Do not use this in production." Also, it's just a start; the existing "secrets" implementation needs to be completely scrapped in the name of security.
- metral 10y agoAdding integration of Vault into Secrets is an active discussion in sig-auth: https://github.com/kubernetes/kubernetes/issues/10439#issuecomment-263954184 https://github.com/kubernetes/kubernetes/issues/10439#issuec... If you'd like to participate in these discussions, here are the details for the sig-auth's meetings & agenda items: https://github.com/kubernetes/community/tree/master/sig-auth https://github.com/kubernetes/community/tree/master/sig-auth
- otterley 10y agoThanks!
- elsonrodriguez 10y agoA prototype is a great way to get the conversation started. Most people don't even know why Vault should be considered, let alone why it's better than K8s secrets (which in their world works fine).
- otterley 10y agoI agree, it's a start. At its core, I'm having some difficulty wrapping my brain around the idea that any organization that's large and complex enough to demand to run their own compute platform (e.g. K8S) isn't also large and complex enough to demand a secure configuration system by default.
- meddlepal 10y agoFines are cheap. Engineering is expensive. There's your answer. I don't like it any more than you do, but that's the world we live in.
- otterley 10y agoIt's not just fines; it's the potential loss of your certifications upon which customers condition their business. That's a Big Deal.
- iamdeedubs 10y agoMaybe that's the disconnect here. There are plenty of small orgs using kubernetes for the ease of use and portability
- markbnj 10y agoThe API refers to these objects as secrets. How does it improve understanding to pick some other name because you feel the current one is undeserved? I'm not disagreeing with that assessment, but it seems odd to just refuse to use it and chide others for doing so. It that supposed to be a protest or something?
- otterley 10y agoIt's more a criticism of the implementation. Kubernetes deserves a secrets store, but the one it's got isn't worthy of the name. I'm advocating for engineering efforts to be prioritized over other projects on making it a secure implementation, because people are likely to be misled into thinking the existing implementation is already secure.
- kelseyhightower 10y agoFirst, thanks for taking the time to offer feedback regarding what we call "secrets" in Kubernetes. As you've rightfully pointed out the current implementation of "secrets" are stored unencrypted in etcd; something like Hashicorp's Vault [1] would be a much better choice for the given use-case. I've already began exploring how an integration between Vault and Kubernetes would work [2]. There are some things to work out, but those discussions are under way [3]. The current prototype/example demonstrates one way of leveraging Vault from applications deployed on Kubernetes, without changes to the Kubernetes core. While the vault-controller [2] works, we can do better. One idea is to consider what a deeper Vault integration looks like. Ideally we can modify parts of Kubernetes, mainly the kubelet agent that runs on every node, to handled the secure generation and renewal of unique Vault tokens for each Pod during the Pod creation phase. Another idea would be to rethink "secrets" altogether and rebuild the entire feature on top of the Vault API. We can "hide" Vault under the current "secrets" API, which would let us remain backwards compatible. That would be phase one. Phase two could introduce new Secret extensions, which would enable users to declaratively manage Vault tokens, backends, and secret renewals through the Kubernetes API. [1] https://www.vaultproject.io https://www.vaultproject.io [2] https://github.com/kelseyhightower/vault-controller https://github.com/kelseyhightower/vault-controller [3] https://github.com/kubernetes/kubernetes/issues/10439#issuecomment-263954184 https://github.com/kubernetes/kubernetes/issues/10439#issuec...
- otterley 10y agoI appreciate your efforts here, Kelsey. What I'm looking for (and I realize I'm not your manager; this is just my opinion borne of a year of experience with K8S) is a focus on strengthening the core first, then going back to the veneer. Great edifices start with a strong foundation. Also, the engineering teams need to do a better job of integrating community PRs that address implementor demands. For example, Consul K/V support has been asked for since 2014 [1], yet nobody seems to be in a hurry to integrate the (generously-provided) functionality [2]. Finally, I'd like to see deadlines and decision-makers appointed for making decisions such as the one you discuss, so we can avoid endless debates and make forward progress quickly. [1] https://github.com/kubernetes/kubernetes/issues/1957 https://github.com/kubernetes/kubernetes/issues/1957 [2] https://github.com/kubernetes/kubernetes/pull/31622 https://github.com/kubernetes/kubernetes/pull/31622
- TheDong 10y agoIf someone can access your etcd, they can write new pods to run into it which will then be distributed to nodes to run. Those new pods could reference existing secrets, be in any namespace (e.g. kube-system), and are exactly the same as root on all k8s nodes. Because of that, having access to etcd is equivalent to having admin cluster access. The secrets are no more secret if they're encrypted in etcd or not since anyone with access to etcd can launch pods that will decrypt arbitrary secrets. This is true iwth vault as well. Once secrets are in vault, access to etcd will still result in all secrets being available to the attacker. The threat model is no better. There are plans to encrypt them in etcd for the sake of them not appearing in backups and because people like you constantly bitch about it, but it doesn't mater and saying they shouldn't be called secrets because of this is FUD.
- otterley 10y agoFirst, I'm offended by your characterization of my argument as "bitching," as though our concerns are unjustified. At any rate, you bring up two points: First, that etcd is insecure, which is a problem in itself that needs to be addressed. Communications between K8S and etcd and between applications and etcd need to be secured, and K8S itself needs role-based ACLs in the core. Second, for the Vault use case, the best practice is not to place Vault tokens in etcd that grant direct access to any secrets. Instead, the best practice (as Kelsey's vault-controller project does) is to pass a "wrapper token," which is a single-use token. The application consumes the wrapper token (hopefully quickly) and exchanges it for a longer-lived token, which can then be used to access the secret data. Once the wrapper token has been used, it is subsequently worthless to someone who has direct etcd access. And Vault's auditing capabilities can help you detect misuses of wrapper tokens.
- soamv 10y agoThe comment you're replying to doesn't say that etcd is insecure; just that access to etcd is equivalent in power to Kubernetes API access. It's hard to have a productive discussion in such a confrontational tone, especially when you don't really have all the facts. If you have questions about how to secure Kubernetes communication with etcd, feel free to ask!