8 ms·

Show HN: konfd – Manage Kubernetes secrets and configmaps with Go templates

gabrtv 10y ago
More great stuff from Kelsey... My tldr; konfd writes out k8s configmaps based on other k8s resources like secrets, configmaps, etc. Really useful for writing out complete config files into a pod namespace without relying on external config backends. Question: While looping on a syncInterval is certainly clean and understandable, it feels suboptimal when all the templates sources are themselves watchable with a k8s client. Benefits of switching to a watch model: 1. Speed: It'd be nice to have the template rendering fire immediately after a source secret changed versus waiting for syncInterval. 2. Resource Utilization: Switching from syncInterval to a watch should save significant cycles by avoiding reprocessing templates when config hasn't changed.
- kelseyhightower 10y ago
  This is great feedback. I'm still on the fence about a watch mode for the following reasons: 1. The template configmap will not change as often as the secrets and configmap key/value pairs they reference. This means you'll always need a sync loop to regenerate the template and compare it with the current generated version. This is how most configuration management systems work. Maybe we can watch the data and uses changes as triggers? 2. Watch is a bit racy when a template references multiple secrets and configmaps. It's really hard to prevent a partial config. One idea is to scale the konfd replicaset to 0 before making changing a bunch of secrets and configmaps, then scaling konfd back to 1 to trigger the processing of the templates to pick up the new values. The other option is to wait the "sync-interval", but you'll run the risk of partial configs or needing multiple runs to generate the complete config. One possible solution to prevent 1 and 2 is to only reference one external configmap or secret for each template resource. I'm still testing these patterns in real life and will update the docs on which patterns work best for each situation. With that said, I think I'll still add a watch mode so others can test and provide feedback.
  - andrewstuart2 10y ago
    The second one has plagued me with a couple kubernetes clients I've written, but something about your restatement of the problem struck me with (IMO, of course) a nice idiomatic solution: a "debounce" fan-in (ish) channel between the ResultChan() and consumer that bundles up watch notifications after some configurable wait period. And actually, it's a common enough use case (I'd definitely use it too) that I'll bet it would be useful to have in client-go itself. And secondly, I would love to see this kind of concept upgraded to a first-class citizen in kubernetes. Essentially, I think an equivalent of `/etc/environment` for your cluster would be a powerful way of generalizing app components. It would be great if just placing resources in your cluster allows them to pick up at least some of their required configuration, much like customizing `/etc/environment` does for *nix machines.
otterley 10y ago
Kelsey, I (and others) would kindly appreciate it if you'd stop calling anything in Kubernetes as it exists today a "secret" and writing new code to further encourage its use. People should be aware in no uncertain terms that the "secrets" store in etcd is totally unencrypted and insecure. If you'd like to find a place to help, I'd suggest focusing your efforts on connecting Kubernetes to Hashicorp Vault, which is truly secure, and deprecating the old unencrypted etcd-backed implementation.
- coleca 10y ago
  Didn't he already do this? https://github.com/kelseyhightower/vault-controller https://github.com/kelseyhightower/vault-controller
  - otterley 10y ago
    "This is a prototype. Do not use this in production." Also, it's just a start; the existing "secrets" implementation needs to be completely scrapped in the name of security.
    - metral 10y ago
      Adding integration of Vault into Secrets is an active discussion in sig-auth: https://github.com/kubernetes/kubernetes/issues/10439#issuecomment-263954184 https://github.com/kubernetes/kubernetes/issues/10439#issuec... If you'd like to participate in these discussions, here are the details for the sig-auth's meetings & agenda items: https://github.com/kubernetes/community/tree/master/sig-auth https://github.com/kubernetes/community/tree/master/sig-auth
      - otterley 10y ago
        Thanks!
    - elsonrodriguez 10y ago
      A prototype is a great way to get the conversation started. Most people don't even know why Vault should be considered, let alone why it's better than K8s secrets (which in their world works fine).
fcantournet 10y ago
Hi kelsey, this is pretty nice ! Is there any way to bribe you into slowing down on the awesome sauce delivery pipeline so people can catch up !? asking for a friend. If you find the time can you elaborate on why you elected to use ConfigMaps for the templates too instead of building a 3rd party ressource ? This way you wouldn't have to use annotations and the definition might be a little more terse (or not) Is there something about configmaps that make implementation easier ? or provides additional behavior wrt to pod lifecycle or something like that ?
- kelseyhightower 10y ago
  ConfigMaps offer everything I need for a tool like konfd, and ConfigMaps work with the majority of clusters deployed today. While I could have used a ThirdPartyResource, that would require me to design a custom scheme and write a little more code up front. Also, there are a few issues [1] with ThirdPartyResource objects including the lack of validation and inconsistencies when interacting with ThirdPartyResource objects through the Kubernetes API [2]. That's not to say ThirdPartyResource's should not be used, but it's not a decision take lightly. Now, if konfd were to grow or add more features, then I think a ThirdPartyResource would be the way to go -- mainly because ConfigMaps require the use of "reserved" keys and annotations to configure behavior. [1] https://github.com/kubernetes/kubernetes/issues/22768 https://github.com/kubernetes/kubernetes/issues/22768 [2] https://github.com/kubernetes/kubernetes/issues/29542 https://github.com/kubernetes/kubernetes/issues/29542
  - fcantournet 10y ago
    That answers my question. Thank you for taking the time. :)
tonyhb 10y ago
Aren't secrets stored unecrypted in kube? I'd hesitate to call them secrets, though config looks good.
kozikow 10y ago
I propose an alternative to templates: Use go objects to define your config. Better dynamicity, readability and typesafety. See my post https://kozikow.com/2016/09/02/using-go-to-autogenerate-kubernetes-configs/ https://kozikow.com/2016/09/02/using-go-to-autogenerate-kube... .

More great stuff from Kelsey... My tldr; konfd writes out k8s configmaps based on other k8s resources like secrets, configmaps, etc. Really useful for writing out complete config files into a pod namespace without relying on external config backends. Question: While looping on a syncInterval is certainly clean and understandable, it feels suboptimal when all the templates sources are themselves watchable with a k8s client. Benefits of switching to a watch model: 1. Speed: It'd be nice to have the template rendering fire immediately after a source secret changed versus waiting for syncInterval. 2. Resource Utilization: Switching from syncInterval to a watch should save significant cycles by avoiding reprocessing templates when config hasn't changed.

kelseyhightower 10y ago
This is great feedback. I'm still on the fence about a watch mode for the following reasons: 1. The template configmap will not change as often as the secrets and configmap key/value pairs they reference. This means you'll always need a sync loop to regenerate the template and compare it with the current generated version. This is how most configuration management systems work. Maybe we can watch the data and uses changes as triggers? 2. Watch is a bit racy when a template references multiple secrets and configmaps. It's really hard to prevent a partial config. One idea is to scale the konfd replicaset to 0 before making changing a bunch of secrets and configmaps, then scaling konfd back to 1 to trigger the processing of the templates to pick up the new values. The other option is to wait the "sync-interval", but you'll run the risk of partial configs or needing multiple runs to generate the complete config. One possible solution to prevent 1 and 2 is to only reference one external configmap or secret for each template resource. I'm still testing these patterns in real life and will update the docs on which patterns work best for each situation. With that said, I think I'll still add a watch mode so others can test and provide feedback.
- andrewstuart2 10y ago
  The second one has plagued me with a couple kubernetes clients I've written, but something about your restatement of the problem struck me with (IMO, of course) a nice idiomatic solution: a "debounce" fan-in (ish) channel between the ResultChan() and consumer that bundles up watch notifications after some configurable wait period. And actually, it's a common enough use case (I'd definitely use it too) that I'll bet it would be useful to have in client-go itself. And secondly, I would love to see this kind of concept upgraded to a first-class citizen in kubernetes. Essentially, I think an equivalent of `/etc/environment` for your cluster would be a powerful way of generalizing app components. It would be great if just placing resources in your cluster allows them to pick up at least some of their required configuration, much like customizing `/etc/environment` does for *nix machines.

otterley 10y ago

Kelsey, I (and others) would kindly appreciate it if you'd stop calling anything in Kubernetes as it exists today a "secret" and writing new code to further encourage its use. People should be aware in no uncertain terms that the "secrets" store in etcd is totally unencrypted and insecure. If you'd like to find a place to help, I'd suggest focusing your efforts on connecting Kubernetes to Hashicorp Vault, which is truly secure, and deprecating the old unencrypted etcd-backed implementation.

coleca 10y ago
Didn't he already do this? https://github.com/kelseyhightower/vault-controller https://github.com/kelseyhightower/vault-controller
- otterley 10y ago
  "This is a prototype. Do not use this in production." Also, it's just a start; the existing "secrets" implementation needs to be completely scrapped in the name of security.
  - metral 10y ago
    Adding integration of Vault into Secrets is an active discussion in sig-auth: https://github.com/kubernetes/kubernetes/issues/10439#issuecomment-263954184 https://github.com/kubernetes/kubernetes/issues/10439#issuec... If you'd like to participate in these discussions, here are the details for the sig-auth's meetings & agenda items: https://github.com/kubernetes/community/tree/master/sig-auth https://github.com/kubernetes/community/tree/master/sig-auth
    - otterley 10y ago
      Thanks!
  - elsonrodriguez 10y ago
    A prototype is a great way to get the conversation started. Most people don't even know why Vault should be considered, let alone why it's better than K8s secrets (which in their world works fine).

fcantournet 10y ago

Hi kelsey, this is pretty nice ! Is there any way to bribe you into slowing down on the awesome sauce delivery pipeline so people can catch up !? asking for a friend. If you find the time can you elaborate on why you elected to use ConfigMaps for the templates too instead of building a 3rd party ressource ? This way you wouldn't have to use annotations and the definition might be a little more terse (or not) Is there something about configmaps that make implementation easier ? or provides additional behavior wrt to pod lifecycle or something like that ?

kelseyhightower 10y ago
ConfigMaps offer everything I need for a tool like konfd, and ConfigMaps work with the majority of clusters deployed today. While I could have used a ThirdPartyResource, that would require me to design a custom scheme and write a little more code up front. Also, there are a few issues [1] with ThirdPartyResource objects including the lack of validation and inconsistencies when interacting with ThirdPartyResource objects through the Kubernetes API [2]. That's not to say ThirdPartyResource's should not be used, but it's not a decision take lightly. Now, if konfd were to grow or add more features, then I think a ThirdPartyResource would be the way to go -- mainly because ConfigMaps require the use of "reserved" keys and annotations to configure behavior. [1] https://github.com/kubernetes/kubernetes/issues/22768 https://github.com/kubernetes/kubernetes/issues/22768 [2] https://github.com/kubernetes/kubernetes/issues/29542 https://github.com/kubernetes/kubernetes/issues/29542
- fcantournet 10y ago
  That answers my question. Thank you for taking the time. :)

tonyhb 10y ago

Aren't secrets stored unecrypted in kube? I'd hesitate to call them secrets, though config looks good.

kozikow 10y ago

I propose an alternative to templates: Use go objects to define your config. Better dynamicity, readability and typesafety. See my post https://kozikow.com/2016/09/02/using-go-to-autogenerate-kubernetes-configs/ https://kozikow.com/2016/09/02/using-go-to-autogenerate-kube... .