6 ms·

A Kubernetes operator to sync secrets from AWS Secrets Manager

68 points by stefanjudis 6y ago

wdb 6y ago
Interesting, wondering if it's portable to make work with Google Secrets Manager
- tecleandor 6y ago
  Seems like the article author doesn't like it much, but GoDaddy's Kubernetes External Secrets seems to support Google Secrets Manager: https://github.com/godaddy/kubernetes-external-secrets https://github.com/godaddy/kubernetes-external-secrets
- riccardomc 6y ago
  Please, take a also a look at: https://github.com/ContainerSolutions/externalsecret-operator https://github.com/ContainerSolutions/externalsecret-operato...
gouggoug 6y ago
It's worth noting that the "Kubernetes External Secrets"[0] project from Godaddy is now supplanted by "Secret-Manager"[1]. I've been using Secret-Manager and it works very well. The authors of "kube-secret-syncer" mention "[other solutions] lack either in security, caching or flexibility". When it comes to "secret-manager", although I can not vouch for its security, the codebase is very small and probably easily auditable. It's also very flexible. It supports "SecretStores", currently AWS, GCP and Vault out of the box, and it's easy to add more. Not sure why "caching" is mentioned in the mix. I'm surprised they decided to re-invent the wheel instead of improving secret-manager. [0]: https://github.com/godaddy/kubernetes-external-secrets https://github.com/godaddy/kubernetes-external-secrets [1]: https://github.com/itscontained/secret-manager https://github.com/itscontained/secret-manager
- tecleandor 6y ago
  Why is it supplanted? Is it a fork? I still see commits on GoDaddy's repository. Secret-Manager docs are, ahem, limited.
  - gouggoug 6y ago
    I could be wrong; I had originally started using "external-secrets" then I believe found about "secret-manager" from the Godaddy repository. I've used both solutions, and ultimately, I think itscontained/secret-manager is better than external-secrets. Their doc was re-jiggled a few days ago and I agree its made it look like it's inexistant. There's not a _ton_ of it, but it's there[0][1] [0]https://github.com/itscontained/secret-manager/tree/master/docs https://github.com/itscontained/secret-manager/tree/master/d... [1]https://github.com/itscontained/secret-manager/blob/master/docs/usage.md https://github.com/itscontained/secret-manager/blob/master/d... edit: found the link in the Godaddy repo to "secret-manager": https://github.com/godaddy/kubernetes-external-secrets/issues/423#issuecomment-677749973 https://github.com/godaddy/kubernetes-external-secrets/issue... I was mistaken when I said "secret-manager supplanted external-secrets". It's a Golang rewrite from a user.
    - tecleandor 6y ago
      Oh, I see!
    - riccardomc 6y ago
      Indeed is not supplanted, it is just another solution. Speaking of which, I would like to put our solution on your radar as well: https://github.com/ContainerSolutions/externalsecret-operator https://github.com/ContainerSolutions/externalsecret-operato...
    - Keunic 6y ago
      Yeah, we should probably include in Github readme, but the itscontained/secret-manager is an implementation of the standardization that a few of us have been working on in https://github.com/godaddy/kubernetes-external-secrets/pull/477 https://github.com/godaddy/kubernetes-external-secrets/pull/.... There has been talks of moving one of these solutions to https://github.com/external-secrets https://github.com/external-secrets ownership, but nothing has happened around that yet.
- krthkv 6y ago
  Hey there, I work on the infra team at Contentful and wanted to expand on the caching. Polling AWS Secrets Manager often can incur considerable costs since it is priced by API calls. We've tried to alleviate this by caching the list of secrets and their values in the process.
nuker 6y ago
Just switch from kube to ECS already, if you're in AWS. Much better integration and support.
- captn3m0 6y ago
  Really? I'd found even simple things like volume mounted secrets a pain to use. https://github.com/aws/containers-roadmap/issues/56 https://github.com/aws/containers-roadmap/issues/56
  - nuker 6y ago
    Secrets to be fetched by CFN of the Service. "define a config file as a "volume" and mount into the container" is very unusual. Store it in S3, and give your Task's IAM Role permission to fetch it.
    - dtech 6y ago
      > define a config file as a "volume" and mount into the container That is how kubernetes secrets work so I wouldn't call it unusual
      - nuker 6y ago
        > That is how kubernetes secrets work As I said, try ECS.
      - captn3m0 6y ago
        And so do so many container based applications that expect a secrets to be present in a file. Kubernetes provides an easy-to-use abstraction for the same, which ECS does not.
        nuker 6y ago
        > which ECS does not. It actually does. You may, if you wish, have a volume and mount it is ECS tasks [0][1]. The issue above does not seem legit. [0] https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_data_volumes.html https://docs.aws.amazon.com/AmazonECS/latest/developerguide/... [1] https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ecs-taskdefinition.html#cfn-ecs-taskdefinition-volumes https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui...
hurricaneSlider 6y ago
If you're looking for a gitops alternative that can make use of AWS KMS, Azure KeyVault and Google Cloud KMS, would highly recommend kamus (https://kamus.soluto.io/ https://kamus.soluto.io/). Allows secrets to be safely managed in source control and also has an AES mode ideal for local development.
riccardomc 6y ago
I am the founder of a very similar project that supports both AWS Secrets Manager and Google Secrets Manager which actually predates this and GoDaddy's solutions[1]. The proliferation of these type of projects clearly shows the need for secret handling. While I think that more solutions for the same problem is not a bad thing, I also believe that we could benefit from a coordinated effort. My colleagues are actively working with GoDaddy's maintainers to find a common way forward by standardizing the "ExternalSecret" CRD and eventually merging the projects[2]. [1]https://github.com/ContainerSolutions/externalsecret-operator https://github.com/ContainerSolutions/externalsecret-operato... [2]https://github.com/godaddy/kubernetes-external-secrets/issues/47 https://github.com/godaddy/kubernetes-external-secrets/issue...
- peterwwillis 6y ago
  This is why we need standard protocols and data formats for these common systems. If there were just a "secrets protocol" or "secrets data format", any program could just implement it for input and output, along with a standard interface to perform the actions (a single URL for example). It used to be commonplace to just write an RFC for what you were doing and then other people would use the RFC. Which wasn't perfect, but nothing ever is... Instead, the standard solution today is custom integration, which leads to a lot of reinventing the wheel and incompatibility with extremely similar products.
  - riccardomc 6y ago
    I will run the risk of sounding condescending here, but I sense some negativity in your comment that I fail to find a justification for. People need solutions to their problems and they develop them asynchronously and in isolation from each other. Turns out that some problems are more universal than others and could benefit from a common effort. Suddenly, solutions collide and collaboration happens. What you describe is exactly what they are trying to do now: https://github.com/godaddy/kubernetes-external-secrets/pull/477 https://github.com/godaddy/kubernetes-external-secrets/pull/... So, rejoice! The magic of open source and internet enabled collaboration is happening right before your eyes! :)
thehermit 6y ago
I'm looking into this myself and ran across what looks to be a more "official" tool for this [1]. Kubernetes is a fast moving ecosystem and using random OSS operators has bit us in the past. [1] https://github.com/kubernetes-sigs/secrets-store-csi-driver https://github.com/kubernetes-sigs/secrets-store-csi-driver
deleted 6y ago
[deleted]
acamillo 6y ago
I wrote this[1] operator over one year ago to learn some Go lang. The project is a Kubernetes operator that automatically creates and updates Kubernetes secrets according to what is stored in AWS Secrets Manager (SM). A custom resource, named AWSSecret, maps an AWS SM entry to a K8S Secret resource. [1] https://github.com/acamillo/aws-secret-operator https://github.com/acamillo/aws-secret-operator
cbanek 6y ago
Kind of a different problem, but I've had really good experience with using Hashicorp's vault, which is excellent, paired with the vault-secrets-operator for kubernetes to do my secrets management. It will sync secrets from a vault path and create a kubernetes secret that you can use like any other secret. At least this way I feel like there's less lockin to a cloud provider (and some of the places I run this have on-prem kubernetes, so I have to have something that works outside of the cloud, and sometimes without internet). https://www.vaultproject.io/ https://www.vaultproject.io/ https://github.com/ricoberger/vault-secrets-operator https://github.com/ricoberger/vault-secrets-operator
Keunic 6y ago
The entire Kubernetes secret space is a bit immature with no standard solutions. Many of the larger solutions are vendor specific and don't solve the problem in a generic way, see AWS[1] or Vault[2][3]. I've been discussing the problem-space with the Godaddy External Secret maintainers and they seem a bit burnt-out. There is work on standardization here https://github.com/godaddy/kubernetes-external-secrets/pull/477 https://github.com/godaddy/kubernetes-external-secrets/pull/..., but this more covers creating Kubernetes Secrets from external sources, work still remains around a generic pod injector solution. A few of us have started work on what the implementation of this would look like over at https://github.com/itscontained/secret-manager https://github.com/itscontained/secret-manager. [1] https://github.com/mumoshu/aws-secret-operator https://github.com/mumoshu/aws-secret-operator [2] https://github.com/hashicorp/vault-k8s https://github.com/hashicorp/vault-k8s [3] https://banzaicloud.com/blog/inject-secrets-into-pods-vault-revisited/ https://banzaicloud.com/blog/inject-secrets-into-pods-vault-...

68 points by stefanjudis 6y ago

wdb 6y ago

Interesting, wondering if it's portable to make work with Google Secrets Manager

tecleandor 6y ago
Seems like the article author doesn't like it much, but GoDaddy's Kubernetes External Secrets seems to support Google Secrets Manager: https://github.com/godaddy/kubernetes-external-secrets https://github.com/godaddy/kubernetes-external-secrets
riccardomc 6y ago
Please, take a also a look at: https://github.com/ContainerSolutions/externalsecret-operator https://github.com/ContainerSolutions/externalsecret-operato...

gouggoug 6y ago

It's worth noting that the "Kubernetes External Secrets"[0] project from Godaddy is now supplanted by "Secret-Manager"[1]. I've been using Secret-Manager and it works very well. The authors of "kube-secret-syncer" mention "[other solutions] lack either in security, caching or flexibility". When it comes to "secret-manager", although I can not vouch for its security, the codebase is very small and probably easily auditable. It's also very flexible. It supports "SecretStores", currently AWS, GCP and Vault out of the box, and it's easy to add more. Not sure why "caching" is mentioned in the mix. I'm surprised they decided to re-invent the wheel instead of improving secret-manager. [0]: https://github.com/godaddy/kubernetes-external-secrets https://github.com/godaddy/kubernetes-external-secrets [1]: https://github.com/itscontained/secret-manager https://github.com/itscontained/secret-manager

tecleandor 6y ago
Why is it supplanted? Is it a fork? I still see commits on GoDaddy's repository. Secret-Manager docs are, ahem, limited.
- gouggoug 6y ago
  I could be wrong; I had originally started using "external-secrets" then I believe found about "secret-manager" from the Godaddy repository. I've used both solutions, and ultimately, I think itscontained/secret-manager is better than external-secrets. Their doc was re-jiggled a few days ago and I agree its made it look like it's inexistant. There's not a _ton_ of it, but it's there[0][1] [0]https://github.com/itscontained/secret-manager/tree/master/docs https://github.com/itscontained/secret-manager/tree/master/d... [1]https://github.com/itscontained/secret-manager/blob/master/docs/usage.md https://github.com/itscontained/secret-manager/blob/master/d... edit: found the link in the Godaddy repo to "secret-manager": https://github.com/godaddy/kubernetes-external-secrets/issues/423#issuecomment-677749973 https://github.com/godaddy/kubernetes-external-secrets/issue... I was mistaken when I said "secret-manager supplanted external-secrets". It's a Golang rewrite from a user.
  - tecleandor 6y ago
    Oh, I see!
  - riccardomc 6y ago
    Indeed is not supplanted, it is just another solution. Speaking of which, I would like to put our solution on your radar as well: https://github.com/ContainerSolutions/externalsecret-operator https://github.com/ContainerSolutions/externalsecret-operato...
  - Keunic 6y ago
    Yeah, we should probably include in Github readme, but the itscontained/secret-manager is an implementation of the standardization that a few of us have been working on in https://github.com/godaddy/kubernetes-external-secrets/pull/477 https://github.com/godaddy/kubernetes-external-secrets/pull/.... There has been talks of moving one of these solutions to https://github.com/external-secrets https://github.com/external-secrets ownership, but nothing has happened around that yet.
krthkv 6y ago
Hey there, I work on the infra team at Contentful and wanted to expand on the caching. Polling AWS Secrets Manager often can incur considerable costs since it is priced by API calls. We've tried to alleviate this by caching the list of secrets and their values in the process.

nuker 6y ago

Just switch from kube to ECS already, if you're in AWS. Much better integration and support.

captn3m0 6y ago
Really? I'd found even simple things like volume mounted secrets a pain to use. https://github.com/aws/containers-roadmap/issues/56 https://github.com/aws/containers-roadmap/issues/56
- nuker 6y ago
  Secrets to be fetched by CFN of the Service. "define a config file as a "volume" and mount into the container" is very unusual. Store it in S3, and give your Task's IAM Role permission to fetch it.
  - dtech 6y ago
    > define a config file as a "volume" and mount into the container That is how kubernetes secrets work so I wouldn't call it unusual
    - nuker 6y ago
      > That is how kubernetes secrets work As I said, try ECS.
    - captn3m0 6y ago
      And so do so many container based applications that expect a secrets to be present in a file. Kubernetes provides an easy-to-use abstraction for the same, which ECS does not.
      - nuker 6y ago
        > which ECS does not. It actually does. You may, if you wish, have a volume and mount it is ECS tasks [0][1]. The issue above does not seem legit. [0] https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_data_volumes.html https://docs.aws.amazon.com/AmazonECS/latest/developerguide/... [1] https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ecs-taskdefinition.html#cfn-ecs-taskdefinition-volumes https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui...

hurricaneSlider 6y ago

If you're looking for a gitops alternative that can make use of AWS KMS, Azure KeyVault and Google Cloud KMS, would highly recommend kamus (https://kamus.soluto.io/ https://kamus.soluto.io/). Allows secrets to be safely managed in source control and also has an AES mode ideal for local development.

riccardomc 6y ago

I am the founder of a very similar project that supports both AWS Secrets Manager and Google Secrets Manager which actually predates this and GoDaddy's solutions[1]. The proliferation of these type of projects clearly shows the need for secret handling. While I think that more solutions for the same problem is not a bad thing, I also believe that we could benefit from a coordinated effort. My colleagues are actively working with GoDaddy's maintainers to find a common way forward by standardizing the "ExternalSecret" CRD and eventually merging the projects[2]. [1]https://github.com/ContainerSolutions/externalsecret-operator https://github.com/ContainerSolutions/externalsecret-operato... [2]https://github.com/godaddy/kubernetes-external-secrets/issues/47 https://github.com/godaddy/kubernetes-external-secrets/issue...

peterwwillis 6y ago
This is why we need standard protocols and data formats for these common systems. If there were just a "secrets protocol" or "secrets data format", any program could just implement it for input and output, along with a standard interface to perform the actions (a single URL for example). It used to be commonplace to just write an RFC for what you were doing and then other people would use the RFC. Which wasn't perfect, but nothing ever is... Instead, the standard solution today is custom integration, which leads to a lot of reinventing the wheel and incompatibility with extremely similar products.
- riccardomc 6y ago
  I will run the risk of sounding condescending here, but I sense some negativity in your comment that I fail to find a justification for. People need solutions to their problems and they develop them asynchronously and in isolation from each other. Turns out that some problems are more universal than others and could benefit from a common effort. Suddenly, solutions collide and collaboration happens. What you describe is exactly what they are trying to do now: https://github.com/godaddy/kubernetes-external-secrets/pull/477 https://github.com/godaddy/kubernetes-external-secrets/pull/... So, rejoice! The magic of open source and internet enabled collaboration is happening right before your eyes! :)

thehermit 6y ago

I'm looking into this myself and ran across what looks to be a more "official" tool for this [1]. Kubernetes is a fast moving ecosystem and using random OSS operators has bit us in the past. [1] https://github.com/kubernetes-sigs/secrets-store-csi-driver https://github.com/kubernetes-sigs/secrets-store-csi-driver

deleted 6y ago

[deleted]

acamillo 6y ago

I wrote this[1] operator over one year ago to learn some Go lang. The project is a Kubernetes operator that automatically creates and updates Kubernetes secrets according to what is stored in AWS Secrets Manager (SM). A custom resource, named AWSSecret, maps an AWS SM entry to a K8S Secret resource. [1] https://github.com/acamillo/aws-secret-operator https://github.com/acamillo/aws-secret-operator

cbanek 6y ago

Kind of a different problem, but I've had really good experience with using Hashicorp's vault, which is excellent, paired with the vault-secrets-operator for kubernetes to do my secrets management. It will sync secrets from a vault path and create a kubernetes secret that you can use like any other secret. At least this way I feel like there's less lockin to a cloud provider (and some of the places I run this have on-prem kubernetes, so I have to have something that works outside of the cloud, and sometimes without internet). https://www.vaultproject.io/ https://www.vaultproject.io/ https://github.com/ricoberger/vault-secrets-operator https://github.com/ricoberger/vault-secrets-operator

Keunic 6y ago

The entire Kubernetes secret space is a bit immature with no standard solutions. Many of the larger solutions are vendor specific and don't solve the problem in a generic way, see AWS[1] or Vault[2][3]. I've been discussing the problem-space with the Godaddy External Secret maintainers and they seem a bit burnt-out. There is work on standardization here https://github.com/godaddy/kubernetes-external-secrets/pull/477 https://github.com/godaddy/kubernetes-external-secrets/pull/..., but this more covers creating Kubernetes Secrets from external sources, work still remains around a generic pod injector solution. A few of us have started work on what the implementation of this would look like over at https://github.com/itscontained/secret-manager https://github.com/itscontained/secret-manager. [1] https://github.com/mumoshu/aws-secret-operator https://github.com/mumoshu/aws-secret-operator [2] https://github.com/hashicorp/vault-k8s https://github.com/hashicorp/vault-k8s [3] https://banzaicloud.com/blog/inject-secrets-into-pods-vault-revisited/ https://banzaicloud.com/blog/inject-secrets-into-pods-vault-...