4 ms·

Why are all the various "secrets vault" approaches so splintered and proprietary, anyway? Why is there a separate tool I have to install for: • AWS secrets, GC

by derefr 2y ago

Why are all the various "secrets vault" approaches so splintered and proprietary, anyway? Why is there a separate tool I have to install for: • AWS secrets, GCP secrets, Azure secrets... each has its own API • secrets in a HashiCorp Vault install • secrets from whatever cloud password manager • "ambient" secrets from env-vars, or the local .netrc, or the local macOS Keychain • k8s Secrets resources (when you're a k8s CRD controller) • secrets stored in SOPS files, in turn encrypted by keys held in any of the above Why haven't we seen a generic "secrets client" library, with pluggable adapters for handling all of these cases through the same library API / CLI tooling? Or better yet, why not a generic stub secrets client, that speaks to an also-generic "caching middleware proxy" like this AWS one — where the proxy has the pluggable backend adapters + connection config for them?

karmajunkie 2y ago
at least in kubernetes-land, external-secrets.io provides this.
- dayjah 2y ago
  external-secrets really is great! Pointing this out here, because big evil companies generally don’t get praise when it’s due: godaddy built this!
cbsmith 2y ago
The stub secrets client is just a key->value API, so the value of a proxy is pretty limited. It's not a hard enough problem that anyone is interested in having a separate product for it.
- derefr 2y ago
  The point of the proxy, is that it would talk to these fifteen different backends and convert them into a generic key-value API. And also, as with the AWS solution above, do TTL-based cache refresh of the secrets, cache-invalidation when it loses connection to the backend, etc. Also, the "stub" client wouldn't really be a stub, as all the "ambient environment" secrets adapters would necessarily be local to the client rather than to the proxy. The client library would be a bit like using dnsmasq(1) as a local "stub" DNS resolver — where it reads your /etc/hosts and so forth, but for most things is deferring to a configured upstream DNS server.
  - abadpoli 2y ago
    The extent of the “conversion” required would pretty much just be taking one form of JSON output and transforming it into a different JSON output, which is pretty easy to do in a few lines of Python or a single jq command. It would likely be more work and hassle to have to install and manage a secrets proxy, rather than just writing a few transformation lines, or better yet just using the SDK of the service you’re using. Even this secrets manager proxy that the OP is about is explicitly to be used in legacy situations where you can’t use the AWS SDK, which is preferred because it does all of the stuff you mentioned for you. However, this is Hacker News! If you think you see a problem that can be solved that other people don’t, why don’t you build it?
  - cbsmith 2y ago
    I mean, that's kind of what KeyWhiz does, no?
throw1230 2y ago
Because a security vulnerability in the common library will have a much larger impact. It also increases the potential attack surface by adding more components. Companies value the secrets they keep and want to make sure they have 100% vertical control, where they can audit everything. Also, at any project with a sane architecture, you're using 1 vault and maybe 1-2 ambient strategies to pass the data. You won't use all the vaults at the same time anyway
- derefr 2y ago
  > Also, at any project with a sane architecture, you're using 1 vault and maybe 1-2 ambient strategies to pass the data. You won't use all the vaults at the same time anyway You're assuming the secrets here are managed by infra+glue added by a DevOps team when deploying an app. I'm talking about use-cases where the secret-handling is designed into e.g. a cluster-scale deployable virtual appliance, where you configure the app through its UI or deployment-time config files to access your "secrets provider" of choice. (Think "deployable PaaS.")
pxc 2y ago
> secrets stored in SOPS files, in turn encrypted by keys held in any of the above SOPS does already work this way, right? You don't have to use local GPG keys or whatever with SOPS, you can use keys from AWS KMS or stored in HashiCorp Vault or whatever
antihero 2y ago
I think 1password can be used for aws at least
throwaway8r8te 2y ago
> Why haven't we seen a generic "secrets client" library, with pluggable adapters "Spring" has that. You can define a property that can be populated from pluggable sources, like vaults, yaml, environment variables and others. You just add an annotation @Value("${foo.bar}") to a field or constructor parameter, and it will be filled from the appropriate source automatically.

by derefr 2y ago

karmajunkie 2y ago

at least in kubernetes-land, external-secrets.io provides this.

dayjah 2y ago
external-secrets really is great! Pointing this out here, because big evil companies generally don’t get praise when it’s due: godaddy built this!

cbsmith 2y ago

The stub secrets client is just a key->value API, so the value of a proxy is pretty limited. It's not a hard enough problem that anyone is interested in having a separate product for it.

derefr 2y ago
The point of the proxy, is that it would talk to these fifteen different backends and convert them into a generic key-value API. And also, as with the AWS solution above, do TTL-based cache refresh of the secrets, cache-invalidation when it loses connection to the backend, etc. Also, the "stub" client wouldn't really be a stub, as all the "ambient environment" secrets adapters would necessarily be local to the client rather than to the proxy. The client library would be a bit like using dnsmasq(1) as a local "stub" DNS resolver — where it reads your /etc/hosts and so forth, but for most things is deferring to a configured upstream DNS server.
- abadpoli 2y ago
  The extent of the “conversion” required would pretty much just be taking one form of JSON output and transforming it into a different JSON output, which is pretty easy to do in a few lines of Python or a single jq command. It would likely be more work and hassle to have to install and manage a secrets proxy, rather than just writing a few transformation lines, or better yet just using the SDK of the service you’re using. Even this secrets manager proxy that the OP is about is explicitly to be used in legacy situations where you can’t use the AWS SDK, which is preferred because it does all of the stuff you mentioned for you. However, this is Hacker News! If you think you see a problem that can be solved that other people don’t, why don’t you build it?
- cbsmith 2y ago
  I mean, that's kind of what KeyWhiz does, no?

throw1230 2y ago

Because a security vulnerability in the common library will have a much larger impact. It also increases the potential attack surface by adding more components. Companies value the secrets they keep and want to make sure they have 100% vertical control, where they can audit everything. Also, at any project with a sane architecture, you're using 1 vault and maybe 1-2 ambient strategies to pass the data. You won't use all the vaults at the same time anyway

derefr 2y ago
> Also, at any project with a sane architecture, you're using 1 vault and maybe 1-2 ambient strategies to pass the data. You won't use all the vaults at the same time anyway You're assuming the secrets here are managed by infra+glue added by a DevOps team when deploying an app. I'm talking about use-cases where the secret-handling is designed into e.g. a cluster-scale deployable virtual appliance, where you configure the app through its UI or deployment-time config files to access your "secrets provider" of choice. (Think "deployable PaaS.")

pxc 2y ago

> secrets stored in SOPS files, in turn encrypted by keys held in any of the above SOPS does already work this way, right? You don't have to use local GPG keys or whatever with SOPS, you can use keys from AWS KMS or stored in HashiCorp Vault or whatever

antihero 2y ago

I think 1password can be used for aws at least

throwaway8r8te 2y ago

> Why haven't we seen a generic "secrets client" library, with pluggable adapters "Spring" has that. You can define a property that can be populated from pluggable sources, like vaults, yaml, environment variables and others. You just add an annotation @Value("${foo.bar}") to a field or constructor parameter, and it will be filled from the appropriate source automatically.