6 ms·

AWS Secrets Manager Agent

112 points by plurby 2y ago

slaughtr 2y ago
This seems like quite a lot of setup and hassle for what could be handled some other way with less fuss, like chamber[0] or Doppler[1]. Heck, even the classic .env seems like a better choice in every way. What are the advantages to a configuration like this? Seems the HTTP interface with non-encrypted cache and separate agent situation isn’t something secure enough to satisfy most companies these days. [0] https://github.com/segmentio/chamber https://github.com/segmentio/chamber [1] https://www.doppler.com/ https://www.doppler.com/
- mac-chaffee 2y ago
  The use-case seems to be intentionally narrow: > The Secrets Manager Agent provides compatibility for legacy applications that access secrets through an existing agent or that need caching for languages not supported through other solutions.
- gurchik 2y ago
  I think the audience for this is someone who is already using AWS Secrets Manager, but wants to reduce their API usage (perhaps due to cost). Chamber uses SSM Parameter Store, which for many cases is similar, but some people might have a preference for Secrets Manager. For example, a team might like the automatic RDS password rotation for Secrets Manager and decide to put everything there for consistency. For Doppler, well maybe someone doesn't want to pay for it, or they'd rather control access to their secrets via IAM instead of through a separate tool.
  - drodgers 2y ago
    Chamber can also use S3 + KMS as a backend, which reduces the API costs to ~0 and massively improves the scalability (since SSM has annoyingly low rate limits, or at least it did a few years ago when we last tried it).
  - SamuelAdams 2y ago
    Yes, we use something similar for debugging lambdas locally. We use Dotnet, and this library: https://github.com/Kralizek/AWSSecretsManagerConfigurationExtensions https://github.com/Kralizek/AWSSecretsManagerConfigurationEx... Normally Boto uses the current account context to get secrets, but if we run a lambda as a local build, it uses this library to pull secrets from the actual dev AWS account. This makes it easier to onboard new developers, reduces problems of figuring out what secrets to get for each lambda, etc. Also if secrets are rotated in dev, local stacks get them automatically. I am curious to see if this tool is remarkably different.
  - banku_brougham 2y ago
    Its no joke that AWS Secrets Manager calls add up. At my medium-size US web company, for our data lake account last month, KMS is the second highest line item after s3 service cost. S3 at 94% of total, KMS at 4% of total with Tax and Kinesis the remaining sizable components.
- globular-toast 2y ago
  I was going to say you can rotate secrets in secrets manager without redeploying all your services. But this caches the secrets so you'll still get stale results for up to 5 minutes by default. Not sure what the point is then.
- lukeschlather 2y ago
  This sounds an awful lot like an internal Amazon tool that predates AWS secret manager. It was actually really nice to use; the advantage comes if you always can rely on the daemon being available and you can just say "these machines have access to this secret." If you had to set up and configure the VM, maybe pointless, but it's intended for situations where you're deploying 1000s of VMs with many teams and some centralized team is preparing the machine images you're using.
- ak217 2y ago
  > even the classic .env seems like a better choice in every way That's a pretty thorough misunderstanding of the value that secrets management services provide. We can start with the idea of never storing secrets in files. I think most companies also understand the difference between plain HTTP localhost loopback and transmitting secrets in plaintext over the network. There are many services that rely on localhost loopbacks for handling all kinds of sensitive data. Chamber is great but generally relies on transmitting secrets via environment variables to the enclosed process and assumes that they will remain valid for the lifetime of that process. Part of the point of this tool is to provide a secrets cache with a TTL.
420official 2y ago
This is really cool, I've been running something similar to simplify rotating database credentials for legacy projects.
WatchDog 2y ago
So the point of this is just to cache secrets, to avoid caching them in your app memory? Seems like kinda a niche threat model, if your app is already compromised to the point where it's secret cache can be read, it seems likely that the attacker could also pivot to just read from the cache, or use the instance credentials to read from secrets manager itself.
- deleted 2y ago
  [deleted]
- bruce343434 2y ago
  No, the point is to get sensitive data out of the env variables, which nowadays get stored in plaintext in an .env file or similar. This is a solution for storing and retrieving secrets using AWS credentials. Essentially an online password manager for your application.
  - kriops 2y ago
    What about the credentials used to access AWS credentials? I think there's a good case for centralised credentials where they are shared across applications, though I would seriously question the need to share them across applications. But what you're achieving here as far as I can tell is just making secret retrieval more convoluted (for both devs and hypothetical attackers). Not to beat the dead horse, but obscurity != security.
    - mixxit 2y ago
      Ah that's where we have the Credentials for AWS Credentials Service Agent Just simply pass it a credential and it will provide you the necessary credentials to access the Credentials for AWS Credentials Service
      - bruce343434 2y ago
        [flagged]
      - 2y ago
wrs 2y ago
FYI, there is an AWS-provided Lambda layer similar in principle to this, also including access to Parameter Store. https://aws.amazon.com/blogs/compute/using-the-aws-parameter-and-secrets-lambda-extension-to-cache-parameters-and-secrets/ https://aws.amazon.com/blogs/compute/using-the-aws-parameter...
webprofusion 2y ago
So a bit like Hashicorp Vault (in that it has a locally accessed secrets store) but backed by AWS Secrets Manager.
Sparkyte 2y ago
I got to use secrets manager a while back it was a breath of fresh air as it was all of those things you seeking in vault without all of the problems of it being hashicorp. No offense hashicorp. I rather blame AWS than a self-managed solution.
- Salgat 2y ago
  The auth alone makes it so much simpler. We initially were going to setup a self-hosted vault and setup all the auth to integrate into our EC2s and on a whim I spent a few hours setting it all up with AWS Secrets Manager with implicit auth through an IAM role attached to the EC2s and it was dead simple and done. Best part is, I don't have to care how AWS Secrets Manager is hosted and my services don't care how to authenticate against it, it's all implicit through a simple api.
  - Sparkyte 2y ago
    Yep delegating it all to IAM is another huge win.
lijok 2y ago
I'm going to say this as nicely as I can. Secrets Manager can fuck right off with their $.50/mo/secret pricing. Moved all our secrets to S3 a long time ago and haven't looked back.
- syrgian 2y ago
  If you don't need the granularity, you can store all the credentials that will be used by a specific caller(s) in a single JSON object and it will cost you only those $0.50. You can easily fit a thousand, maximum size is 64kb.
- perpil 2y ago
  DynamoDB also makes for a nice fine grained secrets manager with their new table resource policies: https://speedrun.nobackspacecrew.com/blog/2024/06/27/using-dynamodb-as-a-secrets-manager.html https://speedrun.nobackspacecrew.com/blog/2024/06/27/using-d...
- trallnag 2y ago
  I use Parameter Store instead
perryizgr8 2y ago
How is this different from calling Secrets Manager directly? The only benefit I can think of is caching. So your secrets can be fetched a bit faster. But that is such a niche use-case, and you can easily cache it yourself if you need to.
- rfoo 2y ago
  Sometimes you just want a daemon to fetch secrets / config files containing secret for whatever code you don't own. For example you spin up nginx and setup HTTP basic auth quickly and don't bother writing your own script to periodically update user list from SSM.
- rirze 2y ago
  apparently it has to do with pricing per API call.
  - Salgat 2y ago
    AWS directly contacted us to warn us about pricing because we were pulling secrets so much across all our deployments. Caching is definitely important for that reason alone.
micahbule 2y ago
One particular use case that I might try this for is for (very) restrictive environments. One such case was with my previous work where we had to develop services for the client but we can only do it in a remote desktop with certain network and application restrictions. Instead of having conditions for the environment to load certain config, we can simply retrieve the secrets stored in AWS (ex. RDS credentials) via the agent.
gtirloni 2y ago
This should come in handy with SOPS and git log.
thedougd 2y ago
What I really want is a consul-template for AWS Secrets Manager. As I wrote this I googled and found a plugin: https://github.com/chrissav/consul-template-plugin-secretsmanager https://github.com/chrissav/consul-template-plugin-secretsma... I didn't realize consul-template supported plugins.
derefr 2y ago
Why are all the various "secrets vault" approaches so splintered and proprietary, anyway? Why is there a separate tool I have to install for: • AWS secrets, GCP secrets, Azure secrets... each has its own API • secrets in a HashiCorp Vault install • secrets from whatever cloud password manager • "ambient" secrets from env-vars, or the local .netrc, or the local macOS Keychain • k8s Secrets resources (when you're a k8s CRD controller) • secrets stored in SOPS files, in turn encrypted by keys held in any of the above Why haven't we seen a generic "secrets client" library, with pluggable adapters for handling all of these cases through the same library API / CLI tooling? Or better yet, why not a generic stub secrets client, that speaks to an also-generic "caching middleware proxy" like this AWS one — where the proxy has the pluggable backend adapters + connection config for them?
- karmajunkie 2y ago
  at least in kubernetes-land, external-secrets.io provides this.
  - dayjah 2y ago
    external-secrets really is great! Pointing this out here, because big evil companies generally don’t get praise when it’s due: godaddy built this!
- cbsmith 2y ago
  The stub secrets client is just a key->value API, so the value of a proxy is pretty limited. It's not a hard enough problem that anyone is interested in having a separate product for it.
  - derefr 2y ago
    The point of the proxy, is that it would talk to these fifteen different backends and convert them into a generic key-value API. And also, as with the AWS solution above, do TTL-based cache refresh of the secrets, cache-invalidation when it loses connection to the backend, etc. Also, the "stub" client wouldn't really be a stub, as all the "ambient environment" secrets adapters would necessarily be local to the client rather than to the proxy. The client library would be a bit like using dnsmasq(1) as a local "stub" DNS resolver — where it reads your /etc/hosts and so forth, but for most things is deferring to a configured upstream DNS server.
symlinkk 2y ago
Who cares? People are only upvoting this because it’s written in Rust. The actual tool seems useless
shironandonon_ 2y ago
this feels more like Azure Secrets which has been a superior product.
SunnyW 2y ago
For senior developers who are ready to write code, integrating the appropriate AWS SDK library for your programming language and writing a few lines of code might seem straightforward, and may not take more than half a day. However, consider a large company with thousands of applications—like in my case—where this effort is multiplied a thousandfold. Moreover, these applications are developed in over 10 different languages, some of which may not even have an available AWS SDK. Therefore, using an agent that simplifies these operations into a single HTTP call to a sidecar service truly adds value. Another consideration is operation; imagine that there are 10 different libraries maintained for this purpose, and if there is a new feature, say, you need all logs going to one place, making sure it is available in all languages would require a team with different programming skills to do so. Secrets agent, being language agnostic, you only need to change at one place, and someone else may have already done it for it or ready to do it, as it is open source project. When it comes to cost saving, imagine scenarios where a junior developer improperly implements secret retrieval in a Lambda function, with retrieval occurring at every function invocation and each function handling 100 transactions per second. Such a single oversight can cost $1,000 a month, and if left unnoticed for a year—a common occurrence when the function appears to work—people often overlook further scrutiny as long as it functions.

112 points by plurby 2y ago

This seems like quite a lot of setup and hassle for what could be handled some other way with less fuss, like chamber[0] or Doppler[1]. Heck, even the classic .env seems like a better choice in every way. What are the advantages to a configuration like this? Seems the HTTP interface with non-encrypted cache and separate agent situation isn’t something secure enough to satisfy most companies these days. [0] https://github.com/segmentio/chamber https://github.com/segmentio/chamber [1] https://www.doppler.com/ https://www.doppler.com/

mac-chaffee 2y ago
The use-case seems to be intentionally narrow: > The Secrets Manager Agent provides compatibility for legacy applications that access secrets through an existing agent or that need caching for languages not supported through other solutions.
gurchik 2y ago
I think the audience for this is someone who is already using AWS Secrets Manager, but wants to reduce their API usage (perhaps due to cost). Chamber uses SSM Parameter Store, which for many cases is similar, but some people might have a preference for Secrets Manager. For example, a team might like the automatic RDS password rotation for Secrets Manager and decide to put everything there for consistency. For Doppler, well maybe someone doesn't want to pay for it, or they'd rather control access to their secrets via IAM instead of through a separate tool.
- drodgers 2y ago
  Chamber can also use S3 + KMS as a backend, which reduces the API costs to ~0 and massively improves the scalability (since SSM has annoyingly low rate limits, or at least it did a few years ago when we last tried it).
- SamuelAdams 2y ago
  Yes, we use something similar for debugging lambdas locally. We use Dotnet, and this library: https://github.com/Kralizek/AWSSecretsManagerConfigurationExtensions https://github.com/Kralizek/AWSSecretsManagerConfigurationEx... Normally Boto uses the current account context to get secrets, but if we run a lambda as a local build, it uses this library to pull secrets from the actual dev AWS account. This makes it easier to onboard new developers, reduces problems of figuring out what secrets to get for each lambda, etc. Also if secrets are rotated in dev, local stacks get them automatically. I am curious to see if this tool is remarkably different.
- banku_brougham 2y ago
  Its no joke that AWS Secrets Manager calls add up. At my medium-size US web company, for our data lake account last month, KMS is the second highest line item after s3 service cost. S3 at 94% of total, KMS at 4% of total with Tax and Kinesis the remaining sizable components.
globular-toast 2y ago
I was going to say you can rotate secrets in secrets manager without redeploying all your services. But this caches the secrets so you'll still get stale results for up to 5 minutes by default. Not sure what the point is then.
lukeschlather 2y ago
This sounds an awful lot like an internal Amazon tool that predates AWS secret manager. It was actually really nice to use; the advantage comes if you always can rely on the daemon being available and you can just say "these machines have access to this secret." If you had to set up and configure the VM, maybe pointless, but it's intended for situations where you're deploying 1000s of VMs with many teams and some centralized team is preparing the machine images you're using.
ak217 2y ago
> even the classic .env seems like a better choice in every way That's a pretty thorough misunderstanding of the value that secrets management services provide. We can start with the idea of never storing secrets in files. I think most companies also understand the difference between plain HTTP localhost loopback and transmitting secrets in plaintext over the network. There are many services that rely on localhost loopbacks for handling all kinds of sensitive data. Chamber is great but generally relies on transmitting secrets via environment variables to the enclosed process and assumes that they will remain valid for the lifetime of that process. Part of the point of this tool is to provide a secrets cache with a TTL.

420official 2y ago

This is really cool, I've been running something similar to simplify rotating database credentials for legacy projects.

WatchDog 2y ago

So the point of this is just to cache secrets, to avoid caching them in your app memory? Seems like kinda a niche threat model, if your app is already compromised to the point where it's secret cache can be read, it seems likely that the attacker could also pivot to just read from the cache, or use the instance credentials to read from secrets manager itself.

deleted 2y ago
[deleted]
bruce343434 2y ago
No, the point is to get sensitive data out of the env variables, which nowadays get stored in plaintext in an .env file or similar. This is a solution for storing and retrieving secrets using AWS credentials. Essentially an online password manager for your application.
- kriops 2y ago
  What about the credentials used to access AWS credentials? I think there's a good case for centralised credentials where they are shared across applications, though I would seriously question the need to share them across applications. But what you're achieving here as far as I can tell is just making secret retrieval more convoluted (for both devs and hypothetical attackers). Not to beat the dead horse, but obscurity != security.
  - mixxit 2y ago
    Ah that's where we have the Credentials for AWS Credentials Service Agent Just simply pass it a credential and it will provide you the necessary credentials to access the Credentials for AWS Credentials Service
    - bruce343434 2y ago
      [flagged]
    - 2y ago

wrs 2y ago

FYI, there is an AWS-provided Lambda layer similar in principle to this, also including access to Parameter Store. https://aws.amazon.com/blogs/compute/using-the-aws-parameter-and-secrets-lambda-extension-to-cache-parameters-and-secrets/ https://aws.amazon.com/blogs/compute/using-the-aws-parameter...

webprofusion 2y ago

So a bit like Hashicorp Vault (in that it has a locally accessed secrets store) but backed by AWS Secrets Manager.

Sparkyte 2y ago

I got to use secrets manager a while back it was a breath of fresh air as it was all of those things you seeking in vault without all of the problems of it being hashicorp. No offense hashicorp. I rather blame AWS than a self-managed solution.

Salgat 2y ago
The auth alone makes it so much simpler. We initially were going to setup a self-hosted vault and setup all the auth to integrate into our EC2s and on a whim I spent a few hours setting it all up with AWS Secrets Manager with implicit auth through an IAM role attached to the EC2s and it was dead simple and done. Best part is, I don't have to care how AWS Secrets Manager is hosted and my services don't care how to authenticate against it, it's all implicit through a simple api.
- Sparkyte 2y ago
  Yep delegating it all to IAM is another huge win.

lijok 2y ago

I'm going to say this as nicely as I can. Secrets Manager can fuck right off with their $.50/mo/secret pricing. Moved all our secrets to S3 a long time ago and haven't looked back.

syrgian 2y ago
If you don't need the granularity, you can store all the credentials that will be used by a specific caller(s) in a single JSON object and it will cost you only those $0.50. You can easily fit a thousand, maximum size is 64kb.
perpil 2y ago
DynamoDB also makes for a nice fine grained secrets manager with their new table resource policies: https://speedrun.nobackspacecrew.com/blog/2024/06/27/using-dynamodb-as-a-secrets-manager.html https://speedrun.nobackspacecrew.com/blog/2024/06/27/using-d...
trallnag 2y ago
I use Parameter Store instead

perryizgr8 2y ago

How is this different from calling Secrets Manager directly? The only benefit I can think of is caching. So your secrets can be fetched a bit faster. But that is such a niche use-case, and you can easily cache it yourself if you need to.

rfoo 2y ago
Sometimes you just want a daemon to fetch secrets / config files containing secret for whatever code you don't own. For example you spin up nginx and setup HTTP basic auth quickly and don't bother writing your own script to periodically update user list from SSM.
rirze 2y ago
apparently it has to do with pricing per API call.
- Salgat 2y ago
  AWS directly contacted us to warn us about pricing because we were pulling secrets so much across all our deployments. Caching is definitely important for that reason alone.

micahbule 2y ago

One particular use case that I might try this for is for (very) restrictive environments. One such case was with my previous work where we had to develop services for the client but we can only do it in a remote desktop with certain network and application restrictions. Instead of having conditions for the environment to load certain config, we can simply retrieve the secrets stored in AWS (ex. RDS credentials) via the agent.

gtirloni 2y ago

This should come in handy with SOPS and git log.

thedougd 2y ago

What I really want is a consul-template for AWS Secrets Manager. As I wrote this I googled and found a plugin: https://github.com/chrissav/consul-template-plugin-secretsmanager https://github.com/chrissav/consul-template-plugin-secretsma... I didn't realize consul-template supported plugins.

derefr 2y ago

Why are all the various "secrets vault" approaches so splintered and proprietary, anyway? Why is there a separate tool I have to install for: • AWS secrets, GCP secrets, Azure secrets... each has its own API • secrets in a HashiCorp Vault install • secrets from whatever cloud password manager • "ambient" secrets from env-vars, or the local .netrc, or the local macOS Keychain • k8s Secrets resources (when you're a k8s CRD controller) • secrets stored in SOPS files, in turn encrypted by keys held in any of the above Why haven't we seen a generic "secrets client" library, with pluggable adapters for handling all of these cases through the same library API / CLI tooling? Or better yet, why not a generic stub secrets client, that speaks to an also-generic "caching middleware proxy" like this AWS one — where the proxy has the pluggable backend adapters + connection config for them?

karmajunkie 2y ago
at least in kubernetes-land, external-secrets.io provides this.
- dayjah 2y ago
  external-secrets really is great! Pointing this out here, because big evil companies generally don’t get praise when it’s due: godaddy built this!
cbsmith 2y ago
The stub secrets client is just a key->value API, so the value of a proxy is pretty limited. It's not a hard enough problem that anyone is interested in having a separate product for it.
- derefr 2y ago
  The point of the proxy, is that it would talk to these fifteen different backends and convert them into a generic key-value API. And also, as with the AWS solution above, do TTL-based cache refresh of the secrets, cache-invalidation when it loses connection to the backend, etc. Also, the "stub" client wouldn't really be a stub, as all the "ambient environment" secrets adapters would necessarily be local to the client rather than to the proxy. The client library would be a bit like using dnsmasq(1) as a local "stub" DNS resolver — where it reads your /etc/hosts and so forth, but for most things is deferring to a configured upstream DNS server.

symlinkk 2y ago

Who cares? People are only upvoting this because it’s written in Rust. The actual tool seems useless

shironandonon_ 2y ago

this feels more like Azure Secrets which has been a superior product.

SunnyW 2y ago

For senior developers who are ready to write code, integrating the appropriate AWS SDK library for your programming language and writing a few lines of code might seem straightforward, and may not take more than half a day. However, consider a large company with thousands of applications—like in my case—where this effort is multiplied a thousandfold. Moreover, these applications are developed in over 10 different languages, some of which may not even have an available AWS SDK. Therefore, using an agent that simplifies these operations into a single HTTP call to a sidecar service truly adds value. Another consideration is operation; imagine that there are 10 different libraries maintained for this purpose, and if there is a new feature, say, you need all logs going to one place, making sure it is available in all languages would require a team with different programming skills to do so. Secrets agent, being language agnostic, you only need to change at one place, and someone else may have already done it for it or ready to do it, as it is open source project. When it comes to cost saving, imagine scenarios where a junior developer improperly implements secret retrieval in a Lambda function, with retrieval occurring at every function invocation and each function handling 100 transactions per second. Such a single oversight can cost $1,000 a month, and if left unnoticed for a year—a common occurrence when the function appears to work—people often overlook further scrutiny as long as it functions.