13 ms·
AWS Secrets Manager – Store, Distribute, and Rotate Credentials Securely
- marvinpinto 9y agoI'm still trying to figure out how different this is from their "parameter store" offering in AWS Systems Manager. The main thing I guess is you get more control over key rotation.
- epberry 9y agoBesides the secrets rotation how is this different from EC2 Parameter Store? I’m genuinely curious and will move away from parameter store if this provides some benefits.
- deleted 9y ago[deleted]
- neuronexmachina 9y agoSkimming the blog post, the main difference seems that it allows you to basically store a dict of key/value pairs for each secret. So for example, you could store all the user/pass/host/port for a DB connection as a secret. If I recall correctly, Parameter Store could only store a single SecureString for each secret.
- scrollaway 9y agoParameterStore lets you store String, StringList or SecureString. But there's no limit to SecureString. A SecureString can be `postgres://admin:hunter2@localhost:5432/db`. It can also be `{"username": "admin", "password": "hunter2", "host": "localhost", "port": "5432"}`.
- neuronexmachina 9y agoSure you could do that if you don't mind adding in your own string->dict conversion. I wouldn't be surprised if the internal implementation is near-identical. Secrets Manager seems like a slightly more easy-to-use version of Parameter Store that's also visible, instead of hidden away inside Systems Manager.
- epberry 9y agoIt did take me a minute to actually find Parameter Store. There's so much stuff hidden away in the EC2 menus.
- jknoepfler 9y ago$$$
- chatmasta 9y agoHow is this different than KMS? “Key Management Service” is practically synonymous with the name of this new product, so how exactly do the two differ/interact?
- jvanderhoof 9y agoI believe KMS is primarily for storing encryption keys (to replace an HSM). AWS Secrets Manager looks like it's much easier to integrate into credential best practices, like periodic rotation.
- kondro 9y agoKMS is a shared HSM, managed encryption product. AWS Secrets Manager is for storing credentials to products (e.g. DBs) and have processes for automatically rotating them on a regular basis.
- 0xCMP 9y agoKMS is the service which performs encryption where this stores the secrets using a specified key (provided or from KMS). KMS provides an alternative to an HSM in the cloud where this service is alternative to running a Vault cluster with a backing Consul cluster.
- cavisne 9y agoA key difference is in kms you never know the actual secret (kms stores this on your behalf) . This is for use cases where you need the secret in application code, like db creds. Looks like a cool product I noticed they dont include a revoke workflow though, the trickiest bit of key rotation
- mratzloff 9y agoWell, you can use KMS to do something similar, but without the auto-rotation. 1. Use a KMS key to encrypt a secrets file (obviously, never check this into source control) 2. Store the encrypted secrets file in an S3 bucket 3. Tie a new IAM role with kms:Decrypt and s3:GetObject policies for the relevant resources to your EC2 instance 4. On app start, get the KMS key and secrets file, decrypt, and set environment variables In practice, rotating using this scheme just means creating a new KMS key, re-encrypting the file and pushing the updated copy to S3, and updating the IAM role's kms:Decrypt policy. It's not too bad unless you have a million services.
- whateveracct 9y agoThis looks like an AWS equivalent of the Amazon-internal secret management tool called Odin. Which is very nice because Odin was pretty much universally loved from what I saw.
- josefdlange 9y agoI miss having ODIN.
- deleted 9y ago[deleted]
- valar_m 9y agoOdin works well, it's just a hassle to share credentials with non-developers who don't have a dev desktop and therefore can't access Odin. It's an issue I've encountered several times recently with no clear solution, it seems.
- qxi 9y agoOdin works really well but but only because it integrates so tightly with with the rest of the internal tooling. Without Apollo (another internal tool) to manage and orchestrate the deployment of packages that need Odin's secrets, it's a lot clumsier to use.
- ryanianian 9y agoThere's still a bootstrap issue with AWS Secrets Manager - you have to set up enough tooling to be able to call the API. The killer-app of odin imho was/is the on-machine http server that let all manner of applications very easily get credentials. You could just do something like MY_API_KEY="$(curl http://localhost:5000?key=my-api)" and boom your shell script or whatever was very easily using secrets. The fact that Amazon systems bootstrapped EC2 instances meant they could easily enable this org-wide. The odin back-end then owned all the lifecycles around those secrets including what kinds of hosts they would go to, if/how they would rotate, etc etc. It has its annoyances including the fact you have to use http to localhost which can get saturated if you're not doing things right, but overall it really made secrets-management a non-issue.
- kondro 9y agoAt $0.40 per secret per month and $0.05 per 10,000 requests this is much more expensive than the practically free SSM Parameter Store product, even if you factor in the auto-rotating bits.
- bpicolo 9y agoYeah, I'm surprised they charge per secret. I guess it's a blip when you have big aws spend
- xkjkls 9y agoYeah, I don't think too many people have tens of thousands of secrets, generally. The addition of key rotation is a great benefit.
- 0xCMP 9y agoIt'd be cheaper than running Vault with a backing Consul cluster which also provides rotation and other features. There is a point where Vault is more cost effective, but I believe it'd require a ton of requests and secrets to justify min 6 machines of at least t2.micro that also need to managed and secured.
- eropple 9y agoIt's definitely not cheaper than KMS and DynamoDB via Credstash, though.
- orthecreedence 9y agoWhat if you're already running a nomad/consul cluster? Is vault a particularly hard thing to implement/scale at that point?
- 0xCMP 9y agoYea it that case then it's not that hard or extra cost. But I imagine most people aren't already running Nomad or Consul and can benefit from this. Lambdas, Wordpress, etc. can now get rotating secrets which is pretty nice possibility now with a lot less operational overhead.
- 0xCMP 9y agoThis will be nice for systems and budgets which can't afford a consul+vault cluster to handle this for you.
- scrollaway 9y agoFirst reaction: Holy crap! They finally turned ParameterStore into a proper product! Second reaction: Holy shit that's expensive [for what it does]. ParameterStore is free (minus the KMS component). The only value-add is secret rotation and that's not something that most of the time makes sense to use. [Edit: I'm not advocating for no rotation; see replies] Edit: Had more time to think about it. Someone enlighten me: What's the difference between writing a rotation lambda for this new product, vs. writing a rotation lambda for ParameterStore that you then cron? The pricing really doesn't make sense.
- toomuchtodo 9y ago> The only value-add is secret rotation and that's not something that most of the time makes sense to use. From a security perspective, you should be rotating secrets somewhere between annually and every 90 days, depending on your business/security/compliance requirements and the nature of the data secured by the secret.
- scrollaway 9y agoYou're not necessarily your own source of secret (and even when you are, you don't necessarily have the option to use AWS-sourced rotation). In other words, yes, you should be rotating what you can rotate, but this doesn't always help. Furthermore, it doesn't justify the pricing. It would make sense if this were, say, "Hey, you can now auto-rotate SSM-PS secrets for an extra $0.40/secret". Right now it just seems weird. [Edit: I just saw the custom rotation bit of the article. Cool; but if you're at the point of setting up lambdas for the rotation, you might as well cron a lambda on top of ParameterStore...]
- deleted 9y ago[deleted]
- mlrtime 9y agoDoes it matter? How many secrets do you have where this is even close to your ec2/storage costs?
- Thaxll 9y agoLooks like the equivalent of Vault? Anyone can compare the two?
- thepumpkin1979 9y agoKinda similar to what Hashicorp's Vault does for secret management but hosted.
- weej 9y agoI believe that's what HC's Vault Enterprise is (hosted/SaaS): https://www.hashicorp.com/products/vault https://www.hashicorp.com/products/vault
- ryanSrich 9y agoThere's really no comparison. You could use Vault to build a PKI. You couldn't say the same for a proprietary, vendor locked solution like this.
- jtwaleson 9y agoAs others have noted, the pricing is really the odd thing here. AWS seems to be moving to value based pricing rather than cost based pricing for some of its niche products. I used to think that Cloudwatch metrics were very expensive at $.50 per custom metric per month, but this seems waaayyyy cheaper to store.
- 013a 9y ago> $0.40/secret/month WOWZER. I get having a managed solution is great, but you don't have to store many secrets before running your own Vault server makes sense.
- seanieb 9y agoFTFU - Valut service.... You need more than one server, it'll need to have 100% uptime too, backed up and available in multiple regions. No security team want to own this infrastructure.
- Florin_Andrei 9y agoYeah, it's a minimum of 2 instances per region, and you should also run Consul or some other replicated backend, etc. Been there, done that; it gets complicated pretty quickly. That being said, if the security traffic and/or amount of data is huge, Vault might be cheaper, even with the extra work. I guess it depends on the scenario.
- weej 9y agoNot to mention the Hardware Security Module(s) behind this are very expensive, especially when scaled out for redundancy and availability.
- crazypyro 9y agoAt my work, we have a vault service, but it takes 3 servers for vault plus another 3 for consul and they are not very cheap servers (m4.larges). Our rough calculations, using a 3x3 vault/consul architecture, estimated you'd need over 1100-1300 secrets to make it worth implementing vault (not including development/maintenance cost which could be significant. and if you have multiple, independent environments, it gets worse. The real issue for some of the people at my work is the vendor lock-in versus vault.
- outworlder 9y agoConsul only makes sense if you are already using it for something else. If just Vault, you can use DynamoDB (with full HA) or S3 (without HA support), same with Azure and Google Cloud (although without HA). There are other backends, like PostgreSQL.
- spullara 9y ago$0.40 per secret per month is outrageous.
- pageandrew 9y agoDoes the Secret Rotation for the RDS-integrated credential store actually update the password for the user in the SQL database? If so, thats pretty damn cool.
- ranman 9y agoYes!
- rconti 9y agoOh, I misread it as a secret for Twitter (which, I guess, makes no sense, because tweets are generally public).. and I was wondering how the password got updated on the Twitter side. So I guess, naturally, secrets can only be rotated for AWS services that support it.
- mwarkentin 9y agoYou can write the custom lambdas to rotate 3rd party creds.
- kerng 9y agoAnyone know how this compares to Azure Key Vault?
- spydum 9y agoVault already did secrets. Nothing stops you from writing an azure function to handle rotation, but you gotta build it yourself. AWS has BUILT IN support for RDS. However, the biggest delta is: I'm like 90% sure Azure key vault doesn't have fine-grained access policy per VAULT. That kind of stinks.. you need a vault per role ideally.
- sowbug 9y agoFor those of us living in the service-development stone ages, is the idea that a secret-manager service replaces any number of ad-hoc local secret-storage and configuration mechanisms with a single robust mechanism that takes only a single root credential to retrieve all the individual secrets that your service needs? You do still have to figure out a way to securely provide the root credential to your service so that it can fetch the secrets from the secret manager, correct? Otherwise this would be magic of a kind I think is impossible. If my questions aren't too far off in the weeds, then this service sounds like a personal password manager but for a service rather than a person, though I'm sure AWS's service has finer-grained controls than just the all-or-nothing master passphrase. Similar risks apply: an attacker obtaining the master passphrase is a major issue, losing the master passphrase is devastating (though recoverable here because you probably didn't lose your personal AWS login credentials), and unavailability of the password database is catastrophic. But the usability benefits of having everything in one secure place, behind a service managed by experts, should outweigh those risks. I have more questions about the credential-rotation feature, but this is enough for now.
- lmartel 9y agoThe big missing piece is roles. No service uses a root access key directly. Instead, there's a webserver role with access to a relevant secrets group but no access to data warehouse secrets, for example. Access keys can be provisioned and downloaded straight onto the box from the service. Sure, a compromise is bad, but only exposes the secrets that would be available on the pwned box regardless.
- sowbug 9y agoOK, so "root" wasn't really the right term. I get an X credential so I can be an X, and nobody needs to worry that I also got enough to be any part of a Y. Thanks.
- notoverthere 9y agoThis sort of model also fits nicely with the AWS ecosystem. EC2 instances (virtual machines) can be given an IAM Role when they boot-up. An IAM Role is essentially an automatically generated access key which is unique to that EC2 instance, and has pre-determined permissions. So in other words – a unique key is generated every time a virtual machine is created. It's fully automated, never shared between instances, and never needs to be handled manually. That key will give the virtual machine permission to access other AWS services, in this case the AWS Secrets Manager. So as long as you're using EC2 instances, you won't need to worry about securely passing a 'master password' to your VMs in order for them to access secrets.
- outworlder 9y ago"Not just for secrets" Yay, we can go and store all types of secrets! But not with this price per secret...
- talawahdotnet 9y agoAt $0.40 per secret it would have been nice if they had a 5 secret free tier. That would get smaller users to start using it instead of parameter store and eventually realize the value of automated and audited secret rotation
- kungito 9y agoYeah I'm thinking about starting to use it for personal projects but don't want to bother with these 2 dollar bills. I'm pretty sure they can benefit from me playing with it first and then advocating for it at work
- ppierald 9y agoThere is a lot of commentary about the use of vault as an alternative, number of secrets needed, etc. I think the inclusion of Secrets Manager is a great addition for AWS and will definitely help people get better control over their secrets, however, vault contains richer functionality than just secrets key/value storage. It can provision users to backends like SSH, databases, cloud providers, and such. Use is audited, can be revoked, and has a TTL associated. Additionally, vault contains a full "crypto-in-a-box" implementation that allows for sign/verify, hmac/verify, encrypt/decrypt, random number generation, and other functions. So I applaud AWS for doing this and hope the will continue developing KMS/HSM/Parameter Store/Secret Store/??? in the future and innovating, but evaluating Secret Store vs. Vault simply on price may be a short sighted comparison. Disclaimers: Employer is an AWS customer using vault
- drodgers 9y agoI was hoping to use this for RDS parameters (to get automatic rotation) and leave everything else in ParameterStore. Unfortunately — although I'm sure it's built on-top of ParameterStore internally — I just checked and you can't see SecretsManager secrets in ParameterStore, so an application would need to read from both and merge them, or switch entirely to SecretsManager to take advantage of the automatic rotation.
- bestbear 9y agoCompare this to the cost of Hashicorp Vault. You might be surprised at the difference.
- netvisao 9y agoI am curious as to how it handles those race conditions where a connection is made with the older credentials just after the time the rds master key rotates, or a connection is made with the newer credentials just before the time the rds client key rotates. Short of using two credentials accounts ...
- ranman 9y agothere’s a pretty nifty 4 stage rotation mechanism that solves most, but not all, of those race conditions. More info on it in the post and the docs.
- netvisao 9y agoAre we talking about these? createSecret setSecret testSecret finishSecret These were mentioned around the custom rotation strategy, any pointers to docs describing how it is done if the secret is an RDS Secret?
- tyingq 9y agoAu revoir Cyberark. I won't miss you.