8 ms·

Confidant: an open-source secret management service

245 points by woodrow 11y ago

luisrudge 11y ago
That's very nice. If you're using Microsoft Azure though, you might as well use Azure Key Vault[0]. [0] https://azure.microsoft.com/en-us/services/key-vault/ https://azure.microsoft.com/en-us/services/key-vault/
- deleted 11y ago
  [deleted]
jvehent 11y ago
An alternative approach, if you don't want to be 100% tied to KMS, is https://github.com/mozilla/sops https://github.com/mozilla/sops . It uses KMS, but also PGP, and potentially anything that comes up in the future.
- teddyfinkins 11y ago
  A big +1 for SOPS. We've used it with great success.
justadan 11y ago
"KMS provides access to master encryption keys,... but doesn’t provide direct access to the master key itself, so it can’t be stolen." Doesn't Amazon KMS have access to the master key? And therefore, it can be stolen from them?
- ryan_lane 11y ago
  https://aws.amazon.com/kms/faqs/ https://aws.amazon.com/kms/faqs/ "AWS KMS is designed so that no one has access to your master keys."
  - justadan 11y ago
    "...never storing plaintext master keys on disk, not persisting them in memory, and limiting which systems can connect to the device..." Translation: We promise to not look at your keys. Granted, this is most likely better than nothing, especially if you trust AWS, but ideally, you want only the client to have the key material.
    - ryan_lane 11y ago
      If you're running in AWS with any secret management service that isn't using CloudHSM, you're trusting AWS with your key material. There surely is some amount of trust there, though.
      - ppierald 11y ago
        My undertanding was that the Customer Master Key was stored in an HSM, but your customer-generated keys were not. I might be wrong about that. So if true, AWS employees would not have access to your root key material, but the definitely the intermediary key material. It's a cost trade-off
        jamiesonbecker 11y ago
        Either way, you have to bootstrap secret material onto your instances somehow, so you've got to trust Amazon somewhere. Not to diminish your point -- the how/where/when really does matter. Locking your house key in your car is still better than leaving it on your front step, but also not as good as in your pocket.
squidlogic 11y ago
We use ZeroTier to encrypt our AWS microservices traffic. Way easier to setup and just ... works.
- ryan_lane 11y ago
  This is a bit different from network encryption (which is really valuable). This is a centralized location for services to store and retrieve secrets, like passwords to external services, SSL keys, etc..
austinjp 11y ago
How does this compare to Vault? [1] Looks like Confidant is tied to AWS whereas Vault can use various backends..? [1] https://www.vaultproject.io/ https://www.vaultproject.io/
- ryan_lane 11y ago
  FAQ in the post discussed this: "The main difference between Confidant and the others is that Confidant is purposely not cloud agnostic, choosing to use AWS’s features to deliver a more integrated experience. By leveraging AWS’s KMS service, Confidant is able to ensure the master encryption key can’t be stolen, that authentication credentials for Confidant don’t need to be distributed to clients, and that authentication credentials for Confidant clients don’t need to generated and trusted through quasi-trustable metadata." Confidant is purposely tied to AWS so that it can rely on KMS for its master key and for authentication.
  - austinjp 11y ago
    Indeed, thanks. I just found a little more on Vault's take on this: https://www.vaultproject.io/intro/vs/kms.html https://www.vaultproject.io/intro/vs/kms.html
- bkeroack 11y ago
  Vault is a nice piece of engineering (we use it), but it has what I call serious "backend-itis". Everything is pluggable, which makes it a bit of a nightmare to understand and use. For example, "secret backends" and "storage backends" are entirely separate things, but the docs aren't super clear about it (not to mention auth backends, audit backends, listeners ...). Unsurprisingly it buys completely into the Hashicorp ecosystem (Consul, weirdo HCL configs, etc) which is a plus or minus depending upon your perspective I guess. Vault servers are also stateful, which complicates deployment (you can't just stick N Vault servers behind a load balancer). If I had infinite time I'd consider creating a "SimpleVault" fork consisting of exactly one secret and storage backend, one way to auth, one curated workflow and make it run in a stateless manner. I'd probably also remove all the clever secret generation stuff, since 90% of applications just require a secure place to store secret blobs of data, as opposed to creating dynamic time-limited IAM credentials on the fly or whatever.
jonesb6 11y ago
Nice way for Lyft to fire back after that iOS reverse-engineering video [1] revealed that they were showing off one of their keys in a production client. I don't know if this was intentional, and I believe whatever exploit they had was mild, but it restores (at first glance) my faith in them a bit :). [1]: https://realm.io/news/conrad-kramer-reverse-engineering-ios-apps-lyft/ https://realm.io/news/conrad-kramer-reverse-engineering-ios-...
bpicolo 11y ago
Another one of those problems every company seems to try to solve on their own. :P
- adzuci 11y ago
  I totally agree (we did), though Confidant is in some ways trying to be an extension of existing AWS tools. I'm curious what you're using though since I've felt for a while now that every one of these secrets stores have faults or are a giant pain to work with. Confidant may be the least offender in that regard & anyone that is trying to make being secure easier or trying to make better tools gets an A in my book.
  - bpicolo 11y ago
    My company is using Vault as far as I know. I think HashiCorp is doing a decent job identifying the sort of things every company is trying to reinvent
bascule 11y ago
So they basically reinvented this? https://github.com/codahale/sneaker https://github.com/codahale/sneaker
- ryan_lane 11y ago
  We wrote Confidant in the beginning of 2015, so it's not a reinvention of sneaker. That said, sneaker is very similar in design, as is credstash: https://github.com/fugue/credstash https://github.com/fugue/credstash Both of those are really great projects and I recommend them if they fit your use-case. One of the reasons we originally avoided S3 was because it can occasionally be very eventually consistent, which we wanted to avoid. We also wanted to avoid fanning out secrets to services when the secret needed to be shared between multiple services. We also wanted to have a UI that was simple and easy to use for everyone.
vangale 11y ago
This is the first time I've seen a nicely documented requirements.txt and I like! https://github.com/lyft/confidant/blob/master/requirements.txt https://github.com/lyft/confidant/blob/master/requirements.t...
- babl-yc 11y ago
  No more "pip freeze | sort -f > requirements.txt", explicit dependencies are way more maintainable!
- viraptor 11y ago
  + for readability, - for precise versions. Good luck receiving critical updates this way...
  - ryan_lane 11y ago
    We watch CVEs and update accordingly. Assuming you're using the latest stable release of the docker image or are using the latest stable release of Confidant (we're still working on making releases through github) you should be using a version with secure dependencies. Using >= doesn't ensure security, but it does ensure less stability, and part of security is availability.
  - catshirt 11y ago
    i mean, aren't specified versions crucial for compatibility? what am i missing? on second thought, i guess you are suggesting to specify the major, with an ambiguous minor? but doesn't this require that no package will introduce a breaking change in a minor version? not sure you can rely on that assumption.
    - tedmiston 11y ago
      Yes, and yes. It's the best practice to have flexible subversions of dependencies (or dependencies of dependencies) when you're shipping a package with setup.py (http://python-packaging-user-guide.readthedocs.org/en/latest/requirements/ http://python-packaging-user-guide.readthedocs.org/en/latest...). However in requirements.txt for an app that runs by itself, their approach is the best practice.
AndrewWright 11y ago
Another alternative developed for AWS deployments, written in Python and uses KMS: Credstash https://github.com/fugue/credstash https://github.com/fugue/credstash
- ryan_lane 11y ago
  The only downside of credstash is that it doesn't have the ability to restrict sets of credentials to different IAM roles. The access is all-or-nothing, per dynamo table. Otherwise the general design of credstash is very similar to Confidant.
  - vogan_mother 11y ago
    It is possible to use fine grained access control with dynamodb in order to restrict access within a ddb table
normanjoyner 11y ago
It's unfortunate that this is tied to AWS, while the industry is moving towards a cloud agnostic approach to hosting. That being said, it seems like an interesting project to keep an eye on.
- mpdehaan2 11y ago
  I'd say the opposite - the industry is in general being a lot more comfortable with investing in AWS to the point of lockin being acceptable now. More people are treating cloud providers less like a big pool of IP addresses. Not saying it's a good or bad thing, but attend AWS reInvent or something and it's pretty easy to percieve huge uptake. If you adopt extra management stacks running on or against your cloud you lose a lot of the benefits.
  - normanjoyner 11y ago
    That's an interesting perspective and observation. I've always found lock-in to be a bit scary, and I've heard this echoed by a number of colleagues & companies. No one can deny the usefulness of tools provided by hosting providers, namely AWS; however, to me, architecting a system in such a way where you are literally unable to jump ship when the need arises (compliance, cost, etc) is unacceptable. With hybrid / multi cloud more of a reality with the existence of tools like Docker (and others), there needs to exist sets of tools capable of handling such variability. I digress, this really belongs in an entirely different thread.
    - yeukhon 11y ago
      We tried running on three cloud service providers while connecting back to our own data center (corporate stuff) at the same time, for a couple years, because of the whole "cloud agnostic". The code requires us to be compatible with OpenStack and AWS. Trust me, no matter what people say about the compatibility of some of the components between OpenStack and AWS, you cannot do the AWS-way 100% in OpenStack. In the end I got a pile of shit code with a lot of workaround and condition. Sure Heat is like a comparable version of CloudFormation, but you can't do what you did in AWS and translate to Heat. OpenStack doesn't have all the services that AWS has. Not every product (and not every API in a product of AWS) works 100% with CloudFormation. If you are running an old release of OpenStack, you are more fucked in that case. We are now moving our cloud infrastructure to AWS entirely so no more "it works in this environment", or "they sort of work and sort of different because of X and Y tool not comparable." The right attitude should be "do one thing well, and improve." Architect "right" and than unlock yourself whenever possible. The same goes to people migrating to Docker or any container technology. Dockerfile is a lock-in. You still have to move back to regular shell script later if you drop Docker.
Merkur 11y ago
Please correct me if I am wrong, but I think there is no secure way to store stuff in an virtual environment. I wish I am wrong - cause my heart always bleeds if I see db passwords in configuration files! But As long as there is a hypervisor you do not control access to - you must trust the owner of the bare metal to (1) honor your privacy (2) be competent to secure his system. Trust is nice, but it is not security. granted - Confidant and KMS seem better solution than most. Will look into it at more detail. thx for open sourcing it and moving the solution forward.
- spacehome 11y ago
  https://en.wikipedia.org/wiki/Homomorphic_encryption https://en.wikipedia.org/wiki/Homomorphic_encryption
  - Merkur 11y ago
    I did not know such thing is possible. very cool! thx for pointing it out! :) but still..some rhetorical questions: Is KMS based on this concept? Is it's implementation open sourced? Is it verified to run unchanged in the instance I pay? Who is verifying it? Who is paying the one verifying it? Can I meet him / her to drink a coffee and build trust between us? pls, do not misunderstand me. I got more trust in the amazon techs than in my own ability to admin a complex linux system on bare metal. But while all that is great progress toward security we still need to make a leap to find a solution. don't we?
    - viraptor 11y ago
      > I did not know such thing is possible. It's theoretically possible, but right now it's too slow to be practical in usual situations. Just storing and single operations are maybe possible. But for example homomorphic AES takes multiple seconds per each block. And that's still AES-128.
      - e12e 11y ago
        Wait, are you saying HE is feasible for 128 bytes of data, if you can allow seconds for de/encryption? That'd be plenty usable for read-at-startup 128 bit api keys for micro-services, disk encryption keys etc? (Granted, in this context, having a de-crypted disk at run time under a vm would be considered insecure - but still better (for some use cases) than man alternatives)
nemothekid 11y ago
I'd love for someone to explain what you get from using a secret management service other than encrypted at rest blobs. Ex. You store your AWS Master key in a config file, and you have Microservice A that reads that key from the file. Microservice A is compromised (or its VM is compromised). How does having a secret store help you here? Couldn't the attacker just inspect the code of Microservice A and see that you are just reading from disk/reading from Vault? In short, what do services like this protect from me (other than accidentally checking in my code to a public repo?)
- ryan_lane 11y ago
  They let you not check secrets into repos, they let you update and rotate them in a centralized place, they let you easily share them between services, and they let you store them encrypted at rest. Also, in this particular case, thanks to KMS you can also keep the stored at-rest on microservice A, and decrypt them only in memory as well. Note that you don't store a master key in a config file, but instead you use the KMS master key to encrypt/decrypt things. You never get direct access to the master key.
dcosson 11y ago
I have a genuine question, why not use S3 alone for secret management? One selling point of Confidant is using IAM roles to bootstrap authentication to the secret store. You can also do that with S3, put each secret into an individual text file and give each IAM role permission to access the secrets it needs. Set the S3 bucket to encrypt the data at rest, it uses KMS behind the scenes and automatically rotates encryption keys. Rotation of the secrets themselves could be scripted or manual, that part would be basically the same process as using Confidant or any other tool. And I believe S3 access can even be auditable with CloudWatch logs. Also, S3 now offers either eventually consistent or read-after-write consistency. EDIT: actually, it looks like new object PUTS can be read-after-write consistent but updates are not. So this could be a downside, if you rotate a key getting the new one is eventually consistent. In practice this might not be a big deal though, there's already going to be a gap between when you activate the new key and when your app gets reconfigured to start using the new key. I'm very curious what the downsides might be of doing this. For all the various secret management tools that have been released in the past year or two, I'm kind of surprised I've never heard anyone talk about using raw S3.
- ryan_lane 11y ago
  Encrypted S3 is definitely an option. The difficulty there is how to manage secrets that may be shared across services and how to ensure that there's no race conditions between updates of secrets. Confidant provides access to credentials by IAM role. The key here is that you can have credentials A, B, and C, and IAM roles example1-production, example2-production and example3-production, then map credentials A and B to example1-production, A and C to example2-production and A to example3-production. Confidant just stores mapping info for this. If you update A, you don't need to fan out an update to all three services. Race conditions can come in when A and B or C are updated at the same time. S3 is very eventually consistent and there's no locking. read-after-write consistency is only available in one region at this point. Of course, this is all based on your assumptions and how you design your system. Based on our needs KMS + DynamoDB made more sense for us. Something like sneaker (https://github.com/codahale/sneaker https://github.com/codahale/sneaker) may fit your needs better. Another interesting bit about Confidant is that it manages the KMS grants on the auth KMS key, which lets you use the key from any service to any service to pass secrets, or to do general service to service auth. Also, there's a bit more coming down the road for automatic management of other common secrets that are annoying to manage in AWS, like SSH host keys for autoscale groups: https://github.com/lyft/confidant/issues/1 https://github.com/lyft/confidant/issues/1

245 points by woodrow 11y ago

That's very nice. If you're using Microsoft Azure though, you might as well use Azure Key Vault[0]. [0] https://azure.microsoft.com/en-us/services/key-vault/ https://azure.microsoft.com/en-us/services/key-vault/

deleted 11y ago
[deleted]

jvehent 11y ago

An alternative approach, if you don't want to be 100% tied to KMS, is https://github.com/mozilla/sops https://github.com/mozilla/sops . It uses KMS, but also PGP, and potentially anything that comes up in the future.

teddyfinkins 11y ago
A big +1 for SOPS. We've used it with great success.

justadan 11y ago

"KMS provides access to master encryption keys,... but doesn’t provide direct access to the master key itself, so it can’t be stolen." Doesn't Amazon KMS have access to the master key? And therefore, it can be stolen from them?

ryan_lane 11y ago
https://aws.amazon.com/kms/faqs/ https://aws.amazon.com/kms/faqs/ "AWS KMS is designed so that no one has access to your master keys."
- justadan 11y ago
  "...never storing plaintext master keys on disk, not persisting them in memory, and limiting which systems can connect to the device..." Translation: We promise to not look at your keys. Granted, this is most likely better than nothing, especially if you trust AWS, but ideally, you want only the client to have the key material.
  - ryan_lane 11y ago
    If you're running in AWS with any secret management service that isn't using CloudHSM, you're trusting AWS with your key material. There surely is some amount of trust there, though.
    - ppierald 11y ago
      My undertanding was that the Customer Master Key was stored in an HSM, but your customer-generated keys were not. I might be wrong about that. So if true, AWS employees would not have access to your root key material, but the definitely the intermediary key material. It's a cost trade-off
      - jamiesonbecker 11y ago
        Either way, you have to bootstrap secret material onto your instances somehow, so you've got to trust Amazon somewhere. Not to diminish your point -- the how/where/when really does matter. Locking your house key in your car is still better than leaving it on your front step, but also not as good as in your pocket.

squidlogic 11y ago

We use ZeroTier to encrypt our AWS microservices traffic. Way easier to setup and just ... works.

ryan_lane 11y ago
This is a bit different from network encryption (which is really valuable). This is a centralized location for services to store and retrieve secrets, like passwords to external services, SSL keys, etc..

austinjp 11y ago

How does this compare to Vault? [1] Looks like Confidant is tied to AWS whereas Vault can use various backends..? [1] https://www.vaultproject.io/ https://www.vaultproject.io/

ryan_lane 11y ago
FAQ in the post discussed this: "The main difference between Confidant and the others is that Confidant is purposely not cloud agnostic, choosing to use AWS’s features to deliver a more integrated experience. By leveraging AWS’s KMS service, Confidant is able to ensure the master encryption key can’t be stolen, that authentication credentials for Confidant don’t need to be distributed to clients, and that authentication credentials for Confidant clients don’t need to generated and trusted through quasi-trustable metadata." Confidant is purposely tied to AWS so that it can rely on KMS for its master key and for authentication.
- austinjp 11y ago
  Indeed, thanks. I just found a little more on Vault's take on this: https://www.vaultproject.io/intro/vs/kms.html https://www.vaultproject.io/intro/vs/kms.html
bkeroack 11y ago
Vault is a nice piece of engineering (we use it), but it has what I call serious "backend-itis". Everything is pluggable, which makes it a bit of a nightmare to understand and use. For example, "secret backends" and "storage backends" are entirely separate things, but the docs aren't super clear about it (not to mention auth backends, audit backends, listeners ...). Unsurprisingly it buys completely into the Hashicorp ecosystem (Consul, weirdo HCL configs, etc) which is a plus or minus depending upon your perspective I guess. Vault servers are also stateful, which complicates deployment (you can't just stick N Vault servers behind a load balancer). If I had infinite time I'd consider creating a "SimpleVault" fork consisting of exactly one secret and storage backend, one way to auth, one curated workflow and make it run in a stateless manner. I'd probably also remove all the clever secret generation stuff, since 90% of applications just require a secure place to store secret blobs of data, as opposed to creating dynamic time-limited IAM credentials on the fly or whatever.

jonesb6 11y ago

Nice way for Lyft to fire back after that iOS reverse-engineering video [1] revealed that they were showing off one of their keys in a production client. I don't know if this was intentional, and I believe whatever exploit they had was mild, but it restores (at first glance) my faith in them a bit :). [1]: https://realm.io/news/conrad-kramer-reverse-engineering-ios-apps-lyft/ https://realm.io/news/conrad-kramer-reverse-engineering-ios-...

bpicolo 11y ago

Another one of those problems every company seems to try to solve on their own. :P

adzuci 11y ago
I totally agree (we did), though Confidant is in some ways trying to be an extension of existing AWS tools. I'm curious what you're using though since I've felt for a while now that every one of these secrets stores have faults or are a giant pain to work with. Confidant may be the least offender in that regard & anyone that is trying to make being secure easier or trying to make better tools gets an A in my book.
- bpicolo 11y ago
  My company is using Vault as far as I know. I think HashiCorp is doing a decent job identifying the sort of things every company is trying to reinvent

bascule 11y ago

So they basically reinvented this? https://github.com/codahale/sneaker https://github.com/codahale/sneaker

ryan_lane 11y ago
We wrote Confidant in the beginning of 2015, so it's not a reinvention of sneaker. That said, sneaker is very similar in design, as is credstash: https://github.com/fugue/credstash https://github.com/fugue/credstash Both of those are really great projects and I recommend them if they fit your use-case. One of the reasons we originally avoided S3 was because it can occasionally be very eventually consistent, which we wanted to avoid. We also wanted to avoid fanning out secrets to services when the secret needed to be shared between multiple services. We also wanted to have a UI that was simple and easy to use for everyone.

vangale 11y ago

This is the first time I've seen a nicely documented requirements.txt and I like! https://github.com/lyft/confidant/blob/master/requirements.txt https://github.com/lyft/confidant/blob/master/requirements.t...

babl-yc 11y ago
No more "pip freeze | sort -f > requirements.txt", explicit dependencies are way more maintainable!
viraptor 11y ago
+ for readability, - for precise versions. Good luck receiving critical updates this way...
- ryan_lane 11y ago
  We watch CVEs and update accordingly. Assuming you're using the latest stable release of the docker image or are using the latest stable release of Confidant (we're still working on making releases through github) you should be using a version with secure dependencies. Using >= doesn't ensure security, but it does ensure less stability, and part of security is availability.
- catshirt 11y ago
  i mean, aren't specified versions crucial for compatibility? what am i missing? on second thought, i guess you are suggesting to specify the major, with an ambiguous minor? but doesn't this require that no package will introduce a breaking change in a minor version? not sure you can rely on that assumption.
  - tedmiston 11y ago
    Yes, and yes. It's the best practice to have flexible subversions of dependencies (or dependencies of dependencies) when you're shipping a package with setup.py (http://python-packaging-user-guide.readthedocs.org/en/latest/requirements/ http://python-packaging-user-guide.readthedocs.org/en/latest...). However in requirements.txt for an app that runs by itself, their approach is the best practice.

AndrewWright 11y ago

Another alternative developed for AWS deployments, written in Python and uses KMS: Credstash https://github.com/fugue/credstash https://github.com/fugue/credstash

ryan_lane 11y ago
The only downside of credstash is that it doesn't have the ability to restrict sets of credentials to different IAM roles. The access is all-or-nothing, per dynamo table. Otherwise the general design of credstash is very similar to Confidant.
- vogan_mother 11y ago
  It is possible to use fine grained access control with dynamodb in order to restrict access within a ddb table

normanjoyner 11y ago

It's unfortunate that this is tied to AWS, while the industry is moving towards a cloud agnostic approach to hosting. That being said, it seems like an interesting project to keep an eye on.

mpdehaan2 11y ago
I'd say the opposite - the industry is in general being a lot more comfortable with investing in AWS to the point of lockin being acceptable now. More people are treating cloud providers less like a big pool of IP addresses. Not saying it's a good or bad thing, but attend AWS reInvent or something and it's pretty easy to percieve huge uptake. If you adopt extra management stacks running on or against your cloud you lose a lot of the benefits.
- normanjoyner 11y ago
  That's an interesting perspective and observation. I've always found lock-in to be a bit scary, and I've heard this echoed by a number of colleagues & companies. No one can deny the usefulness of tools provided by hosting providers, namely AWS; however, to me, architecting a system in such a way where you are literally unable to jump ship when the need arises (compliance, cost, etc) is unacceptable. With hybrid / multi cloud more of a reality with the existence of tools like Docker (and others), there needs to exist sets of tools capable of handling such variability. I digress, this really belongs in an entirely different thread.
  - yeukhon 11y ago
    We tried running on three cloud service providers while connecting back to our own data center (corporate stuff) at the same time, for a couple years, because of the whole "cloud agnostic". The code requires us to be compatible with OpenStack and AWS. Trust me, no matter what people say about the compatibility of some of the components between OpenStack and AWS, you cannot do the AWS-way 100% in OpenStack. In the end I got a pile of shit code with a lot of workaround and condition. Sure Heat is like a comparable version of CloudFormation, but you can't do what you did in AWS and translate to Heat. OpenStack doesn't have all the services that AWS has. Not every product (and not every API in a product of AWS) works 100% with CloudFormation. If you are running an old release of OpenStack, you are more fucked in that case. We are now moving our cloud infrastructure to AWS entirely so no more "it works in this environment", or "they sort of work and sort of different because of X and Y tool not comparable." The right attitude should be "do one thing well, and improve." Architect "right" and than unlock yourself whenever possible. The same goes to people migrating to Docker or any container technology. Dockerfile is a lock-in. You still have to move back to regular shell script later if you drop Docker.

Merkur 11y ago

Please correct me if I am wrong, but I think there is no secure way to store stuff in an virtual environment. I wish I am wrong - cause my heart always bleeds if I see db passwords in configuration files! But As long as there is a hypervisor you do not control access to - you must trust the owner of the bare metal to (1) honor your privacy (2) be competent to secure his system. Trust is nice, but it is not security. granted - Confidant and KMS seem better solution than most. Will look into it at more detail. thx for open sourcing it and moving the solution forward.

spacehome 11y ago
https://en.wikipedia.org/wiki/Homomorphic_encryption https://en.wikipedia.org/wiki/Homomorphic_encryption
- Merkur 11y ago
  I did not know such thing is possible. very cool! thx for pointing it out! :) but still..some rhetorical questions: Is KMS based on this concept? Is it's implementation open sourced? Is it verified to run unchanged in the instance I pay? Who is verifying it? Who is paying the one verifying it? Can I meet him / her to drink a coffee and build trust between us? pls, do not misunderstand me. I got more trust in the amazon techs than in my own ability to admin a complex linux system on bare metal. But while all that is great progress toward security we still need to make a leap to find a solution. don't we?
  - viraptor 11y ago
    > I did not know such thing is possible. It's theoretically possible, but right now it's too slow to be practical in usual situations. Just storing and single operations are maybe possible. But for example homomorphic AES takes multiple seconds per each block. And that's still AES-128.
    - e12e 11y ago
      Wait, are you saying HE is feasible for 128 bytes of data, if you can allow seconds for de/encryption? That'd be plenty usable for read-at-startup 128 bit api keys for micro-services, disk encryption keys etc? (Granted, in this context, having a de-crypted disk at run time under a vm would be considered insecure - but still better (for some use cases) than man alternatives)

nemothekid 11y ago

I'd love for someone to explain what you get from using a secret management service other than encrypted at rest blobs. Ex. You store your AWS Master key in a config file, and you have Microservice A that reads that key from the file. Microservice A is compromised (or its VM is compromised). How does having a secret store help you here? Couldn't the attacker just inspect the code of Microservice A and see that you are just reading from disk/reading from Vault? In short, what do services like this protect from me (other than accidentally checking in my code to a public repo?)

ryan_lane 11y ago
They let you not check secrets into repos, they let you update and rotate them in a centralized place, they let you easily share them between services, and they let you store them encrypted at rest. Also, in this particular case, thanks to KMS you can also keep the stored at-rest on microservice A, and decrypt them only in memory as well. Note that you don't store a master key in a config file, but instead you use the KMS master key to encrypt/decrypt things. You never get direct access to the master key.

dcosson 11y ago

I have a genuine question, why not use S3 alone for secret management? One selling point of Confidant is using IAM roles to bootstrap authentication to the secret store. You can also do that with S3, put each secret into an individual text file and give each IAM role permission to access the secrets it needs. Set the S3 bucket to encrypt the data at rest, it uses KMS behind the scenes and automatically rotates encryption keys. Rotation of the secrets themselves could be scripted or manual, that part would be basically the same process as using Confidant or any other tool. And I believe S3 access can even be auditable with CloudWatch logs. Also, S3 now offers either eventually consistent or read-after-write consistency. EDIT: actually, it looks like new object PUTS can be read-after-write consistent but updates are not. So this could be a downside, if you rotate a key getting the new one is eventually consistent. In practice this might not be a big deal though, there's already going to be a gap between when you activate the new key and when your app gets reconfigured to start using the new key. I'm very curious what the downsides might be of doing this. For all the various secret management tools that have been released in the past year or two, I'm kind of surprised I've never heard anyone talk about using raw S3.

ryan_lane 11y ago
Encrypted S3 is definitely an option. The difficulty there is how to manage secrets that may be shared across services and how to ensure that there's no race conditions between updates of secrets. Confidant provides access to credentials by IAM role. The key here is that you can have credentials A, B, and C, and IAM roles example1-production, example2-production and example3-production, then map credentials A and B to example1-production, A and C to example2-production and A to example3-production. Confidant just stores mapping info for this. If you update A, you don't need to fan out an update to all three services. Race conditions can come in when A and B or C are updated at the same time. S3 is very eventually consistent and there's no locking. read-after-write consistency is only available in one region at this point. Of course, this is all based on your assumptions and how you design your system. Based on our needs KMS + DynamoDB made more sense for us. Something like sneaker (https://github.com/codahale/sneaker https://github.com/codahale/sneaker) may fit your needs better. Another interesting bit about Confidant is that it manages the KMS grants on the auth KMS key, which lets you use the key from any service to any service to pass secrets, or to do general service to service auth. Also, there's a bit more coming down the road for automatic management of other common secrets that are annoying to manage in AWS, like SSH host keys for autoscale groups: https://github.com/lyft/confidant/issues/1 https://github.com/lyft/confidant/issues/1