22 ms·

Vault – A tool for managing secrets

440 points by eloycoto 11y ago

carllerche 11y ago
Looks interesting. There is definitely a need for something secure to manage secrets easily & efficiently. I'm not super familiar with Shamir's secret sharing. The post says that the master key is split up across a number of people. One thing that I am confused about is if Vault goes down (possibly in the middle of the night), does it require a quorum of humans to boot it back up?
- limaoscarjuliet 11y ago
  Yes, otherwise - when keys travel with secrets - there would be absolutely no security. This is akin to disk encryption where you have to type password on boot, but then data is accessible later.
- viraptor 11y ago
  Haven't tried it yet, but most likely yes. But if you actually need vault and do spread the key to more than 2 people, you likely have more than one vault server running at a time.
- mitchellh 11y ago
  It does. The solution to this problem is to run Vault in high availability mode[1]. This will have a set of stand-bys that can take over if the leader goes down. The idea is to unseal all the stand bys, and if the leader goes down, the standbys take over, and the sealed one can be unsealed at some other future point. Security-sensitive things have been an interesting UX challenge for us, but this compromise comes at an expense of UX for a much better security promise. As one of the other comments says around this comment: without this feature there would be no real security for the stored data. [1]: http://vaultproject.io/docs/concepts/ha.html http://vaultproject.io/docs/concepts/ha.html
  - carllerche 11y ago
    Looks good. The post did not mention HA (that I saw), which caused the confusion.
  - odiroot 11y ago
    Would you also recommend to set the lease duration to something higher in order to allow people to react? This way even if Vault dies during the night (and you don't have pager duty) at least some clients (e.g. your cloud instances) can live through?
  - SEJeff 11y ago
    Can you call mlock in go to prevent any secrets from being paged to disk?
    - mitchellh 11y ago
      We already do this. And yes, you can (using the "syscall" package).
- btilly 11y ago
  Shamir's secret sharing is actually fairly simple. You want to have a secret shared by n people, any m of whom can decode it. You create a polynomial of degree m, and give each of those people the value of said polynomial at a different point. The secret will be the value of the polynomial at yet another point. From the value of the polynomial at any m points you can figure out the polynomial and the secret. But the value at any m-1 points does not let you predict its value anywhere else. (Detail. You would make the polynomial be over a finite field instead of the real numbers. But the math is all the same.)
deleted 11y ago
[deleted]
- robszumski 11y ago
  Check out the ACL RFC [1] which has been developed and is now available for testing in the first alpha of etcd v2.1.0 [2], we'd love your feedback. [1] https://coreos.com/etcd/docs/2.0.8/rfc/api_security.html https://coreos.com/etcd/docs/2.0.8/rfc/api_security.html [2] https://github.com/coreos/etcd/releases/tag/v2.1.0-alpha.0 https://github.com/coreos/etcd/releases/tag/v2.1.0-alpha.0
- mpdehaan2 11y ago
  (EDIT: I appear to be replying to a edited comment that was mentioning a possible naming confusion with ansible vault, I'll keep the post here even though the original is different now) re: naming (and disclaimer): I wrote Ansible and designed Ansible's vault (and no longer work on Ansible), but to be clear, Ansible's vault is a nicer CLI (IMHO) around an idea that Chef users were already used to: encrypted data bags. It was nothing new. It took about two weeks to implement, but we had a few security things to shake out after that made it take a bit longer before it got right. (Some people also got picky about whole-data-document encryption vs leaf-node-in-datastructure encryption, and I see points for doing it both ways, but went with the way that revealed the least amount of data) I think calling it vault is perfectly fair, as it's a very generic word, and words are in short supply. This is also doing a LOT more. I like what Mitchell and crew have done around revocation here, though I do somewhat wonder about the mode in the docs that sets secrets and allows them to show in bash history, which is why ansible vault spawned an editor. I'm sure it's got editor options too though. I'd probably also change the phrasing "HTTP API" to "HTTPS API" :) The dynamic secret stuff also looks pretty interesting. One of the interesting ideas might be to whether it allows sharing of secret values to some applications without letting someone read it (asking if a user can use X without letting the user READ x), which is something we did in tower - though it seems the ACL system might allow something like that. An example of that was the system could memorize an SSH key and let a team use it, but never give it to anyone, to avoid rotation scenarios. Anyway, all the lease ideas are very cool and it seems natural to want this decoupled. For a time I considered decoupling what what in Ansible vault from Ansible, though really it already operates on generic files so that's easy enough if someone wanted to take a stab at it. I suspect it wouldn't be impossible to make Ansible Vault have a hashicorp-vault-mode where it encrypts a file "into" the vault, and the contents that remain in your "ansible vault" file are nothing more than a pointer to a secret in Hashicorp Vault. In this way, it would just be another cipher type. Maybe that's crazy though - but I think I like that better than storing secrets in version control, where they exist forever unless you prune history. (And I never thought storing vault files in public source was a good idea). However, avoiding a server or database was always one of Ansible's design goals - for better or worse, that limited some of the things the core app could do, in ways that were both good and also occasionally limiting. Anyway, keep up the forward progress with the new things! I really envy the time to architect/design something so detailed here.
steveb 11y ago
I'm pretty excited about this release. We just added deploying Vault on top of consul in the microservice/realtime data platform we're building: https://github.com/CiscoCloud/microservices-infrastructure https://github.com/CiscoCloud/microservices-infrastructure We were already using consul as our service discovery engine and as a data store for dynamic configurations via consul template. We feel Vault will help us manage secrets on our infrastructure and inside containers.
- jgrowl 11y ago
  Neat! I'll have to see how you ended up using Vault since your project uses a lot of the same tech my personal webrtc server project uses. I'm also using ansible, docker, consul, registrator. Excuse the mess, I'm still working on it: https://github.com/jgrowl/livehq https://github.com/jgrowl/livehq
deleted 11y ago
[deleted]
mentat 11y ago
Any reason why AWS dynamic credentials use IAM users instead of STS and role assumption? That seems to be a somewhat more widely accepted way to do dynamic credentials / role proxying.
- setheron 11y ago
  that would definitely be more idiomatic in AWS source: I am an Amazon employee
- ukd1 11y ago
  came here to see how it was done; was hoping for STS :(
  - mentat 11y ago
    https://github.com/dump247/docker-ec2-metadata https://github.com/dump247/docker-ec2-metadata and https://github.com/AdRoll/hologram https://github.com/AdRoll/hologram exist as examples of this, just FYI.
christop 11y ago
Interesting indeed! But I couldn't find anything on the website or documentation about how secrets are encrypted? The security page seems to be a generic one about reporting vulnerabilities: http://vaultproject.io/security.html http://vaultproject.io/security.html Edit: Found it under the threat model section here: https://vaultproject.io/docs/internals/security.html https://vaultproject.io/docs/internals/security.html
- andrewmwatson 11y ago
  the blog post mentioned AES-GCM 256bit
- weitzj 11y ago
  See: http://vaultproject.io/docs/internals/security.html http://vaultproject.io/docs/internals/security.html
  - christop 11y ago
    Found it, thanks.
ankushio 11y ago
What are some of the existing tools/solutions which people use for managing secrets in a production environment?
- amenghra 11y ago
  https://corner.squareup.com/2015/04/keywhiz.html https://corner.squareup.com/2015/04/keywhiz.html
- ubercow 11y ago
  slightly different scope, but solves a similar problem https://github.com/cloudflare/redoctober https://github.com/cloudflare/redoctober good writeup about it here https://blog.cloudflare.com/red-october-cloudflares-open-source-implementation-of-the-two-man-rule/ https://blog.cloudflare.com/red-october-cloudflares-open-sou...
- dustinrcollins 11y ago
  http://conjur.net/ http://conjur.net/
jvehent 11y ago
There are many solutions out there that attempt to solve the secrets distribution problem. Vault is the newest one. None of them solve the fundamental problem of the secret zero: provisioning the first secret on an untrusted system the automated way is hard, and without that secret zero, there is no way to authenticate the new system. With Vault, you still need to figure out how to push secret zero (here a client authentication token). If you can solve that, why not push the secret you need to push directly, and bypass Vault entirely?
- mitchellh 11y ago
  We worked with a handful of companies to help us design the "App ID" authentication backend that does exactly that: https://vaultproject.io/docs/auth/app-id.html https://vaultproject.io/docs/auth/app-id.html It allows you to have a non-sensitive single factor within things like configuration management, and have the second factor come from a machine-local location (instance ID, MAC address, TSM module, etc.). The idea is that another team out of band sets the 2nd factor that developers and config management never get to see. The result is you have full automation without secret zero issues. As an additional security parameter, you can bind the two factors together to a single CIDR block, so you can have an additional factor restricting access from that machine's IP. The URL itself above goes into a lot more detail, but our beta users are successfully automated in an elastic environment this way.
  - jvehent 11y ago
    That's definitely a trade-off compared to strong authentication, one that I wouldn't want to make in critical environments. But indeed a very nice feature for 99% of low/medium risk applications.
  - nutate 11y ago
    hard problem, but you clearly address it. There is a typo though: "in client" should probably read "if a client" or "if the client"
  - acveilleux 11y ago
    So if I read that correctly, the UserID (I find that naming confusing since users here aren't necessarily humans) creation has a human in the loop to add the UserID to Vault and therefore make the authorization decision?
coherentpony 11y ago
How do I know I can trust this?
- vially 11y ago
  Read the source code: https://github.com/hashicorp/vault https://github.com/hashicorp/vault
  - coherentpony 11y ago
    Thank you. I read the article but it wasn't clear that it was open source.
rmetzler 11y ago
I know it is very new, but I would love to read a Vault vs Keywhiz [1] article. [1] https://square.github.io/keywhiz/ https://square.github.io/keywhiz/
- hbbio 11y ago
  One is implemented in Java (Keywhiz), the other in Go?
  - cwp 11y ago
    Surely there's a little more to it than that.
outworlder 11y ago
> "Vault is the sixth such tool we've built. We've also built Vagrant, Packer, Serf, Consul, and Terraform. " Sigh. This is the first time that I actually feel overwhelmed. That nagging feeling that I'm doing something in some way solely because of habits. And that someone somewhere has created an awesome tool that would have allowed me to go home two hours earlier. The amount of new tools cropping in the last couple of years alone is the 'culprit'. These are awesome times, of course. Still, a part of my brain misses the time where you could ftp crap away to a server (even if automated by a Capistrano tool or similar) and spend the rest of the time arguing about the merits of puppet vs chef.
- debaserab2 11y ago
  Use the tools that are useful to you. Plenty of people hack out great things on old toolsets. If you don't want to be overwhelmed, don't be.
- ars 11y ago
  Don't feel that way. Very few tools like this stand the test of time. I avoid using new tools in production. I wait a few years first to see if they last - after all I'm going to have to support the system for years, I don't want the tools to be the weak spot.
- IgorPartola 11y ago
  Obligatory quote from that XKCD about automating things. Remember, most of these tools simply automate what you already do. If you don't spend much time performing certain things, you don't need the tool. The cargo cult of "I should use Packer to create my Vagrant boxes, so I can have dev/prod parity" only works to a point. Instead, look at where you spend lots of time, and then look for tools that automate that task. Personally, I try to be pretty conservative with tools that I start relying on. I know by painful experience that some tools get abandoned, some change drastically, making it impossible to upgrade, and some just turn out to be very buggy and horrible once you start putting them through their paces. In lots of cases, it's better to be critical than permissive when it comes to your infrastructure (and that's what these tools are aimed at). In almost all cases, your edge is in knowing the tools you use, and knowing about other tools. It's not in knowing all and using everything under the sun.
perlgeek 11y ago
It sounds very exciting. But, the first example made me flinch: $ vault write secret/foo value=bar Don't ever pass passwords on the command line. On linux, other users can see your command line (ps aux), and it usually ends up in a shell's history file.
- odiroot 11y ago
  Put a space before the command and it's gone from the history, AFAIR.
  - ams6110 11y ago
    Yes but very easy to forget to do. BTDT.
    - msutherl 11y ago
      Could you alias "vault" to " vault"?
      - gknoy 11y ago
        I doubt it. My history file shows all the git-related aliases I use (st, gd, gd-) rather than the expansions (git status, git diff, git diff --cached).
  - lenish 11y ago
    This is only true in bash if HISTCONTROL contains ignorespace or ignoreboth (ignoreboth implies ignorespace and ignoredups), so you should check that before relying on this functionality.
- mitchellh 11y ago
  Absolutely true! We support values coming from the command-line, stdin, and files: https://vaultproject.io/docs/commands/read-write.html https://vaultproject.io/docs/commands/read-write.html So you should use what you feel is most secure given the data you're entering.
dorfsmay 11y ago
Got to mention blackbox, it's based on gpg, and meant for teams interacting within version control systems: https://github.com/StackExchange/blackbox https://github.com/StackExchange/blackbox
- metavida 11y ago
  I'm not terribly familiar with blackbox, but based on the blackbox Readme it seems that this critique from Vault's website [1] addresses why HashiCorp thinks Vault is a better solution. > Chef, Puppet, etc. all solve this in a similar way: single-key encrypted storage. Chef has encrypted data bags, Puppet has encrypted Hiera, an so on. The encrypted data is always one secret (a password, a key, etc.) away from being decrypted, and this secret is generally not well protected since in an elastic environment, every server needs to somehow get this secret to decrypt the data. Additionally, access to the encrypted data isn't always logged, so if there is an intrusion, it isn't clear what data has been accessed and by who. [1]: https://vaultproject.io/intro/vs/chef-puppet-etc.html https://vaultproject.io/intro/vs/chef-puppet-etc.html
mikewhy 11y ago
this may be an obvious question, but how do things like Vault/Keywhiz differ from Consul/Etcd? Is it simply that the former is more encrypted than the latter?
- mikewhy 11y ago
  ah, as per http://vaultproject.io/intro/vs/consul.html http://vaultproject.io/intro/vs/consul.html > While Consul can be used to store secret information and gate access using ACLs, it is not designed for that purpose. As such, data is not encrypted in transit nor at rest, it does not have pluggable authentication mechanisms, and there is no per-request auditing mechanism. I guess my question now is why not add E2E encryption on top of Consul/Etcd instead of a whole new system?
  - acveilleux 11y ago
    It's desirable to keep secret managements simple and as self-contained and self-sufficient as possible. That way, it's much easier to drastically lock it down without impacting normal users too much. Your organization might have, say, 30 people who routinely need access to the orchestration / deployment infrastructure but maybe 3 who need access to cryptovariables.
malandrew 11y ago
What's coming next from hashicorp? The one infrastructure piece for which good open-source solutions still lack is ACL with solid auditing and variety of permissions approval models. There are a bunch of good ideas in this RFC, https://tools.ietf.org/html/rfc2904 https://tools.ietf.org/html/rfc2904 Something like a host-your-own Amazon IAM would be pretty dope. Add support for 2fa and especially u2f and you've got yourself a winner.
flowerpot 11y ago
Neat! I've been working on something similar for about the past six months, although my approach is a little more minimalistic, but also based on shamir's secret sharing. I was planning to release a preview the next couple of days. If any one is interested https://github.com/duse-io/api#how-it-works https://github.com/duse-io/api#how-it-works //edit My solution focuses more on the sharing aspect between people, but can just as well be used like Vault
hexedpackets 11y ago
This is really awesome! I've been passively looking for something like Vault for a while, can't wait to start integrating. I just finished tossing together a Chef cookbook to get Vault installed/configured up to the init/unseal step. Here's the link in case its useful to anyone: https://github.com/hexedpackets/chef-vault https://github.com/hexedpackets/chef-vault
AndrewWright 11y ago
There must be something in the air regarding tools for managing secrets. First Keywiz, then Credstash, a little python utility for managing secrets using AWS KMS https://github.com/LuminalOSS/credstash https://github.com/LuminalOSS/credstash
BuildTheRobots 11y ago
> Vault is one giant leap forward for practical security in a cloud environment. My biggest security concern with a cloud (or VPS) provider is that they can very easily read any keys or unencrypted data from memory. Still a worry when renting a dedicated box, but it's a damn site easier when it's virtualised. I don't see Vault helping me with this... but then I'm probably both paranoid and missing the point.
dserodio 11y ago
I haven't finished reading the docs yet, but this part[1] caught my attention: > When a backend is unmounted, all of its secrets are revoked and its data is deleted. If a backend is similar to a filesystem as mentioned in the docs, that's quite a surprise. If a backend is like a filesystem, the secrets (files) should become unavailable when unmounted, but should become available again after remounting the backend (filesystem). Are the secrets actually _deleted_ (ie, can't be recovered again), or am I misunderstanding? [1] http://vaultproject.io/intro/getting-started/secret-backends.html http://vaultproject.io/intro/getting-started/secret-backends...
- mitchellh 11y ago
  I'm responding a little late here, but hopefully you'll still see this. Yes, when you unmount a backend, all the data is deleted. Vault mostly behaves like a filesystem, but not in this scenario. The raw data is not preserved. If you want to preserve the raw data, you can remount (move the mount point). We didn't finish it for 0.1, but we already built in secret enumeration into the core. We plan to expose this at some point. If you want to unmount and preserve your secrets, you can always enumerate them and back them up elsewhere (unencrypted). This isn't yet available.
dustinrcollins 11y ago
The policies look like they'd be a bear to manage since they are path- and not role-based. In order to implement least privilege with multiple actors you'd have to be really careful with your paths. Example: 2 apps. App 1 needs secrets A B and D. App 2 need secrets A B and C. So we need to set up our paths in a way that App 1 can get A and B and C, but not D. App 2 needs C but not D. Now when you want to modify secret access from your apps you have to rethink how your paths are set up. When you're instead assigning permissions to roles this is a lot easier. An example: http://blog.conjur.net/what-is-a-devops-secrets-server http://blog.conjur.net/what-is-a-devops-secrets-server
MCRed 11y ago
Can someone explain hashicorps business model to me? Love all the projects. Are they contracting out features to companies? Selling support contracts? Very curious-- I think if people, especially individuals, can make a living from open source projects, that's great for everybody. But do you have to be "famous" to do it?
- druiid 11y ago
  To the best of my knowledge it comes from contracting/support and mostly their Vagrant Cloud (now Atlas) features that are business/enterprise only, which they charge money for. For example private hosting of Vagrant box images is a paid feature. The costs for that are per-user/per-month.
kgilpin 11y ago
It's pretty amazing to see how many new secrets tools for DevOps and cloud have suddenly materialized in the last few weeks. At Conjur, we have been in production in this space for almost two years. If you're wondering how to stack up these different tools against each other, here are a few facts about our solution: * At Conjur, we care deeply about three things : (1) the overall user experience for developers, operations, and security personnel (2) the security of our code (3) high availability * Conjur implements full-featured role-based access control for both "carbon" (people) and "silicon" (VMs, servers, containers, and other code) * Conjur solves not just secrets management, but also SSH access management and software-defined firewall (granular access control for microservices) * The Conjur core services are not open source, but our cryptography is (https://github.com/conjurinc/slosilo https://github.com/conjurinc/slosilo), as are many of our client libraries and connectors (for example: https://github.com/conjurinc/api-ruby https://github.com/conjurinc/api-ruby, https://github.com/conjur-cookbooks/conjur https://github.com/conjur-cookbooks/conjur). And Conjur cryptography has been audited by a respected 3rd party. * Conjur features packaged integrations with Chef (https://supermarket.chef.io/cookbooks/conjur https://supermarket.chef.io/cookbooks/conjur), Puppet (https://forge.puppetlabs.com/conjur/conjur https://forge.puppetlabs.com/conjur/conjur), Docker (http://developer.conjur.net/product_demo/secrets.html http://developer.conjur.net/product_demo/secrets.html), Heroku (https://github.com/conjurinc/heroku-buildpack-conjur https://github.com/conjurinc/heroku-buildpack-conjur), AWS and more. New integrations are easy to build using the Conjur HTTPS and LDAPS APIs (http://developer.conjur.net/reference/services http://developer.conjur.net/reference/services) We have also gone to great lengths to make it easy to try Conjur yourself, with automatically provisioned, self-service demo appliances featuring the Conjur server and client, Docker and Git: https://demo-factory-conjur.herokuapp.com/request/secrets https://demo-factory-conjur.herokuapp.com/request/secrets https://demo-factory-conjur.herokuapp.com/request/ssh https://demo-factory-conjur.herokuapp.com/request/ssh
2bluesc 11y ago
Would be cool if there was Yubikey support using Challenge-Response mechanism to seal the vault. If only I had time...
saintfiends 11y ago
Slightly OT, but is there such a tool (to store secrets securely and accessibly) for personal use that runs well with Linux and can be easily moved around (portable).
- vruiz 11y ago
  Try http://www.passwordstore.org/ http://www.passwordstore.org/
- notum 11y ago
  https://www.keepassx.org/ https://www.keepassx.org/ but I'm unsure if it has any sort of a CLI. It was mentioned in the EFF's updated SSL article.
LukeHoersten 11y ago
I haven't been to HashiCorp's website basically since they released Vagrant. I use vagrant all the time though. A few reactions. First: amazing design. Awesome documentation. Second: wow they have an awesome little suite of high-impact devops tools. I spent all day learning about them. The unified style of websites they have organized into "intro" and "documentation" is awesome and easy to really digest what's going on quickly. All of the tools seem powerful, well thought out, and complementary. Vagrant's been awesome to me and I'm excited to start using some of their tools like Packer, Consul, and Terraform in production!

440 points by eloycoto 11y ago

Looks interesting. There is definitely a need for something secure to manage secrets easily & efficiently. I'm not super familiar with Shamir's secret sharing. The post says that the master key is split up across a number of people. One thing that I am confused about is if Vault goes down (possibly in the middle of the night), does it require a quorum of humans to boot it back up?

limaoscarjuliet 11y ago
Yes, otherwise - when keys travel with secrets - there would be absolutely no security. This is akin to disk encryption where you have to type password on boot, but then data is accessible later.
viraptor 11y ago
Haven't tried it yet, but most likely yes. But if you actually need vault and do spread the key to more than 2 people, you likely have more than one vault server running at a time.
mitchellh 11y ago
It does. The solution to this problem is to run Vault in high availability mode[1]. This will have a set of stand-bys that can take over if the leader goes down. The idea is to unseal all the stand bys, and if the leader goes down, the standbys take over, and the sealed one can be unsealed at some other future point. Security-sensitive things have been an interesting UX challenge for us, but this compromise comes at an expense of UX for a much better security promise. As one of the other comments says around this comment: without this feature there would be no real security for the stored data. [1]: http://vaultproject.io/docs/concepts/ha.html http://vaultproject.io/docs/concepts/ha.html
- carllerche 11y ago
  Looks good. The post did not mention HA (that I saw), which caused the confusion.
- odiroot 11y ago
  Would you also recommend to set the lease duration to something higher in order to allow people to react? This way even if Vault dies during the night (and you don't have pager duty) at least some clients (e.g. your cloud instances) can live through?
- SEJeff 11y ago
  Can you call mlock in go to prevent any secrets from being paged to disk?
  - mitchellh 11y ago
    We already do this. And yes, you can (using the "syscall" package).
btilly 11y ago
Shamir's secret sharing is actually fairly simple. You want to have a secret shared by n people, any m of whom can decode it. You create a polynomial of degree m, and give each of those people the value of said polynomial at a different point. The secret will be the value of the polynomial at yet another point. From the value of the polynomial at any m points you can figure out the polynomial and the secret. But the value at any m-1 points does not let you predict its value anywhere else. (Detail. You would make the polynomial be over a finite field instead of the real numbers. But the math is all the same.)

deleted 11y ago

[deleted]

robszumski 11y ago
Check out the ACL RFC [1] which has been developed and is now available for testing in the first alpha of etcd v2.1.0 [2], we'd love your feedback. [1] https://coreos.com/etcd/docs/2.0.8/rfc/api_security.html https://coreos.com/etcd/docs/2.0.8/rfc/api_security.html [2] https://github.com/coreos/etcd/releases/tag/v2.1.0-alpha.0 https://github.com/coreos/etcd/releases/tag/v2.1.0-alpha.0
mpdehaan2 11y ago
(EDIT: I appear to be replying to a edited comment that was mentioning a possible naming confusion with ansible vault, I'll keep the post here even though the original is different now) re: naming (and disclaimer): I wrote Ansible and designed Ansible's vault (and no longer work on Ansible), but to be clear, Ansible's vault is a nicer CLI (IMHO) around an idea that Chef users were already used to: encrypted data bags. It was nothing new. It took about two weeks to implement, but we had a few security things to shake out after that made it take a bit longer before it got right. (Some people also got picky about whole-data-document encryption vs leaf-node-in-datastructure encryption, and I see points for doing it both ways, but went with the way that revealed the least amount of data) I think calling it vault is perfectly fair, as it's a very generic word, and words are in short supply. This is also doing a LOT more. I like what Mitchell and crew have done around revocation here, though I do somewhat wonder about the mode in the docs that sets secrets and allows them to show in bash history, which is why ansible vault spawned an editor. I'm sure it's got editor options too though. I'd probably also change the phrasing "HTTP API" to "HTTPS API" :) The dynamic secret stuff also looks pretty interesting. One of the interesting ideas might be to whether it allows sharing of secret values to some applications without letting someone read it (asking if a user can use X without letting the user READ x), which is something we did in tower - though it seems the ACL system might allow something like that. An example of that was the system could memorize an SSH key and let a team use it, but never give it to anyone, to avoid rotation scenarios. Anyway, all the lease ideas are very cool and it seems natural to want this decoupled. For a time I considered decoupling what what in Ansible vault from Ansible, though really it already operates on generic files so that's easy enough if someone wanted to take a stab at it. I suspect it wouldn't be impossible to make Ansible Vault have a hashicorp-vault-mode where it encrypts a file "into" the vault, and the contents that remain in your "ansible vault" file are nothing more than a pointer to a secret in Hashicorp Vault. In this way, it would just be another cipher type. Maybe that's crazy though - but I think I like that better than storing secrets in version control, where they exist forever unless you prune history. (And I never thought storing vault files in public source was a good idea). However, avoiding a server or database was always one of Ansible's design goals - for better or worse, that limited some of the things the core app could do, in ways that were both good and also occasionally limiting. Anyway, keep up the forward progress with the new things! I really envy the time to architect/design something so detailed here.

steveb 11y ago

I'm pretty excited about this release. We just added deploying Vault on top of consul in the microservice/realtime data platform we're building: https://github.com/CiscoCloud/microservices-infrastructure https://github.com/CiscoCloud/microservices-infrastructure We were already using consul as our service discovery engine and as a data store for dynamic configurations via consul template. We feel Vault will help us manage secrets on our infrastructure and inside containers.

jgrowl 11y ago
Neat! I'll have to see how you ended up using Vault since your project uses a lot of the same tech my personal webrtc server project uses. I'm also using ansible, docker, consul, registrator. Excuse the mess, I'm still working on it: https://github.com/jgrowl/livehq https://github.com/jgrowl/livehq

deleted 11y ago

[deleted]

mentat 11y ago

Any reason why AWS dynamic credentials use IAM users instead of STS and role assumption? That seems to be a somewhat more widely accepted way to do dynamic credentials / role proxying.

setheron 11y ago
that would definitely be more idiomatic in AWS source: I am an Amazon employee
ukd1 11y ago
came here to see how it was done; was hoping for STS :(
- mentat 11y ago
  https://github.com/dump247/docker-ec2-metadata https://github.com/dump247/docker-ec2-metadata and https://github.com/AdRoll/hologram https://github.com/AdRoll/hologram exist as examples of this, just FYI.

christop 11y ago

Interesting indeed! But I couldn't find anything on the website or documentation about how secrets are encrypted? The security page seems to be a generic one about reporting vulnerabilities: http://vaultproject.io/security.html http://vaultproject.io/security.html Edit: Found it under the threat model section here: https://vaultproject.io/docs/internals/security.html https://vaultproject.io/docs/internals/security.html

andrewmwatson 11y ago
the blog post mentioned AES-GCM 256bit
weitzj 11y ago
See: http://vaultproject.io/docs/internals/security.html http://vaultproject.io/docs/internals/security.html
- christop 11y ago
  Found it, thanks.

ankushio 11y ago

What are some of the existing tools/solutions which people use for managing secrets in a production environment?

amenghra 11y ago
https://corner.squareup.com/2015/04/keywhiz.html https://corner.squareup.com/2015/04/keywhiz.html
ubercow 11y ago
slightly different scope, but solves a similar problem https://github.com/cloudflare/redoctober https://github.com/cloudflare/redoctober good writeup about it here https://blog.cloudflare.com/red-october-cloudflares-open-source-implementation-of-the-two-man-rule/ https://blog.cloudflare.com/red-october-cloudflares-open-sou...
dustinrcollins 11y ago
http://conjur.net/ http://conjur.net/

jvehent 11y ago

There are many solutions out there that attempt to solve the secrets distribution problem. Vault is the newest one. None of them solve the fundamental problem of the secret zero: provisioning the first secret on an untrusted system the automated way is hard, and without that secret zero, there is no way to authenticate the new system. With Vault, you still need to figure out how to push secret zero (here a client authentication token). If you can solve that, why not push the secret you need to push directly, and bypass Vault entirely?

mitchellh 11y ago
We worked with a handful of companies to help us design the "App ID" authentication backend that does exactly that: https://vaultproject.io/docs/auth/app-id.html https://vaultproject.io/docs/auth/app-id.html It allows you to have a non-sensitive single factor within things like configuration management, and have the second factor come from a machine-local location (instance ID, MAC address, TSM module, etc.). The idea is that another team out of band sets the 2nd factor that developers and config management never get to see. The result is you have full automation without secret zero issues. As an additional security parameter, you can bind the two factors together to a single CIDR block, so you can have an additional factor restricting access from that machine's IP. The URL itself above goes into a lot more detail, but our beta users are successfully automated in an elastic environment this way.
- jvehent 11y ago
  That's definitely a trade-off compared to strong authentication, one that I wouldn't want to make in critical environments. But indeed a very nice feature for 99% of low/medium risk applications.
- nutate 11y ago
  hard problem, but you clearly address it. There is a typo though: "in client" should probably read "if a client" or "if the client"
- acveilleux 11y ago
  So if I read that correctly, the UserID (I find that naming confusing since users here aren't necessarily humans) creation has a human in the loop to add the UserID to Vault and therefore make the authorization decision?

coherentpony 11y ago

How do I know I can trust this?

vially 11y ago
Read the source code: https://github.com/hashicorp/vault https://github.com/hashicorp/vault
- coherentpony 11y ago
  Thank you. I read the article but it wasn't clear that it was open source.

rmetzler 11y ago

I know it is very new, but I would love to read a Vault vs Keywhiz [1] article. [1] https://square.github.io/keywhiz/ https://square.github.io/keywhiz/

hbbio 11y ago
One is implemented in Java (Keywhiz), the other in Go?
- cwp 11y ago
  Surely there's a little more to it than that.

outworlder 11y ago

> "Vault is the sixth such tool we've built. We've also built Vagrant, Packer, Serf, Consul, and Terraform. " Sigh. This is the first time that I actually feel overwhelmed. That nagging feeling that I'm doing something in some way solely because of habits. And that someone somewhere has created an awesome tool that would have allowed me to go home two hours earlier. The amount of new tools cropping in the last couple of years alone is the 'culprit'. These are awesome times, of course. Still, a part of my brain misses the time where you could ftp crap away to a server (even if automated by a Capistrano tool or similar) and spend the rest of the time arguing about the merits of puppet vs chef.

debaserab2 11y ago
Use the tools that are useful to you. Plenty of people hack out great things on old toolsets. If you don't want to be overwhelmed, don't be.
ars 11y ago
Don't feel that way. Very few tools like this stand the test of time. I avoid using new tools in production. I wait a few years first to see if they last - after all I'm going to have to support the system for years, I don't want the tools to be the weak spot.
IgorPartola 11y ago
Obligatory quote from that XKCD about automating things. Remember, most of these tools simply automate what you already do. If you don't spend much time performing certain things, you don't need the tool. The cargo cult of "I should use Packer to create my Vagrant boxes, so I can have dev/prod parity" only works to a point. Instead, look at where you spend lots of time, and then look for tools that automate that task. Personally, I try to be pretty conservative with tools that I start relying on. I know by painful experience that some tools get abandoned, some change drastically, making it impossible to upgrade, and some just turn out to be very buggy and horrible once you start putting them through their paces. In lots of cases, it's better to be critical than permissive when it comes to your infrastructure (and that's what these tools are aimed at). In almost all cases, your edge is in knowing the tools you use, and knowing about other tools. It's not in knowing all and using everything under the sun.

perlgeek 11y ago

It sounds very exciting. But, the first example made me flinch: $ vault write secret/foo value=bar Don't ever pass passwords on the command line. On linux, other users can see your command line (ps aux), and it usually ends up in a shell's history file.

odiroot 11y ago
Put a space before the command and it's gone from the history, AFAIR.
- ams6110 11y ago
  Yes but very easy to forget to do. BTDT.
  - msutherl 11y ago
    Could you alias "vault" to " vault"?
    - gknoy 11y ago
      I doubt it. My history file shows all the git-related aliases I use (st, gd, gd-) rather than the expansions (git status, git diff, git diff --cached).
- lenish 11y ago
  This is only true in bash if HISTCONTROL contains ignorespace or ignoreboth (ignoreboth implies ignorespace and ignoredups), so you should check that before relying on this functionality.
mitchellh 11y ago
Absolutely true! We support values coming from the command-line, stdin, and files: https://vaultproject.io/docs/commands/read-write.html https://vaultproject.io/docs/commands/read-write.html So you should use what you feel is most secure given the data you're entering.

dorfsmay 11y ago

Got to mention blackbox, it's based on gpg, and meant for teams interacting within version control systems: https://github.com/StackExchange/blackbox https://github.com/StackExchange/blackbox

metavida 11y ago
I'm not terribly familiar with blackbox, but based on the blackbox Readme it seems that this critique from Vault's website [1] addresses why HashiCorp thinks Vault is a better solution. > Chef, Puppet, etc. all solve this in a similar way: single-key encrypted storage. Chef has encrypted data bags, Puppet has encrypted Hiera, an so on. The encrypted data is always one secret (a password, a key, etc.) away from being decrypted, and this secret is generally not well protected since in an elastic environment, every server needs to somehow get this secret to decrypt the data. Additionally, access to the encrypted data isn't always logged, so if there is an intrusion, it isn't clear what data has been accessed and by who. [1]: https://vaultproject.io/intro/vs/chef-puppet-etc.html https://vaultproject.io/intro/vs/chef-puppet-etc.html

mikewhy 11y ago

this may be an obvious question, but how do things like Vault/Keywhiz differ from Consul/Etcd? Is it simply that the former is more encrypted than the latter?

mikewhy 11y ago
ah, as per http://vaultproject.io/intro/vs/consul.html http://vaultproject.io/intro/vs/consul.html > While Consul can be used to store secret information and gate access using ACLs, it is not designed for that purpose. As such, data is not encrypted in transit nor at rest, it does not have pluggable authentication mechanisms, and there is no per-request auditing mechanism. I guess my question now is why not add E2E encryption on top of Consul/Etcd instead of a whole new system?
- acveilleux 11y ago
  It's desirable to keep secret managements simple and as self-contained and self-sufficient as possible. That way, it's much easier to drastically lock it down without impacting normal users too much. Your organization might have, say, 30 people who routinely need access to the orchestration / deployment infrastructure but maybe 3 who need access to cryptovariables.

malandrew 11y ago

What's coming next from hashicorp? The one infrastructure piece for which good open-source solutions still lack is ACL with solid auditing and variety of permissions approval models. There are a bunch of good ideas in this RFC, https://tools.ietf.org/html/rfc2904 https://tools.ietf.org/html/rfc2904 Something like a host-your-own Amazon IAM would be pretty dope. Add support for 2fa and especially u2f and you've got yourself a winner.

flowerpot 11y ago

Neat! I've been working on something similar for about the past six months, although my approach is a little more minimalistic, but also based on shamir's secret sharing. I was planning to release a preview the next couple of days. If any one is interested https://github.com/duse-io/api#how-it-works https://github.com/duse-io/api#how-it-works //edit My solution focuses more on the sharing aspect between people, but can just as well be used like Vault

hexedpackets 11y ago

This is really awesome! I've been passively looking for something like Vault for a while, can't wait to start integrating. I just finished tossing together a Chef cookbook to get Vault installed/configured up to the init/unseal step. Here's the link in case its useful to anyone: https://github.com/hexedpackets/chef-vault https://github.com/hexedpackets/chef-vault

AndrewWright 11y ago

There must be something in the air regarding tools for managing secrets. First Keywiz, then Credstash, a little python utility for managing secrets using AWS KMS https://github.com/LuminalOSS/credstash https://github.com/LuminalOSS/credstash

BuildTheRobots 11y ago

> Vault is one giant leap forward for practical security in a cloud environment. My biggest security concern with a cloud (or VPS) provider is that they can very easily read any keys or unencrypted data from memory. Still a worry when renting a dedicated box, but it's a damn site easier when it's virtualised. I don't see Vault helping me with this... but then I'm probably both paranoid and missing the point.

dserodio 11y ago

I haven't finished reading the docs yet, but this part[1] caught my attention: > When a backend is unmounted, all of its secrets are revoked and its data is deleted. If a backend is similar to a filesystem as mentioned in the docs, that's quite a surprise. If a backend is like a filesystem, the secrets (files) should become unavailable when unmounted, but should become available again after remounting the backend (filesystem). Are the secrets actually _deleted_ (ie, can't be recovered again), or am I misunderstanding? [1] http://vaultproject.io/intro/getting-started/secret-backends.html http://vaultproject.io/intro/getting-started/secret-backends...

mitchellh 11y ago
I'm responding a little late here, but hopefully you'll still see this. Yes, when you unmount a backend, all the data is deleted. Vault mostly behaves like a filesystem, but not in this scenario. The raw data is not preserved. If you want to preserve the raw data, you can remount (move the mount point). We didn't finish it for 0.1, but we already built in secret enumeration into the core. We plan to expose this at some point. If you want to unmount and preserve your secrets, you can always enumerate them and back them up elsewhere (unencrypted). This isn't yet available.

dustinrcollins 11y ago

The policies look like they'd be a bear to manage since they are path- and not role-based. In order to implement least privilege with multiple actors you'd have to be really careful with your paths. Example: 2 apps. App 1 needs secrets A B and D. App 2 need secrets A B and C. So we need to set up our paths in a way that App 1 can get A and B and C, but not D. App 2 needs C but not D. Now when you want to modify secret access from your apps you have to rethink how your paths are set up. When you're instead assigning permissions to roles this is a lot easier. An example: http://blog.conjur.net/what-is-a-devops-secrets-server http://blog.conjur.net/what-is-a-devops-secrets-server

MCRed 11y ago

Can someone explain hashicorps business model to me? Love all the projects. Are they contracting out features to companies? Selling support contracts? Very curious-- I think if people, especially individuals, can make a living from open source projects, that's great for everybody. But do you have to be "famous" to do it?

druiid 11y ago
To the best of my knowledge it comes from contracting/support and mostly their Vagrant Cloud (now Atlas) features that are business/enterprise only, which they charge money for. For example private hosting of Vagrant box images is a paid feature. The costs for that are per-user/per-month.

kgilpin 11y ago

It's pretty amazing to see how many new secrets tools for DevOps and cloud have suddenly materialized in the last few weeks. At Conjur, we have been in production in this space for almost two years. If you're wondering how to stack up these different tools against each other, here are a few facts about our solution: * At Conjur, we care deeply about three things : (1) the overall user experience for developers, operations, and security personnel (2) the security of our code (3) high availability * Conjur implements full-featured role-based access control for both "carbon" (people) and "silicon" (VMs, servers, containers, and other code) * Conjur solves not just secrets management, but also SSH access management and software-defined firewall (granular access control for microservices) * The Conjur core services are not open source, but our cryptography is (https://github.com/conjurinc/slosilo https://github.com/conjurinc/slosilo), as are many of our client libraries and connectors (for example: https://github.com/conjurinc/api-ruby https://github.com/conjurinc/api-ruby, https://github.com/conjur-cookbooks/conjur https://github.com/conjur-cookbooks/conjur). And Conjur cryptography has been audited by a respected 3rd party. * Conjur features packaged integrations with Chef (https://supermarket.chef.io/cookbooks/conjur https://supermarket.chef.io/cookbooks/conjur), Puppet (https://forge.puppetlabs.com/conjur/conjur https://forge.puppetlabs.com/conjur/conjur), Docker (http://developer.conjur.net/product_demo/secrets.html http://developer.conjur.net/product_demo/secrets.html), Heroku (https://github.com/conjurinc/heroku-buildpack-conjur https://github.com/conjurinc/heroku-buildpack-conjur), AWS and more. New integrations are easy to build using the Conjur HTTPS and LDAPS APIs (http://developer.conjur.net/reference/services http://developer.conjur.net/reference/services) We have also gone to great lengths to make it easy to try Conjur yourself, with automatically provisioned, self-service demo appliances featuring the Conjur server and client, Docker and Git: https://demo-factory-conjur.herokuapp.com/request/secrets https://demo-factory-conjur.herokuapp.com/request/secrets https://demo-factory-conjur.herokuapp.com/request/ssh https://demo-factory-conjur.herokuapp.com/request/ssh

2bluesc 11y ago

Would be cool if there was Yubikey support using Challenge-Response mechanism to seal the vault. If only I had time...

saintfiends 11y ago

Slightly OT, but is there such a tool (to store secrets securely and accessibly) for personal use that runs well with Linux and can be easily moved around (portable).

vruiz 11y ago
Try http://www.passwordstore.org/ http://www.passwordstore.org/
notum 11y ago
https://www.keepassx.org/ https://www.keepassx.org/ but I'm unsure if it has any sort of a CLI. It was mentioned in the EFF's updated SSL article.

LukeHoersten 11y ago

I haven't been to HashiCorp's website basically since they released Vagrant. I use vagrant all the time though. A few reactions. First: amazing design. Awesome documentation. Second: wow they have an awesome little suite of high-impact devops tools. I spent all day learning about them. The unified style of websites they have organized into "intro" and "documentation" is awesome and easy to really digest what's going on quickly. All of the tools seem powerful, well thought out, and complementary. Vagrant's been awesome to me and I'm excited to start using some of their tools like Packer, Consul, and Terraform in production!