9 ms·

Hashicorp Vault v1.0

409 points by blopeur 8y ago

KerrickStaley 8y ago
Confidant is another open-source product in this space https://lyft.github.io/confidant/ https://lyft.github.io/confidant/ Disclosure: I work at Lyft.
- athenot 8y ago
  That looks nice but it does requires things to live in DynamoDB, so if you use multiple cloud providers and/or on prem, that might be limited.
  - gbrayut 8y ago
    Agreed... Neat project, but a hard dependency on AWS is an issue for us.
stevecalifornia 8y ago
Went to use Vault for Enterprise and heard we got an invoice for half a million dollars. Went with another solution.
- gigatexal 8y ago
  Quality software and developers cost money.
  - leetrout 8y ago
    That sounds very reasonable for what you get (assuming it's not node / core locked) and for a moderately sized org. I wish companies would publish their "whale sized" pricing. I know they're coming up with different pricing for different customers but it'd be great to be able to put a stake in the ground to make a judgement of utility. $40k / month with a warranty or SLA on patches is a lot cheaper than a team of "devops" maintaining it on their own (assuming it's core to the business ops).
    - gigatexal 8y ago
      Yeah. I totally agree. I understand why companies have a call us message on their enterprise pricing pages (don’t want to give your competitors an edge). But at the same time I just want the matrix of features are in the community edition and ones that aren’t. And then if we still want we will call and negotiate a good price. The guys at Cockroach Labs did just this but again their pricing is hidden behind an email to sales. That’s their prerogative but a bit annoying.
      - Already__Taken 8y ago
        You're talking in general I presume but if not it's further down https://www.hashicorp.com/products/vault/enterprise https://www.hashicorp.com/products/vault/enterprise
- dev_dull 8y ago
  I guess we ca that style “Medical bill” invoicing.
- mitchellh 8y ago
  Hey, I'm one of the founders of HashiCorp. We'd like to make more of Vault Enterprise available to smaller companies with lower palatable price points. This will be reflected in certain features omitted as well as a lower level of support. For now, please understand that our target _enterprise/commercial_ customer at the moment are Global 2000-esque companies. We currently have almost 10% of the global 2K as paying customers of Vault Enterprise. The features we've built along with the support you get reflect that (dedicated TAMs and so on). But we recognize that certain features of Vault Enterprise would be useful to smaller companies (replication and so forth). To that end, we're currently planning some new packaging/pricing aimed at this type of user. I have no timelines on when we'd publish that, but its something we're actively doing now. This should make Vault Enterprise more affordable by smaller companies (think 5-figures/year instead of 6-figures/year). Meanwhile, we're also making more "quality of life" features like auto-unseal available in Open Source. As we continue to add more features and value aimed directly at the Enterprise user, this lets us reevaluate and move more features into OSS and we have continued to do so throughout the life of Vault Enterprise. We hope this helps smaller companies adopt Vault successfully. And note that this is a great example of the model working: our success in Vault has funded growth in Vault staffing and that growth in Vault staffing has directly led to more OSS features and the growth in funding has led to our ability to more confidently make more features free. The community plays a huge role here, too. This is exactly how we intend for it to work! One thing we learned rather painfully is that selling to the 4-figure vs 5-figure vs 6-figure vs. 7-figure/year customer is each a _very_ different company-building exercise. The expectations of the 7-figure customer (and we have a number of those) is dedicated TAMs, dedicated support reps, quarterly in-person meetings about the state of the install, high impact on the roadmap, and much much more. That of course requires a certain kind of staffing. Very often this staffing scales "down" to lower price points but very rarely does the staffing at lower price points scale "up" to higher price points. As a company, we chose to go after the large enterprise deals first. This of course alienates some of the smaller deals since the large deals suck the air out of your org a bit. But as we've acquired more and more customers, grown, acquired more funding, etc. we're moving in that direction rapidly. So, hopefully we'll get there soon! I'm sorry that you were quoted at a point that didn't work for us mutually and I'm glad you find an alternate solution that worked for you. For others in a similar position: we're working on it!
- ah- 8y ago
  What was your motivation to go with Enterprise? It seems like most features are OSS now.
kitotik 8y ago
Congrats on the big milestone. I’ve been extremely happy working with HashiCorp tools for the past several years. Vault provides sooooo much out of the box, it’s hard for me to imagine spinning up a new project without it anymore. Which leads to my biggest fear...my jaded-self is expecting an ‘unfriendly’ acquisition (Microsoft, Alphabet) and/or some onerous licensing/pricing changes.
- hangonhn 8y ago
  I've been really happy with most of the Hashicorp tools too (Nomad is a bit iffy at the moment for me). Vault has given us some pretty seamless ways for managing our secrets sharing. I can't rave enough about Consul. It solved some seriously difficult CS problems (although a lot of credit goes to Raft too). Congrats Hashicorp. Huge fan of your work!
  - SteveNuts 8y ago
    What issues have you had with Nomad? We've been using OSS Nomad in production for a year now with very few problems, if any.
    - hangonhn 8y ago
      Oh interesting... a couple problems we've had: 1. The Nomad agent either dies or the process continues to run but in a way that seems unresponsive to requests. 2. When conditions change on the host (low disk space but we free up some disk space) Nomad doesn't seem to pick up on this and retry a failed deployment. This is after we've told it to re-evaluate. Perhaps we misunderstood what it's suppose to do under those conditions.
- mlangenberg 8y ago
  Which features of Vault are common across your projects? I've been looking at Vault for implementing envelope encryption and using Amazon KMS's API for encrypting keys is very similar.
  - 8y ago
SureshG 8y ago
We use https://square.github.io/keywhiz/ https://square.github.io/keywhiz/. It provide secrets as files in a directory, securely. So no special API, client libraries required to access it.
- jefferai 8y ago
  You can use Consul-Template (https://github.com/hashicorp/consul-template https://github.com/hashicorp/consul-template) -- yes, it really needs a rename -- to do this with Vault (or Consul).
honkycat 8y ago
Vault looks great, but I always balk at the operational overhead. Also the cost is significant. I'm at a tiny org, though. For smaller orgs and projects, Mozilla Sops is really great: https://github.com/mozilla/sops https://github.com/mozilla/sops It encrypts your secrets at rest using Google KMS, Amazon KMS, and various other cloud provider key services. You can then put those secrets into your code repository, cloud file storage, etc. and give your build pipeline a service account with the ability to decrypt the secret files. Scales like crap, but is quick and dirty when you need it.
- sethvargo 8y ago
  I agree on the operational load, but Vault is free. Did you mean Vault Enterprise? There's two versions - OSS and Enterprise, but a lot of small and mid-sized companies are able to use the free version at scale. Especially if you're a smaller org, OSS should be fine, so your costs are the VMs it's running on and the humans running it.
  - atmosx 8y ago
    I think he is talking about the burden of running a SPoF in highly available fashion, with backups, DR plans, monitoring, logging and all the baggage that comes with it.
    - sk5t 8y ago
      Right on. This is also what prevents me from using (unmanaged) Vault--not having the time to invest in learning the failure modes and internals, backup and restore strategies, in addition to the best ways to use it for the 5-10 patterns where it would become important.
- mlangenberg 8y ago
  As Cloud Auto Unseal is now part of the Open Source version, that mitigates the biggest operational overhead, doesn't it?
  - 8y ago
aerovistae 8y ago
My understanding of vault is not ironclad, but from what I have read it seems it allows ephemeral passwords that allow your application to get access to a service at time of initialization, and then the password ceases to be valid. Which means your application has access, but there's no credentials floating around anywhere that they could be compromised later. If anyone could correct me if I'm wrong, that would be great.
- no_one_ever 8y ago
  You've more or less got it. You authorize to the vault and store the secrets in memory. So no passwords on-disk/in-source 'floating around'.
  - xhrpost 8y ago
    But how do you authorize to the vault then? Aren't there credentials for that part?
    - ReidZB 8y ago
      Yes, there is the "initial secret" problem. There are a variety of different ways Vault handles that (look up the auth plugin docs), but at my org we use AWS IAM auth. So, app servers authenticate via their IAM instance profile (provided/managed by AWS) while developers assume a specific IAM role and then authenticate via that (which is how we enforce MFA for Vault without paying the crazy enterprise pricing). Note that with AWS IAM auth, AWS is a trusted third party, and accounts with high-powered IAM access (think AWS admins) end up having a great deal of authority in Vault, too. But for us, at least, these assumptions are reasonable.
- sethvargo 8y ago
  That's one piece of the functionality, but there's quite a few more. It's a pluggable system that can also generate more than "passwords" (like cloud IAM credentials, database credentials, PKI)
baby 8y ago
There's so much detailed craft and love poured into Hashicorp's codebases, this is great. Congrats Mitchell!
Dowwie 8y ago
What are the real-world ephemeral workloads that batch tokens are intended to address?
- mitchellh 8y ago
  This is a feature we built to specifically address large scale serverless workloads that are requesting a huge number of short lived tokens. And this isn’t theoretical: it was directly driven by multiple paying customers. Fun!
xhrpost 8y ago
Looking at some of these encryption-as-a-service providers, I'm a bit confused on one of the selling points. From my limited understanding: with a traditional system, you encrypt in the database but your encryption key likely exists on your main server, possibly in an environment variable. Attacker compromises main system and has access to both encrypted data and encryption keys. So, you instead use something like Vault to request an encryption key in real-time from a remote service and thus don't need to store it on your server. So, one of their selling points on their site is that Vault is better because two systems would have to be compromised by an attacker in order to decrypt sensitive data. The part I don't understand though is, if an attacker has compromised my server, could they not just initiate a request to Vault for a decryption key at that point? I feel like I'm missing something because this sounds like it remains a single point of failure.
- avar 8y ago
  You have N number of encryption keys for different parts of your data, then a single client server can only decrypt a small part of the data. Since you need to request the key each time to decrypt things like key rotation become easier since you can do them centrally. Even if you can decrypt data X on a client node the key you got an hour ago for X doesn't work anymore, you must request a new one. Then the server can do things like impose rate-limits. Maybe you can decrypt A-F, but only at a certain imposed rate, and alarms will be raised if the rate is exceeded.
- zimbatm 8y ago
  Vault provides an audit log. Since each process gets their own token it allows to correlate which machine/container got compromised. I think we will see more tooling being developed around that in the future to make the search more efficient. The other aspect is that if secret rotation is well exercised, it means that in case of a breach, taking an instance out of commission and rotating all the secrets is not a risky task anymore. If developers copy-and-paste tokens on slack, the window of attack is limited to the TTL of the secret. It also prevents developers from using the same secret for another unrelated service.
- jedberg 8y ago
  It's part of security in depth. Yes, if they own the server, they could keep requesting decryption keys. The main thing this protects against is someone getting a copy of the encrypted data, then breaking into a server and getting a key that is good forever. By using this system, it renders their copy of the encrypted data useless. Then you have to add in defenses for the active attack, such as rate limits, anomaly detection on access patterns for decryption keys, and the usual host and network based intrusion detection. It's one part of a complete security strategy.
chiu 8y ago
"Expanded Alibaba Cloud Integration" haha.
jteppinette 8y ago
I spent a few months of side work time working on a "secure-deployment-seed" project, https://github.com/jteppinette/secure-deployment-seed https://github.com/jteppinette/secure-deployment-seed. It is a set of Ansible playbooks/roles that have Vault/Consul at the center of a standard web deployment where privacy/security is taken to the Nth degree of perfectionist driven insanity. I ended up never using it, because it never really felt "perfect" to me.. There are so many circular dependencies between systems (DNS/Consul-Template/Consul/Vault/Ansible) and bootstrapping is just complete hell. Dive into that repo and witness it for yourself. I can see myself using this setup if I was ever just doing Ops work, but when you are also doing everything else, it is just too much. Anyways, congrats to the Hashicorp team. Your stuff really is topnotch.
deleted 8y ago
[deleted]

409 points by blopeur 8y ago

Confidant is another open-source product in this space https://lyft.github.io/confidant/ https://lyft.github.io/confidant/ Disclosure: I work at Lyft.

athenot 8y ago
That looks nice but it does requires things to live in DynamoDB, so if you use multiple cloud providers and/or on prem, that might be limited.
- gbrayut 8y ago
  Agreed... Neat project, but a hard dependency on AWS is an issue for us.

stevecalifornia 8y ago

Went to use Vault for Enterprise and heard we got an invoice for half a million dollars. Went with another solution.

gigatexal 8y ago
Quality software and developers cost money.
- leetrout 8y ago
  That sounds very reasonable for what you get (assuming it's not node / core locked) and for a moderately sized org. I wish companies would publish their "whale sized" pricing. I know they're coming up with different pricing for different customers but it'd be great to be able to put a stake in the ground to make a judgement of utility. $40k / month with a warranty or SLA on patches is a lot cheaper than a team of "devops" maintaining it on their own (assuming it's core to the business ops).
  - gigatexal 8y ago
    Yeah. I totally agree. I understand why companies have a call us message on their enterprise pricing pages (don’t want to give your competitors an edge). But at the same time I just want the matrix of features are in the community edition and ones that aren’t. And then if we still want we will call and negotiate a good price. The guys at Cockroach Labs did just this but again their pricing is hidden behind an email to sales. That’s their prerogative but a bit annoying.
    - Already__Taken 8y ago
      You're talking in general I presume but if not it's further down https://www.hashicorp.com/products/vault/enterprise https://www.hashicorp.com/products/vault/enterprise
dev_dull 8y ago
I guess we ca that style “Medical bill” invoicing.
mitchellh 8y ago
Hey, I'm one of the founders of HashiCorp. We'd like to make more of Vault Enterprise available to smaller companies with lower palatable price points. This will be reflected in certain features omitted as well as a lower level of support. For now, please understand that our target _enterprise/commercial_ customer at the moment are Global 2000-esque companies. We currently have almost 10% of the global 2K as paying customers of Vault Enterprise. The features we've built along with the support you get reflect that (dedicated TAMs and so on). But we recognize that certain features of Vault Enterprise would be useful to smaller companies (replication and so forth). To that end, we're currently planning some new packaging/pricing aimed at this type of user. I have no timelines on when we'd publish that, but its something we're actively doing now. This should make Vault Enterprise more affordable by smaller companies (think 5-figures/year instead of 6-figures/year). Meanwhile, we're also making more "quality of life" features like auto-unseal available in Open Source. As we continue to add more features and value aimed directly at the Enterprise user, this lets us reevaluate and move more features into OSS and we have continued to do so throughout the life of Vault Enterprise. We hope this helps smaller companies adopt Vault successfully. And note that this is a great example of the model working: our success in Vault has funded growth in Vault staffing and that growth in Vault staffing has directly led to more OSS features and the growth in funding has led to our ability to more confidently make more features free. The community plays a huge role here, too. This is exactly how we intend for it to work! One thing we learned rather painfully is that selling to the 4-figure vs 5-figure vs 6-figure vs. 7-figure/year customer is each a _very_ different company-building exercise. The expectations of the 7-figure customer (and we have a number of those) is dedicated TAMs, dedicated support reps, quarterly in-person meetings about the state of the install, high impact on the roadmap, and much much more. That of course requires a certain kind of staffing. Very often this staffing scales "down" to lower price points but very rarely does the staffing at lower price points scale "up" to higher price points. As a company, we chose to go after the large enterprise deals first. This of course alienates some of the smaller deals since the large deals suck the air out of your org a bit. But as we've acquired more and more customers, grown, acquired more funding, etc. we're moving in that direction rapidly. So, hopefully we'll get there soon! I'm sorry that you were quoted at a point that didn't work for us mutually and I'm glad you find an alternate solution that worked for you. For others in a similar position: we're working on it!
ah- 8y ago
What was your motivation to go with Enterprise? It seems like most features are OSS now.

kitotik 8y ago

Congrats on the big milestone. I’ve been extremely happy working with HashiCorp tools for the past several years. Vault provides sooooo much out of the box, it’s hard for me to imagine spinning up a new project without it anymore. Which leads to my biggest fear...my jaded-self is expecting an ‘unfriendly’ acquisition (Microsoft, Alphabet) and/or some onerous licensing/pricing changes.

hangonhn 8y ago
I've been really happy with most of the Hashicorp tools too (Nomad is a bit iffy at the moment for me). Vault has given us some pretty seamless ways for managing our secrets sharing. I can't rave enough about Consul. It solved some seriously difficult CS problems (although a lot of credit goes to Raft too). Congrats Hashicorp. Huge fan of your work!
- SteveNuts 8y ago
  What issues have you had with Nomad? We've been using OSS Nomad in production for a year now with very few problems, if any.
  - hangonhn 8y ago
    Oh interesting... a couple problems we've had: 1. The Nomad agent either dies or the process continues to run but in a way that seems unresponsive to requests. 2. When conditions change on the host (low disk space but we free up some disk space) Nomad doesn't seem to pick up on this and retry a failed deployment. This is after we've told it to re-evaluate. Perhaps we misunderstood what it's suppose to do under those conditions.
mlangenberg 8y ago
Which features of Vault are common across your projects? I've been looking at Vault for implementing envelope encryption and using Amazon KMS's API for encrypting keys is very similar.
- 8y ago

SureshG 8y ago

We use https://square.github.io/keywhiz/ https://square.github.io/keywhiz/. It provide secrets as files in a directory, securely. So no special API, client libraries required to access it.

jefferai 8y ago
You can use Consul-Template (https://github.com/hashicorp/consul-template https://github.com/hashicorp/consul-template) -- yes, it really needs a rename -- to do this with Vault (or Consul).

honkycat 8y ago

Vault looks great, but I always balk at the operational overhead. Also the cost is significant. I'm at a tiny org, though. For smaller orgs and projects, Mozilla Sops is really great: https://github.com/mozilla/sops https://github.com/mozilla/sops It encrypts your secrets at rest using Google KMS, Amazon KMS, and various other cloud provider key services. You can then put those secrets into your code repository, cloud file storage, etc. and give your build pipeline a service account with the ability to decrypt the secret files. Scales like crap, but is quick and dirty when you need it.

sethvargo 8y ago
I agree on the operational load, but Vault is free. Did you mean Vault Enterprise? There's two versions - OSS and Enterprise, but a lot of small and mid-sized companies are able to use the free version at scale. Especially if you're a smaller org, OSS should be fine, so your costs are the VMs it's running on and the humans running it.
- atmosx 8y ago
  I think he is talking about the burden of running a SPoF in highly available fashion, with backups, DR plans, monitoring, logging and all the baggage that comes with it.
  - sk5t 8y ago
    Right on. This is also what prevents me from using (unmanaged) Vault--not having the time to invest in learning the failure modes and internals, backup and restore strategies, in addition to the best ways to use it for the 5-10 patterns where it would become important.
mlangenberg 8y ago
As Cloud Auto Unseal is now part of the Open Source version, that mitigates the biggest operational overhead, doesn't it?
- 8y ago

aerovistae 8y ago

My understanding of vault is not ironclad, but from what I have read it seems it allows ephemeral passwords that allow your application to get access to a service at time of initialization, and then the password ceases to be valid. Which means your application has access, but there's no credentials floating around anywhere that they could be compromised later. If anyone could correct me if I'm wrong, that would be great.

no_one_ever 8y ago
You've more or less got it. You authorize to the vault and store the secrets in memory. So no passwords on-disk/in-source 'floating around'.
- xhrpost 8y ago
  But how do you authorize to the vault then? Aren't there credentials for that part?
  - ReidZB 8y ago
    Yes, there is the "initial secret" problem. There are a variety of different ways Vault handles that (look up the auth plugin docs), but at my org we use AWS IAM auth. So, app servers authenticate via their IAM instance profile (provided/managed by AWS) while developers assume a specific IAM role and then authenticate via that (which is how we enforce MFA for Vault without paying the crazy enterprise pricing). Note that with AWS IAM auth, AWS is a trusted third party, and accounts with high-powered IAM access (think AWS admins) end up having a great deal of authority in Vault, too. But for us, at least, these assumptions are reasonable.
sethvargo 8y ago
That's one piece of the functionality, but there's quite a few more. It's a pluggable system that can also generate more than "passwords" (like cloud IAM credentials, database credentials, PKI)

baby 8y ago

There's so much detailed craft and love poured into Hashicorp's codebases, this is great. Congrats Mitchell!

Dowwie 8y ago

What are the real-world ephemeral workloads that batch tokens are intended to address?

mitchellh 8y ago
This is a feature we built to specifically address large scale serverless workloads that are requesting a huge number of short lived tokens. And this isn’t theoretical: it was directly driven by multiple paying customers. Fun!

xhrpost 8y ago

Looking at some of these encryption-as-a-service providers, I'm a bit confused on one of the selling points. From my limited understanding: with a traditional system, you encrypt in the database but your encryption key likely exists on your main server, possibly in an environment variable. Attacker compromises main system and has access to both encrypted data and encryption keys. So, you instead use something like Vault to request an encryption key in real-time from a remote service and thus don't need to store it on your server. So, one of their selling points on their site is that Vault is better because two systems would have to be compromised by an attacker in order to decrypt sensitive data. The part I don't understand though is, if an attacker has compromised my server, could they not just initiate a request to Vault for a decryption key at that point? I feel like I'm missing something because this sounds like it remains a single point of failure.

avar 8y ago
You have N number of encryption keys for different parts of your data, then a single client server can only decrypt a small part of the data. Since you need to request the key each time to decrypt things like key rotation become easier since you can do them centrally. Even if you can decrypt data X on a client node the key you got an hour ago for X doesn't work anymore, you must request a new one. Then the server can do things like impose rate-limits. Maybe you can decrypt A-F, but only at a certain imposed rate, and alarms will be raised if the rate is exceeded.
zimbatm 8y ago
Vault provides an audit log. Since each process gets their own token it allows to correlate which machine/container got compromised. I think we will see more tooling being developed around that in the future to make the search more efficient. The other aspect is that if secret rotation is well exercised, it means that in case of a breach, taking an instance out of commission and rotating all the secrets is not a risky task anymore. If developers copy-and-paste tokens on slack, the window of attack is limited to the TTL of the secret. It also prevents developers from using the same secret for another unrelated service.
jedberg 8y ago
It's part of security in depth. Yes, if they own the server, they could keep requesting decryption keys. The main thing this protects against is someone getting a copy of the encrypted data, then breaking into a server and getting a key that is good forever. By using this system, it renders their copy of the encrypted data useless. Then you have to add in defenses for the active attack, such as rate limits, anomaly detection on access patterns for decryption keys, and the usual host and network based intrusion detection. It's one part of a complete security strategy.

chiu 8y ago

"Expanded Alibaba Cloud Integration" haha.

jteppinette 8y ago

I spent a few months of side work time working on a "secure-deployment-seed" project, https://github.com/jteppinette/secure-deployment-seed https://github.com/jteppinette/secure-deployment-seed. It is a set of Ansible playbooks/roles that have Vault/Consul at the center of a standard web deployment where privacy/security is taken to the Nth degree of perfectionist driven insanity. I ended up never using it, because it never really felt "perfect" to me.. There are so many circular dependencies between systems (DNS/Consul-Template/Consul/Vault/Ansible) and bootstrapping is just complete hell. Dive into that repo and witness it for yourself. I can see myself using this setup if I was ever just doing Ops work, but when you are also doing everything else, it is just too much. Anyways, congrats to the Hashicorp team. Your stuff really is topnotch.

deleted 8y ago

[deleted]