5 ms·

Vault looks great, but I always balk at the operational overhead. Also the cost is significant. I'm at a tiny org, though. For smaller orgs and projects, Mozil

Vault looks great, but I always balk at the operational overhead. Also the cost is significant. I'm at a tiny org, though. For smaller orgs and projects, Mozilla Sops is really great: https://github.com/mozilla/sops https://github.com/mozilla/sops It encrypts your secrets at rest using Google KMS, Amazon KMS, and various other cloud provider key services. You can then put those secrets into your code repository, cloud file storage, etc. and give your build pipeline a service account with the ability to decrypt the secret files. Scales like crap, but is quick and dirty when you need it.

sethvargo 8y ago
I agree on the operational load, but Vault is free. Did you mean Vault Enterprise? There's two versions - OSS and Enterprise, but a lot of small and mid-sized companies are able to use the free version at scale. Especially if you're a smaller org, OSS should be fine, so your costs are the VMs it's running on and the humans running it.
- atmosx 8y ago
  I think he is talking about the burden of running a SPoF in highly available fashion, with backups, DR plans, monitoring, logging and all the baggage that comes with it.
  - sk5t 8y ago
    Right on. This is also what prevents me from using (unmanaged) Vault--not having the time to invest in learning the failure modes and internals, backup and restore strategies, in addition to the best ways to use it for the 5-10 patterns where it would become important.
mlangenberg 8y ago
As Cloud Auto Unseal is now part of the Open Source version, that mitigates the biggest operational overhead, doesn't it?
- sethvargo 8y ago
  Correct, but it doesn't automate creation of new clusters (for dev, qa, stage, etc). You'd still need github.com/sethvargo/vault-init or similar to automate initialization, but unsealing can be automated in OSS now.
  - keeler 8y ago
    You can use terraform to deploy Vault clusters. There's also a Helm chart available for deploying Vault onto Kubernetes.
    - grogenaut 8y ago
      It is still infrastructure that you run, this is a real cost for lots of people. If you're running other people's terraform or whatever it's also a black box of software YOU are on the hook for but that you don't know anything about until you have to (3-5 hours after the service has been down).
      - keeler 8y ago
        But nobody's saying to run "other people's terraform or whatever," or that you should be running a sensitive service that you "don't know anything about until you have to." Common sense doesn't go out the window just because we're talking about hosting Vault within your infrastructure.
        grogenaut 8y ago
        You would think, but that's exactly what tons of people do. Many people who do not include the cost of running a service in the cost. It's very easy when the install is nice and easy to just say "sweet it's running" and pat hands "it's done". Then you realize you have no idea what's going on when it's totes on fire. At least with hashi products they're open source so you have a chance. And there's enterprise support, but you're still bleeding till you can get them in to help you out and understand what insanity you did with their product. This is why saas is preferable in a lot of situations. If you're not great at ops and make bad decisions, hopefully the SAAS folks are better at this than you. If you're really good at ops and think a ton about this stuff, then running it yourself makes sense a lot of time. And yes with SAAS now you have lockin and other problems which has their own set of solutions you should make sure you are doing, like layers of abstraction. Then you get into the self-fulfilling-infrastructure scenario. We're a vault shop, everyone use vault even for stuff that makes no sense to use vault for. Then rinse and repeat. Or you get into the sunk cost fallacy with your ops team... "what will they do if we replace this with a SAAS", so you keep services around just to not fire people, not because they're the best solution anymore. Lots of places to make bad decisions.
autrilla 8y ago
Could you go further into why SOPS scales badly?
- honkycat 8y ago
  Replied here: https://news.ycombinator.com/item?id=18605767 https://news.ycombinator.com/item?id=18605767
markpapadakis 8y ago
Apologies for the blatant promotion attempt but you may want to check out our simple (in pretty much every sense) KMS implementation. https://github.com/phaistos-networks/KMS https://github.com/phaistos-networks/KMS It’s inspired by vault and Google KMS and scales horizontally.
sytse 8y ago
I think the unique value of Vault is not that it can change the locks so that old keys no longer work. For this it needs to understand how to reset keys on for example every database [0]. I don't think Sops does that. I think there is a network effect to Vault that the more locks it can change the more people will use it and the more plugins will be added to it. I think Vault is hard to use and we're thinking of integrating it in GitLab to give it a friendly UI [1]. 0. https://www.vaultproject.io/docs/secrets/databases/index.html https://www.vaultproject.io/docs/secrets/databases/index.htm... 1. https://gitlab.com/gitlab-org/gitlab-ee/issues/7569 https://gitlab.com/gitlab-org/gitlab-ee/issues/7569
zimbatm 8y ago
Anther solution is to store secrets in the Terraform state and use a secured state backend (eg: S3+KMS) to share them. It makes key sharing and rotation a bit easier while not promoting copy-and-paste. You can use the "random" provider to generate secrets, the "tls" provider to generate certs and I also wrote a "secret" provider[1] to hold arbitrary secrets. The best part is that it doesn't require to maintain a long-running process. For small shops it's a good upgrade over storing credentials in code or using git-crypt. [1]: https://github.com/tweag/terraform-provider-secret https://github.com/tweag/terraform-provider-secret
- yeukhon 8y ago
  You don't want to store secrets in state files. Imagine you run a CI/CD system where you run terraform plan. Now secrets is all public.
  - toomuchtodo 8y ago
    https://github.com/hashicorp/terraform/issues/516 https://github.com/hashicorp/terraform/issues/516 TLDR: Terraform should use Vault for storing secrets in state, but does not support it yet. (it is occasionally unavoidable storing secrets in state due to resources orchestrated)
  - zimbatm 8y ago
    That's fine. Secrets are market as "sensitive" and not displayed in the plan output.
meddlepal 8y ago
Why does it scale like crap? Sops is on my shortlist to eval because we have a secrets mgmt problem.
- honkycat 8y ago
  A few reasons. SOPS is fine, I was more referring to my implementation. The reason my pipeline scales poorly is because it requires a full build and deployment cycle to update my dev / stg / prod configurations. Also, if you store the encrypted files in our git repos as I do, you get constant merge conflicts and basically useless git history. It is an extremely lazy implementation and literally the bare minimum I could do to get get my application configurations updating in my CI/CD stack.
  - autrilla 8y ago
    We (Mozilla) store our SOPS-encrypted files in git repos and we don't really get merge conflicts, but that's just because we structure secret files in a way such that two people don't edit the same file at the same time often. Git history is also fine. Have you considered configuring SOPS as a differ for git?

sethvargo 8y ago

I agree on the operational load, but Vault is free. Did you mean Vault Enterprise? There's two versions - OSS and Enterprise, but a lot of small and mid-sized companies are able to use the free version at scale. Especially if you're a smaller org, OSS should be fine, so your costs are the VMs it's running on and the humans running it.

atmosx 8y ago
I think he is talking about the burden of running a SPoF in highly available fashion, with backups, DR plans, monitoring, logging and all the baggage that comes with it.
- sk5t 8y ago
  Right on. This is also what prevents me from using (unmanaged) Vault--not having the time to invest in learning the failure modes and internals, backup and restore strategies, in addition to the best ways to use it for the 5-10 patterns where it would become important.

mlangenberg 8y ago

As Cloud Auto Unseal is now part of the Open Source version, that mitigates the biggest operational overhead, doesn't it?

sethvargo 8y ago
Correct, but it doesn't automate creation of new clusters (for dev, qa, stage, etc). You'd still need github.com/sethvargo/vault-init or similar to automate initialization, but unsealing can be automated in OSS now.
- keeler 8y ago
  You can use terraform to deploy Vault clusters. There's also a Helm chart available for deploying Vault onto Kubernetes.
  - grogenaut 8y ago
    It is still infrastructure that you run, this is a real cost for lots of people. If you're running other people's terraform or whatever it's also a black box of software YOU are on the hook for but that you don't know anything about until you have to (3-5 hours after the service has been down).
    - keeler 8y ago
      But nobody's saying to run "other people's terraform or whatever," or that you should be running a sensitive service that you "don't know anything about until you have to." Common sense doesn't go out the window just because we're talking about hosting Vault within your infrastructure.
      - grogenaut 8y ago
        You would think, but that's exactly what tons of people do. Many people who do not include the cost of running a service in the cost. It's very easy when the install is nice and easy to just say "sweet it's running" and pat hands "it's done". Then you realize you have no idea what's going on when it's totes on fire. At least with hashi products they're open source so you have a chance. And there's enterprise support, but you're still bleeding till you can get them in to help you out and understand what insanity you did with their product. This is why saas is preferable in a lot of situations. If you're not great at ops and make bad decisions, hopefully the SAAS folks are better at this than you. If you're really good at ops and think a ton about this stuff, then running it yourself makes sense a lot of time. And yes with SAAS now you have lockin and other problems which has their own set of solutions you should make sure you are doing, like layers of abstraction. Then you get into the self-fulfilling-infrastructure scenario. We're a vault shop, everyone use vault even for stuff that makes no sense to use vault for. Then rinse and repeat. Or you get into the sunk cost fallacy with your ops team... "what will they do if we replace this with a SAAS", so you keep services around just to not fire people, not because they're the best solution anymore. Lots of places to make bad decisions.

autrilla 8y ago

Could you go further into why SOPS scales badly?

honkycat 8y ago
Replied here: https://news.ycombinator.com/item?id=18605767 https://news.ycombinator.com/item?id=18605767

markpapadakis 8y ago

Apologies for the blatant promotion attempt but you may want to check out our simple (in pretty much every sense) KMS implementation. https://github.com/phaistos-networks/KMS https://github.com/phaistos-networks/KMS It’s inspired by vault and Google KMS and scales horizontally.

sytse 8y ago

I think the unique value of Vault is not that it can change the locks so that old keys no longer work. For this it needs to understand how to reset keys on for example every database [0]. I don't think Sops does that. I think there is a network effect to Vault that the more locks it can change the more people will use it and the more plugins will be added to it. I think Vault is hard to use and we're thinking of integrating it in GitLab to give it a friendly UI [1]. 0. https://www.vaultproject.io/docs/secrets/databases/index.html https://www.vaultproject.io/docs/secrets/databases/index.htm... 1. https://gitlab.com/gitlab-org/gitlab-ee/issues/7569 https://gitlab.com/gitlab-org/gitlab-ee/issues/7569

zimbatm 8y ago

Anther solution is to store secrets in the Terraform state and use a secured state backend (eg: S3+KMS) to share them. It makes key sharing and rotation a bit easier while not promoting copy-and-paste. You can use the "random" provider to generate secrets, the "tls" provider to generate certs and I also wrote a "secret" provider[1] to hold arbitrary secrets. The best part is that it doesn't require to maintain a long-running process. For small shops it's a good upgrade over storing credentials in code or using git-crypt. [1]: https://github.com/tweag/terraform-provider-secret https://github.com/tweag/terraform-provider-secret

yeukhon 8y ago
You don't want to store secrets in state files. Imagine you run a CI/CD system where you run terraform plan. Now secrets is all public.
- toomuchtodo 8y ago
  https://github.com/hashicorp/terraform/issues/516 https://github.com/hashicorp/terraform/issues/516 TLDR: Terraform should use Vault for storing secrets in state, but does not support it yet. (it is occasionally unavoidable storing secrets in state due to resources orchestrated)
- zimbatm 8y ago
  That's fine. Secrets are market as "sensitive" and not displayed in the plan output.

meddlepal 8y ago

Why does it scale like crap? Sops is on my shortlist to eval because we have a secrets mgmt problem.

honkycat 8y ago
A few reasons. SOPS is fine, I was more referring to my implementation. The reason my pipeline scales poorly is because it requires a full build and deployment cycle to update my dev / stg / prod configurations. Also, if you store the encrypted files in our git repos as I do, you get constant merge conflicts and basically useless git history. It is an extremely lazy implementation and literally the bare minimum I could do to get get my application configurations updating in my CI/CD stack.
- autrilla 8y ago
  We (Mozilla) store our SOPS-encrypted files in git repos and we don't really get merge conflicts, but that's just because we structure secret files in a way such that two people don't edit the same file at the same time often. Git history is also fine. Have you considered configuring SOPS as a differ for git?