4 ms·

I am progressively adopting Hashicorp Vault as the secrets manager of choice. It can be used by a variety of different scenarios -- directly into the applicatio

I am progressively adopting Hashicorp Vault as the secrets manager of choice. It can be used by a variety of different scenarios -- directly into the application using AppRole, with Terraform using it's secrets provider, by developers, during vault authentication when they get their secrets and access regenerated. This way I am not bound to docker swarm, or keywhiz, or god forbid AWS Secrets Manager. As of now, I am still exposing secrets with Env Vars, but the next step is to use Vault directly. Vault has been pretty reliable so far. It is using AWS KMS for managing the master key and a scalable DynamoDB table for high availability backend.

gpmcadam 6y ago
How do you share the token to access Vault to your code?
- linkdd 6y ago
  Using AppRoles and consul-template. And if you're on Kubernetes, you can use https://github.com/sethvargo/vault-kubernetes-authenticator https://github.com/sethvargo/vault-kubernetes-authenticator in an init container.
  - gpmcadam 6y ago
    And how does k8s pass that value to the app?
    - linkdd 6y ago
      You can use consul-template to generate a config file that lives in-memory (volumes.myvolume.emptyDir.medium = InMemory in a PodSpec).
carlosf 6y ago
Vault looks cool, but looking at the reference architecture [1] my guts tell me it's much easier to fuck up setting up a Vault cluster than environment variables. [1] https://learn.hashicorp.com/vault/operations/ops-reference-architecture#network-connectivity-details https://learn.hashicorp.com/vault/operations/ops-reference-a...
- gchamonlive 6y ago
  You can always use their official Terraform module for AWS if you are not comfortable with setting it all up by yourself: https://registry.terraform.io/modules/hashicorp/vault/aws/0.0.9/submodules/vault-cluster https://registry.terraform.io/modules/hashicorp/vault/aws/0.... But if you cut Consul for backend, if you are not using consul for other service discovery and Nomad etc..., you can simplify that deployment a whole lot. You make sure you open the cluster communication port, setup a Application Load Balancer in front of the cluster to balance traffic and serve SSL, configure auto-unseal using AWS KMS (since you are not using it too often, 1$ is OK to have AWS manage your master key), deploy Vault on every cluster instance, and use something managed like DynamoDB as your backend. I think this is a pretty simple yet scalable setup. The amount of cloud lock-in is pretty minimal and can easily be replaced with HAProxy setup.
- omgbear 6y ago
  If you are in AWS, you can use S3 for data storage and DynamoDB for HA. Our first install was set up using their 'best practices' and consul, being stateful, is frustrating to run in kubernetes. We migrated to using S3/Dynamo and now have fewer moving parts and haven't seen any issues.
  - carlosf 6y ago
    Thanks for the advice!
jen20 6y ago
> This way I am not bound to [...] keywhiz No, but you’re bound to Vault? I agree that Vault is likely the best solution to this (dynamic secrets are super useful), but “not being coupled to something” is not a benefit you get from it.
- gchamonlive 6y ago
  You are right, I think that was a little harsh on my part. I can't really say much about keywhiz, I should have left it out of my comment. Yes, you are tied to Hashicorp Vault once you adopt it, but it isn't much of a problem because you can easily redeploy it in a different cloud. It might be true to keywhiz, so my comment wouldn't be valid to this point.
oceanfloor 6y ago
What's wrong with AWS Secrets Manager? If I'm already working 100% in Amazon, I'm tempted to use Secrets Manager rather than justify the cost in hours to deploy and maintain a Vault cluster. Interested in your opinion.
- gchamonlive 6y ago
  Cloud lock-in (not being able for example to manage cross-platform credentials) and cost. By itself AWS Secrets Manager is fine, but as with everything AWS, billing is pretty opaque. If you couple this with a poorly planed Lambda setup, costs can potentially uncontrollably scale. For what you are paying, AWS Secrets Manager seems too barebones/rigid.
  - tyingq 6y ago
    Lock in shouldn't be too bad. It's create, get, set, list, delete, describe. I imagine the API footprint isn't much different if you switch over to something else. Most apps will just use get().
    - gchamonlive 6y ago
      It shouldn't, but it can be very bad. And with vault you potentially circumvent that and end up with a tool that is way more flexible than Secrets Manager.
- katnegermis 6y ago
  Not OP, but I've recently worked with AWS Parameter Store a lot, and I've been very happy with it so far. We store all configuration (including secrets) using it. We started by using https://github.com/segmentio/chamber/ https://github.com/segmentio/chamber/, but because of the way we decided to structure our secrets we decided to write a clone of it ourselves (https://github.com/micvbang/confman-go https://github.com/micvbang/confman-go). In practice, we have a client written in Python that grabs configuration from Parameter Store and puts it into the process' ENV variables at startup. This allows us to avoid vendor lockin in the rest of our code, since we still just fetch all config from the environment. I like it so far :)

gpmcadam 6y ago

How do you share the token to access Vault to your code?

linkdd 6y ago
Using AppRoles and consul-template. And if you're on Kubernetes, you can use https://github.com/sethvargo/vault-kubernetes-authenticator https://github.com/sethvargo/vault-kubernetes-authenticator in an init container.
- gpmcadam 6y ago
  And how does k8s pass that value to the app?
  - linkdd 6y ago
    You can use consul-template to generate a config file that lives in-memory (volumes.myvolume.emptyDir.medium = InMemory in a PodSpec).

carlosf 6y ago

Vault looks cool, but looking at the reference architecture [1] my guts tell me it's much easier to fuck up setting up a Vault cluster than environment variables. [1] https://learn.hashicorp.com/vault/operations/ops-reference-architecture#network-connectivity-details https://learn.hashicorp.com/vault/operations/ops-reference-a...

gchamonlive 6y ago
You can always use their official Terraform module for AWS if you are not comfortable with setting it all up by yourself: https://registry.terraform.io/modules/hashicorp/vault/aws/0.0.9/submodules/vault-cluster https://registry.terraform.io/modules/hashicorp/vault/aws/0.... But if you cut Consul for backend, if you are not using consul for other service discovery and Nomad etc..., you can simplify that deployment a whole lot. You make sure you open the cluster communication port, setup a Application Load Balancer in front of the cluster to balance traffic and serve SSL, configure auto-unseal using AWS KMS (since you are not using it too often, 1$ is OK to have AWS manage your master key), deploy Vault on every cluster instance, and use something managed like DynamoDB as your backend. I think this is a pretty simple yet scalable setup. The amount of cloud lock-in is pretty minimal and can easily be replaced with HAProxy setup.
omgbear 6y ago
If you are in AWS, you can use S3 for data storage and DynamoDB for HA. Our first install was set up using their 'best practices' and consul, being stateful, is frustrating to run in kubernetes. We migrated to using S3/Dynamo and now have fewer moving parts and haven't seen any issues.
- carlosf 6y ago
  Thanks for the advice!

jen20 6y ago

> This way I am not bound to [...] keywhiz No, but you’re bound to Vault? I agree that Vault is likely the best solution to this (dynamic secrets are super useful), but “not being coupled to something” is not a benefit you get from it.

gchamonlive 6y ago
You are right, I think that was a little harsh on my part. I can't really say much about keywhiz, I should have left it out of my comment. Yes, you are tied to Hashicorp Vault once you adopt it, but it isn't much of a problem because you can easily redeploy it in a different cloud. It might be true to keywhiz, so my comment wouldn't be valid to this point.

oceanfloor 6y ago

What's wrong with AWS Secrets Manager? If I'm already working 100% in Amazon, I'm tempted to use Secrets Manager rather than justify the cost in hours to deploy and maintain a Vault cluster. Interested in your opinion.

gchamonlive 6y ago
Cloud lock-in (not being able for example to manage cross-platform credentials) and cost. By itself AWS Secrets Manager is fine, but as with everything AWS, billing is pretty opaque. If you couple this with a poorly planed Lambda setup, costs can potentially uncontrollably scale. For what you are paying, AWS Secrets Manager seems too barebones/rigid.
- tyingq 6y ago
  Lock in shouldn't be too bad. It's create, get, set, list, delete, describe. I imagine the API footprint isn't much different if you switch over to something else. Most apps will just use get().
  - gchamonlive 6y ago
    It shouldn't, but it can be very bad. And with vault you potentially circumvent that and end up with a tool that is way more flexible than Secrets Manager.
katnegermis 6y ago
Not OP, but I've recently worked with AWS Parameter Store a lot, and I've been very happy with it so far. We store all configuration (including secrets) using it. We started by using https://github.com/segmentio/chamber/ https://github.com/segmentio/chamber/, but because of the way we decided to structure our secrets we decided to write a clone of it ourselves (https://github.com/micvbang/confman-go https://github.com/micvbang/confman-go). In practice, we have a client written in Python that grabs configuration from Parameter Store and puts it into the process' ENV variables at startup. This allows us to avoid vendor lockin in the rest of our code, since we still just fetch all config from the environment. I like it so far :)