6 ms·

This just pushed the problem further down the stack. You should have keys to unlock vault when it is restarted. How do you secure those keys?

mmm_grayons 6y ago
I can't seem to understand how a "secrets manager" helps things. Could someone who does ELI5 why it's better than a config file with permissions locked down?
- bvandewalle 6y ago
  It makes sense at scale. If you are a company of two there are probably better solutions. At scale, you can very granularly define policies for each secret. When a secret is accessed, it is done so through a user or application identity. Each access is also logged.
  - scott00 6y ago
    So then how do you manage the secret that authenticates an application's identity? And what good is the logging if after an application has the secret it can do whatever it wants with it?
    - bvandewalle 6y ago
      if it is an instance on the cloud, GCP and AWS let you define ServiceAccounts that get populated on the Instance at boot time. you should only let the instance access the secret it requires.
      - rdslw 6y ago
        and how do you manage secrets that let you define that ServiceAccounts? As OP wrote, you did not solve it, just moved it to a different level.
- closeparen 6y ago
  In a monolithic application they're pretty close. In a microservices architecture, suppose you have 100 services and schedule 6 service containers per host. That is 94 services worth of secrets that don't need to be there, expanding the blast radius of single host compromise. There may be a credential on that host that can be exchanged for more secrets, but audit logs can tell you whether it actually was, and you can revoke it. Secrets may still need to go to config files for third party stuff, but you can write your in-house applications to fetch secrets at runtime and hold them only in memory. That's less opportunity for compromise vs. both memory and disk. Also have heard that Linux's process address space isolation is less prone to vulnerabilities than user account separation or filesystem permissions. Not really sure how true that is.
- LASR 6y ago
  The real difference over a file is that you some intelligence(the manager) running that can offer all kinds of additional security functionality. A typical deployment might involve placing the manager on a secure host that has access to generate and rotate keys, for example. The manager can then configured to re-generate keys and vend them on-demand to instances that need require them. You can configure these keys with very limited access, and also make them expiring. The manager then becomes effectively a keystore that can never export master keys, but only vends out some limited-scope keys to other instances. Other instances would have to authenticate using some pre-configured host keys or even be authenticated directly though the cloud provider. If your instances are compromised, the worst someone can do is to get access to a limited-scope key that will expire. Hopefully you have other measures in place that would prevent and detect someone from just sitting on an instance and sucking up all your data and exporting them.
  - mmm_grayons 6y ago
    Thanks, this makes a lot more sense and seems much more useful than the justifications I've heard others give. I appreciate it.
bvandewalle 6y ago
If you use Vault, you should use it as an RBAC system as well. That means that each application got a ServiceAccount (SA) and each user got a username/password. Based on your identity, you get access to specific secrets from Vault.
goodoldneon 6y ago
Store them in a separate instance of Vault.
closeparen 6y ago
You could do a lot worse than to have the operators store their shares in their separate password managers or on paper in safe places. It does admit the possibility that an operator's share could be copied. To work around that you can get a proper HSM that needs a quorum of smart cards presented to unlock. (The offline, low QPS, root of trust-oriented ones are not exactly cheap, but much cheaper than the network-attached ones targeting high QPS transactions). Vault Enterprise has PKCS#11 integration. With the Thales nShield stuff, you can replicate key material from one to another for redundancy while allegedly still preserving the "can't ever get keys out in plaintext" property. Not sure about others.
ruuda 6y ago
We put them in a password manager (1Password) to which multiple accounts have access. Each account is secured with a key, passphrase, and 2FA.
jmarcher 6y ago
You can use GCPKMS (probably AWS KMS too) to unseal Vault automatically. https://github.com/sethvargo/vault-on-gke/blob/master/terraform/k8s.tf#L203-L208 https://github.com/sethvargo/vault-on-gke/blob/master/terraf... The KMS ring itself is only accessible by Vault. People with high enough privileges for our Vault GCP project could technically grant themselves access to it, but on day-to-day business, nobody can view the project. At some other place, where we were using AWS, I wrote a script that would store encrypt unseal keys (need multiple due to shamir) via pgp using their keybase public key. IIRC you can store encrypted keys in Vault that can be accessed only for unsealing purposes (please correct me on this). When needing an unseal, the script on the user side would then decrypt the key and submit it to Vault for unsealing. It worked well enough, and it felt like being in the movie/game GoldenEye, but nowhere as slick as auto unseal. Regardless of the setup, yes, at the end of the day any solution is really just pushing the problem further down the stack.
perlgeek 6y ago
There are actually two answers to that: * You can split the key between different persons, and you can even implement "n of k" schemes, like you specify (at key creation time) that you need any 4 out of 9 shards to unseal the vault. You can then keep those shards on separate operator's laptops, in separate backup systems etc. * You can use a hardware security module to unseal the vault (support for that is not included in the free version, IIRC). But even if the vault wasn't stored encrypted, it'd still be a huge improvement over "keys on NFS", because only machine administrators get access to the whole DB, and you can limit and audit the access of everybody else in a sane manner.
greyhair 6y ago
We also use an HSM. Smart cards are held by two separate people, each card set is different. Both have to be present to restart the HSM. Our HSM can require up to five different cards be required for certain operations. For us, we only require two for normal operations. For HSM management (key generation, card authentication, etc..) we require a third card set member to be present. This protects against accidental key erasure, or fraud.
- deleted 6y ago
  [deleted]

mmm_grayons 6y ago

I can't seem to understand how a "secrets manager" helps things. Could someone who does ELI5 why it's better than a config file with permissions locked down?

bvandewalle 6y ago
It makes sense at scale. If you are a company of two there are probably better solutions. At scale, you can very granularly define policies for each secret. When a secret is accessed, it is done so through a user or application identity. Each access is also logged.
- scott00 6y ago
  So then how do you manage the secret that authenticates an application's identity? And what good is the logging if after an application has the secret it can do whatever it wants with it?
  - bvandewalle 6y ago
    if it is an instance on the cloud, GCP and AWS let you define ServiceAccounts that get populated on the Instance at boot time. you should only let the instance access the secret it requires.
    - rdslw 6y ago
      and how do you manage secrets that let you define that ServiceAccounts? As OP wrote, you did not solve it, just moved it to a different level.
closeparen 6y ago
In a monolithic application they're pretty close. In a microservices architecture, suppose you have 100 services and schedule 6 service containers per host. That is 94 services worth of secrets that don't need to be there, expanding the blast radius of single host compromise. There may be a credential on that host that can be exchanged for more secrets, but audit logs can tell you whether it actually was, and you can revoke it. Secrets may still need to go to config files for third party stuff, but you can write your in-house applications to fetch secrets at runtime and hold them only in memory. That's less opportunity for compromise vs. both memory and disk. Also have heard that Linux's process address space isolation is less prone to vulnerabilities than user account separation or filesystem permissions. Not really sure how true that is.
LASR 6y ago
The real difference over a file is that you some intelligence(the manager) running that can offer all kinds of additional security functionality. A typical deployment might involve placing the manager on a secure host that has access to generate and rotate keys, for example. The manager can then configured to re-generate keys and vend them on-demand to instances that need require them. You can configure these keys with very limited access, and also make them expiring. The manager then becomes effectively a keystore that can never export master keys, but only vends out some limited-scope keys to other instances. Other instances would have to authenticate using some pre-configured host keys or even be authenticated directly though the cloud provider. If your instances are compromised, the worst someone can do is to get access to a limited-scope key that will expire. Hopefully you have other measures in place that would prevent and detect someone from just sitting on an instance and sucking up all your data and exporting them.
- mmm_grayons 6y ago
  Thanks, this makes a lot more sense and seems much more useful than the justifications I've heard others give. I appreciate it.

bvandewalle 6y ago

If you use Vault, you should use it as an RBAC system as well. That means that each application got a ServiceAccount (SA) and each user got a username/password. Based on your identity, you get access to specific secrets from Vault.

goodoldneon 6y ago

Store them in a separate instance of Vault.

closeparen 6y ago

You could do a lot worse than to have the operators store their shares in their separate password managers or on paper in safe places. It does admit the possibility that an operator's share could be copied. To work around that you can get a proper HSM that needs a quorum of smart cards presented to unlock. (The offline, low QPS, root of trust-oriented ones are not exactly cheap, but much cheaper than the network-attached ones targeting high QPS transactions). Vault Enterprise has PKCS#11 integration. With the Thales nShield stuff, you can replicate key material from one to another for redundancy while allegedly still preserving the "can't ever get keys out in plaintext" property. Not sure about others.

ruuda 6y ago

We put them in a password manager (1Password) to which multiple accounts have access. Each account is secured with a key, passphrase, and 2FA.

jmarcher 6y ago

You can use GCPKMS (probably AWS KMS too) to unseal Vault automatically. https://github.com/sethvargo/vault-on-gke/blob/master/terraform/k8s.tf#L203-L208 https://github.com/sethvargo/vault-on-gke/blob/master/terraf... The KMS ring itself is only accessible by Vault. People with high enough privileges for our Vault GCP project could technically grant themselves access to it, but on day-to-day business, nobody can view the project. At some other place, where we were using AWS, I wrote a script that would store encrypt unseal keys (need multiple due to shamir) via pgp using their keybase public key. IIRC you can store encrypted keys in Vault that can be accessed only for unsealing purposes (please correct me on this). When needing an unseal, the script on the user side would then decrypt the key and submit it to Vault for unsealing. It worked well enough, and it felt like being in the movie/game GoldenEye, but nowhere as slick as auto unseal. Regardless of the setup, yes, at the end of the day any solution is really just pushing the problem further down the stack.

perlgeek 6y ago

There are actually two answers to that: * You can split the key between different persons, and you can even implement "n of k" schemes, like you specify (at key creation time) that you need any 4 out of 9 shards to unseal the vault. You can then keep those shards on separate operator's laptops, in separate backup systems etc. * You can use a hardware security module to unseal the vault (support for that is not included in the free version, IIRC). But even if the vault wasn't stored encrypted, it'd still be a huge improvement over "keys on NFS", because only machine administrators get access to the whole DB, and you can limit and audit the access of everybody else in a sane manner.

greyhair 6y ago

We also use an HSM. Smart cards are held by two separate people, each card set is different. Both have to be present to restart the HSM. Our HSM can require up to five different cards be required for certain operations. For us, we only require two for normal operations. For HSM management (key generation, card authentication, etc..) we require a third card set member to be present. This protects against accidental key erasure, or fraud.

deleted 6y ago
[deleted]