4 ms·

There are many solutions out there that attempt to solve the secrets distribution problem. Vault is the newest one. None of them solve the fundamental problem o

by jvehent 11y ago

There are many solutions out there that attempt to solve the secrets distribution problem. Vault is the newest one. None of them solve the fundamental problem of the secret zero: provisioning the first secret on an untrusted system the automated way is hard, and without that secret zero, there is no way to authenticate the new system. With Vault, you still need to figure out how to push secret zero (here a client authentication token). If you can solve that, why not push the secret you need to push directly, and bypass Vault entirely?

mitchellh 11y ago
We worked with a handful of companies to help us design the "App ID" authentication backend that does exactly that: https://vaultproject.io/docs/auth/app-id.html https://vaultproject.io/docs/auth/app-id.html It allows you to have a non-sensitive single factor within things like configuration management, and have the second factor come from a machine-local location (instance ID, MAC address, TSM module, etc.). The idea is that another team out of band sets the 2nd factor that developers and config management never get to see. The result is you have full automation without secret zero issues. As an additional security parameter, you can bind the two factors together to a single CIDR block, so you can have an additional factor restricting access from that machine's IP. The URL itself above goes into a lot more detail, but our beta users are successfully automated in an elastic environment this way.
- jvehent 11y ago
  That's definitely a trade-off compared to strong authentication, one that I wouldn't want to make in critical environments. But indeed a very nice feature for 99% of low/medium risk applications.
- nutate 11y ago
  hard problem, but you clearly address it. There is a typo though: "in client" should probably read "if a client" or "if the client"
- acveilleux 11y ago
  So if I read that correctly, the UserID (I find that naming confusing since users here aren't necessarily humans) creation has a human in the loop to add the UserID to Vault and therefore make the authorization decision?
- kgilpin 11y ago
  Using an instance id or MAC address as a secret 2nd factor seems surprising to me. This is what banks are asking for? It's nice that there are two factors to the authentication, but neither one seems to be securely random.
  - mitchellh 11y ago
    No, it is usually one of those values salted with a value that is only available offline.
zhoutong 11y ago
We've had the same issues at CoinJar, and we used SSH agent forwarding to solve it. This is how it can work: Every new instance of application server is provisioned automatically by a trusted server (which holds the key to a credentials server). During orchestration/deployment, the application server has a temporary permission to fetch the secrets it needed, through SSH agent forwarding. Once the deployment is done, the session will end and the application server can never read new secrets until the next deployment. This way we don't really need a solution like Vault. SSH is mature enough to provide authentication, and encryption is easy once you've figured out how to distribute keys automatically.
- otterley 11y ago
  How does SSH agent forwarding work if there's no interactive user driving the process? Otherwise the concurrency is going to be limited to however many ssh sessions the client can spawn; and the orchestration/deployment cannot be a handsfree process.
  - ntucker 11y ago
    I'm not the person you're asking, but in our system, our secret access is tied to an SSH identity, not an SSH agent. This means that you have flexibility to require a personal user's ssh agent to be present for access to some things, or an identity file (optionally encrypted) stored on the server for less-sensitive things. In the case of things that need to happen unattended, we use an in-the-clear identity file, which might seem a little counterintuitive since security-wise, it's basically equivalent to just having the secret sitting there, but the indirection adds ease of rotating secrets along with revocability if we were to suspect we lost control of such a machine. But for most of our secrets, we tie them to individual user's identities which must be forwarded via an agent. We consider this an acceptable constraint because deployment and app restarts are always driven by a human in our operations. It's not a perfect system or a panacea, but it's far better than having a bunch of passwords directly stored in config files, in my opinion. We also have a wrapper tool that uses your SSH agent access to fetch secrets for the mysql suite of tools as needed, so nobody is ever tempted to create a personal .my.cnf on prod servers. Anybody who actually has access to a given mysql password is still on the honor system not to fetch it and write it down somewhere, but removing the need to ever even see them eliminates the temptation to put it in a .my.cnf or in your clipboard, which eliminates a whole class of mistakes you can make in handling them.
IgorPartola 11y ago
Why not do it at the network level? In some infrastructures this is really easy. For example in AWS you can set up all your servers in a VPC. Then put Vault on a machine on a specific subnetwork, and firewall it off such that only the machines from your VPC may access it. You have now created a walled garden where anything inside your VPC has access to the secrets without holding an authentication token, while anything outside does not. Personally, I think this is your only line of defense anyways. Someone with access to your private network will likely be able to do some harm, and if they get even a non-root user on your server, it's game over, since you are just one local exploit away from them gaining root. Remember, once they gain root on one box, it's just a matter of time before they replace /usr/bin/sudo and /usr/bin/login with a script that emails them your password and/or steals your ssh-agent socket.
- acveilleux 11y ago
  Because you want authorization and authentication by application. So a web server / proxy needs the public TLS cert but maybe an application server doesn't. Meanwhile the app server needs DB credentials but the web server doesn't. In your scenario, if something happens to a front facing web server, nothing prevents the attacker to just enumerate all secrets and retrieve them, whether the server should have had access or not. Defense in depth is a thing.
  - IgorPartola 11y ago
    I believe what you are describing can be built using the primitives I described. Vault, or another secrets manager, would only release secrets to severs on specific subnets. The front-end would be on your front-end subnet, the application server would be on the backend subnet, and the application server would not be authorized to access the private TLS key because it's on the wrong subnet for that particular secret.
    - acveilleux 11y ago
      To some extent, that's true. The granularity is a bit lacking however. A bigger worry would be that the ability to spin up an host in a specific subnet is the effective authentication. It's not something I'd be comfortable with myself but I guess some people would be.

by jvehent 11y ago

mitchellh 11y ago

We worked with a handful of companies to help us design the "App ID" authentication backend that does exactly that: https://vaultproject.io/docs/auth/app-id.html https://vaultproject.io/docs/auth/app-id.html It allows you to have a non-sensitive single factor within things like configuration management, and have the second factor come from a machine-local location (instance ID, MAC address, TSM module, etc.). The idea is that another team out of band sets the 2nd factor that developers and config management never get to see. The result is you have full automation without secret zero issues. As an additional security parameter, you can bind the two factors together to a single CIDR block, so you can have an additional factor restricting access from that machine's IP. The URL itself above goes into a lot more detail, but our beta users are successfully automated in an elastic environment this way.

jvehent 11y ago
That's definitely a trade-off compared to strong authentication, one that I wouldn't want to make in critical environments. But indeed a very nice feature for 99% of low/medium risk applications.
nutate 11y ago
hard problem, but you clearly address it. There is a typo though: "in client" should probably read "if a client" or "if the client"
acveilleux 11y ago
So if I read that correctly, the UserID (I find that naming confusing since users here aren't necessarily humans) creation has a human in the loop to add the UserID to Vault and therefore make the authorization decision?
kgilpin 11y ago
Using an instance id or MAC address as a secret 2nd factor seems surprising to me. This is what banks are asking for? It's nice that there are two factors to the authentication, but neither one seems to be securely random.
- mitchellh 11y ago
  No, it is usually one of those values salted with a value that is only available offline.

zhoutong 11y ago

We've had the same issues at CoinJar, and we used SSH agent forwarding to solve it. This is how it can work: Every new instance of application server is provisioned automatically by a trusted server (which holds the key to a credentials server). During orchestration/deployment, the application server has a temporary permission to fetch the secrets it needed, through SSH agent forwarding. Once the deployment is done, the session will end and the application server can never read new secrets until the next deployment. This way we don't really need a solution like Vault. SSH is mature enough to provide authentication, and encryption is easy once you've figured out how to distribute keys automatically.

otterley 11y ago
How does SSH agent forwarding work if there's no interactive user driving the process? Otherwise the concurrency is going to be limited to however many ssh sessions the client can spawn; and the orchestration/deployment cannot be a handsfree process.
- ntucker 11y ago
  I'm not the person you're asking, but in our system, our secret access is tied to an SSH identity, not an SSH agent. This means that you have flexibility to require a personal user's ssh agent to be present for access to some things, or an identity file (optionally encrypted) stored on the server for less-sensitive things. In the case of things that need to happen unattended, we use an in-the-clear identity file, which might seem a little counterintuitive since security-wise, it's basically equivalent to just having the secret sitting there, but the indirection adds ease of rotating secrets along with revocability if we were to suspect we lost control of such a machine. But for most of our secrets, we tie them to individual user's identities which must be forwarded via an agent. We consider this an acceptable constraint because deployment and app restarts are always driven by a human in our operations. It's not a perfect system or a panacea, but it's far better than having a bunch of passwords directly stored in config files, in my opinion. We also have a wrapper tool that uses your SSH agent access to fetch secrets for the mysql suite of tools as needed, so nobody is ever tempted to create a personal .my.cnf on prod servers. Anybody who actually has access to a given mysql password is still on the honor system not to fetch it and write it down somewhere, but removing the need to ever even see them eliminates the temptation to put it in a .my.cnf or in your clipboard, which eliminates a whole class of mistakes you can make in handling them.

IgorPartola 11y ago

Why not do it at the network level? In some infrastructures this is really easy. For example in AWS you can set up all your servers in a VPC. Then put Vault on a machine on a specific subnetwork, and firewall it off such that only the machines from your VPC may access it. You have now created a walled garden where anything inside your VPC has access to the secrets without holding an authentication token, while anything outside does not. Personally, I think this is your only line of defense anyways. Someone with access to your private network will likely be able to do some harm, and if they get even a non-root user on your server, it's game over, since you are just one local exploit away from them gaining root. Remember, once they gain root on one box, it's just a matter of time before they replace /usr/bin/sudo and /usr/bin/login with a script that emails them your password and/or steals your ssh-agent socket.

acveilleux 11y ago
Because you want authorization and authentication by application. So a web server / proxy needs the public TLS cert but maybe an application server doesn't. Meanwhile the app server needs DB credentials but the web server doesn't. In your scenario, if something happens to a front facing web server, nothing prevents the attacker to just enumerate all secrets and retrieve them, whether the server should have had access or not. Defense in depth is a thing.
- IgorPartola 11y ago
  I believe what you are describing can be built using the primitives I described. Vault, or another secrets manager, would only release secrets to severs on specific subnets. The front-end would be on your front-end subnet, the application server would be on the backend subnet, and the application server would not be authorized to access the private TLS key because it's on the wrong subnet for that particular secret.
  - acveilleux 11y ago
    To some extent, that's true. The granularity is a bit lacking however. A bigger worry would be that the ability to spin up an host in a specific subnet is the effective authentication. It's not something I'd be comfortable with myself but I guess some people would be.