5 ms·

The advice suggesting storing config in environment variables isn’t great security-wise if your config contains secrets. From systemd docs: > Note that environ

The advice suggesting storing config in environment variables isn’t great security-wise if your config contains secrets. From systemd docs: > Note that environment variables are not suitable for passing secrets (such as passwords, key material, …) to service processes. Environment variables set for a unit are exposed to unprivileged clients via D-Bus IPC, and generally not understood as being data that requires protection. Moreover, environment variables are propagated down the process tree, including across security boundaries (such as setuid/setgid executables), and hence might leak to processes that should not have access to the secret data. The former reason is systemd-specific, but the latter isn’t. There are some more reasons in https://movingfast.io/articles/environment-variables-considered-harmful/ https://movingfast.io/articles/environment-variables-conside....

SonOfLilit 4y ago
They are also exposed to anyone who can read from /proc/PID/environ, which means a simple directory traversal vulnerability can translate to database credentials.
- tptacek 4y ago
  This is effectively saying that they're exposed to root or the owner of the PID, which is the same story as storing secrets in files. Root also has other ways to get secrets out of remote processes.
  - SonOfLilit 4y ago
    My problem is with owner-of-PID, if you have directory traversal as root... good luck I guess. You can store secrets in a process belonging to a different PID and get them with some kind of RPC.
    - tptacek 4y ago
      Now that process has to get its secret from somewhere. At some point, you're just recapitulating the design of Hashicorp Vault. But even if you're using Vault, you're most likely injecting secrets as environment variables.
      - SonOfLilit 4y ago
        Belonging to a different user* It can get it from a file that isn't readable by the user that runs remote-accessible code.
lioeters 4y ago
I was curious what alternative is suggested besides using env variables. From the linked article: > Our solution thus has been to use a configuration file which is managed by the server's configuration management software (chef in our case). This way we can store secret keys outside of the code repository, and manage them using the same tool used to configure our servers. This solution, while not perfect, doesn't expose our secret keys to child processes and requires explicit access, which also communicates how developers should work with it.
- tptacek 4y ago
  I mean, pick your poison. If you store config secrets in files, those files are targets (and you have to provision them). If you store them in environment variables, you can inject them with something like Vault. There's no perfect answer here.
withinboredom 4y ago
If your machine is compromised, it’s only a matter of time until they get the credentials anyway. Whether you have more time than them is ultimately the question. I don’t think the vulnerability you mention here is “worth working around” for most apps, where you can trust the hardware and OS you’re running on. For those that can’t trust the OS and hardware they are running on, this may be legitimate. Even then, if you can’t trust that, what are you doing running software on it?
- jesterpm 4y ago
  My bigger concern with credentials in environment variables is that environment variables are broadly accessed through may different frameworks, libraries, etc. If you can convince a library to give you an arbitrary (or all) environment variables, your credentials are leaked. The developer of that particular library may have never even considered that environment variables might contain sensitive data. PHP's phpinfo() function comes to mind immediately: https://www.php.net/manual/en/function.phpinfo.php https://www.php.net/manual/en/function.phpinfo.php
  - trinovantes 4y ago
    If your threat model is executing untrusted 3rd party code, then you should be vetting them to ensure they're not reading your environment data or anything else nefarious. Otherwise they've already accomplished remote code execution. Why would they need to read your env when they can open a backdoor?
    - paulgb 4y ago
      It used to be not-impossible to find public phpinfo.php pages out there put up for debugging purposes. Hopefully people have smartened up to that, but it’s an example of how the data could be exposed without remote code execution. In any case, defense in depth seems like reason alone to do this.
    - zmmmmm 4y ago
      > Why would they need to read your env when they can open a backdoor? It's a good question. But consider, containers are (in modern infra) transient and usually sit in a constrained security context where access to other resources is pretty limited. Meanwhile the login credentials and security tokens are often long-lived and (usually) give broad access to a wide security context. If you give me a back door to a container, the first thing I would be trying to do is use it to level up my credentials before the container dies. One of the first things I would try is dumping the environment! But if the secrets were learned via an ephemeral file or through a temporary network connection, and even better if they were exchanged for time limited or single use type tokens, I am really left with both a challenging task to dig them out and / or something of limited value anyway.
nonameiguess 4y ago
This is already in the article: > This factor is now obsolete in one respect. As much as possible, secrets (passwords, private keys, et c) should be stored using a secrets management system such as Hashicorp Vault or Azure Key Vault. Particularly where we can rely on the infrastructure to authenticate the calling process (e.g. via a Kubernetes service account) access to secrets will not directly require credentials. The existence of the secret is under version control, but the actual secret content is immaterial.
wryun 4y ago
One approach I like (tell me if I'm mistaken!) is to pass things in the environment but read and remove them immediately on startup. This prevents leaking your secrets accidentally if you spawn another process etc., and has the advantage that you can use a secret store in production but raw env vars in development.
aceBacker 4y ago
What do you think about secret volumes like kubernetes does? https://kubernetes.io/docs/concepts/configuration/secret/#consuming-secret-values-from-volumes https://kubernetes.io/docs/concepts/configuration/secret/#co...
majkinetor 4y ago
While theoretically valid, in practice, nothing beats env vars and I have yet to see problems. If it leaks, you can change it anyway. If somebody is on the server, env vars are the least of your problems (also they are session wise and not saved locally). You should always use both - keep passwords in secret manager and during build communicate them to relevant parts of app via env vars. That way, you can easily reproduce it locally without secret manager available or slowing you down (maybe you need VPN to access it?, GitLab access to secrets etc.). Also, you can't easily change them for testing purposes or store variations of them - perhaps you mix several production with several testing environments. Contrary to that, you can keep dozen of them and easily change them in a single file while developing. Thee are bunch of tools to hide leaking environment vars too by masking them. So, no, this is not obsolete. Maybe it doesn't make sense in some scenarios but such cases shouldn't be the norm.
- preseinger 4y ago
  This is straight-up bad advice. Shuttling secrets via env vars expose you to strictly more risk vectors than through properly permissioned files on disk. Point blank, no argument. It's not acceptable to use env vars for secrets.
- CraigJPerry 4y ago
  I wouldn't want to change the usage of env vars, they're ergonomic and only the current user or root can see the env vars of a running process which makes it easy for us to architect secure usage patterns. But... it'd be cool if we had better support for avoiding accidental leaks. E.g. a way to identify another process as my user that ends up reading /proc/<pid>/environ unexpectedly. You could model that with SELinux but it could be a bear to own and not worth it for outside of large/important/high-value target companies. I suppose that's an apt description of SELinux generally... Maybe we could have a "light" version at the code level, maybe use the type system to throw an exception if you ever try to .toString() a secret. I'm trying to think how i would do that in Java without binding someone to an awkward type hierarchy (yuk!). This is not (currently) valid Java: interface Secret { default sealed String toString() { throw new IllegalAccessException() } } By this i mean a type that if i compose with, then it overrides (not currently supported) my toString() and blocks me changing the impl of that (sealed), then i can't accidentally log it for example. I think the way to do this in Java today would need to be via a dedicated class, i don't think i could use a composable interface. I mean that's fine-ish but i feel allergic to constraining other developers to my abstract class Secret { ... } EDIT: just blocking /proc/<pid>/environ wouldn't be enough, the other process could ptrace(2) for example.
seer 4y ago
I’m a huge fan of the berglas approach where you pass “location” of the secrets in env vars and the app loads them by itself. This allows you to keep permission grants separate, and you can safely send “.env” files and (gasp) save them in version control. You can change where and how you store and encrypt them, rotate keys etc. all transparently.
- mLuby 4y ago
  Am I understanding correctly that everyone with read access to your version control also has your production secrets?
  - tnzk 4y ago
    I guess it's about to check in just .env file, not files that contain secrets.
  - austinjp 4y ago
    Not the secrets themselves, but their location, eg /opt/secrets/some-database.sqlite3 I'm not parent commenter, but that's how I've done it before. The file on disk has the appropriate permissions etc. I'm sure there are issues with this, so if anyone can enlighten me I'd be interested.
  - sph 4y ago
    That's a unix access problem. It's a little harder to do on containers, but the UNIX way is having the secrets file root-readable only, your daemon spawns as root, reads the configurations and drops down to a less privileged UID.

SonOfLilit 4y ago

They are also exposed to anyone who can read from /proc/PID/environ, which means a simple directory traversal vulnerability can translate to database credentials.

tptacek 4y ago
This is effectively saying that they're exposed to root or the owner of the PID, which is the same story as storing secrets in files. Root also has other ways to get secrets out of remote processes.
- SonOfLilit 4y ago
  My problem is with owner-of-PID, if you have directory traversal as root... good luck I guess. You can store secrets in a process belonging to a different PID and get them with some kind of RPC.
  - tptacek 4y ago
    Now that process has to get its secret from somewhere. At some point, you're just recapitulating the design of Hashicorp Vault. But even if you're using Vault, you're most likely injecting secrets as environment variables.
    - SonOfLilit 4y ago
      Belonging to a different user* It can get it from a file that isn't readable by the user that runs remote-accessible code.

lioeters 4y ago

I was curious what alternative is suggested besides using env variables. From the linked article: > Our solution thus has been to use a configuration file which is managed by the server's configuration management software (chef in our case). This way we can store secret keys outside of the code repository, and manage them using the same tool used to configure our servers. This solution, while not perfect, doesn't expose our secret keys to child processes and requires explicit access, which also communicates how developers should work with it.

tptacek 4y ago
I mean, pick your poison. If you store config secrets in files, those files are targets (and you have to provision them). If you store them in environment variables, you can inject them with something like Vault. There's no perfect answer here.

withinboredom 4y ago

If your machine is compromised, it’s only a matter of time until they get the credentials anyway. Whether you have more time than them is ultimately the question. I don’t think the vulnerability you mention here is “worth working around” for most apps, where you can trust the hardware and OS you’re running on. For those that can’t trust the OS and hardware they are running on, this may be legitimate. Even then, if you can’t trust that, what are you doing running software on it?

jesterpm 4y ago
My bigger concern with credentials in environment variables is that environment variables are broadly accessed through may different frameworks, libraries, etc. If you can convince a library to give you an arbitrary (or all) environment variables, your credentials are leaked. The developer of that particular library may have never even considered that environment variables might contain sensitive data. PHP's phpinfo() function comes to mind immediately: https://www.php.net/manual/en/function.phpinfo.php https://www.php.net/manual/en/function.phpinfo.php
- trinovantes 4y ago
  If your threat model is executing untrusted 3rd party code, then you should be vetting them to ensure they're not reading your environment data or anything else nefarious. Otherwise they've already accomplished remote code execution. Why would they need to read your env when they can open a backdoor?
  - paulgb 4y ago
    It used to be not-impossible to find public phpinfo.php pages out there put up for debugging purposes. Hopefully people have smartened up to that, but it’s an example of how the data could be exposed without remote code execution. In any case, defense in depth seems like reason alone to do this.
  - zmmmmm 4y ago
    > Why would they need to read your env when they can open a backdoor? It's a good question. But consider, containers are (in modern infra) transient and usually sit in a constrained security context where access to other resources is pretty limited. Meanwhile the login credentials and security tokens are often long-lived and (usually) give broad access to a wide security context. If you give me a back door to a container, the first thing I would be trying to do is use it to level up my credentials before the container dies. One of the first things I would try is dumping the environment! But if the secrets were learned via an ephemeral file or through a temporary network connection, and even better if they were exchanged for time limited or single use type tokens, I am really left with both a challenging task to dig them out and / or something of limited value anyway.

nonameiguess 4y ago

This is already in the article: > This factor is now obsolete in one respect. As much as possible, secrets (passwords, private keys, et c) should be stored using a secrets management system such as Hashicorp Vault or Azure Key Vault. Particularly where we can rely on the infrastructure to authenticate the calling process (e.g. via a Kubernetes service account) access to secrets will not directly require credentials. The existence of the secret is under version control, but the actual secret content is immaterial.

wryun 4y ago

One approach I like (tell me if I'm mistaken!) is to pass things in the environment but read and remove them immediately on startup. This prevents leaking your secrets accidentally if you spawn another process etc., and has the advantage that you can use a secret store in production but raw env vars in development.

aceBacker 4y ago

What do you think about secret volumes like kubernetes does? https://kubernetes.io/docs/concepts/configuration/secret/#consuming-secret-values-from-volumes https://kubernetes.io/docs/concepts/configuration/secret/#co...

majkinetor 4y ago

While theoretically valid, in practice, nothing beats env vars and I have yet to see problems. If it leaks, you can change it anyway. If somebody is on the server, env vars are the least of your problems (also they are session wise and not saved locally). You should always use both - keep passwords in secret manager and during build communicate them to relevant parts of app via env vars. That way, you can easily reproduce it locally without secret manager available or slowing you down (maybe you need VPN to access it?, GitLab access to secrets etc.). Also, you can't easily change them for testing purposes or store variations of them - perhaps you mix several production with several testing environments. Contrary to that, you can keep dozen of them and easily change them in a single file while developing. Thee are bunch of tools to hide leaking environment vars too by masking them. So, no, this is not obsolete. Maybe it doesn't make sense in some scenarios but such cases shouldn't be the norm.

preseinger 4y ago
This is straight-up bad advice. Shuttling secrets via env vars expose you to strictly more risk vectors than through properly permissioned files on disk. Point blank, no argument. It's not acceptable to use env vars for secrets.
CraigJPerry 4y ago
I wouldn't want to change the usage of env vars, they're ergonomic and only the current user or root can see the env vars of a running process which makes it easy for us to architect secure usage patterns. But... it'd be cool if we had better support for avoiding accidental leaks. E.g. a way to identify another process as my user that ends up reading /proc/<pid>/environ unexpectedly. You could model that with SELinux but it could be a bear to own and not worth it for outside of large/important/high-value target companies. I suppose that's an apt description of SELinux generally... Maybe we could have a "light" version at the code level, maybe use the type system to throw an exception if you ever try to .toString() a secret. I'm trying to think how i would do that in Java without binding someone to an awkward type hierarchy (yuk!). This is not (currently) valid Java: interface Secret { default sealed String toString() { throw new IllegalAccessException() } } By this i mean a type that if i compose with, then it overrides (not currently supported) my toString() and blocks me changing the impl of that (sealed), then i can't accidentally log it for example. I think the way to do this in Java today would need to be via a dedicated class, i don't think i could use a composable interface. I mean that's fine-ish but i feel allergic to constraining other developers to my abstract class Secret { ... } EDIT: just blocking /proc/<pid>/environ wouldn't be enough, the other process could ptrace(2) for example.

seer 4y ago

I’m a huge fan of the berglas approach where you pass “location” of the secrets in env vars and the app loads them by itself. This allows you to keep permission grants separate, and you can safely send “.env” files and (gasp) save them in version control. You can change where and how you store and encrypt them, rotate keys etc. all transparently.

mLuby 4y ago
Am I understanding correctly that everyone with read access to your version control also has your production secrets?
- tnzk 4y ago
  I guess it's about to check in just .env file, not files that contain secrets.
- austinjp 4y ago
  Not the secrets themselves, but their location, eg /opt/secrets/some-database.sqlite3 I'm not parent commenter, but that's how I've done it before. The file on disk has the appropriate permissions etc. I'm sure there are issues with this, so if anyone can enlighten me I'd be interested.
- sph 4y ago
  That's a unix access problem. It's a little harder to do on containers, but the UNIX way is having the secrets file root-readable only, your daemon spawns as root, reads the configurations and drops down to a less privileged UID.