6 ms·

One thing that I think could be updated in the 12F approach is the use of env vars for config, especially secrets. I’ve generally found it much better to mount

One thing that I think could be updated in the 12F approach is the use of env vars for config, especially secrets. I’ve generally found it much better to mount these into the filesystem and read from there. It helps with, for example, secret rotation for long running processes. Relying on process restarts can be ugly in some setups, especially if startup time is expensive.

zach_garwood 4y ago
Seconded, but porque no los dos? I tend to put my applications' env vars in `/etc/environment`.
- numbsafari 4y ago
  It’s a good question and idea. I’ve been trending towards treating secrets and non-secrets differently. For non-secrets, I tend to have a config discovery mechanism that lets me read config from files, then from env, then from cli args as final fallback. This is mostly for ergonomics. I prefer to use files for deployed processes, and env vars or cli params when debugging or working interactively. Env vars are handy for cli args I’m going to be repeating a lot. But for secrets I’ve tended to only fetching those from files, but allowing the file location to be specified by env var or cli arg for quick changes. It sounds kinda complicated, but you write that logic into a library and then everything behaves the way you expect consistently.
  - dfsdf4sds 4y ago
    For what it's worth at $dayjob (public company, 1B+ revenue) we do the exact same, although either I misunderstood you or we have the exact opposite priority: args always win, if they're not defined then env wins, then config file, then default config value in code which sometimes is an actual value, and sometimes it defaults to throwing an exception and never passing a readiness check, depending on the behavior of the config in question. This last one is mostly a relic of the past and we believe it's better to blow up if there's no default configuration value in the chart in all cases; it also makes it way easier not to have to chase defaults through the code. We use these for running locally and debugging as you say, but come deployment time we ship all applications as helm charts with default config values, store overrides and static secrets (with SOPS) in a different, company-global repo, and use kubernetes secrets to mount a tmpfs volume to pass them in to the application. There's no way to pass args or env there.
- Parrot_NTheKing 4y ago
  One pattern that we use at our company is that we have multiple env files, and we launch programs using `run_with_config <config-file> <command>`. The script loads environment variables from config-file and then launches command.
  - zach_garwood 4y ago
    I like to do something similar with my personal projects: `set -o allexport && source /ect/environment && set +o allexport; command`
timmahoney 4y ago
In kubernetes I run external secrets, which is nice. I store The secrets in key value or file format in AWS Secrets Manager, which gets synchronized to the cluster into a secret. From there it gets mounted into the running pod via the envFrom or volume mount method.
- thinkmassive 4y ago
  External secrets are great, especially if your app can read them directly from k8s and avoid ever having them mounted as a volume (or in env var). If your app can’t interface directly w/k8s, but it can read secrets from a file, you can use a small init program to fetch the k8s secret and write it to a named pipe. This is advantageous compared to mounting as a volume, because the pipe disappears after both ends close their connection to it.
eriner_ 4y ago
I agree completely. Storing secrets in environment variables is wrong: https://blog.forcesunseen.com/stop-storing-secrets-in-environment-variables https://blog.forcesunseen.com/stop-storing-secrets-in-enviro... Even unsetting environment variables leaves them in /proc/self/environ. There isn't a thread-safe way to unset environment variables in POSIX, so even if you `unset TOP_SECRET` from within a program the contents of /proc/self/environ will remain unchanged and available.
- josephcsible 4y ago
  But /proc/*/environ is only readable by the user the process is running as and root, so if you can read it, you're already on the other side of the airtight hatchway.
  - eriner_ 4y ago
    Arbitrary file read vulnerabilities are extremely common.
    - josephcsible 4y ago
      Sure, but if you have one of them, then you can read the secret files out of /vault or whatever too.
      - hedora 4y ago
        Or, if you only have access to proc, from the process's heap.
      - eriner_ 4y ago
        Correct! This is why I explicitly said the following in the blog post I linked above: > Once the application hits “readiness” status (as determined by health endpoints or the load balancer), the secrets volume should be unmounted and made inaccessible.
        4y ago
infamousjoeg 4y ago
Check out the Secretless Broker at https://secretless.io https://secretless.io. It's a cool open source project that allows applications to not need to know secrets which adheres to 12-factor app guidelines.
- cphoover 4y ago
  hmmm.... im trying to understand the benefit of secretless broker... if someone compromises thisnwouldnt they have access to all credentials for everything? now we are just moving from trusting a bunch of distinct services to trusting this single broker... just moving the responsibility of trust to a single point of potential failure no? Also dont credentials have to be passed to secretless broker? how does it know the application has access to the service? isnt that still at risk of being leaked. i like the idea of not thinking about secrets but it seems to good to be true.
- numbsafari 4y ago
  I’ll have to dig into it to see how it compares, but https://spiffe.io/ https://spiffe.io/ is what I look to in this area. Not having long lived secrets is the ultimate destination, but we all live with the legacy around us.
emmelaich 4y ago
It's the part of 12 factor that makes me giggle a bit; like it how does it get into the environment? Reminds of the "Front fell off" comic sketch. "No, we towed it out of the environment" https://www.youtube.com/watch?v=3m5qxZm_JqM https://www.youtube.com/watch?v=3m5qxZm_JqM
- kortex 4y ago
  It's environment variables all the way down.
- mindcrime 4y ago
  "Well what's out there?" "Nothing's out there." "Well there must be something out there." "There is nothing out there. All there is, is sea, and birds, and fish." "And?" "And 20,000 tons of crude oil." "And what else"? "And a fire." "And anything else?" "And the part of the ship that the front fell off. But there's nothing else out there. It's a complete void."
tdfx 4y ago
> I’ve generally found it much better to mount these into the filesystem and read from there. Is there a standardized/best practice way to do this? Some convention for file names or format?
- kortex 4y ago
  I'm fond of the XDG path convention. Which basically boils down to putting configs in ~/.config/your_app/ and secrets in ~/.config/your_app/secrets. If you use pydantic, it supports a secrets dir with a predefined path. https://specifications.freedesktop.org/desktop-entry-spec/desktop-entry-spec-latest.html https://specifications.freedesktop.org/desktop-entry-spec/de...
- hakre 4y ago
  There is docker secrets and some specialized convention for example to share the ssh control socket during build time etc.
arinlen 4y ago
> One thing that I think could be updated in the 12F approach is the use of env vars for config, especially secrets. The 12F section on config already mentions quite prominently the use of config files. Taken from https://12factor.net/config https://12factor.net/config > "Another approach to config is the use of config files which are not checked into revision control, such as config/database.yml in Rails. This is a huge improvement over using constants which are checked into the code repo, but still has weaknesses: it’s easy to mistakenly check in a config file to the repo; there is a tendency for config files to be scattered about in different places and different formats, making it hard to see and manage all the config in one place. Further, these formats tend to be language- or framework-specific."
danenania 4y ago
EnvKey[1] can help with process reloading, and can facilitate both restarts and hot reload updates. (Disclaimer: I’m the founder.) The pros/cons of environment variables vs. files (or other approaches) is also something I’ve thought about a lot while working on EnvKey. We use environment variables as a default approach, since it seems to be the most common way to pass secrets/config to a process in the wild and we want to meet people where they’re at. But we also make it easy to use files or system calls instead (I think system calls are actually the most secure.) One thing it’s always important to remember though in security: if you make the “secure way” too hard, people will route around it, making matters worse in practice. There’s always a balance to be struck. Honestly, I’m skeptical that threat models where environments are exposed but files are safe are realistic enough to be worth worrying about. At that point, it seems like rearranging deck chairs on the Titanic. It seems simpler to say “our last line of defense is the OS boundary.” You trust the host and go from there. If the host is breached, you’re screwed in a plethora of ways. There’s no point in sweating the particulars. Just don’t let it happen in the first place! And when it comes to concerns about leaking the environment to sub-processes, this seems like a deeper problem. Even if you don’t store secrets in the environment, that doesn’t mean it’s safe to just send env vars off wherever. At this point, like it or not, environments are sensitive, because enough people and programs treat them as sensitive that it’s a self-fulfilling prophecy. If they might be leaked, then the leak is the security problem in my eyes, not the data in the environment. 1 - https://envkey.com https://envkey.com
- zmmmmm 4y ago
  EnvKey looks great! Only one thing strikes me as unfortunate, which is it seems like it fragments the development process because now your code history is tracked in git but your config history is in EnvKey. It means everything I do is duplicated (for example, tags, branches etc). It isn't clear to me how I would take, for example common operations such as rebasing one git branch on another and replicate the same on the two EnvKey branches involved. We track all our non-sensitive config in git for this reason - then merging two branches is literally merging the config changes. However, we don't have any of the nice features EnvKey is doing like service restarting, inheritance, secret management etc. Just curious if I'm missing something here or you have a strategy for what I discuss above?
pjbster 4y ago
The best approach I've seen is to have the secrets in a vault and to keep the vault URL in an env var (so different vaults can be used for dev and prod, for example). The secrets location within the vault is held in a variable defined in an application config file which is checked in to source control. At runtime, a piece of code consults the variable and queries the vault using the provided location and then dynamically configures the db connection with the returned credentials. The credentials are dumped from memory once the transaction completes. (I should also point out that this is in a batch ETL system.) The nice thing about this approach is that it is virtually impossible for secrets to end up in source control. The risk is that the vault becomes a major single point of failure.
- Someone 4y ago
  It also makes cycling credentials easy. All you have to do is update the values in the vault. No need to hunt down all those places where you set environment variables. I think you don’t even need to redeploy any long-running tasks. If they have a connection open, it should continue working, and if they don’t, they should consult the vault when they open one and get the new credentials from it (this may be different for different kinds of credentials) There will be a race condition between creating credentials and storing them in the vault, but even that can be avoided: - create new credentials with same rights as the old ones - update vault - delete old credentials Another race where credentials get changed between reading them from the vault and using them can’t be avoided, but if you wait a few minutes before you delete the old credentials, it’s extremely unlikely you’ll hit it.
- de6u99er 4y ago
  This is the way!
- vojtad 4y ago
  Do you have any particular vault in mind? We are looking for a simple solution which would work exactly like you described. We just need couple of different sets of variables for our web server, workers and other services.
  - 4y ago
TheRealDunkirk 4y ago
<Laughs in Rails.> Credentials.yml.enc/master.key sorts this pretty well. I have been very grateful for its addition to the stack.
Cthulhu_ 4y ago
I wouldn't put them on a filesystem; do you have any guarantees the secrets are wiped clean after your application is shut down? At least env vars are in memory (...I... think...?), and will be offloaded if the server shuts down. A secrets manager may be a better option, these days. I don't believe they were a thing in 2011 though. Also 2011 was 11 years ago, and I still keep this webpage in mind when building server-side software <_<.

zach_garwood 4y ago

Seconded, but porque no los dos? I tend to put my applications' env vars in `/etc/environment`.

numbsafari 4y ago
It’s a good question and idea. I’ve been trending towards treating secrets and non-secrets differently. For non-secrets, I tend to have a config discovery mechanism that lets me read config from files, then from env, then from cli args as final fallback. This is mostly for ergonomics. I prefer to use files for deployed processes, and env vars or cli params when debugging or working interactively. Env vars are handy for cli args I’m going to be repeating a lot. But for secrets I’ve tended to only fetching those from files, but allowing the file location to be specified by env var or cli arg for quick changes. It sounds kinda complicated, but you write that logic into a library and then everything behaves the way you expect consistently.
- dfsdf4sds 4y ago
  For what it's worth at $dayjob (public company, 1B+ revenue) we do the exact same, although either I misunderstood you or we have the exact opposite priority: args always win, if they're not defined then env wins, then config file, then default config value in code which sometimes is an actual value, and sometimes it defaults to throwing an exception and never passing a readiness check, depending on the behavior of the config in question. This last one is mostly a relic of the past and we believe it's better to blow up if there's no default configuration value in the chart in all cases; it also makes it way easier not to have to chase defaults through the code. We use these for running locally and debugging as you say, but come deployment time we ship all applications as helm charts with default config values, store overrides and static secrets (with SOPS) in a different, company-global repo, and use kubernetes secrets to mount a tmpfs volume to pass them in to the application. There's no way to pass args or env there.
Parrot_NTheKing 4y ago
One pattern that we use at our company is that we have multiple env files, and we launch programs using `run_with_config <config-file> <command>`. The script loads environment variables from config-file and then launches command.
- zach_garwood 4y ago
  I like to do something similar with my personal projects: `set -o allexport && source /ect/environment && set +o allexport; command`

timmahoney 4y ago

In kubernetes I run external secrets, which is nice. I store The secrets in key value or file format in AWS Secrets Manager, which gets synchronized to the cluster into a secret. From there it gets mounted into the running pod via the envFrom or volume mount method.

thinkmassive 4y ago
External secrets are great, especially if your app can read them directly from k8s and avoid ever having them mounted as a volume (or in env var). If your app can’t interface directly w/k8s, but it can read secrets from a file, you can use a small init program to fetch the k8s secret and write it to a named pipe. This is advantageous compared to mounting as a volume, because the pipe disappears after both ends close their connection to it.

eriner_ 4y ago

I agree completely. Storing secrets in environment variables is wrong: https://blog.forcesunseen.com/stop-storing-secrets-in-environment-variables https://blog.forcesunseen.com/stop-storing-secrets-in-enviro... Even unsetting environment variables leaves them in /proc/self/environ. There isn't a thread-safe way to unset environment variables in POSIX, so even if you `unset TOP_SECRET` from within a program the contents of /proc/self/environ will remain unchanged and available.

josephcsible 4y ago
But /proc/*/environ is only readable by the user the process is running as and root, so if you can read it, you're already on the other side of the airtight hatchway.
- eriner_ 4y ago
  Arbitrary file read vulnerabilities are extremely common.
  - josephcsible 4y ago
    Sure, but if you have one of them, then you can read the secret files out of /vault or whatever too.
    - hedora 4y ago
      Or, if you only have access to proc, from the process's heap.
    - eriner_ 4y ago
      Correct! This is why I explicitly said the following in the blog post I linked above: > Once the application hits “readiness” status (as determined by health endpoints or the load balancer), the secrets volume should be unmounted and made inaccessible.
      - 4y ago

infamousjoeg 4y ago

Check out the Secretless Broker at https://secretless.io https://secretless.io. It's a cool open source project that allows applications to not need to know secrets which adheres to 12-factor app guidelines.

cphoover 4y ago
hmmm.... im trying to understand the benefit of secretless broker... if someone compromises thisnwouldnt they have access to all credentials for everything? now we are just moving from trusting a bunch of distinct services to trusting this single broker... just moving the responsibility of trust to a single point of potential failure no? Also dont credentials have to be passed to secretless broker? how does it know the application has access to the service? isnt that still at risk of being leaked. i like the idea of not thinking about secrets but it seems to good to be true.
numbsafari 4y ago
I’ll have to dig into it to see how it compares, but https://spiffe.io/ https://spiffe.io/ is what I look to in this area. Not having long lived secrets is the ultimate destination, but we all live with the legacy around us.

emmelaich 4y ago

It's the part of 12 factor that makes me giggle a bit; like it how does it get into the environment? Reminds of the "Front fell off" comic sketch. "No, we towed it out of the environment" https://www.youtube.com/watch?v=3m5qxZm_JqM https://www.youtube.com/watch?v=3m5qxZm_JqM

kortex 4y ago
It's environment variables all the way down.
mindcrime 4y ago
"Well what's out there?" "Nothing's out there." "Well there must be something out there." "There is nothing out there. All there is, is sea, and birds, and fish." "And?" "And 20,000 tons of crude oil." "And what else"? "And a fire." "And anything else?" "And the part of the ship that the front fell off. But there's nothing else out there. It's a complete void."

tdfx 4y ago

> I’ve generally found it much better to mount these into the filesystem and read from there. Is there a standardized/best practice way to do this? Some convention for file names or format?

kortex 4y ago
I'm fond of the XDG path convention. Which basically boils down to putting configs in ~/.config/your_app/ and secrets in ~/.config/your_app/secrets. If you use pydantic, it supports a secrets dir with a predefined path. https://specifications.freedesktop.org/desktop-entry-spec/desktop-entry-spec-latest.html https://specifications.freedesktop.org/desktop-entry-spec/de...
hakre 4y ago
There is docker secrets and some specialized convention for example to share the ssh control socket during build time etc.

arinlen 4y ago

> One thing that I think could be updated in the 12F approach is the use of env vars for config, especially secrets. The 12F section on config already mentions quite prominently the use of config files. Taken from https://12factor.net/config https://12factor.net/config > "Another approach to config is the use of config files which are not checked into revision control, such as config/database.yml in Rails. This is a huge improvement over using constants which are checked into the code repo, but still has weaknesses: it’s easy to mistakenly check in a config file to the repo; there is a tendency for config files to be scattered about in different places and different formats, making it hard to see and manage all the config in one place. Further, these formats tend to be language- or framework-specific."

danenania 4y ago

EnvKey[1] can help with process reloading, and can facilitate both restarts and hot reload updates. (Disclaimer: I’m the founder.) The pros/cons of environment variables vs. files (or other approaches) is also something I’ve thought about a lot while working on EnvKey. We use environment variables as a default approach, since it seems to be the most common way to pass secrets/config to a process in the wild and we want to meet people where they’re at. But we also make it easy to use files or system calls instead (I think system calls are actually the most secure.) One thing it’s always important to remember though in security: if you make the “secure way” too hard, people will route around it, making matters worse in practice. There’s always a balance to be struck. Honestly, I’m skeptical that threat models where environments are exposed but files are safe are realistic enough to be worth worrying about. At that point, it seems like rearranging deck chairs on the Titanic. It seems simpler to say “our last line of defense is the OS boundary.” You trust the host and go from there. If the host is breached, you’re screwed in a plethora of ways. There’s no point in sweating the particulars. Just don’t let it happen in the first place! And when it comes to concerns about leaking the environment to sub-processes, this seems like a deeper problem. Even if you don’t store secrets in the environment, that doesn’t mean it’s safe to just send env vars off wherever. At this point, like it or not, environments are sensitive, because enough people and programs treat them as sensitive that it’s a self-fulfilling prophecy. If they might be leaked, then the leak is the security problem in my eyes, not the data in the environment. 1 - https://envkey.com https://envkey.com

zmmmmm 4y ago
EnvKey looks great! Only one thing strikes me as unfortunate, which is it seems like it fragments the development process because now your code history is tracked in git but your config history is in EnvKey. It means everything I do is duplicated (for example, tags, branches etc). It isn't clear to me how I would take, for example common operations such as rebasing one git branch on another and replicate the same on the two EnvKey branches involved. We track all our non-sensitive config in git for this reason - then merging two branches is literally merging the config changes. However, we don't have any of the nice features EnvKey is doing like service restarting, inheritance, secret management etc. Just curious if I'm missing something here or you have a strategy for what I discuss above?

pjbster 4y ago

The best approach I've seen is to have the secrets in a vault and to keep the vault URL in an env var (so different vaults can be used for dev and prod, for example). The secrets location within the vault is held in a variable defined in an application config file which is checked in to source control. At runtime, a piece of code consults the variable and queries the vault using the provided location and then dynamically configures the db connection with the returned credentials. The credentials are dumped from memory once the transaction completes. (I should also point out that this is in a batch ETL system.) The nice thing about this approach is that it is virtually impossible for secrets to end up in source control. The risk is that the vault becomes a major single point of failure.

Someone 4y ago
It also makes cycling credentials easy. All you have to do is update the values in the vault. No need to hunt down all those places where you set environment variables. I think you don’t even need to redeploy any long-running tasks. If they have a connection open, it should continue working, and if they don’t, they should consult the vault when they open one and get the new credentials from it (this may be different for different kinds of credentials) There will be a race condition between creating credentials and storing them in the vault, but even that can be avoided: - create new credentials with same rights as the old ones - update vault - delete old credentials Another race where credentials get changed between reading them from the vault and using them can’t be avoided, but if you wait a few minutes before you delete the old credentials, it’s extremely unlikely you’ll hit it.
de6u99er 4y ago
This is the way!
vojtad 4y ago
Do you have any particular vault in mind? We are looking for a simple solution which would work exactly like you described. We just need couple of different sets of variables for our web server, workers and other services.
- 4y ago

TheRealDunkirk 4y ago

<Laughs in Rails.> Credentials.yml.enc/master.key sorts this pretty well. I have been very grateful for its addition to the stack.

Cthulhu_ 4y ago

I wouldn't put them on a filesystem; do you have any guarantees the secrets are wiped clean after your application is shut down? At least env vars are in memory (...I... think...?), and will be offloaded if the server shuts down. A secrets manager may be a better option, these days. I don't believe they were a thing in 2011 though. Also 2011 was 11 years ago, and I still keep this webpage in mind when building server-side software <_<.