23 ms·
Don't use ENV variables for secret data (2017)
- fomine3 6y agoI'm curious why such opinion is unpopular. ENV is too global and easily exposed in accident. I think people just don't want to think to leave 12factor (including me).
- xiwenc 6y agoFrom experience, it's good to support multiple ways of configuring an app. Depending on your case, this could become hard or requires naming conventions. This way you can move secret data to files if needed depending on the deployment choice. My own rule of thumb is: 1) sensible default value 2) read from file 3) read from env Examples: - https://github.com/spf13/viper https://github.com/spf13/viper - https://docs.spring.io/spring-boot/docs/current/reference/html/spring-boot-features.html#boot-features-external-config https://docs.spring.io/spring-boot/docs/current/reference/ht...
- djsumdog 6y agoWhen I was doing Scala development, we went the same route using TypeSafe Config. Default/file in the src/resources which could be overridden by ENV. It seems like secrets aren't orchestration agnostic. You can't seem to use Docker secrets without being in swarm mode, and k8s has its own secrets management system (or if you're running on AWS, you can use pod2iam and ssm to store/get encrypted parameters). I've been at places that use k8s+vault as well.
- gchamonlive 6y agoI am progressively adopting Hashicorp Vault as the secrets manager of choice. It can be used by a variety of different scenarios -- directly into the application using AppRole, with Terraform using it's secrets provider, by developers, during vault authentication when they get their secrets and access regenerated. This way I am not bound to docker swarm, or keywhiz, or god forbid AWS Secrets Manager. As of now, I am still exposing secrets with Env Vars, but the next step is to use Vault directly. Vault has been pretty reliable so far. It is using AWS KMS for managing the master key and a scalable DynamoDB table for high availability backend.
- gpmcadam 6y agoHow do you share the token to access Vault to your code?
- linkdd 6y agoUsing AppRoles and consul-template. And if you're on Kubernetes, you can use https://github.com/sethvargo/vault-kubernetes-authenticator https://github.com/sethvargo/vault-kubernetes-authenticator in an init container.
- carlosf 6y agoVault looks cool, but looking at the reference architecture [1] my guts tell me it's much easier to fuck up setting up a Vault cluster than environment variables. [1] https://learn.hashicorp.com/vault/operations/ops-reference-architecture#network-connectivity-details https://learn.hashicorp.com/vault/operations/ops-reference-a...
- gchamonlive 6y agoYou can always use their official Terraform module for AWS if you are not comfortable with setting it all up by yourself: https://registry.terraform.io/modules/hashicorp/vault/aws/0.0.9/submodules/vault-cluster https://registry.terraform.io/modules/hashicorp/vault/aws/0.... But if you cut Consul for backend, if you are not using consul for other service discovery and Nomad etc..., you can simplify that deployment a whole lot. You make sure you open the cluster communication port, setup a Application Load Balancer in front of the cluster to balance traffic and serve SSL, configure auto-unseal using AWS KMS (since you are not using it too often, 1$ is OK to have AWS manage your master key), deploy Vault on every cluster instance, and use something managed like DynamoDB as your backend. I think this is a pretty simple yet scalable setup. The amount of cloud lock-in is pretty minimal and can easily be replaced with HAProxy setup.
- mrkeen 6y agoFor those in the comments suggesting to use another system to store credentials instead of ENV: Do you need a password to access that system? Why not?
- deleted 6y ago[deleted]
- hellofunk 6y agoEvery production product I’ve ever worked on, the entire team put database credentials in the environment variables.
- _wldu 6y agoIt's way better than hard coding them into the code.
- willis936 6y agoWhy is that? Also, as an aside: The very premise of plaintext credentials for computer-computer database connections always seemed strange to me. Maybe I'm just not knowledgeable enough here, but I wish the standard for database credentials was key-based.
- ebg13 6y ago> Why is that? You don't want to accidentally commit your credentials to github and have the world see them. At least if they're in ENV they stay private as long as your environment does.
- chrisfinazzo 6y agoWhat about the global .gitconfig? One of the basic key value pairs is the GPG signingkey, which is usually stored in cleartext in the aforementioned file. Although the credential.helper is in my Keychain (iCloud-backed 2FA) In theory someone could copy this and try to sign commits as me, but I have to think this value is unique and they would get rejected if they tried to use it. My login credentials are 2FA as well, at least on unknown machines, so they would be prompted there as well. Personal Access Tokens for the CLI would be another way to prevent nefarious things from happening.
- mberning 6y agoNow every developer has access to any db credential that was in source control. If your project has had hundreds or thousands of developers that is a security concern.
- hankchinaski 6y agoWe have recently started removing credentials in env vars and started using google secret manager (previously berglas) and its been amazing so far. AWS and azure should have the same. More challenging when you don’t have all the infrastructure and services in place (ie. when working on plain VPS or other systems without those tools)
- seer 6y agoHm not sure I understand how google secret manager relates to berglas to be honest, thought those two were separate apis... As for berglas itself, we also use it and have been very happy with it. Since you put just the names of your secrets into the ENV files, not the secrets themselves, they can be easily stored in version control, passed around in chat and you can just do whatever you want with them. Instead of: ENV_PASS=my-secret-pass You do: ENV_PASS=berglas://bucket/secret-id And it will be decrypted at the last possible moment - e.g. when the system starts. Or even later if you need to, if you use the apis provided. Funny enough we had implemented the almost the same approach with AWS SSM apis ourselves (https://github.com/ovotech/ssm-env-secrets https://github.com/ovotech/ssm-env-secrets). But I think it should be possible to use berglas in AWS directly without issue.
- giberti 6y agoAmazon has a similar offering called Secrets Manager which can be used for sensitive secrets as well as configuration values. https://aws.amazon.com/secrets-manager/ https://aws.amazon.com/secrets-manager/
- ablanco 6y agoI don't agree at all. The reasons in the article all seem like "envs are bad because if you make a mistake you can expose them". This is not exclusive to envs, it applies to all secrets, independent of the medium used to make it available to the process using it. In my experience, if you prevent using envs for secrets (as docker swarm does) all you get is a disgruntled programmer reading the contents of a secret file to an env in the entrypoint.
- laumars 6y agoThere are actually key/value stores that solve this securely, such as Hashicorp Vault. The issue isn’t that it can’t be done but more that most people either don’t already know it can be done or don’t want to invest in the infrastructure to do it. Regarding the latter point, for self hosted solutions I can sympathise a little and it’s really a question of risk analysis. But most cloud computing services do offer their own secrets management service. (not affiliated with Hashicorp and other services exist).
- ablanco 6y agoSure, and they are great. But in some cases, it's inevitable to read some secrets from the secret management service to envs. This is what docker swarm doesn't allow with the 'docker secret' command
- falcolas 6y agoThe problem with Hashicorp Vault (and their peers): Your application still need a secret to access values made available to your application's role. The values might not be in the immediate container space (well, aside from being in program memory), but they're only one (likely well documented internally to the container) hop away.
- laumars 6y ago> The problem with Hashicorp Vault (and their peers): Your application still need a secret to access values made available to your application's role. True but those credentials can be decoupled from the application (like env vars are) so you satisfy the developer problem I was addressing.
- RedShift1 6y agoAnd fix it by making it dependent on something docker specific?
- hobbescotch 6y agoThe way I got around this was to store secrets in Google KMS encrypted files in Google Cloud Storage. The KMS key and encrypted files share the same name and can be accessed by that name programmatically. This secret storage method works really well for me and lets you easily access & manage secrets across all environments. It's so convenient, I sometimes even use this system as a simple key/value store.
- cameronbrown 6y agoWhy not use KMS for storing keys directly?
- hobbescotch 6y agoNot sure if it's still like this, but I got in on KMS early on and I think you could only store Google generated keys. However, I needed to be able to store API keys, passwords, etc... So I used the KMS generated keys to encrypt GCS files who themselves contained the API key, passwords, etc... that needed storing.
- ravenstine 6y agoThat seems nice, but won't you be pwned the day that GCS accidentally loses your files, or your entire Google account gets locked because its algorithm mistakes it as a bot? The danger of that sounds greater than the possibility that your environment variables get leaked.
- alchemism 6y agoNeither of those scenarios quite applies to the cloud side of Google.
- spyspy 6y agoGCS seems like overkill, not sure why the op went that route. You can just store the encrypted secrets in the code and decrypt them with kms at start up.
- hobbescotch 6y ago
- kwhitefoot 6y agoGreat :-( Had to read to the end of the description of why environment variables are bad to discover that it is effectively an advertisement for Docker. I don't use Docker so the article told me pretty much nothing that wasn't fairly obvious already, although it is a valuable reminder.
- cimnine 6y agoThis is not Docker related. If an application spawns a sub-process, that sub-process will inherit all environment variables. Which might be fine or might not be, e.g. if the spawned application is user controlled. Also tools such as Airbrake or Sentry often send all your ENV variables to the error collection server, effectively exposing your secret values. Most such tools offer to filter variables, but that's in my experience almost always not done pro-actively. The only thing that's Docker related in that post is that it does offer a turn-key solution for Docker-based projects. The solution's principle can be applied to other projects though, i.e. secrets should be read from a config file (or something like Vault).
- linkdd 6y agoI wonder why you would run a user supplied application without sandboxing it (reset env, user with almost no permissions, ...)
- bluejekyll 6y ago> If an application spawns a sub-process, that sub-process will inherit all environment variables. ...“by default”, specifying the environment is something you do when creating a new process.
- chme 6y ago> If an application spawns a sub-process, that sub-process will inherit all environment variables. Which might be fine or might not be, e.g. if the spawned application is user controlled. Right. But isn't that well known? Of course you have to explicitly define which environment variables are passed through to the process you are going to spawn, just as you would have to drop permissions to (configuration) files and possible limit capabilities if you start spawning untrusted processes. IDK... For me it make sense that you should not give secrets over the command line, because they will appear in the program listings, but environment variable is pretty much ok in many cases.
- quotemstr 6y agoThe advice is good. But why is ENV capitalized? The term is just "environment variable". One heuristic I use for evaluating technical material is orthography: if you spell or capitalize or spell something improperly, it's likely you'll get a lot else wrong too. As for secret storage: didn't we solve this problem with keyrings? If I must put a secret in long-term plaintext storage, I might as well put it in a file, where I can see, access-control it, and audit it. Where's the audit log for someone reading an environment variable value?
- mightypirate 6y agobecause traditionally envs are all caps. I think it conveys that extremelly well
- cholmon 6y agoMaybe he's got a history of working with PHP? $_ENV is an autoglobal array that gets filled with environment variables. https://www.php.net/manual/en/reserved.variables.environment.php https://www.php.net/manual/en/reserved.variables.environment...
- irjustin 6y agoIt's also accessed that way in Ruby. ENV['MY_VAR']
- jeffbee 6y agoIt's also %ENV in Perl, and ENVIRON in AWK.
- deleted 6y ago[deleted]
- Izkata 6y agoThe example they use at the top of the article is in bash/shell, and seeing it written as in the article (as "ENV variables") is kinda like a code smell, indicating the author isn't really familiar with what they're criticizing. I think that's what GP is getting at, and it threw me as well - the article really only makes sense when looking at it from the perspective of the author not knowing how env vars are scoped.
- y7 6y agoSo the author offers two alternatives: 1. Using docker-secret inside of a Docker swarm 2. Using Keywhiz [1], a Java server together with a FUSE client. This seems overkill for a lot of cases. If environment variables are such a security problem, why not just use a config file (not checked into the source code repository) with proper permissions set? [1] https://developer.squareup.com/blog/protecting-infrastructure-secrets-with-keywhiz/ https://developer.squareup.com/blog/protecting-infrastructur...
- thesimon 6y ago> not checked into the source code repository Or checked-in, for easy distribution, just encrypted: https://github.com/sobolevn/git-secret https://github.com/sobolevn/git-secret
- stanmancan 6y agoI mean anything encrypted needs to be decrypted, meaning you have to... have the key stored in an environment variable on the server?
- kevmo314 6y agoThis would at least alleviate having to know n different env secrets that need to be set.
- stanmancan 6y agoYeah, it’s convenient but it doesn’t solve the topic of the post, which is “you shouldn’t use environment variables for secret data”
- R0b0t1 6y agoInstall the secret for production. For testing you just lock/unlock the secret.
- rad_gruchalski 6y ago
- arnaudsm 6y agoWhat's a best practice that doesn't use this Docker-specific feature ?
- wolco 6y agoUse a config file not checked into git
- jimktrains2 6y agoThis comes back to where does the config file live and how is it managed?
- linkdd 6y agoThe plain config file must live in memory (tmpfs). The encrypted config file on a hard drive that is accessed only to decrypt the file to memory.
- jimktrains2 6y agoNo, I mean management of the secrets. Though it is a rather moot point as the same problems apply to most systems anyway.
- wolco 6y agoWiki protected pages by role. When roles change so do passwords.
- yakshaving_jgt 6y agoThe proposed solution appears to be "use Docker", which is a bit lame.
- agentultra 6y agoIf migrating your infrastructure to swarm is not feasible: - make sure to sanitize the environment before spawning any child processes. - Be sure to `set +x` (or your shell's equivalent) in your CI process - that your secrets never get interpolated into a string through your scripting language.
- tonetheman 6y agoMaybe if all of your stuff is in docker does this article make sense. Otherwise it is just directly wrong. Use ENV variables they work.
- fortran77 6y agoSo to mitigate the security risk of ENV variables I should run a JavaVM, a Java Server, and a FUSE client? No thanks.
- sahoo 6y agoThere is secret manager is AWS and gcp.
- irjustin 6y agoI'm surprised at the amount of responses here that hate this article. It's not without cause/justification. Any stack tracking software (PagerDuty, Rollbar, NewRelic...) gets huge amounts of secrets pumped out to them regularly because of things like this. And the author is not wrong, the environment doesn't need the secret. The application does. Sure, you may not like the proposed solutions, but there are plenty out there, he just named 2.
- shuringai 6y agoenv vars are not collected by default within containers. if OP's alternative is to use docker for every service then the whole problem is nonexistent since containers cannot access the host env.
- biznickman 6y agoRails already solved this with credentials files...
- worldhistory 6y agoI use and recommend Mozilla SOPS https://github.com/mozilla/sops https://github.com/mozilla/sops
- tyingq 6y agoIs transferring them to memory in your startup routine, then doing unsetenv() a reasonable mitigation? It seems like it addresses several of the listed concerns. It's not perfect, of course, but perhaps better, and straightforward.
- ptx 6y agoI don't understand why we have to put them into the environment in the first place (and then make sure we scrub it). Isn't it just as easy to read the secret from a file?
- djsumdog 6y agoAre secrets specific to docker-swarm? If you're using plain docker, I imagine this wouldn't work? k8s has its own secrets system built into deployments. I haven't used it though. I've been at shops that use k8s+vault, and other places that uses marathon/DCOS+consoul. If you're on AWS you can use pod2iam in a k8s cluster and then use the SSM parameter store to encrypt/decrypt secrets based on pod roles. I'm sure Google Cloud must have similar services. The most agnostic way would be to mount in a file or volume at runtime. It's still accessible to the process, but just via the filesystem and not via environment variables. You still need to program with security in mind, but it's less likely for inadvertent leaks; basic layers of security. From there you could use something that encrypts that mount at rest and decrypt it when you start the container. > Environment variables are passed down to child processes, which allows for unintended access. Doesn't this depend on how you create the new process? fork() would keep a copy of the env in both parent/child processes and exec would keep the env because it replaces the current process. But if you start a process using something like the subprocessing module in Python, it would give you a fresh shell for that process, right?
- eihli 6y agoI came across a blogpost describing this workflow recently and I'm curious to hear HN opinions about it. Any pitfalls? https://matthewdowney.github.io/encrypting-keys-in-clojure-applications.html https://matthewdowney.github.io/encrypting-keys-in-clojure-a... 1. Generate a new set of API keys. 2. Read my encrypted map of keys from disk, decrypt it with a passphrase, assoc in the new key & secret, encrypt it again, and write it to disk. 3. At the entry point for my application, use (.readPassword (System/console)) to securely read in the passphrase, and then use it to decrypt the key file and read it into a Clojure map. 4. Instead of passing the key map around (allowing it to potentially escape into a debug log, or be printed at the REPL if I do something dumb), the top level code of my application passes the credentials into a signer-factory for each api that closes over the credentials. ;; The factory is shaped something like this (defn request-signer-factory [{:keys [key secret]] (fn [request-to-sign] (sign-request request-to-sign key secret))) ;; Then an API endpoint looks like this (defn place-order! [signer {:keys [price qty side market post-only?]}] (let [request (comment "Format the order data for the exchange") signed (singer request)] (do-http-request! signed))) I like this workflow more than others which are centered around only encrypting credentials inside of your Git repository, and decrypting them when you clone / pull, because it means that not even on my development machine are keys just sitting around in plaintext.
- kazinator 6y ago> At a previous job I helped solve this problem with a really elegant solution That distinction might go to something packaged in a single .h and .c file, if it passes additional smell tests.
- gfiorav 6y agoI mean, if the argument against it is: what happens if I run arbitrary code in your server? The lease of your problems is having secrets in your env...
- photon12 6y agoAlso this pattern makes path traversal vulnerabilities (a thing not uncommon in web frameworks) have the potential to allow for privilege escalation on Linux via the /proc/self/environ file. I've been on a pentest where a recently disclosed path traversal bug in Rails was not patched in the environment I was testing and I thought I would get at least some credentials from at least one service, but every host used a dedicated API for secret retrieval and there was nothing sensitive exposed via any system. Maybe your threat model doesn't care, just adding a data point.
- joshspankit 6y agoAny chance they documented that setup publicly? Would be interested to dive in to how all that works and gain any new insights
- photon12 6y agoIt's a large company that built their own bespoke internal credentials service running over a TCP port to the application with another proprietary protocol to push key material to hosts. Can't say much more due to NDAs. Edit: this service handles credentials and rotation for hundreds of thousands to millions of hosts.
- joshspankit 6y ago! Wild. I guess with custom protocols there’s not much that can be learned from that setup. Presumably wrap the request in security, handshake to verify authority, use the custom protocol to deliver the secret which is also wrapped in security. Too bad they keep it close to the vest, but I certainly don’t begrudge them for it.
- photon12 6y agoI mean the same company built a service with better architecture that they sell as part of their managed computing environment options. Some people complain about not wanting to use it due to "lock-in."
- joosters 6y agoIf you think that putting secrets in ENV is bad, you probably shouldn't take the advice of running 'docker service create --secret="secure-secret" redis:alpine' either! Putting secrets in a command line makes them just as visible, in fact more so, to other processes. It also makes the secrets available to anyone regardless of permissions, since everyone can monitor the currently running processes and their args (e.g. a 'ps waux' or 'cat /proc/12345/cmdline')
- photon12 6y agoAren't these equivalent, given that the /proc/x/environ file exists?
- joosters 6y ago/proc/x/environ is similar, but has more restrictive permissions: user-readable only, whereas /proc/x/cmdline is world-readable.
- photon12 6y agoFair point. Though I feel like in any modern app deployment scenario that isn't going to be a meaningful defensible boundary. The answer is to use neither, especially given the number of times I've used a path traversal vulnerability to expose /proc/self/environ on a pentest, and the one time I was frustrated on a pentest when every app in the environment used a dedicated API for secret retrieval and the path traversal vulnerability I had access to was worthless.
- imtringued 6y agoHow do docker secrets solve the problem then? According to the article they store the secret in a file as well. You could access the docker secret just as easily as environment variables. The only meaningful difference is that /proc/self/environ will return all secrets at once meanwhile with docker secrets you have to know the file path.
- joshspankit 6y agoOddly not talked about much: In a lot of cases, if an attacker gets even limited access to the application environment, dumping the ENV variables is trivial as they were never intended to be secure. `process.env`, `printenv` (including php attacks), `ENV`, etc
- makethetick 6y agoI just did a write up about how we use a secrets manager to load our environments allowing an easy centralised management across multiple projects/envs. https://news.ycombinator.com/item?id=23822681 https://news.ycombinator.com/item?id=23822681