9 ms·

Environment variables are a legacy mess: Let's dive deep into them

234 points by signa11 1y ago

deleted 1y ago
[deleted]
rsyring 1y ago
Environment variables are often used to pass secrets around. But, despite its ubiquity, I believe that's a bad practice: - On Linux systems, any user process can inspect any other process of that same user for it's environment variables. We can argue about threat model but, especially for a developer's system, there are A LOT of processes running as the same user as the developer. - IMO, this has become an even more pronounced problem with the popularity of running non-containerized LLM agents in the same user space as the developer's primary OS user. It's a secret exfiltration exploiter's dream. - Environment variables are usually passed down to other spawned processes instead of being contained to the primary process which is often the only one that needs it. - Systemd exposes unit environment variables to all system clients through DBUS and warns against using environment variables for secrets[1]. I believe this means non-root users have access to environment variables set for root-only units/services. I could be wrong, I haven't tested it yet. But if this is the case, I bet that's a HUGE surprise to many system admins. I think ephemeral file sharing between a secret managing process (e.g. 1Password's op cli tool) and the process that needs the secrets (flask, terraform, etc.) is likely the only solution that keeps secrets out of files and also out of environment variables. This is how Systemd's credentials system works. But it's far from widely supported. Any good solutions for passing secrets around that don't involve environment variables or regular plain text files? Edit: I think 1Password's op client has a good start in that each new "session" requires my authorization. So I can enable that tool for a cli sessions where I need some secrets but a rogue process that just tries to use the `op` binary isn't going to get to piggyback on that authorization. I'd get a new popup. But this is only step 1. Step 2 is...how to share that secret with the process that needs it and we are now back to the discussion above. 1: https://www.freedesktop.org/software/systemd/man/latest/systemd.exec.html#Environment= https://www.freedesktop.org/software/systemd/man/latest/syst...
- deleted 1y ago
  [deleted]
- robertlagrant 1y ago
  > On Linux systems, any user process can inspect any other process of that same user for it's environment variables. We can argue about threat model but, especially for a developer's system, there are A LOT of processes running as the same user as the developer. This is a very good point I'd never realised! I'm not sure how you get around it, though, as if that program can even find a credential and decrypt a file, if it runs as the user then everything else can go find that credential as well.
  - rsyring 1y ago
    I added a note to my original post about how 1Password's cli tool work. Access to the binary doesn't mean automatic access to any authorized session. I think that's a good start. In my case, I have it tied into my fingerprint reader, so if something want's access to a secret using `op` I get a prompt to authorize. If I haven't just called `op` myself, I know it's a suspect request.
  - chasil 1y ago
    If a root exploit is leveraged, then /proc/*/environ of all processes is visible to the adversary. The classical alternative has been to store (FTP) credentials in a .netrc file (also used by curl). I have some custom code to pull passwords out of a SQLite database. For people who are really concerned with this, a "secrecy manger" is more appropriate, such as Cyberark conjur and summon, or Hashicorp Vault.
    - jkrejcha 1y ago
      If a full root exploit is leveraged, it's already game over already, basically anything at that point is just going to be rearranging deck chairs.
    - robertlagrant 1y ago
      > If a root exploit is leveraged, then /proc/*/environ of all processes is visible to the adversary. > The classical alternative has been to store (FTP) credentials in a .netrc file (also used by curl). You're not suggesting that a netrc file would stop a root exploit?
simonask 1y ago
Interesting read. Another interesting fact is that `setenv()` is fundamentally broken on POSIX, and should essentially never be called in library code. In application code, it should be called only in the absence of any alternative, and certainly before any threads have started. The reason is that `getenv()` hands out raw pointers to the variables, so overwriting a variable using `setenv()` is impossible to guard against. Treat with extreme caution.
- wmf 1y ago
  AFAIK Solaris solved that problem but Linux refuses to copy their solution.
  - foofoo12 1y ago
    Link?
  - ori_b 1y ago
    Their solution was that setenv leaks memory rather than overwriting in place. FreeBSD does the same. See here for discussion: https://freebsd-current.freebsd.narkive.com/NwqZQDWm/fix-for-memory-leak-in-setenv-unsetenv https://freebsd-current.freebsd.narkive.com/NwqZQDWm/fix-for...
    - monocasa 1y ago
      Sure. But practically the amount you leak is infinitesimal. For a lot of applications that's the right call given the rest of the posix semantics you're constrained to and the kinds and frequency of data you pass via env vars.
      - cbsmith 1y ago
        An infinitesimal leak becomes a problem if it is done an infinite number of times... execve seems like the preferable choice on a lot of grounds.
shmerl 1y ago
> POSIX-specified utilities use uppercase envvars, but that’s not prescriptive for your programs. Quite the contrary: you’re encouraged to use lowercase for your envvars so they don’t collide with the standard tools. > But in reality, not many applications use lowercase. The proper etiquette in software development is to use ALL_UPPERCASE I always prefer lower case for env variables in scripts. Thanks for pointing out that it can help reduce clashes with standard tools.
amelius 1y ago
Another legacy mess: Argument list too long It's absolutely crazy that this isn't a dynamically resizable vector.
- formerly_proven 1y ago
  It is, there’s just a limit set by the kernel for the number of pages the command line as a whole can occupy or something like that.
  - amelius 1y ago
    No, it's still an artificial limit, even if you can change it (usually after your command has failed).
- chasil 1y ago
  The xargs command was designed to address this. From this perspective, it is a kludge.
  - amelius 1y ago
    Yes, and xargs isn't the same thing. It splits a single command into multiple commands, which can lead to surprises. The issue also breaks scripts left and right. It really needs to be solved.
the__alchemist 1y ago
I get anxiety whenever I need to set an environment var on Linux. There are (somewhat distro-specific) ways that work properly, but the usual procedures you find online stops working once you reboot (or close the terminal I think?). They should add a simple env var GUI like Windows has that just works, and isn't terminal-specific. Windows has the annoyance of needing to restart the the terminal (or open a new one) for changes to take effect, but works well other than that.
- zdragnar 1y ago
  > but the usual procedures you find online stops working once you reboot (or close the terminal I think?) The environment isn't persistent between sessions. That means you need to make the change in a way that runs on every new session (login or new terminal window). Depending on how your system is configured: .bash_profile gets run once on every login .bashrc gets run once on every non-login new session (i.e. a new terminal window) It's typical, if using these files, to do something like this: if [ -f ~/.bashrc ]; then source ~/.bashrc fi in the .bash_profile file, putting most other things in .bashrc, such that you don't have to worry about the distinction. If you're not even using bash or bash-likes at all, but instead something like Zsh, fish, etc you'll need to set things the way they want to be set there. > They should add a simple env var GUI like Windows has that just works, and isn't terminal-specific This doesn't exist in linux, because there isn't "one place" that all possible terminals draw from. Conceivably, it's possible to write a GUI tool that reads your .bashrc file, looks for anything that resembles a variable, parses the bash code to look for ways it is set, and then present a GUI tool that could update it in common cases, but... it's way easier to just write the change in a text editor.
  - petre 1y ago
    > Conceivably, it's possible to write a GUI tool that reads your .bashrc file What's wrong with an envfile or envdir? The envdir is kind of annoying but at least you can set permissions on the files inside it.
    - zdragnar 1y ago
bjourne 1y ago
We're almost rid of dotfiles in $HOME. Can we please also get rid of the abuse of environment variables for configuration? There is no reason your program needs to muck up the environment table of every process by defining MY_PROGRAM_API_KEY instead of taking a command line argument or reading a configuration file. It's not "secure" just because it's not on the command line. And it will mess up for users because it's nigh impossible to ensure you have the same environment variables over all login types. The variable might be there when you run locally, but not in an ssh session or cron. Some ephemeral configuration such as LD_LIBRARY_PATH and PWD is difficult to handle without environment variables, but those are rare exceptions and not the norm. Contribute to a clean environment! Don't use environment variables for configuration! </rant>
- 1718627440 1y ago
  This problem is also removed by just not exporting the variables. Also you can just pass an envp to execvE.
  - bjourne 1y ago
    "MY_APP_SECRET_KEY=bla myapp" hardly accomplishes anything substantial over "myapp --secret-key=blah" It's just a less robust and less well-supported command-line interface. It's not supported by most gui launchers, PowerShell, nor many cron implementations, for example.
    - akvadrako 1y ago
      The command line can be read by any user on the host (with `ps auxww` for example) while the environment cannot. You should never pass secrets on the command line.
      - bjourne 1y ago
        Environment variables are not more secure than command line parameters! It's such a common misconception that because environment variables are "not seen" they can serve as a secure channel. They emphatically cannot.
aleksandrm 1y ago
That was a hard to read font.
- lemontheme 1y ago
  Huh... Guess it's what you're used to. First thing that struck me about the site is how beautiful I found it. I even inspected the font: Iosevka Web, apparently.
- db48x 1y ago
  Change your browser settings. You can tell the browser what font you prefer to use, and set minimum font sizes. You can also turn off remote fonts, so that pages can only use fonts you already have. Then you can just uninstall any fonts you don’t like or that you cannot read easily.
- jml7c5 1y ago
  I'm surprised you say that. Iosevka is quite beloved as a monospace font. I use it for all my terminals, etc.
  - layer8 1y ago
    Prose in monospace is harder to read regardless of the specific font.
WalterBright 1y ago
I gave up on environment variables long ago. Now my compilers read a dmd.conf file that is in the same directory as the compiler executable.
- deleted 1y ago
  [deleted]
m3047 1y ago
...and people have to find special ways to set them. I didn't even know that pam_env existed until a recent security vulnerability announcement. It's never come to my attention before, I don't think I've ever seen it utilized. I've now made a runbook item to disable it. It's a shame that "configuration" includes undoing the fetishes and helpfulness [0][1] of others. pam-config -d --env [0] Crowded elevator atrium. Multiple elevators running. Elevator wants to close, another one is coming (oh! I heard it "ding"!). Somebody is holding the elevator which wants to depart and trying to wave me in. Why doesn't somebody push them out? [1] I'm at a stop sign. Some complete idiot is trying to turn left onto the street I'm leaving and waving me to turn left in front of them. Fuck no! I turn in front of you, somebody rearends you, you fly forward into me: my fault! You should be able to make this turn, if you can't go around the block! [2] [2] I could go out of my way and turn right. Or I can just wait and see what happens.
- bandrami 1y ago
  If you need realtime scheduling pam_env was the only way to do it until rtkit came out (and it still does it better than rtkit does, annoyingly).
stackskipton 1y ago
SRE/Sysadmin/DevOps/Whatever here, while blog didn't talk about doing anything difficult but setting ENVVAR standards, I will point out all replacements are just as frustrating especially when talking about secrets. Anything involving vaults where Application reaches out to specific secret vault like Hashicorp Vault/OpenBao/Secrets Manager quickly becomes massive vendor lock in where replacement is very difficult due to library replacement and makes vault uptime extremely important. This puts Ops in extremely difficult place when it becomes time to upgrade or do maintenance. Config files have problem of you have secrets, how do you get them into config file since config files are generally kept in public systems? Most of the time it's some form of either "Template replacement by privileged system before handing it to application" or "Entire Config File gets loaded into secret vault and passed into application". Templating can be error prone and loading entire config files into secret manager is frustrating as well since someone could screw up the load. Speaking of config files, since most systems are running containers, and unless you are at Ops discipline company, these config files are never in the right place, it becomes error prone for Ops to screw up the mounting. Also, whatever format you use, JSON/YAML/TOML is ripe for some weird config file bug to emerge like Norway problem in YAML. Getting secrets from Kubernetes Secrets API I've seen done but lock in again. I'd strongly not recommend this approach unless you are designing a Kubernetes operator or other such type system. I will say I've seen Subprocess thing bite people but I've seen less and less subprocess generation these days. Most teams go with message bus type system instead of sub processing since it's more robust and allows independent scaling.
- jppittma 1y ago
  How is the kubernetes secret API lock in? Genuinely wondering - were you trying to use that deployment yaml for something other than a kubernetes deployment? For most applications, you should be mounting the secret on your application, then you can inject it as either an environment variable or a json file that your application reads in an environment agnostic way. Then, on the backend, you can configure etcd to use whatever KMS provider you like for encryption.
  - stackskipton 1y ago
    Because you can't run the container, even for development outside Kubernetes. Yes, you can mount Secrets as Volumes or Env Var in Kubernetes which is fine but I'm not talking about "How you get env var/secret" but "Methods of dealing with config."
jhallenworld 1y ago
Eh, they are just more command line arguments, ones that go on the left side of the command instead of the right. I guess an alternative is something the Windows registry, but I'm not seeing that as a great improvement since it's less direct.
- compiler-guy 1y ago
  Command-line arguments aren't passed to subprocesseses, can't be inspected by arbitrary functions in the same process, and don't leak memory if they are changed.
  - jhallenworld 1y ago
    Yes, I'm just highlighting that it's just as convenient to change them as command line parameters on a per-invocation basis. If there is a proposal for something else, it should retain this capability. Config files and something like the windows registry are much less convenient.
  - zokier 1y ago
    > can't be inspected by arbitrary functions in the same process You can not really rely on that. The initial stack layout is well-defined at least on linux, so digging up argv is not difficult. Or just open /proc/self/cmdline
deleted 1y ago
[deleted]
jonny_eh 1y ago
I can't recommend Varlock [1] enough. A great way to manage env vars in a project. It lets you define which ones are necessary or optional, their types, and where they should be fetched from. [1] https://varlock.dev/ https://varlock.dev/
- esafak 1y ago
  I use https://mise.jdx.dev/ https://mise.jdx.dev/
peter_d_sherman 1y ago
>"Wow, I really enjoyed writing this… …and I hope it wasn’t a boring read." No, it was very interesting actually! An excellent deep-dive into the murky area of Unix/Linux environment variables, how they are set, how they are passed, what's really going on behind the scenes. Basically a must-read for any present or future OS designer... Observation (if I might!) -- environment variables are to programs (especially chains of programs, parent processes, sub processes and sub-sub processes, etc.) what parameters are to functions -- and/or what command-line parameters are... they're sort of like global variables that get passed around a lot... They also can influence program behavior -- and thus the determinism of a program -- and thus the determinism of a chain of programs... Phrased another way -- software that works on one developer's machine might not work on another developer's machine and/or in a production environment because one crucial environment variable was either set or not set, or set to the wrong value... (NixOS seems to understand this pretty well... that "hermetically sealing" or closing or "performing a closure around" (that's probably slightly the wrong language/terminology in "Nix-speak" but bear with me!) a software environment, including the environment variables is the way to create deterministic software builds that run on any machine... but here I'm digressing...) But my main point: I complete agree with the article's author -- environment variables are a legacy mess! Although, if we think about it, environment variables (if we broaden the definition) are a sub-pattern of anything that affects the state of an individual machine or runtime environment -- in other words, things such as the Windows Registry, at least the global aspects of it -- are also in the same category. Future OS's, when they offer environments for programs or chains of programs to run -- should be completely containerized -- that is, the view of the system -- what data/settings/environment variables/registry variables/files/syscalls/global variables it has access to -- should be completely determinable by the user, completely logabble, completely transparent, and completely able to be compared, one environment to another. In this way, software that either a) fails to work at all b) works in a non-deterministic way -- can be more easily debugged/diagnosed/fixed (I'm looking at you, future AI's that will assist humans in doing this!) -- then if all of that information is in various different places, broken, and/or opaque... To reiterate: >"Wow, I really enjoyed writing this… …and I hope it wasn’t a boring read." No, I really enjoyed reading it(!), it's a brilliant article, and thank you for writing it! :-) Upvoted and favorited!
- 1y ago
alexpotato 1y ago
To give an illustration of how bad this can get: At a past firm, I was trying to debug how a particular ENV var was getting set. I started out thinking it was something simple like the user's .bashrc or equivalent. I quickly realized that there were roughly 10 "layers" of env var loadings happening where the first couple layers were: - firm wide - region - business unit - department - team etc etc I ended up having to turn on a bash debug flag so that I could see exactly where the var was getting set in the layer stack.
- jakub_g 1y ago
  Not sure if any other high level languages have it, but Node.js has just added command line flags to trace precisely all env var accesses and modifications: https://nodejs.org/api/cli.html#--trace-env https://nodejs.org/api/cli.html#--trace-env Since you can set/unset/modify env vars in so many ways with different APIs, this sounds super useful in complex debugging scenarios.
- esafak 1y ago
  "One namespace ought to be enough for anybody".
pluto_modadic 1y ago
static secrets that you can even extract and set in a config file (or store in a secrets manager) need to go. TPM secrets that you can't extract, and public keys that you can pin without considering them "secret", along with Oauth / IAM roles, are the way to go.
quotemstr 1y ago
Environment variables also suck due to having zero typo existence. Write FOO_BAR instead of FOO_BARS? Silent failure. V1 of your package recognizes FOO_BAR and V2 changes it to FOO_BARS (because now we can have more than one)? Silent failure.
- sgarland 1y ago
  In a shell script, set -u. It’ll terminate if you reference an unset variable. In other languages, check that the var you pulled in isn’t falsey before proceeding.
  - quotemstr 1y ago
    > In a shell script, set -u Won't help. I'm _running a program_ want to configure it with environment variables, not _writing a program_ that I expect the user to configure for himself. > falsey There are more than two programming languages in the world.
    - sgarland 1y ago
      > There are more than two programming languages in the world. Great, then you know how to check it in your language of choice.
sureglymop 1y ago
On another note, does anyone know how this blog/website was created? I am in general very curious about ssgs hence why I am asking. My guess would be either Jekyll or hand rolled, due to the url structure.
- julianz 1y ago
  The RSS feed has an attribute that suggests Hugo.
deleted 1y ago
[deleted]
trzy 1y ago
Unix is a legacy mess. The sooner we rid ourselves of this scourge, the better. Posted from a productive Windows workstation.
- scuff3d 1y ago
  You couldn't pay me enough to go back to doing dev work on Windows.
- taejavu 1y ago
  As if Windows isn’t its own legacy mess..
- uzerfcwn 1y ago
  I feel that Windows Registry is similar legacy cruft as environment variables. Worse yet, most software doesn't document which registry keys it's using, so you have to find them on some ancient forum comment or do the detective work by yourself.
nrvn 1y ago
One of the worst things about Environment variables among others discussed here is the implicit and opaque nature of them. Majority of applications rely on them in the *nix world. Even if more explicit and obvious ways of configuration files or remote services (consul/etcd, et al.) and command line arguments are supported env vars are traditionally supported as well. But as mentioned in the article it is just a global hashmap that can be cloned and extended for child processes. Maybe in 1979 it was a good design decision. Today it sometimes hurts. For example, kubernetes by default literally pollutes the container’s environment with so-called service links. And you will have fun time debugging a broken application if any of those “default” env vars conflict with the env vars that your app might expect to see. https://kubernetes.io/docs/tutorials/services/connect-applications-service/#environment-variables https://kubernetes.io/docs/tutorials/services/connect-applic... They are ubiquitous and we are living in the world of neo-conservatism in IT where legacy corner cuts are treated as a standard and never challenged (hello /bin, /usr/bin, /lib, /usr/lib)[0] [0] - https://askubuntu.com/a/135679 https://askubuntu.com/a/135679
- oblio 1y ago
  Heh, you can put hjkl in that neo conservatism bucket. Vi hjkl are the way they are because of a dumb terminal from 40+ years ago, which had fewer units sold than the Nokia N9 smartphone.
agumonkey 1y ago
OP: that wasn't a boring read :) thanks
hei-lima 1y ago
Really, really interesting read. I use env vars daily, and it never occurred to me how they actually work. It shows how many things we take for granted have such interesting implementations. :)
ahartmetz 1y ago
My favorite environment variable trivium (that's the singular of trivia) is that PS1 and a few others that "everyone" thinks of as environment variables are not environment variables but shell variables. PS1 doesn't even show up in "env" output!
zzbn00 1y ago
Many comments here focusing on secrets and problems. But here is another way to look at this: Environment variables are the "indefinite scope and dynamic extent" variable bindings to make structured programs out of Unix processes. To compare them to a text file is like writing that a program needs no variables because it can read all the values it needs from an input data file when it needs them. Environment variables are precisely to be used in situation where sub-processes need to be passed information in a way that is not necessarily affecting subsequent calls to that subprocess from another section of a program. Sometime the subprocesses are sequences of nested shells, and sometimes other more complex programs, but the same idea applies.
the_real_cher 1y ago
Programmers: Global Variables are bad Linux admins: Hold my beer
jbverschoor 1y ago
Solved in J2EE

234 points by signa11 1y ago

deleted 1y ago

[deleted]

Environment variables are often used to pass secrets around. But, despite its ubiquity, I believe that's a bad practice: - On Linux systems, any user process can inspect any other process of that same user for it's environment variables. We can argue about threat model but, especially for a developer's system, there are A LOT of processes running as the same user as the developer. - IMO, this has become an even more pronounced problem with the popularity of running non-containerized LLM agents in the same user space as the developer's primary OS user. It's a secret exfiltration exploiter's dream. - Environment variables are usually passed down to other spawned processes instead of being contained to the primary process which is often the only one that needs it. - Systemd exposes unit environment variables to all system clients through DBUS and warns against using environment variables for secrets[1]. I believe this means non-root users have access to environment variables set for root-only units/services. I could be wrong, I haven't tested it yet. But if this is the case, I bet that's a HUGE surprise to many system admins. I think ephemeral file sharing between a secret managing process (e.g. 1Password's op cli tool) and the process that needs the secrets (flask, terraform, etc.) is likely the only solution that keeps secrets out of files and also out of environment variables. This is how Systemd's credentials system works. But it's far from widely supported. Any good solutions for passing secrets around that don't involve environment variables or regular plain text files? Edit: I think 1Password's op client has a good start in that each new "session" requires my authorization. So I can enable that tool for a cli sessions where I need some secrets but a rogue process that just tries to use the `op` binary isn't going to get to piggyback on that authorization. I'd get a new popup. But this is only step 1. Step 2 is...how to share that secret with the process that needs it and we are now back to the discussion above. 1: https://www.freedesktop.org/software/systemd/man/latest/systemd.exec.html#Environment= https://www.freedesktop.org/software/systemd/man/latest/syst...

deleted 1y ago
[deleted]
robertlagrant 1y ago
> On Linux systems, any user process can inspect any other process of that same user for it's environment variables. We can argue about threat model but, especially for a developer's system, there are A LOT of processes running as the same user as the developer. This is a very good point I'd never realised! I'm not sure how you get around it, though, as if that program can even find a credential and decrypt a file, if it runs as the user then everything else can go find that credential as well.
- rsyring 1y ago
  I added a note to my original post about how 1Password's cli tool work. Access to the binary doesn't mean automatic access to any authorized session. I think that's a good start. In my case, I have it tied into my fingerprint reader, so if something want's access to a secret using `op` I get a prompt to authorize. If I haven't just called `op` myself, I know it's a suspect request.
- chasil 1y ago
  If a root exploit is leveraged, then /proc/*/environ of all processes is visible to the adversary. The classical alternative has been to store (FTP) credentials in a .netrc file (also used by curl). I have some custom code to pull passwords out of a SQLite database. For people who are really concerned with this, a "secrecy manger" is more appropriate, such as Cyberark conjur and summon, or Hashicorp Vault.
  - jkrejcha 1y ago
    If a full root exploit is leveraged, it's already game over already, basically anything at that point is just going to be rearranging deck chairs.
  - robertlagrant 1y ago
    > If a root exploit is leveraged, then /proc/*/environ of all processes is visible to the adversary. > The classical alternative has been to store (FTP) credentials in a .netrc file (also used by curl). You're not suggesting that a netrc file would stop a root exploit?

simonask 1y ago

Interesting read. Another interesting fact is that `setenv()` is fundamentally broken on POSIX, and should essentially never be called in library code. In application code, it should be called only in the absence of any alternative, and certainly before any threads have started. The reason is that `getenv()` hands out raw pointers to the variables, so overwriting a variable using `setenv()` is impossible to guard against. Treat with extreme caution.

wmf 1y ago
AFAIK Solaris solved that problem but Linux refuses to copy their solution.
- foofoo12 1y ago
  Link?
- ori_b 1y ago
  Their solution was that setenv leaks memory rather than overwriting in place. FreeBSD does the same. See here for discussion: https://freebsd-current.freebsd.narkive.com/NwqZQDWm/fix-for-memory-leak-in-setenv-unsetenv https://freebsd-current.freebsd.narkive.com/NwqZQDWm/fix-for...
  - monocasa 1y ago
    Sure. But practically the amount you leak is infinitesimal. For a lot of applications that's the right call given the rest of the posix semantics you're constrained to and the kinds and frequency of data you pass via env vars.
    - cbsmith 1y ago
      An infinitesimal leak becomes a problem if it is done an infinite number of times... execve seems like the preferable choice on a lot of grounds.

shmerl 1y ago

> POSIX-specified utilities use uppercase envvars, but that’s not prescriptive for your programs. Quite the contrary: you’re encouraged to use lowercase for your envvars so they don’t collide with the standard tools. > But in reality, not many applications use lowercase. The proper etiquette in software development is to use ALL_UPPERCASE I always prefer lower case for env variables in scripts. Thanks for pointing out that it can help reduce clashes with standard tools.

amelius 1y ago

Another legacy mess: Argument list too long It's absolutely crazy that this isn't a dynamically resizable vector.

formerly_proven 1y ago
It is, there’s just a limit set by the kernel for the number of pages the command line as a whole can occupy or something like that.
- amelius 1y ago
  No, it's still an artificial limit, even if you can change it (usually after your command has failed).
chasil 1y ago
The xargs command was designed to address this. From this perspective, it is a kludge.
- amelius 1y ago
  Yes, and xargs isn't the same thing. It splits a single command into multiple commands, which can lead to surprises. The issue also breaks scripts left and right. It really needs to be solved.

the__alchemist 1y ago

I get anxiety whenever I need to set an environment var on Linux. There are (somewhat distro-specific) ways that work properly, but the usual procedures you find online stops working once you reboot (or close the terminal I think?). They should add a simple env var GUI like Windows has that just works, and isn't terminal-specific. Windows has the annoyance of needing to restart the the terminal (or open a new one) for changes to take effect, but works well other than that.

zdragnar 1y ago
> but the usual procedures you find online stops working once you reboot (or close the terminal I think?) The environment isn't persistent between sessions. That means you need to make the change in a way that runs on every new session (login or new terminal window). Depending on how your system is configured: .bash_profile gets run once on every login .bashrc gets run once on every non-login new session (i.e. a new terminal window) It's typical, if using these files, to do something like this: if [ -f ~/.bashrc ]; then source ~/.bashrc fi in the .bash_profile file, putting most other things in .bashrc, such that you don't have to worry about the distinction. If you're not even using bash or bash-likes at all, but instead something like Zsh, fish, etc you'll need to set things the way they want to be set there. > They should add a simple env var GUI like Windows has that just works, and isn't terminal-specific This doesn't exist in linux, because there isn't "one place" that all possible terminals draw from. Conceivably, it's possible to write a GUI tool that reads your .bashrc file, looks for anything that resembles a variable, parses the bash code to look for ways it is set, and then present a GUI tool that could update it in common cases, but... it's way easier to just write the change in a text editor.
- petre 1y ago
  > Conceivably, it's possible to write a GUI tool that reads your .bashrc file What's wrong with an envfile or envdir? The envdir is kind of annoying but at least you can set permissions on the files inside it.
  - zdragnar 1y ago

bjourne 1y ago

We're almost rid of dotfiles in $HOME. Can we please also get rid of the abuse of environment variables for configuration? There is no reason your program needs to muck up the environment table of every process by defining MY_PROGRAM_API_KEY instead of taking a command line argument or reading a configuration file. It's not "secure" just because it's not on the command line. And it will mess up for users because it's nigh impossible to ensure you have the same environment variables over all login types. The variable might be there when you run locally, but not in an ssh session or cron. Some ephemeral configuration such as LD_LIBRARY_PATH and PWD is difficult to handle without environment variables, but those are rare exceptions and not the norm. Contribute to a clean environment! Don't use environment variables for configuration! </rant>

1718627440 1y ago
This problem is also removed by just not exporting the variables. Also you can just pass an envp to execvE.
- bjourne 1y ago
  "MY_APP_SECRET_KEY=bla myapp" hardly accomplishes anything substantial over "myapp --secret-key=blah" It's just a less robust and less well-supported command-line interface. It's not supported by most gui launchers, PowerShell, nor many cron implementations, for example.
  - akvadrako 1y ago
    The command line can be read by any user on the host (with `ps auxww` for example) while the environment cannot. You should never pass secrets on the command line.
    - bjourne 1y ago
      Environment variables are not more secure than command line parameters! It's such a common misconception that because environment variables are "not seen" they can serve as a secure channel. They emphatically cannot.

aleksandrm 1y ago

That was a hard to read font.

lemontheme 1y ago
Huh... Guess it's what you're used to. First thing that struck me about the site is how beautiful I found it. I even inspected the font: Iosevka Web, apparently.
db48x 1y ago
Change your browser settings. You can tell the browser what font you prefer to use, and set minimum font sizes. You can also turn off remote fonts, so that pages can only use fonts you already have. Then you can just uninstall any fonts you don’t like or that you cannot read easily.
jml7c5 1y ago
I'm surprised you say that. Iosevka is quite beloved as a monospace font. I use it for all my terminals, etc.
- layer8 1y ago
  Prose in monospace is harder to read regardless of the specific font.

WalterBright 1y ago

I gave up on environment variables long ago. Now my compilers read a dmd.conf file that is in the same directory as the compiler executable.

deleted 1y ago
[deleted]

m3047 1y ago

...and people have to find special ways to set them. I didn't even know that pam_env existed until a recent security vulnerability announcement. It's never come to my attention before, I don't think I've ever seen it utilized. I've now made a runbook item to disable it. It's a shame that "configuration" includes undoing the fetishes and helpfulness [0][1] of others. pam-config -d --env [0] Crowded elevator atrium. Multiple elevators running. Elevator wants to close, another one is coming (oh! I heard it "ding"!). Somebody is holding the elevator which wants to depart and trying to wave me in. Why doesn't somebody push them out? [1] I'm at a stop sign. Some complete idiot is trying to turn left onto the street I'm leaving and waving me to turn left in front of them. Fuck no! I turn in front of you, somebody rearends you, you fly forward into me: my fault! You should be able to make this turn, if you can't go around the block! [2] [2] I could go out of my way and turn right. Or I can just wait and see what happens.

bandrami 1y ago
If you need realtime scheduling pam_env was the only way to do it until rtkit came out (and it still does it better than rtkit does, annoyingly).

stackskipton 1y ago

SRE/Sysadmin/DevOps/Whatever here, while blog didn't talk about doing anything difficult but setting ENVVAR standards, I will point out all replacements are just as frustrating especially when talking about secrets. Anything involving vaults where Application reaches out to specific secret vault like Hashicorp Vault/OpenBao/Secrets Manager quickly becomes massive vendor lock in where replacement is very difficult due to library replacement and makes vault uptime extremely important. This puts Ops in extremely difficult place when it becomes time to upgrade or do maintenance. Config files have problem of you have secrets, how do you get them into config file since config files are generally kept in public systems? Most of the time it's some form of either "Template replacement by privileged system before handing it to application" or "Entire Config File gets loaded into secret vault and passed into application". Templating can be error prone and loading entire config files into secret manager is frustrating as well since someone could screw up the load. Speaking of config files, since most systems are running containers, and unless you are at Ops discipline company, these config files are never in the right place, it becomes error prone for Ops to screw up the mounting. Also, whatever format you use, JSON/YAML/TOML is ripe for some weird config file bug to emerge like Norway problem in YAML. Getting secrets from Kubernetes Secrets API I've seen done but lock in again. I'd strongly not recommend this approach unless you are designing a Kubernetes operator or other such type system. I will say I've seen Subprocess thing bite people but I've seen less and less subprocess generation these days. Most teams go with message bus type system instead of sub processing since it's more robust and allows independent scaling.

jppittma 1y ago
How is the kubernetes secret API lock in? Genuinely wondering - were you trying to use that deployment yaml for something other than a kubernetes deployment? For most applications, you should be mounting the secret on your application, then you can inject it as either an environment variable or a json file that your application reads in an environment agnostic way. Then, on the backend, you can configure etcd to use whatever KMS provider you like for encryption.
- stackskipton 1y ago
  Because you can't run the container, even for development outside Kubernetes. Yes, you can mount Secrets as Volumes or Env Var in Kubernetes which is fine but I'm not talking about "How you get env var/secret" but "Methods of dealing with config."

jhallenworld 1y ago

Eh, they are just more command line arguments, ones that go on the left side of the command instead of the right. I guess an alternative is something the Windows registry, but I'm not seeing that as a great improvement since it's less direct.

compiler-guy 1y ago
Command-line arguments aren't passed to subprocesseses, can't be inspected by arbitrary functions in the same process, and don't leak memory if they are changed.
- jhallenworld 1y ago
  Yes, I'm just highlighting that it's just as convenient to change them as command line parameters on a per-invocation basis. If there is a proposal for something else, it should retain this capability. Config files and something like the windows registry are much less convenient.
- zokier 1y ago
  > can't be inspected by arbitrary functions in the same process You can not really rely on that. The initial stack layout is well-defined at least on linux, so digging up argv is not difficult. Or just open /proc/self/cmdline

deleted 1y ago

[deleted]

jonny_eh 1y ago

I can't recommend Varlock [1] enough. A great way to manage env vars in a project. It lets you define which ones are necessary or optional, their types, and where they should be fetched from. [1] https://varlock.dev/ https://varlock.dev/

esafak 1y ago
I use https://mise.jdx.dev/ https://mise.jdx.dev/

peter_d_sherman 1y ago

>"Wow, I really enjoyed writing this… …and I hope it wasn’t a boring read." No, it was very interesting actually! An excellent deep-dive into the murky area of Unix/Linux environment variables, how they are set, how they are passed, what's really going on behind the scenes. Basically a must-read for any present or future OS designer... Observation (if I might!) -- environment variables are to programs (especially chains of programs, parent processes, sub processes and sub-sub processes, etc.) what parameters are to functions -- and/or what command-line parameters are... they're sort of like global variables that get passed around a lot... They also can influence program behavior -- and thus the determinism of a program -- and thus the determinism of a chain of programs... Phrased another way -- software that works on one developer's machine might not work on another developer's machine and/or in a production environment because one crucial environment variable was either set or not set, or set to the wrong value... (NixOS seems to understand this pretty well... that "hermetically sealing" or closing or "performing a closure around" (that's probably slightly the wrong language/terminology in "Nix-speak" but bear with me!) a software environment, including the environment variables is the way to create deterministic software builds that run on any machine... but here I'm digressing...) But my main point: I complete agree with the article's author -- environment variables are a legacy mess! Although, if we think about it, environment variables (if we broaden the definition) are a sub-pattern of anything that affects the state of an individual machine or runtime environment -- in other words, things such as the Windows Registry, at least the global aspects of it -- are also in the same category. Future OS's, when they offer environments for programs or chains of programs to run -- should be completely containerized -- that is, the view of the system -- what data/settings/environment variables/registry variables/files/syscalls/global variables it has access to -- should be completely determinable by the user, completely logabble, completely transparent, and completely able to be compared, one environment to another. In this way, software that either a) fails to work at all b) works in a non-deterministic way -- can be more easily debugged/diagnosed/fixed (I'm looking at you, future AI's that will assist humans in doing this!) -- then if all of that information is in various different places, broken, and/or opaque... To reiterate: >"Wow, I really enjoyed writing this… …and I hope it wasn’t a boring read." No, I really enjoyed reading it(!), it's a brilliant article, and thank you for writing it! :-) Upvoted and favorited!

1y ago

alexpotato 1y ago

To give an illustration of how bad this can get: At a past firm, I was trying to debug how a particular ENV var was getting set. I started out thinking it was something simple like the user's .bashrc or equivalent. I quickly realized that there were roughly 10 "layers" of env var loadings happening where the first couple layers were: - firm wide - region - business unit - department - team etc etc I ended up having to turn on a bash debug flag so that I could see exactly where the var was getting set in the layer stack.

jakub_g 1y ago
Not sure if any other high level languages have it, but Node.js has just added command line flags to trace precisely all env var accesses and modifications: https://nodejs.org/api/cli.html#--trace-env https://nodejs.org/api/cli.html#--trace-env Since you can set/unset/modify env vars in so many ways with different APIs, this sounds super useful in complex debugging scenarios.
esafak 1y ago
"One namespace ought to be enough for anybody".

pluto_modadic 1y ago

static secrets that you can even extract and set in a config file (or store in a secrets manager) need to go. TPM secrets that you can't extract, and public keys that you can pin without considering them "secret", along with Oauth / IAM roles, are the way to go.

quotemstr 1y ago

Environment variables also suck due to having zero typo existence. Write FOO_BAR instead of FOO_BARS? Silent failure. V1 of your package recognizes FOO_BAR and V2 changes it to FOO_BARS (because now we can have more than one)? Silent failure.

sgarland 1y ago
In a shell script, set -u. It’ll terminate if you reference an unset variable. In other languages, check that the var you pulled in isn’t falsey before proceeding.
- quotemstr 1y ago
  > In a shell script, set -u Won't help. I'm _running a program_ want to configure it with environment variables, not _writing a program_ that I expect the user to configure for himself. > falsey There are more than two programming languages in the world.
  - sgarland 1y ago
    > There are more than two programming languages in the world. Great, then you know how to check it in your language of choice.

sureglymop 1y ago

On another note, does anyone know how this blog/website was created? I am in general very curious about ssgs hence why I am asking. My guess would be either Jekyll or hand rolled, due to the url structure.

julianz 1y ago
The RSS feed has an attribute that suggests Hugo.

deleted 1y ago

[deleted]

trzy 1y ago

Unix is a legacy mess. The sooner we rid ourselves of this scourge, the better. Posted from a productive Windows workstation.

scuff3d 1y ago
You couldn't pay me enough to go back to doing dev work on Windows.
taejavu 1y ago
As if Windows isn’t its own legacy mess..
uzerfcwn 1y ago
I feel that Windows Registry is similar legacy cruft as environment variables. Worse yet, most software doesn't document which registry keys it's using, so you have to find them on some ancient forum comment or do the detective work by yourself.

nrvn 1y ago

One of the worst things about Environment variables among others discussed here is the implicit and opaque nature of them. Majority of applications rely on them in the *nix world. Even if more explicit and obvious ways of configuration files or remote services (consul/etcd, et al.) and command line arguments are supported env vars are traditionally supported as well. But as mentioned in the article it is just a global hashmap that can be cloned and extended for child processes. Maybe in 1979 it was a good design decision. Today it sometimes hurts. For example, kubernetes by default literally pollutes the container’s environment with so-called service links. And you will have fun time debugging a broken application if any of those “default” env vars conflict with the env vars that your app might expect to see. https://kubernetes.io/docs/tutorials/services/connect-applications-service/#environment-variables https://kubernetes.io/docs/tutorials/services/connect-applic... They are ubiquitous and we are living in the world of neo-conservatism in IT where legacy corner cuts are treated as a standard and never challenged (hello /bin, /usr/bin, /lib, /usr/lib)[0] [0] - https://askubuntu.com/a/135679 https://askubuntu.com/a/135679

oblio 1y ago
Heh, you can put hjkl in that neo conservatism bucket. Vi hjkl are the way they are because of a dumb terminal from 40+ years ago, which had fewer units sold than the Nokia N9 smartphone.

agumonkey 1y ago

OP: that wasn't a boring read :) thanks

hei-lima 1y ago

Really, really interesting read. I use env vars daily, and it never occurred to me how they actually work. It shows how many things we take for granted have such interesting implementations. :)

ahartmetz 1y ago

My favorite environment variable trivium (that's the singular of trivia) is that PS1 and a few others that "everyone" thinks of as environment variables are not environment variables but shell variables. PS1 doesn't even show up in "env" output!

zzbn00 1y ago

Many comments here focusing on secrets and problems. But here is another way to look at this: Environment variables are the "indefinite scope and dynamic extent" variable bindings to make structured programs out of Unix processes. To compare them to a text file is like writing that a program needs no variables because it can read all the values it needs from an input data file when it needs them. Environment variables are precisely to be used in situation where sub-processes need to be passed information in a way that is not necessarily affecting subsequent calls to that subprocess from another section of a program. Sometime the subprocesses are sequences of nested shells, and sometimes other more complex programs, but the same idea applies.

the_real_cher 1y ago

Programmers: Global Variables are bad Linux admins: Hold my beer

jbverschoor 1y ago

Solved in J2EE