3 ms·

.env desperately needs a formal specification. I wrote a bit of code using some features python_dotenv offers but realized after the fact that other software mi

.env desperately needs a formal specification. I wrote a bit of code using some features python_dotenv offers but realized after the fact that other software might not parse it the same way. I’ve taken to instead doing my templating in Python itself via pydantic BaseSettings

prng2021 3y ago
Can you provide a concrete example? What do you mean by other software not parsing it the same way?
- moojd 3y ago
  Some execute them as a shell script and others parse them as a newline separated list of key value pairs
- 3np 3y ago
  FOO="abc BAR=123" Is this one or two vars? What is the value of FOO?
  - prng2021 3y ago
    I've never seen something like this in an env file. No one should be putting lines like this in an env file unless it's by mistake.
    - 3np 3y ago
      Your personal opinion of how esoteric or weird this is is besides the point - there are fundamentally incompatible interpretations where I can bet that switching will be breaking and potentially dangerous for some out there.
sergiotapia 3y ago
I've only ever used .env as such: export TWILIO_SID="123123123123" export TWILIO_SECRET="123123123123" And then `source .env` in my terminal, then `rails server` or whatever to run my apps. The environment variables will be present to be consumed from within the app. Curious what features you use beyond this.
- koolba 3y ago
  > Curious what features you use beyond this. There’s plenty of outstanding questions. Is the export required? Can variables reference other variables? Can variables reference existing variables in the environment? Can you use other forms of expressions? Can you set overridable defaults for values?
  - sonthonax 3y ago
    It's a shell script that should be executable by any UNIX shell. Parsing it (rather than executing it) is why those questions are raised.
    - claytongulick 3y ago
      In the nodejs community it's normally used via dotenv. So, yes it's parsed and a valid use case.
  - sergiotapia 3y ago
    I've never used/needed those features from my .env files that's why I'm asking for use cases. Are you saying you use those kinds of features in python_dotenv? I'm asking btw, no snark.
    - SOLAR_FIELDS 3y ago
      I use them in python_dotenv for declaring environments that I use to provision infrastructure, as an example: Example, where we have: env_type="dev" env_name="myappenv" primary_subnet_pod="${env_type}-${env_name}-gke-pod-subnet" primary_subnet_service="${env_type}-${env_name}-gke-service-subnet" Which I then might want to use further down, override with defaults, etc and so forth. It strikes a decent balance in that it's a bit more powerful than a static .env file, but doesn't require you to go full-on templating language with Jinja and the like. It's also cross-platform enough in that you could throw a .env in somewhere and expect most languages to have a client library to parse and load it into env. If your use-case is right in that sweet spot it's a pretty good tool. But the behavior, as noted, isn't specced so I can't trust go_dotenv or rust_dotenv or whatever other library to treat it the same way. Interestingly, in practice, it seems like the Node lib is the canonical version that inspired these other ones so in a way it serves as some sort of unofficial spec. It's not something I would write production grade code trusting the unofficial spec on though.
dubcanada 3y ago
Isn't it just environment variables but rather then setting it manually it's in a file? So the spec is the same as regular environment variables?
- SOLAR_FIELDS 3y ago
  No, python_dotenv allows some rudimentary templating in the .env file. So it’s not just specifying env vars
  - dubcanada 3y ago
    That's silly, but I don't think that's solved by a spec. Even with specs there is still JSON-LD, JSON 2.0, JSON API, etc.
    - SahAssar 3y ago
      Nitpick: JSON 2.0 is not a thing. JSON is versionless, maybe you are thinking of JSON5 (which is not a version, it's a superset of JSON with ECMAscript 5 features) or JSONC (also not a version, also a superset of JSON, but this time with comments). JSON-LD and JSON-API are both specs for the contents of JSON, not the dataformat itself.
- panzi 3y ago
  Different dotenv implementations disagree on weather quoting is possible (with quoting comes multi line support). There is more opportunity for disagreement, like weather comments are supported or empty lines are ignored. But all that I used support comments and empty lines (though not all of them support quoted values).
- Too 3y ago
  There is no such thing as a spec for setting “regular” environment variables. Well there is but that’s a much deeper concept in Linux, than what most people get exposed to, in the execve family of functions, where it is defined in C as an array of strings with the key separated from value only by =. Unless you refer to how to do it in shell. That wouldn’t be applicable to a configuration language. Shell has a lot richer syntax and more complex logic, allowing variable templating and even running processes inside a string definition using $(). All the quoting rules around that must be different.
andrew_ 3y ago
https://www.npmjs.com/package/dotenv https://www.npmjs.com/package/dotenv is the gold standard imho. they have a few other packages you can add-on that expand its functionality. I have a custom package which utilizes dotenv, dotenv-expand, and find-up for a truly delightful DX.
RussianCow 3y ago
Or we need to collectively stop using dotenv in favor of other, more formal formats like TOML.
- hotnfresh 3y ago
  If you’re ultimately going to be reading actual env vars, the nice things that toml gives you won’t be there, and having used it as a crutch may leave you with bugs. If you’re reading env as input, eat the pain early or eat it later. If you’re not ultimately going to be reading actual env vars, then yeah—why even consider .env in the first place? Of course don’t use it.
  - RussianCow 3y ago
    > If you’re ultimately going to be reading actual env vars I would argue that most of the time, you don't need to do this in the first place. Far too often people shoehorn things into environment variables that really shouldn't live there, like secrets and app configuration. If you're interfacing with existing software that reads from the environment, obviously you don't have control over that, so using a different format is out of the question (unless you're willing to convert it to env vars at runtime, but that sounds like more effort than it's worth).
- arielcostas 3y ago
  I like ASP.NET Core approach of having an `appsettings.json` where you just specify your settings, and also allowing to have `user-secrets` on a different directory, which prevents you from accidentally adding it to source control (which happens way more than it should). Of course, you can also use environment variables, but creating a JSON file that allows nesting properties is quite nice and probably easier for noobs than figuring out each .env implementation's quirks.

prng2021 3y ago

Can you provide a concrete example? What do you mean by other software not parsing it the same way?

moojd 3y ago
Some execute them as a shell script and others parse them as a newline separated list of key value pairs
3np 3y ago
FOO="abc BAR=123" Is this one or two vars? What is the value of FOO?
- prng2021 3y ago
  I've never seen something like this in an env file. No one should be putting lines like this in an env file unless it's by mistake.
  - 3np 3y ago
    Your personal opinion of how esoteric or weird this is is besides the point - there are fundamentally incompatible interpretations where I can bet that switching will be breaking and potentially dangerous for some out there.

sergiotapia 3y ago

I've only ever used .env as such: export TWILIO_SID="123123123123" export TWILIO_SECRET="123123123123" And then `source .env` in my terminal, then `rails server` or whatever to run my apps. The environment variables will be present to be consumed from within the app. Curious what features you use beyond this.

koolba 3y ago
> Curious what features you use beyond this. There’s plenty of outstanding questions. Is the export required? Can variables reference other variables? Can variables reference existing variables in the environment? Can you use other forms of expressions? Can you set overridable defaults for values?
- sonthonax 3y ago
  It's a shell script that should be executable by any UNIX shell. Parsing it (rather than executing it) is why those questions are raised.
  - claytongulick 3y ago
    In the nodejs community it's normally used via dotenv. So, yes it's parsed and a valid use case.
- sergiotapia 3y ago
  I've never used/needed those features from my .env files that's why I'm asking for use cases. Are you saying you use those kinds of features in python_dotenv? I'm asking btw, no snark.
  - SOLAR_FIELDS 3y ago
    I use them in python_dotenv for declaring environments that I use to provision infrastructure, as an example: Example, where we have: env_type="dev" env_name="myappenv" primary_subnet_pod="${env_type}-${env_name}-gke-pod-subnet" primary_subnet_service="${env_type}-${env_name}-gke-service-subnet" Which I then might want to use further down, override with defaults, etc and so forth. It strikes a decent balance in that it's a bit more powerful than a static .env file, but doesn't require you to go full-on templating language with Jinja and the like. It's also cross-platform enough in that you could throw a .env in somewhere and expect most languages to have a client library to parse and load it into env. If your use-case is right in that sweet spot it's a pretty good tool. But the behavior, as noted, isn't specced so I can't trust go_dotenv or rust_dotenv or whatever other library to treat it the same way. Interestingly, in practice, it seems like the Node lib is the canonical version that inspired these other ones so in a way it serves as some sort of unofficial spec. It's not something I would write production grade code trusting the unofficial spec on though.

dubcanada 3y ago

Isn't it just environment variables but rather then setting it manually it's in a file? So the spec is the same as regular environment variables?

SOLAR_FIELDS 3y ago
No, python_dotenv allows some rudimentary templating in the .env file. So it’s not just specifying env vars
- dubcanada 3y ago
  That's silly, but I don't think that's solved by a spec. Even with specs there is still JSON-LD, JSON 2.0, JSON API, etc.
  - SahAssar 3y ago
    Nitpick: JSON 2.0 is not a thing. JSON is versionless, maybe you are thinking of JSON5 (which is not a version, it's a superset of JSON with ECMAscript 5 features) or JSONC (also not a version, also a superset of JSON, but this time with comments). JSON-LD and JSON-API are both specs for the contents of JSON, not the dataformat itself.
panzi 3y ago
Different dotenv implementations disagree on weather quoting is possible (with quoting comes multi line support). There is more opportunity for disagreement, like weather comments are supported or empty lines are ignored. But all that I used support comments and empty lines (though not all of them support quoted values).
Too 3y ago
There is no such thing as a spec for setting “regular” environment variables. Well there is but that’s a much deeper concept in Linux, than what most people get exposed to, in the execve family of functions, where it is defined in C as an array of strings with the key separated from value only by =. Unless you refer to how to do it in shell. That wouldn’t be applicable to a configuration language. Shell has a lot richer syntax and more complex logic, allowing variable templating and even running processes inside a string definition using $(). All the quoting rules around that must be different.

andrew_ 3y ago

https://www.npmjs.com/package/dotenv https://www.npmjs.com/package/dotenv is the gold standard imho. they have a few other packages you can add-on that expand its functionality. I have a custom package which utilizes dotenv, dotenv-expand, and find-up for a truly delightful DX.

RussianCow 3y ago

Or we need to collectively stop using dotenv in favor of other, more formal formats like TOML.

hotnfresh 3y ago
If you’re ultimately going to be reading actual env vars, the nice things that toml gives you won’t be there, and having used it as a crutch may leave you with bugs. If you’re reading env as input, eat the pain early or eat it later. If you’re not ultimately going to be reading actual env vars, then yeah—why even consider .env in the first place? Of course don’t use it.
- RussianCow 3y ago
  > If you’re ultimately going to be reading actual env vars I would argue that most of the time, you don't need to do this in the first place. Far too often people shoehorn things into environment variables that really shouldn't live there, like secrets and app configuration. If you're interfacing with existing software that reads from the environment, obviously you don't have control over that, so using a different format is out of the question (unless you're willing to convert it to env vars at runtime, but that sounds like more effort than it's worth).
arielcostas 3y ago
I like ASP.NET Core approach of having an `appsettings.json` where you just specify your settings, and also allowing to have `user-secrets` on a different directory, which prevents you from accidentally adding it to source control (which happens way more than it should). Of course, you can also use environment variables, but creating a JSON file that allows nesting properties is quite nice and probably easier for noobs than figuring out each .env implementation's quirks.