5 ms·

Config files and flags should be equivalent and substitutable. There's no data difference between text-at-command-line and text-at-command-time.

by ethbr0 4y ago

bhawks 4y ago
What about when an option is present in both the file and the command line? Which one takes precedent? If an option occurs 2x on the command line typically the last one wins. If an option appears 2x in the file does the last one win too? With restraint and thought you can make the 2 equivalent from a data perspective. That's a ton of effort and if you get it wrong it is pretty hard to change it later on because someone is going to start relying on previous semantics.
- __turbobrew__ 4y ago
  This problem has been solved for a long time now: http://www.catb.org/~esr/writings/taoup/html/ch10s02.html http://www.catb.org/~esr/writings/taoup/html/ch10s02.html
  - bhawks 4y ago
    Multiple decades of unix software configuration and pointless differences between distro configuration would seem to indicate otherwise. Esr even contradicts himself at the end of that link when he says that site wide configs may be immutably set in the system directory.
    - __turbobrew__ 4y ago
      I believe that the problem has been solved, it has just been poorly implemented in some cases. The general gist that more local configs takes precedence over more global configs is a simple and powerful mechanism which has worked. Yes, distros can be a mess where there are five different config directories but I believe that specific problem is orthogonal to how we solve config precedence.
      - lazide 4y ago
        Do you have any examples that aren’t confusing and super complicated?
        __turbobrew__ 4y ago
        I think the aws cli is a decent example. It isn’t perfect by any means but it demonstrates how to handle config files, environment variables, and command line arguments in a usable way: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html#cli-configure-files-where https://docs.aws.amazon.com/cli/latest/userguide/cli-configu...
        __turbobrew__ 4y ago
        Kind of related but not: if you are interested in cli design I recommend this guide as well: https://clig.dev/ https://clig.dev/ I think people are overly focused on how crappy Unix tools from the previous millennia are (ls, sendmail, dd, etc…) but I do truly believe that cli tools can be ergonomic because I have used and built tools which feel ergonomic.
      - bhawks 4y ago
        It is even poorly implemented in such ubiquitous tools as ls. For example if I always want output to be sorted by last modified the _only_ point of configuration is the command line. But I can control colors via either the command line or an environment variable. I've lost count of the numbers of times I've straced commands to see where they pull configuration from or grepped thru codebases looking for getenv calls. Any unix consensus for configuration is in theory only not practice. File location is actually pretty germane to precedence. A more "global" configuration cannot be applied if the program is looking for it in the wrong location in the first place. Let's not even discuss the horrible state of where to store local configurations in a users home directory. As long as it starts with a . it's fine :). I've used Linux exclusively for nearly 20 years - but not because I loved its configuration management story.
        __turbobrew__ 4y ago
reassembled 4y ago
At my current job I wrote a simple Python class to abstract settings being read from either command line args, environment variables or a TOML-formatted config file. The class first reads settings from the config file if it is available, then environment variables if found, and finally from the command line args if set by the user. The last context to provide a setting value is treated as the actual setting to propagate to the app. If desired the app can then save its current settings state back to a file, either the original config or to a new config file. I arrived at this solution after getting tired of both having to track down where the value for a particular setting originated at runtime, as well as having multiple implementations of the same kinds of argparse/os.environ/file-based settings systems across our internal tools and scripts.
- smitty1e 4y ago
  I just put configuration settings in config.py and import them. Problem solved. Plus I have the full programming language available, not some textual subset with Yet Another Dependency to read it.
yjftsjthsd-h 4y ago
In fact, some programs (mpv comes to mind) make them literally the same, with the config file as a text file containing command line options in the exact same format that you would provide from a shell, with the only difference being that newlines are permitted noop characters.
kdeldycke 4y ago
Agreed. Tried to reconcile this for some of my projects a while back and spent much more time I anticipated. This is an unexpected rabbit hole, between type casting, scoping of flags to subcommands and flags > configuration file > environment variables > default values precedence. And TOML / YAML / JSON / XML / INI multi-format support. I made my hacks open-source, so if you have a Click-based Python CLI, try click-extra, contribute back, and help me solves this problem once and for all: https://kdeldycke.github.io/click-extra/config.html https://kdeldycke.github.io/click-extra/config.html
yencabulator 4y ago
Command line flags are miserably at representing anything beyond key=value. A config file will easily handle a hierarchical tree 5 levels deep.

Config files and flags should be equivalent and substitutable. There's no data difference between text-at-command-line and text-at-command-time.

by ethbr0 4y ago

Config files and flags should be equivalent and substitutable. There's no data difference between text-at-command-line and text-at-command-time.

bhawks 4y ago

What about when an option is present in both the file and the command line? Which one takes precedent? If an option occurs 2x on the command line typically the last one wins. If an option appears 2x in the file does the last one win too? With restraint and thought you can make the 2 equivalent from a data perspective. That's a ton of effort and if you get it wrong it is pretty hard to change it later on because someone is going to start relying on previous semantics.

__turbobrew__ 4y ago
This problem has been solved for a long time now: http://www.catb.org/~esr/writings/taoup/html/ch10s02.html http://www.catb.org/~esr/writings/taoup/html/ch10s02.html
- bhawks 4y ago
  Multiple decades of unix software configuration and pointless differences between distro configuration would seem to indicate otherwise. Esr even contradicts himself at the end of that link when he says that site wide configs may be immutably set in the system directory.
  - __turbobrew__ 4y ago
    I believe that the problem has been solved, it has just been poorly implemented in some cases. The general gist that more local configs takes precedence over more global configs is a simple and powerful mechanism which has worked. Yes, distros can be a mess where there are five different config directories but I believe that specific problem is orthogonal to how we solve config precedence.
    - lazide 4y ago
      Do you have any examples that aren’t confusing and super complicated?
      - __turbobrew__ 4y ago
        I think the aws cli is a decent example. It isn’t perfect by any means but it demonstrates how to handle config files, environment variables, and command line arguments in a usable way: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html#cli-configure-files-where https://docs.aws.amazon.com/cli/latest/userguide/cli-configu...
      - __turbobrew__ 4y ago
        Kind of related but not: if you are interested in cli design I recommend this guide as well: https://clig.dev/ https://clig.dev/ I think people are overly focused on how crappy Unix tools from the previous millennia are (ls, sendmail, dd, etc…) but I do truly believe that cli tools can be ergonomic because I have used and built tools which feel ergonomic.
    - bhawks 4y ago
      It is even poorly implemented in such ubiquitous tools as ls. For example if I always want output to be sorted by last modified the _only_ point of configuration is the command line. But I can control colors via either the command line or an environment variable. I've lost count of the numbers of times I've straced commands to see where they pull configuration from or grepped thru codebases looking for getenv calls. Any unix consensus for configuration is in theory only not practice. File location is actually pretty germane to precedence. A more "global" configuration cannot be applied if the program is looking for it in the wrong location in the first place. Let's not even discuss the horrible state of where to store local configurations in a users home directory. As long as it starts with a . it's fine :). I've used Linux exclusively for nearly 20 years - but not because I loved its configuration management story.
      - __turbobrew__ 4y ago

reassembled 4y ago

At my current job I wrote a simple Python class to abstract settings being read from either command line args, environment variables or a TOML-formatted config file. The class first reads settings from the config file if it is available, then environment variables if found, and finally from the command line args if set by the user. The last context to provide a setting value is treated as the actual setting to propagate to the app. If desired the app can then save its current settings state back to a file, either the original config or to a new config file. I arrived at this solution after getting tired of both having to track down where the value for a particular setting originated at runtime, as well as having multiple implementations of the same kinds of argparse/os.environ/file-based settings systems across our internal tools and scripts.

smitty1e 4y ago
I just put configuration settings in config.py and import them. Problem solved. Plus I have the full programming language available, not some textual subset with Yet Another Dependency to read it.

yjftsjthsd-h 4y ago

In fact, some programs (mpv comes to mind) make them literally the same, with the config file as a text file containing command line options in the exact same format that you would provide from a shell, with the only difference being that newlines are permitted noop characters.

kdeldycke 4y ago

Agreed. Tried to reconcile this for some of my projects a while back and spent much more time I anticipated. This is an unexpected rabbit hole, between type casting, scoping of flags to subcommands and flags > configuration file > environment variables > default values precedence. And TOML / YAML / JSON / XML / INI multi-format support. I made my hacks open-source, so if you have a Click-based Python CLI, try click-extra, contribute back, and help me solves this problem once and for all: https://kdeldycke.github.io/click-extra/config.html https://kdeldycke.github.io/click-extra/config.html

yencabulator 4y ago

Command line flags are miserably at representing anything beyond key=value. A config file will easily handle a hierarchical tree 5 levels deep.