5 ms·

Several drawbacks about config files: + Another artifact to ship and deploy + What file format to use? Obviously different teams will use different formats to

by bhawks 4y ago

Several drawbacks about config files: + Another artifact to ship and deploy + What file format to use? Obviously different teams will use different formats too. This compounds as you add engineers. + How do you find the initial file? Whichever mechanism you use besides hardcoding it is another gateway to configuring the program. It will be used in addition to the config file and compose in weird ways. I think the limitations of the command line actually make it scalable especially in the Google context.

ethbr0 4y ago
Config files and flags should be equivalent and substitutable. There's no data difference between text-at-command-line and text-at-command-time.
- bhawks 4y ago
  What about when an option is present in both the file and the command line? Which one takes precedent? If an option occurs 2x on the command line typically the last one wins. If an option appears 2x in the file does the last one win too? With restraint and thought you can make the 2 equivalent from a data perspective. That's a ton of effort and if you get it wrong it is pretty hard to change it later on because someone is going to start relying on previous semantics.
  - __turbobrew__ 4y ago
    This problem has been solved for a long time now: http://www.catb.org/~esr/writings/taoup/html/ch10s02.html http://www.catb.org/~esr/writings/taoup/html/ch10s02.html
    - bhawks 4y ago
      Multiple decades of unix software configuration and pointless differences between distro configuration would seem to indicate otherwise. Esr even contradicts himself at the end of that link when he says that site wide configs may be immutably set in the system directory.
      - __turbobrew__ 4y ago
        I believe that the problem has been solved, it has just been poorly implemented in some cases. The general gist that more local configs takes precedence over more global configs is a simple and powerful mechanism which has worked. Yes, distros can be a mess where there are five different config directories but I believe that specific problem is orthogonal to how we solve config precedence.
        lazide 4y ago
        Do you have any examples that aren’t confusing and super complicated?
        __turbobrew__ 4y ago
        I think the aws cli is a decent example. It isn’t perfect by any means but it demonstrates how to handle config files, environment variables, and command line arguments in a usable way: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html#cli-configure-files-where https://docs.aws.amazon.com/cli/latest/userguide/cli-configu...
        __turbobrew__ 4y ago
        Kind of related but not: if you are interested in cli design I recommend this guide as well: https://clig.dev/ https://clig.dev/ I think people are overly focused on how crappy Unix tools from the previous millennia are (ls, sendmail, dd, etc…) but I do truly believe that cli tools can be ergonomic because I have used and built tools which feel ergonomic.
        bhawks 4y ago
        It is even poorly implemented in such ubiquitous tools as ls. For example if I always want output to be sorted by last modified the _only_ point of configuration is the command line. But I can control colors via either the command line or an environment variable. I've lost count of the numbers of times I've straced commands to see where they pull configuration from or grepped thru codebases looking for getenv calls. Any unix consensus for configuration is in theory only not practice. File location is actually pretty germane to precedence. A more "global" configuration cannot be applied if the program is looking for it in the wrong location in the first place. Let's not even discuss the horrible state of where to store local configurations in a users home directory. As long as it starts with a . it's fine :). I've used Linux exclusively for nearly 20 years - but not because I loved its configuration management story.
- reassembled 4y ago
  At my current job I wrote a simple Python class to abstract settings being read from either command line args, environment variables or a TOML-formatted config file. The class first reads settings from the config file if it is available, then environment variables if found, and finally from the command line args if set by the user. The last context to provide a setting value is treated as the actual setting to propagate to the app. If desired the app can then save its current settings state back to a file, either the original config or to a new config file. I arrived at this solution after getting tired of both having to track down where the value for a particular setting originated at runtime, as well as having multiple implementations of the same kinds of argparse/os.environ/file-based settings systems across our internal tools and scripts.
  - smitty1e 4y ago
    I just put configuration settings in config.py and import them. Problem solved. Plus I have the full programming language available, not some textual subset with Yet Another Dependency to read it.
- yjftsjthsd-h 4y ago
  In fact, some programs (mpv comes to mind) make them literally the same, with the config file as a text file containing command line options in the exact same format that you would provide from a shell, with the only difference being that newlines are permitted noop characters.
- kdeldycke 4y ago
  Agreed. Tried to reconcile this for some of my projects a while back and spent much more time I anticipated. This is an unexpected rabbit hole, between type casting, scoping of flags to subcommands and flags > configuration file > environment variables > default values precedence. And TOML / YAML / JSON / XML / INI multi-format support. I made my hacks open-source, so if you have a Click-based Python CLI, try click-extra, contribute back, and help me solves this problem once and for all: https://kdeldycke.github.io/click-extra/config.html https://kdeldycke.github.io/click-extra/config.html
- yencabulator 4y ago
  Command line flags are miserably at representing anything beyond key=value. A config file will easily handle a hierarchical tree 5 levels deep.
ninkendo 4y ago
These are solvable problems, no?
- bhawks 4y ago
  Are they worth solving though? Flags at command line time can be equivalent to flags in file from a data perspective. If you work on it they can behave with similar semantics too. If you work even harder you can ensure you deploy the config file atomically with the binary. You can also make sure that the file is properly secured, and config values from it are observable and auditable. Or you can just mandate that programs are configured by the command line flags passed to them and not have to do all that extra work.
  - ninkendo 4y ago
    IMO, the advantage of flags is in the fact that a human can copy/paste the command line invocation and get all they need to reproduce an execution. That’s great. But when the command line invocation is so big that you can barely even copy/paste it any more, and it exceeds the max command line length on a stock Linux kernel, you’re way past the point of convenience being the important factor. All the normal places you’d want to see the flags (set +e in a bash script echoing them back to you, etc) are suddenly overloaded with megabytes of text for a command line invocation… who’s gonna look at all that? Who is this serving any more? “Flags for everything” would seem like a good idea taken way too far to me.

by bhawks 4y ago

ethbr0 4y ago

Config files and flags should be equivalent and substitutable. There's no data difference between text-at-command-line and text-at-command-time.

bhawks 4y ago
What about when an option is present in both the file and the command line? Which one takes precedent? If an option occurs 2x on the command line typically the last one wins. If an option appears 2x in the file does the last one win too? With restraint and thought you can make the 2 equivalent from a data perspective. That's a ton of effort and if you get it wrong it is pretty hard to change it later on because someone is going to start relying on previous semantics.
- __turbobrew__ 4y ago
  This problem has been solved for a long time now: http://www.catb.org/~esr/writings/taoup/html/ch10s02.html http://www.catb.org/~esr/writings/taoup/html/ch10s02.html
  - bhawks 4y ago
    Multiple decades of unix software configuration and pointless differences between distro configuration would seem to indicate otherwise. Esr even contradicts himself at the end of that link when he says that site wide configs may be immutably set in the system directory.
    - __turbobrew__ 4y ago
      I believe that the problem has been solved, it has just been poorly implemented in some cases. The general gist that more local configs takes precedence over more global configs is a simple and powerful mechanism which has worked. Yes, distros can be a mess where there are five different config directories but I believe that specific problem is orthogonal to how we solve config precedence.
      - lazide 4y ago
        Do you have any examples that aren’t confusing and super complicated?
        __turbobrew__ 4y ago
        I think the aws cli is a decent example. It isn’t perfect by any means but it demonstrates how to handle config files, environment variables, and command line arguments in a usable way: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html#cli-configure-files-where https://docs.aws.amazon.com/cli/latest/userguide/cli-configu...
        __turbobrew__ 4y ago
        Kind of related but not: if you are interested in cli design I recommend this guide as well: https://clig.dev/ https://clig.dev/ I think people are overly focused on how crappy Unix tools from the previous millennia are (ls, sendmail, dd, etc…) but I do truly believe that cli tools can be ergonomic because I have used and built tools which feel ergonomic.
      - bhawks 4y ago
        It is even poorly implemented in such ubiquitous tools as ls. For example if I always want output to be sorted by last modified the _only_ point of configuration is the command line. But I can control colors via either the command line or an environment variable. I've lost count of the numbers of times I've straced commands to see where they pull configuration from or grepped thru codebases looking for getenv calls. Any unix consensus for configuration is in theory only not practice. File location is actually pretty germane to precedence. A more "global" configuration cannot be applied if the program is looking for it in the wrong location in the first place. Let's not even discuss the horrible state of where to store local configurations in a users home directory. As long as it starts with a . it's fine :). I've used Linux exclusively for nearly 20 years - but not because I loved its configuration management story.
reassembled 4y ago
At my current job I wrote a simple Python class to abstract settings being read from either command line args, environment variables or a TOML-formatted config file. The class first reads settings from the config file if it is available, then environment variables if found, and finally from the command line args if set by the user. The last context to provide a setting value is treated as the actual setting to propagate to the app. If desired the app can then save its current settings state back to a file, either the original config or to a new config file. I arrived at this solution after getting tired of both having to track down where the value for a particular setting originated at runtime, as well as having multiple implementations of the same kinds of argparse/os.environ/file-based settings systems across our internal tools and scripts.
- smitty1e 4y ago
  I just put configuration settings in config.py and import them. Problem solved. Plus I have the full programming language available, not some textual subset with Yet Another Dependency to read it.
yjftsjthsd-h 4y ago
In fact, some programs (mpv comes to mind) make them literally the same, with the config file as a text file containing command line options in the exact same format that you would provide from a shell, with the only difference being that newlines are permitted noop characters.
kdeldycke 4y ago
Agreed. Tried to reconcile this for some of my projects a while back and spent much more time I anticipated. This is an unexpected rabbit hole, between type casting, scoping of flags to subcommands and flags > configuration file > environment variables > default values precedence. And TOML / YAML / JSON / XML / INI multi-format support. I made my hacks open-source, so if you have a Click-based Python CLI, try click-extra, contribute back, and help me solves this problem once and for all: https://kdeldycke.github.io/click-extra/config.html https://kdeldycke.github.io/click-extra/config.html
yencabulator 4y ago
Command line flags are miserably at representing anything beyond key=value. A config file will easily handle a hierarchical tree 5 levels deep.

ninkendo 4y ago

These are solvable problems, no?

bhawks 4y ago
Are they worth solving though? Flags at command line time can be equivalent to flags in file from a data perspective. If you work on it they can behave with similar semantics too. If you work even harder you can ensure you deploy the config file atomically with the binary. You can also make sure that the file is properly secured, and config values from it are observable and auditable. Or you can just mandate that programs are configured by the command line flags passed to them and not have to do all that extra work.
- ninkendo 4y ago
  IMO, the advantage of flags is in the fact that a human can copy/paste the command line invocation and get all they need to reproduce an execution. That’s great. But when the command line invocation is so big that you can barely even copy/paste it any more, and it exceeds the max command line length on a stock Linux kernel, you’re way past the point of convenience being the important factor. All the normal places you’d want to see the flags (set +e in a bash script echoing them back to you, etc) are suddenly overloaded with megabytes of text for a command line invocation… who’s gonna look at all that? Who is this serving any more? “Flags for everything” would seem like a good idea taken way too far to me.