6 ms·

Why would you do this vs. having config files?

by closeparen 4y ago

segmondy 4y ago
scripting/automation
- closeparen 4y ago
  I don't follow, is it easier to automate a ton of args than to automate a config file? Or better yet, put the config file in the bundle you're distributing, since it probably benefits from the same change management flow?
  - hamandcheese 4y ago
    > I don't follow, is it easier to automate a ton of args than to automate a config file? Yes? With just flags, the only artifact that you depend on is the program. With a config file, now there is a separate artifact to maintain for every distinct configuration, which could be a lot. I'm not saying its not solvable, but it does strike me as more complicated.
    - closeparen 4y ago
      I guess if you're distributing standalone binaries, that makes sense. The distribution systems I have seen are all tarballs, packages, or container images, where one more file is no problem. There must be some kind of config system where the flags are being driven from though, unless someone is manually typing them out each time.
    - marcthe12 4y ago
      They could done what bwrap does where there is a special CMD flag that tell program to read the flags from FD. So in flatpak which generally has to give a few kbs of flag to bwrap just pass this via FD. Now whether this works on all supported platform is an another question in it self
  - makeitdouble 4y ago
    On automation, if you are building the config dynamically from external or computed parameters, in the case of a config file you'll need to: - open some descriptor on the system - write the file content - execute the command - get rid of the file Of course you'll also need to deal with the constraints/failure of any of these steps (write the file in a location the program can access, take care it still exists or is not overwritten when you run the command, so have pseudo random names and/or lock the file, etc.) In comparison, passing it all as args is one single step.
    - closeparen 4y ago
      Fair. There are all those "configuration management" frameworks that make templating out config files from data a fairly solved problem, but if you're launching programs from other programs that you also wrote, I guess it's less code to have an arbitrarily long arguments list vs. also dealing with files.
deleted 4y ago
[deleted]
bhawks 4y ago
Several drawbacks about config files: + Another artifact to ship and deploy + What file format to use? Obviously different teams will use different formats too. This compounds as you add engineers. + How do you find the initial file? Whichever mechanism you use besides hardcoding it is another gateway to configuring the program. It will be used in addition to the config file and compose in weird ways. I think the limitations of the command line actually make it scalable especially in the Google context.
- ethbr0 4y ago
  Config files and flags should be equivalent and substitutable. There's no data difference between text-at-command-line and text-at-command-time.
  - bhawks 4y ago
    What about when an option is present in both the file and the command line? Which one takes precedent? If an option occurs 2x on the command line typically the last one wins. If an option appears 2x in the file does the last one win too? With restraint and thought you can make the 2 equivalent from a data perspective. That's a ton of effort and if you get it wrong it is pretty hard to change it later on because someone is going to start relying on previous semantics.
    - __turbobrew__ 4y ago
      This problem has been solved for a long time now: http://www.catb.org/~esr/writings/taoup/html/ch10s02.html http://www.catb.org/~esr/writings/taoup/html/ch10s02.html
      - bhawks 4y ago
        Multiple decades of unix software configuration and pointless differences between distro configuration would seem to indicate otherwise. Esr even contradicts himself at the end of that link when he says that site wide configs may be immutably set in the system directory.
kibwen 4y ago
If you routinely have so many CLI flags that you have to save the CLI invocations to a file, then what you have is an ad-hoc, informally-specified, bug-ridden config file.
- ninkendo 4y ago
  The config flags themselves are ad-hoc, informally-specified and bug-ridden in the first place, so what’s the big deal here?
kyrra 4y ago
/meme why not both! Googler, opinions are my own. Sometimes we use flags on my team for controlling experiments and whatnot, flags tend to be used for more static settings, like the name of other jobs we call. We have a few configuration languages at Google, and many of them will push to production very quickly. We tend to use these for experiment controls. The problem with flags as experiment controls, if you need to roll back quickly. It's a lot slower because you have to wait for all the jobs to restart.
- fruit2020 4y ago
  Depending on the type of app, it could also refresh it’s flags or have values pushed to it.
- closeparen 4y ago
  Maybe it’s used differently at Google, for us experiments are something you roll out to a (sticky) subset of users matching certain criteria. So it cannot be merely data (flag or config), but a library function of user ID and whatever other metadata.
raldi 4y ago
So you can run `top` and see exactly what config the process is running under. If it was started with a config file, that file could’ve changed or disappeared since startup.
- krick 4y ago
  Huh, this is a totally legit reason, but the solution is so horrible it shouldn't justify it. Yet I cannot come up with a "proper" way to handle this. Is there any?
  - humanistbot 4y ago
    A logging overlay that stores configs in a database and monitors each process. If you're testing at that scale, why would you limit yourself to what you can do with just top? To do it manually with flags and monitoring with top is very pets not cattle.
  - raldi 4y ago
    Put the config in a file and have a launcher program that turns it into flags, and then stop thinking about the flags.
  - closeparen 4y ago
    At my work, all services have a "debug" HTTP port with endpoints to, among other things, dump the running configuration. But I'll admit that's less general than flags, since it depends on the server processes being built from our standard frameworks. Also for us, we can't SSH onto hosts, and the scheduler that actually knows about args/pids/etc is fairly deep behind the curtain, but HTTP access is easy. But you could see different companies having different constraints.
  - silisili 4y ago
    That's a silly reason. At simplest, you could a) log the entire config at startup. or b) create a specific file with config used, if logging is too messy. or c) make an API to return the running config. or d) use envvars in a container, that you can just dump from 'env' at will. All of which would be cleaner and saner than browsing top.

Why would you do this vs. having config files?

segmondy 4y ago

scripting/automation

closeparen 4y ago
I don't follow, is it easier to automate a ton of args than to automate a config file? Or better yet, put the config file in the bundle you're distributing, since it probably benefits from the same change management flow?
- hamandcheese 4y ago
  > I don't follow, is it easier to automate a ton of args than to automate a config file? Yes? With just flags, the only artifact that you depend on is the program. With a config file, now there is a separate artifact to maintain for every distinct configuration, which could be a lot. I'm not saying its not solvable, but it does strike me as more complicated.
  - closeparen 4y ago
    I guess if you're distributing standalone binaries, that makes sense. The distribution systems I have seen are all tarballs, packages, or container images, where one more file is no problem. There must be some kind of config system where the flags are being driven from though, unless someone is manually typing them out each time.
  - marcthe12 4y ago
    They could done what bwrap does where there is a special CMD flag that tell program to read the flags from FD. So in flatpak which generally has to give a few kbs of flag to bwrap just pass this via FD. Now whether this works on all supported platform is an another question in it self
- makeitdouble 4y ago
  On automation, if you are building the config dynamically from external or computed parameters, in the case of a config file you'll need to: - open some descriptor on the system - write the file content - execute the command - get rid of the file Of course you'll also need to deal with the constraints/failure of any of these steps (write the file in a location the program can access, take care it still exists or is not overwritten when you run the command, so have pseudo random names and/or lock the file, etc.) In comparison, passing it all as args is one single step.
  - closeparen 4y ago
    Fair. There are all those "configuration management" frameworks that make templating out config files from data a fairly solved problem, but if you're launching programs from other programs that you also wrote, I guess it's less code to have an arbitrarily long arguments list vs. also dealing with files.

deleted 4y ago

[deleted]

bhawks 4y ago

Several drawbacks about config files: + Another artifact to ship and deploy + What file format to use? Obviously different teams will use different formats too. This compounds as you add engineers. + How do you find the initial file? Whichever mechanism you use besides hardcoding it is another gateway to configuring the program. It will be used in addition to the config file and compose in weird ways. I think the limitations of the command line actually make it scalable especially in the Google context.

ethbr0 4y ago
Config files and flags should be equivalent and substitutable. There's no data difference between text-at-command-line and text-at-command-time.
- bhawks 4y ago
  What about when an option is present in both the file and the command line? Which one takes precedent? If an option occurs 2x on the command line typically the last one wins. If an option appears 2x in the file does the last one win too? With restraint and thought you can make the 2 equivalent from a data perspective. That's a ton of effort and if you get it wrong it is pretty hard to change it later on because someone is going to start relying on previous semantics.
  - __turbobrew__ 4y ago
    This problem has been solved for a long time now: http://www.catb.org/~esr/writings/taoup/html/ch10s02.html http://www.catb.org/~esr/writings/taoup/html/ch10s02.html
    - bhawks 4y ago
      Multiple decades of unix software configuration and pointless differences between distro configuration would seem to indicate otherwise. Esr even contradicts himself at the end of that link when he says that site wide configs may be immutably set in the system directory.

kibwen 4y ago

If you routinely have so many CLI flags that you have to save the CLI invocations to a file, then what you have is an ad-hoc, informally-specified, bug-ridden config file.

ninkendo 4y ago
The config flags themselves are ad-hoc, informally-specified and bug-ridden in the first place, so what’s the big deal here?

kyrra 4y ago

/meme why not both! Googler, opinions are my own. Sometimes we use flags on my team for controlling experiments and whatnot, flags tend to be used for more static settings, like the name of other jobs we call. We have a few configuration languages at Google, and many of them will push to production very quickly. We tend to use these for experiment controls. The problem with flags as experiment controls, if you need to roll back quickly. It's a lot slower because you have to wait for all the jobs to restart.

fruit2020 4y ago
Depending on the type of app, it could also refresh it’s flags or have values pushed to it.
closeparen 4y ago
Maybe it’s used differently at Google, for us experiments are something you roll out to a (sticky) subset of users matching certain criteria. So it cannot be merely data (flag or config), but a library function of user ID and whatever other metadata.

raldi 4y ago

So you can run `top` and see exactly what config the process is running under. If it was started with a config file, that file could’ve changed or disappeared since startup.

krick 4y ago
Huh, this is a totally legit reason, but the solution is so horrible it shouldn't justify it. Yet I cannot come up with a "proper" way to handle this. Is there any?
- humanistbot 4y ago
  A logging overlay that stores configs in a database and monitors each process. If you're testing at that scale, why would you limit yourself to what you can do with just top? To do it manually with flags and monitoring with top is very pets not cattle.
- raldi 4y ago
  Put the config in a file and have a launcher program that turns it into flags, and then stop thinking about the flags.
- closeparen 4y ago
  At my work, all services have a "debug" HTTP port with endpoints to, among other things, dump the running configuration. But I'll admit that's less general than flags, since it depends on the server processes being built from our standard frameworks. Also for us, we can't SSH onto hosts, and the scheduler that actually knows about args/pids/etc is fairly deep behind the curtain, but HTTP access is easy. But you could see different companies having different constraints.
- silisili 4y ago
  That's a silly reason. At simplest, you could a) log the entire config at startup. or b) create a specific file with config used, if logging is too messy. or c) make an API to return the running config. or d) use envvars in a container, that you can just dump from 'env' at will. All of which would be cleaner and saner than browsing top.