15 ms·

This just sounds like solving a problem at the wrong level and making it worse. If your config is so complicated that it warrants a full-blown language, the lo

This just sounds like solving a problem at the wrong level and making it worse. If your config is so complicated that it warrants a full-blown language, the logic should go into the main program that reads the config and decides what to do, not left inside config. Having a programmable config is how you end up with horrible interdependencies where function f() does foo, and you can't figure out why it's not blowing up, until you realize the only caller is dynamically configured in staging_env_4.json which is emitted by config_generator.py which always ensures the precondition for foo, but only if it's also generating prod_env.json at the same time, but it's OK because we always do that anyway, and you're not sure if you need another cup of coffee or whiskey.

digitaltrees 7y ago
Definitely this. Configs reflect the architecture, of there is a lot of complexity that means the architecture is overly complex and can likely be cleaned up.
umvi 7y ago
What if it's config for CI (i.e. Jenkins)? In that case your "config" needs to support branching ("build this if master else build that").
- kakwa_ 7y ago
  CI systems tends to be more about defining a workflow than setting simple parameters, specially if they also handle CD. For that reason, something looking more like a programing language feels more appropriate to me. I kind of like the Jenkinsfile syntax for this use case as a CI workflow is not a static definition defined by a configuration file (yaml, json or ini) but the syntax is also not a fully "open" programing language like Python which would authorize too many potential hacks. A specialized DSL in this case seems more appropriate and being the correct balance between flexibility and complexity, at least for most needs.
- yongjik 7y ago
  I don't know Jenkins, but my take is that "DSL is good, dynamic full-language config is bad." If you need a fully programmable environment, then you need a fully programmable environment. The important thing is that the DSL is a well-defined contract between what you specify and what the underlying software does, so if you say "If X then do Y", then that's exactly what it does. You should be able to follow the DSL line-by-line and debug it. (You've probably noticed that I'm not a fan of declarative languages a la Makefile: you can never figure out where things "start" and "end".) A "full language config" (like Python) is much worse than DSL: usually there's no clear semantics about when any snippet is executed, yet it's powerful enough to bring in the whole outside environment, which is what's going to happen in a team with 5+ people.
macromaniac 7y ago
>the logic should go into the main program that reads the config and decides what to do, not left inside config. if i follow your logic i should replace the 5 if debug statements in my config.js with 50 littered throughout my codebase. Also i lose strong typing and in line documentation. edit:ill add im against emitting json from a config script, that seems like additional complexity. just reference a file, you can use json if and only if you need program interop and reference it from the config script
- shawnz 7y ago
  If you really need to process the configuration in multiple places throughout the app, then you could put that processing code in a service class and put the debug statements there.
  - deleted 7y ago
    [deleted]
  - macromaniac 7y ago
    Thats essentially what config.js does, why split up configuration in multiple places? With scripting languages theres no reason to use json for configurtion unless the config is shared between programs. The main reason config files exist is so you can edit the program without recompiling, i feel like people forgot that and kept blindly using json when js or ts is better. My config file is just like 20 lines but putting the extremely simple conditional logic of if its debug if its client if its server there reduces program complexity by a couple hundred sloc.
    - shawnz 7y ago
      Well, yes, admittedly it's true that scripting languages do have this unique advantage where you can basically hardcode the configuration in the program code itself and still have it be flexible enough to be called a configuration file. In a situation like that, I agree, it might not make sense to break into two files what can just be done in one.
    - 7y ago
cortesoft 7y ago
I totally agree... also, this whole article makes it sound like this person is writing their config a by hand. I don't know what their setup is, but most config where I work are generated by one system and consumed by another. I am not writing these configs on my laptop. We have code that pulls data from a service and generates a config for other systems to use. Having that system output executable code seems to be asking for complexity, security vulnerabilities, and unexplained behavior.
EdwardDiego 7y ago
> the only caller is dynamically configured in staging_env_4.json which is emitted by config_generator.py which always ensures the precondition for foo, but only if it's also generating prod_env.json at the same time That sounds specific enough to be based on a very bad day or week at the office. You have my sympathies.
- yongjik 7y ago
  Haha, thanks, it's not directly based on a particular bug, but the way configs are managed did push me several times to wonder "If I walk out, buy a bottle of whiskey, come back and start drinking it, will I get fired?" To be fair, I didn't do it... yet. I have family, I need salary. -_-
  - philipov 7y ago
    When you work from home, the answer to that question is "No."
  - nineteen999 7y ago
    I did devops at a Web 1.9 SaaS shop a couple of years back. One of the product teams asked me to host a retrospective for them one Friday afternoon. Little did I know it was a regular occurrence for them to supply the whiskey during said retrospective sessions. Another project team leader from a company they acquired regularly paid me in bottles of whiskey when I helped his team hit integration milestones for their product, as the main dev teams weren't helping him.
rfoo 7y ago
Until you see the alternative. Wanna take a look at some configurations written for big Kubernetes cluster in a not at all full-blown language a.k.a "mysteriously templated YAML"?
- throwanem 7y ago
  I do that on a regular basis as part of my work, and they're honestly not that bad. Complex, sure, but largely due to the essential complexity of the problem domain. For sure, I don't see how mixing in some occult magic would make them better.
  - rfoo 7y ago
    I agree they are inherently complicated. The point here is you lose your usual tools available for full-blown programming languages (e.g. tests, type checks or other static analysis), which makes it harder to assess the conf, leads to more pain.
    - slifin 7y ago
      It's shocking how little effort is put into developing static analysis compared to the insufferable amount of type system work that goes on Here we're discussing destroying a data format and replacing it with a programming language because we can't be bothered to do static analysis on a data format? LSP should have made this the golden era of static analysis, but instead everyone conflates type systems and static analysis so the industry gets myopic on typescript and friends If you took static analysis out of typescript I wouldn't even give typescript a second look, meanwhile something like this where a given key should be one of a set we don't even consider how to do that without a type system A data format doesn't have a type system and that's partly why they're great, type systems don't work across different programming languages but data does, data is a shared composable language Data as a concept is more important than a type system and static analysis is more important than a type system
      - rfoo 7y ago
amelius 7y ago
I don't agree. Configuration files tell a program what to do. You want expressive power there. Telling a program what to do merely through values only makes things more indirect. To give an example: you can get into the situation that some configuration values are only valid when configuration value X is Y, and otherwise other configuration values are valid. What better way to model this than through an if-statement in a programming language? This makes it immediately apparent which settings have effect.
- Smaug123 7y ago
  I am involuntarily howling internally in anguish, and if the feeling could speak, it would be screaming "parse, don't validate" (https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/ https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-va...). If some config values are valid when X is true, and other config values are valid when X is false, then use a different schema for the two cases. This is what discriminated unions are for: for representing conditionals statically, inspectably, serialisably! If some config values are valid when X is 0.339, and some are valid when X is 0.340, and some are valid when X is 0.341, and so on, then… I don't know how to help, and maybe I must just avert my eyes in shame as I implement the dynamic logic. (But in that case it seems a bit odd to say you're "telling the program what to do" with this configuration; I'd say you're bolting on a little extra program at the start.)
  - saagarjha 7y ago
    Sometimes “valid” doesn’t mean “illegal to have in a configuration file”, it just means that “in this specific case, the configuration should have a certain value”. For example, on my Mac I should prefix my commands with “g” if I want to access GNU tools, while on most Linux systems I don’t need to do this. Trying to run gsed on Ubuntu would be “invalid” in this case but a parser can’t help me here.
    - Smaug123 7y ago
      Yes it can: if your config file is more declarative than simply "lists of command lines". The program can determine what is true of its environment, and can construct command lines appropriately given the data about intended outcome that is stored in the configuration file.
austincheney 7y ago
I would take that a step further and say any config is itself an inherent form of complexity. Some are more complex than others. A simple approach is no config, as in a default working state that executes when called. If there are optional considerations to produce any working state the application is inherently complex, as in there are multiple working states of which none are the default. To be clear complicate means to to make many while the word simplify means to make fewer.
nitely 7y ago
> Having a programmable config is how you end up with horrible interdependencies where function f() does foo, and you can't figure out why it's not blowing up, until you realize the only caller is dynamically configured in staging_env_4.json which is emitted by config_generator.py which always ensures the precondition for foo, but only if it's also generating prod_env.json at the same time, but it's OK because we always do that anyway, and you're not sure if you need another cup of coffee or whiskey. That sounds like the opposite of what the article describes. The point of having configuration as code is having all code related to config in a single file. There is no config generation, nor config loading.
- pdonis 7y ago
  > The point of having configuration as code is having all code related to config in a single file. This makes no sense, because the program has to understand the config in order to use it. That means the code that defines what should be in the config and how it's structured has to be in the program, not the config.
  - danShumway 7y ago
    No -- you're confusing generation and parsing. Your program is still (often, but necessarily) going to consume a static data structure. Properly handled, using code to generate a config at runtime can reduce the number of `if` statements and weird logic you have inside of your main program. Purely as an example, let's say I'm writing a program that wants to consume a list of files in a JSON config. { "files": ["/path/a", "/path/b"] } Now, let's say I want to pass in a wildcard pattern, or I want to exclude a specific directory, or I want to dynamically load the file paths based on a REST request, or some kind of other complicated crap. If I want to have a static JSON file as the config, I have a couple of options: First, I can put logic inside of my program to handle wildcards, and to distinguish between filepaths and URLs, and to have exclude paths. This is not ideal because it makes my program much more complicated, and it's not even complication for a particular useful reason. 99% of the time I'm not going to need those options, so I'm writing all of this code to try and anticipate weird inputs that most people don't need. It also forces me to make opinionated decisions about stupid stuff like, "what should the glob format be? What Regex variant are we using?" None of these are questions I care about or want to answer. The second option, which would be a lot better, is I could have a 3rd-party program generate a config on the fly, write it to disk, and then launch my actual program when it's done. This means I don't need to have a bunch of extra logic in my main program, and everything is a lot less opinionated. BUT, it also means that the logic to launch my program is split up across multiple files, and now I'm adding some weird dependencies to my runtime. It's a hacky solution that's layering a bunch of complexity on top of what should be a simple, "launch the program." ---- So, what's a third option? I write my config file with code, which my program will execute, and which will return the original static config that's nice to consume. module.exports = { files: [ //some big list of files, maybe populated with a glob. //exclude any filepath that contains the letter 'c' ].filter((file) => !file.includes('c')); }; I still need the code to spit out an array of files at the end, but I can do that after making some network requests, or running custom filter functions on that array, or doing really whatever the heck I want. In many cases this does not introduce any new dependencies to my runtime. It's a lot less error prone -- I can launch multiple instances of my app at the same time without worrying about files overwriting each other. It's also a lot cleaner and easier to refactor, because all of my filter functions and Regexes and generation logic lives in one file. And most importantly, for users who aren't taking advantage of the dynamic nature of the config, I'm still ultimately shipping a smaller, simpler program with a less complicated API. I don't need to ship a bunch of error-prone logic dedicated to reconciling 20 different flags or options from random pull requests. ---- Now, taking this a step further, you don't necessarily need to return a static data object. If I'm building a static site generator, maybe instead of embedding a bunch of logic in my system to load/parse Pug templates, instead I just consume a function from the config that takes a string and returns a string. And the reason that might be attractive (beyond what I list above) is that the programmatic APIs for a library like Pug are already just as short and easy to use as any set of YAML options you can come up with. So instead of forcing your users to learn your new APIs to slot in a custom template engine, they just use the APIs they already know and enjoy, and your documentation is shorter and easier to read, and everyone wins. And again, importantly, getting rid of that extra logic has reduced the amount of code in your main project and made stuff like validation and loading easier to reason about and easier to debug. The way you want to think about a config that returns a function is the same way you already think about functions-as-parameters in functional languages like Lisp or Javascript. The trick is realizing that there is no real difference between a config file on a filesystem and an API for a library you're importing at runtime.
SAI_Peregrinus 7y ago
> and you're not sure if you need another cup of coffee or whiskey. The answer to that is always an Irish Coffee.
pdonis 7y ago
> If your config is so complicated that it warrants a full-blown language, the logic should go into the main program that reads the config Exactly. Even the simple toy example in the article illustrates the problem: why is the Person class being defined in the config? If the program is going to use objects representing people, the code defining the structure of those objects should go in the program. If the program is not going to use objects representing people, why are you adding extra code to your config to define Person objects when you could just use a list of plain tuples?
danbmil99 7y ago
was going to downvote as I use config.py everywhere -- but your second paragraph made me laugh (due to familiarity) so you get the upvote. I suggest that all solutions to the config problem gravitate towards major suckage, but creating new micro-languages like yaml just seems like adding more complexity for zero benefit. The advantage of config.py and config.js is that anyone claiming to be a programmer can read them and immediately grok exactly what's going on. That said, starting now I would go with json as it is the lingua franca that every language has to understand -- and it is not a full programming language; you can't do conditionals or loops etc. Put that shit in your code please! Logic should never be platform-dependant.
throwaway894345 7y ago
Lots of things pretty much require dynamic config. Nontrivial docker compose files, Dockerfiles (if you want a static include mechanism, for example), Kubernetes configs, CloudFormation templates, Terraform templates, etc. They need to be dynamic and most of those technologies have one or more popular and hacky solutions for introducing that dynamism (helm charts use ninja templates, CloudFormation adds an AST son too of YAML and drives users toward insane macro lambda functions, terraform keeps extending its “config” language, etc). I’m fine with those platforms preferring a static view of the world so long as there is a sane way to generate the configs dynamically. I would actually prefer a strictly static interface a la Kubernetes config YAML and other targets can generate the corresponding kubernetes configs. The best time I’ve had has actually been using Python to generate this YAML. I even use type hints so I can statically type check my configs, which isn’t something I can do with vanilla YAML (barring some grotesque jsonschema-esque monstrosity).
Waterluvian 7y ago
Ennnnnnhh... I disagree. I've spent a lot of time with Django and it's configuration files are just vanilla Python. It's really great. I especially like being able to add just a little bit of logic here and there.
kabacha 7y ago
Completely disagree here — that's how you get extreme program bloat. Lets take this scenario. I want current weather value be part of config value. This could be pulled/scraped from online source in 2 lines of python. Alternative as you imply every program should have some complex abstraction for scraping now? That's just silly.
- benecollyridam 7y ago
  Why not just let the program get that value? If my config has side effects I wouldn't be confident that my program would run the same each time. To me it seems like this would lead to poor reproducibility.
  - kabacha 7y ago
    How would the program get the value? It has to pack all of the scraping frameworks and abstract it so it could be interacted enough for text configs? Then it would have more scraping code than whatever program intended to do. > To me it seems like this would lead to poor reproducibility. I'm not sure how reproducibility matter here. You're not writing tests - it's a program on your computer for your personal use.
m463 7y ago
Hmmm... I agree with you, except... I think there's a divide between developer and user that could be bridged with a config. Many, if not every piece of software in the world is crippled by what the developer allows you to do. One example is the systemd config files. they are so brain dead that everyone has to go write their own logic to handle anything remotely flexible. On the other hand, they're a clear limited user interface to systemd. bleh. who knows.
haddr 7y ago
No it is usually not the case. There is a real difference between declarative and imperative approach to configuration and such decisions can be traced in many projects. Simplifying things: declarative configuration is usually the best approach if you manage to capture most of your component behaviour in some relatively simple language with limited expressivity. Imperative is where you start mixing some code in. This is usually because of big amount of corner cases and where flexibility is a must. This is seen quite often in build tools: it’s hard to just configure everything, some projects need to be built using really custom pipelines.
GuB-42 7y ago
Well, the idea is not to generate json files, the idea is to not use json files at all. And even if you don't want to implement logic in config file, there are still advantages in having access to a full language. For example: with a simple config file maxAttachmentSize: 33554432, maxTotalSize: 33816576 with a full language maxAttachmentSize = 32 * 1024 * 1024; // 32 MB covers 99% of use cases maxTotalSize = (256 * 1024) + maxAttachmentSize; // 256 kB of text + one big attachment No logic here, just a more readable file. But that uses several things json doesn't support: comments, arithmetic and variables.
- dhagz 7y ago
  I really don't like JSON for configuration. I prefer TOML or YAML for config file definition. But I would also make sure my config item names are descriptive enough to where I know units from the name, e.g. maxAttachmentSizeBytes. Any other comments I usually prefer to put in my code that loads the config files into my program.
  - cogman10 7y ago
    The other option is to pull in the one of many libs it there that can understand suffixes (or even write your own... Really not that hard). As a bonus, your config is more readable.
  - dkersten 7y ago
    I’m a big fan of TOML. It’s not perfect, but as a configuration format, it does the job much better (imho) than the alternatives.
- johannes1234321 7y ago
  With a configuration language one could use maxTotalSize: 1MB Or similar.
  - filleduchaos 7y ago
    And then have to parse and translate that to a workable value in their program...precisely (part of) the problem the article is pointing out
SamuelAdams 7y ago
> the logic should go into the main program that reads the config and decides what to do, not left inside config. SAP has some 3,000 database tables that are all configuration tables. Consultants are paid $200-$250 an hour to do nothing but tweak all these settings. It sucks that SAP was designed this way but god damn its an easy way to make bank.
davydog187 7y ago
Keep your configs small, but write them in real programming languages. You're dismissing a viable solution that works extremely well. Take Elixir for example, which is configured using Elixir https://hexdocs.pm/elixir/master/Config.html https://hexdocs.pm/elixir/master/Config.html Its simple, powerful, and allows you to make it as complicated or not, as you want.
reirob 7y ago
Didn't read the article, but would like to add that there are already programs used and loved by many people, which are using a programming language for configuration. I know just the following, but there are probably many more: - Emacs which uses elisp an embedded Lisp interpreter to configure but as well to implement new functionality. - XMonad, a tiling Windows Manager written in Haskell and it is configured inside the program itself, i.e. you edit the main program. - Guix package manager (and GuixSD, the distribution) that uses Guile (a scheme) to describe the programs to be installed and their configuration. This case is interesting because Guix is using the same concept/functionality to manage the package dependencies which they took from the Nix package manager, but in contrary to Nix decided to use the general purpose scheme programming language instead of a special configuration language.

digitaltrees 7y ago

Definitely this. Configs reflect the architecture, of there is a lot of complexity that means the architecture is overly complex and can likely be cleaned up.

umvi 7y ago

What if it's config for CI (i.e. Jenkins)? In that case your "config" needs to support branching ("build this if master else build that").

kakwa_ 7y ago
CI systems tends to be more about defining a workflow than setting simple parameters, specially if they also handle CD. For that reason, something looking more like a programing language feels more appropriate to me. I kind of like the Jenkinsfile syntax for this use case as a CI workflow is not a static definition defined by a configuration file (yaml, json or ini) but the syntax is also not a fully "open" programing language like Python which would authorize too many potential hacks. A specialized DSL in this case seems more appropriate and being the correct balance between flexibility and complexity, at least for most needs.
yongjik 7y ago
I don't know Jenkins, but my take is that "DSL is good, dynamic full-language config is bad." If you need a fully programmable environment, then you need a fully programmable environment. The important thing is that the DSL is a well-defined contract between what you specify and what the underlying software does, so if you say "If X then do Y", then that's exactly what it does. You should be able to follow the DSL line-by-line and debug it. (You've probably noticed that I'm not a fan of declarative languages a la Makefile: you can never figure out where things "start" and "end".) A "full language config" (like Python) is much worse than DSL: usually there's no clear semantics about when any snippet is executed, yet it's powerful enough to bring in the whole outside environment, which is what's going to happen in a team with 5+ people.

macromaniac 7y ago

>the logic should go into the main program that reads the config and decides what to do, not left inside config. if i follow your logic i should replace the 5 if debug statements in my config.js with 50 littered throughout my codebase. Also i lose strong typing and in line documentation. edit:ill add im against emitting json from a config script, that seems like additional complexity. just reference a file, you can use json if and only if you need program interop and reference it from the config script

shawnz 7y ago
If you really need to process the configuration in multiple places throughout the app, then you could put that processing code in a service class and put the debug statements there.
- deleted 7y ago
  [deleted]
- macromaniac 7y ago
  Thats essentially what config.js does, why split up configuration in multiple places? With scripting languages theres no reason to use json for configurtion unless the config is shared between programs. The main reason config files exist is so you can edit the program without recompiling, i feel like people forgot that and kept blindly using json when js or ts is better. My config file is just like 20 lines but putting the extremely simple conditional logic of if its debug if its client if its server there reduces program complexity by a couple hundred sloc.
  - shawnz 7y ago
    Well, yes, admittedly it's true that scripting languages do have this unique advantage where you can basically hardcode the configuration in the program code itself and still have it be flexible enough to be called a configuration file. In a situation like that, I agree, it might not make sense to break into two files what can just be done in one.
  - 7y ago

cortesoft 7y ago

I totally agree... also, this whole article makes it sound like this person is writing their config a by hand. I don't know what their setup is, but most config where I work are generated by one system and consumed by another. I am not writing these configs on my laptop. We have code that pulls data from a service and generates a config for other systems to use. Having that system output executable code seems to be asking for complexity, security vulnerabilities, and unexplained behavior.

EdwardDiego 7y ago

> the only caller is dynamically configured in staging_env_4.json which is emitted by config_generator.py which always ensures the precondition for foo, but only if it's also generating prod_env.json at the same time That sounds specific enough to be based on a very bad day or week at the office. You have my sympathies.

yongjik 7y ago
Haha, thanks, it's not directly based on a particular bug, but the way configs are managed did push me several times to wonder "If I walk out, buy a bottle of whiskey, come back and start drinking it, will I get fired?" To be fair, I didn't do it... yet. I have family, I need salary. -_-
- philipov 7y ago
  When you work from home, the answer to that question is "No."
- nineteen999 7y ago
  I did devops at a Web 1.9 SaaS shop a couple of years back. One of the product teams asked me to host a retrospective for them one Friday afternoon. Little did I know it was a regular occurrence for them to supply the whiskey during said retrospective sessions. Another project team leader from a company they acquired regularly paid me in bottles of whiskey when I helped his team hit integration milestones for their product, as the main dev teams weren't helping him.

rfoo 7y ago

Until you see the alternative. Wanna take a look at some configurations written for big Kubernetes cluster in a not at all full-blown language a.k.a "mysteriously templated YAML"?

throwanem 7y ago
I do that on a regular basis as part of my work, and they're honestly not that bad. Complex, sure, but largely due to the essential complexity of the problem domain. For sure, I don't see how mixing in some occult magic would make them better.
- rfoo 7y ago
  I agree they are inherently complicated. The point here is you lose your usual tools available for full-blown programming languages (e.g. tests, type checks or other static analysis), which makes it harder to assess the conf, leads to more pain.
  - slifin 7y ago
    It's shocking how little effort is put into developing static analysis compared to the insufferable amount of type system work that goes on Here we're discussing destroying a data format and replacing it with a programming language because we can't be bothered to do static analysis on a data format? LSP should have made this the golden era of static analysis, but instead everyone conflates type systems and static analysis so the industry gets myopic on typescript and friends If you took static analysis out of typescript I wouldn't even give typescript a second look, meanwhile something like this where a given key should be one of a set we don't even consider how to do that without a type system A data format doesn't have a type system and that's partly why they're great, type systems don't work across different programming languages but data does, data is a shared composable language Data as a concept is more important than a type system and static analysis is more important than a type system
    - rfoo 7y ago

amelius 7y ago

I don't agree. Configuration files tell a program what to do. You want expressive power there. Telling a program what to do merely through values only makes things more indirect. To give an example: you can get into the situation that some configuration values are only valid when configuration value X is Y, and otherwise other configuration values are valid. What better way to model this than through an if-statement in a programming language? This makes it immediately apparent which settings have effect.

Smaug123 7y ago
I am involuntarily howling internally in anguish, and if the feeling could speak, it would be screaming "parse, don't validate" (https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/ https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-va...). If some config values are valid when X is true, and other config values are valid when X is false, then use a different schema for the two cases. This is what discriminated unions are for: for representing conditionals statically, inspectably, serialisably! If some config values are valid when X is 0.339, and some are valid when X is 0.340, and some are valid when X is 0.341, and so on, then… I don't know how to help, and maybe I must just avert my eyes in shame as I implement the dynamic logic. (But in that case it seems a bit odd to say you're "telling the program what to do" with this configuration; I'd say you're bolting on a little extra program at the start.)
- saagarjha 7y ago
  Sometimes “valid” doesn’t mean “illegal to have in a configuration file”, it just means that “in this specific case, the configuration should have a certain value”. For example, on my Mac I should prefix my commands with “g” if I want to access GNU tools, while on most Linux systems I don’t need to do this. Trying to run gsed on Ubuntu would be “invalid” in this case but a parser can’t help me here.
  - Smaug123 7y ago
    Yes it can: if your config file is more declarative than simply "lists of command lines". The program can determine what is true of its environment, and can construct command lines appropriately given the data about intended outcome that is stored in the configuration file.

austincheney 7y ago

I would take that a step further and say any config is itself an inherent form of complexity. Some are more complex than others. A simple approach is no config, as in a default working state that executes when called. If there are optional considerations to produce any working state the application is inherently complex, as in there are multiple working states of which none are the default. To be clear complicate means to to make many while the word simplify means to make fewer.

nitely 7y ago

> Having a programmable config is how you end up with horrible interdependencies where function f() does foo, and you can't figure out why it's not blowing up, until you realize the only caller is dynamically configured in staging_env_4.json which is emitted by config_generator.py which always ensures the precondition for foo, but only if it's also generating prod_env.json at the same time, but it's OK because we always do that anyway, and you're not sure if you need another cup of coffee or whiskey. That sounds like the opposite of what the article describes. The point of having configuration as code is having all code related to config in a single file. There is no config generation, nor config loading.

pdonis 7y ago
> The point of having configuration as code is having all code related to config in a single file. This makes no sense, because the program has to understand the config in order to use it. That means the code that defines what should be in the config and how it's structured has to be in the program, not the config.
- danShumway 7y ago
  No -- you're confusing generation and parsing. Your program is still (often, but necessarily) going to consume a static data structure. Properly handled, using code to generate a config at runtime can reduce the number of `if` statements and weird logic you have inside of your main program. Purely as an example, let's say I'm writing a program that wants to consume a list of files in a JSON config. { "files": ["/path/a", "/path/b"] } Now, let's say I want to pass in a wildcard pattern, or I want to exclude a specific directory, or I want to dynamically load the file paths based on a REST request, or some kind of other complicated crap. If I want to have a static JSON file as the config, I have a couple of options: First, I can put logic inside of my program to handle wildcards, and to distinguish between filepaths and URLs, and to have exclude paths. This is not ideal because it makes my program much more complicated, and it's not even complication for a particular useful reason. 99% of the time I'm not going to need those options, so I'm writing all of this code to try and anticipate weird inputs that most people don't need. It also forces me to make opinionated decisions about stupid stuff like, "what should the glob format be? What Regex variant are we using?" None of these are questions I care about or want to answer. The second option, which would be a lot better, is I could have a 3rd-party program generate a config on the fly, write it to disk, and then launch my actual program when it's done. This means I don't need to have a bunch of extra logic in my main program, and everything is a lot less opinionated. BUT, it also means that the logic to launch my program is split up across multiple files, and now I'm adding some weird dependencies to my runtime. It's a hacky solution that's layering a bunch of complexity on top of what should be a simple, "launch the program." ---- So, what's a third option? I write my config file with code, which my program will execute, and which will return the original static config that's nice to consume. module.exports = { files: [ //some big list of files, maybe populated with a glob. //exclude any filepath that contains the letter 'c' ].filter((file) => !file.includes('c')); }; I still need the code to spit out an array of files at the end, but I can do that after making some network requests, or running custom filter functions on that array, or doing really whatever the heck I want. In many cases this does not introduce any new dependencies to my runtime. It's a lot less error prone -- I can launch multiple instances of my app at the same time without worrying about files overwriting each other. It's also a lot cleaner and easier to refactor, because all of my filter functions and Regexes and generation logic lives in one file. And most importantly, for users who aren't taking advantage of the dynamic nature of the config, I'm still ultimately shipping a smaller, simpler program with a less complicated API. I don't need to ship a bunch of error-prone logic dedicated to reconciling 20 different flags or options from random pull requests. ---- Now, taking this a step further, you don't necessarily need to return a static data object. If I'm building a static site generator, maybe instead of embedding a bunch of logic in my system to load/parse Pug templates, instead I just consume a function from the config that takes a string and returns a string. And the reason that might be attractive (beyond what I list above) is that the programmatic APIs for a library like Pug are already just as short and easy to use as any set of YAML options you can come up with. So instead of forcing your users to learn your new APIs to slot in a custom template engine, they just use the APIs they already know and enjoy, and your documentation is shorter and easier to read, and everyone wins. And again, importantly, getting rid of that extra logic has reduced the amount of code in your main project and made stuff like validation and loading easier to reason about and easier to debug. The way you want to think about a config that returns a function is the same way you already think about functions-as-parameters in functional languages like Lisp or Javascript. The trick is realizing that there is no real difference between a config file on a filesystem and an API for a library you're importing at runtime.

SAI_Peregrinus 7y ago

> and you're not sure if you need another cup of coffee or whiskey. The answer to that is always an Irish Coffee.

pdonis 7y ago

> If your config is so complicated that it warrants a full-blown language, the logic should go into the main program that reads the config Exactly. Even the simple toy example in the article illustrates the problem: why is the Person class being defined in the config? If the program is going to use objects representing people, the code defining the structure of those objects should go in the program. If the program is not going to use objects representing people, why are you adding extra code to your config to define Person objects when you could just use a list of plain tuples?

danbmil99 7y ago

was going to downvote as I use config.py everywhere -- but your second paragraph made me laugh (due to familiarity) so you get the upvote. I suggest that all solutions to the config problem gravitate towards major suckage, but creating new micro-languages like yaml just seems like adding more complexity for zero benefit. The advantage of config.py and config.js is that anyone claiming to be a programmer can read them and immediately grok exactly what's going on. That said, starting now I would go with json as it is the lingua franca that every language has to understand -- and it is not a full programming language; you can't do conditionals or loops etc. Put that shit in your code please! Logic should never be platform-dependant.

throwaway894345 7y ago

Lots of things pretty much require dynamic config. Nontrivial docker compose files, Dockerfiles (if you want a static include mechanism, for example), Kubernetes configs, CloudFormation templates, Terraform templates, etc. They need to be dynamic and most of those technologies have one or more popular and hacky solutions for introducing that dynamism (helm charts use ninja templates, CloudFormation adds an AST son too of YAML and drives users toward insane macro lambda functions, terraform keeps extending its “config” language, etc). I’m fine with those platforms preferring a static view of the world so long as there is a sane way to generate the configs dynamically. I would actually prefer a strictly static interface a la Kubernetes config YAML and other targets can generate the corresponding kubernetes configs. The best time I’ve had has actually been using Python to generate this YAML. I even use type hints so I can statically type check my configs, which isn’t something I can do with vanilla YAML (barring some grotesque jsonschema-esque monstrosity).

Waterluvian 7y ago

Ennnnnnhh... I disagree. I've spent a lot of time with Django and it's configuration files are just vanilla Python. It's really great. I especially like being able to add just a little bit of logic here and there.

kabacha 7y ago

Completely disagree here — that's how you get extreme program bloat. Lets take this scenario. I want current weather value be part of config value. This could be pulled/scraped from online source in 2 lines of python. Alternative as you imply every program should have some complex abstraction for scraping now? That's just silly.

benecollyridam 7y ago
Why not just let the program get that value? If my config has side effects I wouldn't be confident that my program would run the same each time. To me it seems like this would lead to poor reproducibility.
- kabacha 7y ago
  How would the program get the value? It has to pack all of the scraping frameworks and abstract it so it could be interacted enough for text configs? Then it would have more scraping code than whatever program intended to do. > To me it seems like this would lead to poor reproducibility. I'm not sure how reproducibility matter here. You're not writing tests - it's a program on your computer for your personal use.

m463 7y ago

Hmmm... I agree with you, except... I think there's a divide between developer and user that could be bridged with a config. Many, if not every piece of software in the world is crippled by what the developer allows you to do. One example is the systemd config files. they are so brain dead that everyone has to go write their own logic to handle anything remotely flexible. On the other hand, they're a clear limited user interface to systemd. bleh. who knows.

haddr 7y ago

No it is usually not the case. There is a real difference between declarative and imperative approach to configuration and such decisions can be traced in many projects. Simplifying things: declarative configuration is usually the best approach if you manage to capture most of your component behaviour in some relatively simple language with limited expressivity. Imperative is where you start mixing some code in. This is usually because of big amount of corner cases and where flexibility is a must. This is seen quite often in build tools: it’s hard to just configure everything, some projects need to be built using really custom pipelines.

GuB-42 7y ago

Well, the idea is not to generate json files, the idea is to not use json files at all. And even if you don't want to implement logic in config file, there are still advantages in having access to a full language. For example: with a simple config file maxAttachmentSize: 33554432, maxTotalSize: 33816576 with a full language maxAttachmentSize = 32 * 1024 * 1024; // 32 MB covers 99% of use cases maxTotalSize = (256 * 1024) + maxAttachmentSize; // 256 kB of text + one big attachment No logic here, just a more readable file. But that uses several things json doesn't support: comments, arithmetic and variables.

dhagz 7y ago
I really don't like JSON for configuration. I prefer TOML or YAML for config file definition. But I would also make sure my config item names are descriptive enough to where I know units from the name, e.g. maxAttachmentSizeBytes. Any other comments I usually prefer to put in my code that loads the config files into my program.
- cogman10 7y ago
  The other option is to pull in the one of many libs it there that can understand suffixes (or even write your own... Really not that hard). As a bonus, your config is more readable.
- dkersten 7y ago
  I’m a big fan of TOML. It’s not perfect, but as a configuration format, it does the job much better (imho) than the alternatives.
johannes1234321 7y ago
With a configuration language one could use maxTotalSize: 1MB Or similar.
- filleduchaos 7y ago
  And then have to parse and translate that to a workable value in their program...precisely (part of) the problem the article is pointing out

SamuelAdams 7y ago

> the logic should go into the main program that reads the config and decides what to do, not left inside config. SAP has some 3,000 database tables that are all configuration tables. Consultants are paid $200-$250 an hour to do nothing but tweak all these settings. It sucks that SAP was designed this way but god damn its an easy way to make bank.

davydog187 7y ago

Keep your configs small, but write them in real programming languages. You're dismissing a viable solution that works extremely well. Take Elixir for example, which is configured using Elixir https://hexdocs.pm/elixir/master/Config.html https://hexdocs.pm/elixir/master/Config.html Its simple, powerful, and allows you to make it as complicated or not, as you want.

reirob 7y ago

Didn't read the article, but would like to add that there are already programs used and loved by many people, which are using a programming language for configuration. I know just the following, but there are probably many more: - Emacs which uses elisp an embedded Lisp interpreter to configure but as well to implement new functionality. - XMonad, a tiling Windows Manager written in Haskell and it is configured inside the program itself, i.e. you edit the main program. - Guix package manager (and GuixSD, the distribution) that uses Guile (a scheme) to describe the programs to be installed and their configuration. This case is interesting because Guix is using the same concept/functionality to manage the package dependencies which they took from the Nix package manager, but in contrary to Nix decided to use the general purpose scheme programming language instead of a special configuration language.