5 ms·

I have seen many of these posts now and I have a simple question. Why do we need special configuration languages? Why not just use existing languages? I unders

I have seen many of these posts now and I have a simple question. Why do we need special configuration languages? Why not just use existing languages? I understand why data formats like JSON and YAML are valuable, and I understand why it would be valuable to use a programming language to automate the generation of such formatted data. But the niche of the configuration language remains mysterious to me.

raphinou 3y ago
I'm guessing here, but maybe you see these configuration languages as templates? However a configuration language is much more than that. For example, jsonnet lets you combine objects and override deeply nested fields [1]. I'm not sure how practical it would be to achieve the same result with a general purpose language. 1: https://jsonnet.org/learning/tutorial.html#oo https://jsonnet.org/learning/tutorial.html#oo
aleksiy123 3y ago
I think because there are some desirable traits for config languages that don't exist in general purpose languages. Not exhaustive list but generally: Usually constrained to reduce complexity and try to eliminate the need for testing config. Interopable between many different programming languages. Readable by programmers working in different languages. I think this usually makes config languages favour declarative over imperative which usually eliminates most general purpose languages. Another topic is why do we use configuration at all and what is the difference between code and config.
- Someone 3y ago
  It also is desirable that you can programmatically process configuration files (example: go through all your docker files to determine which ones use vulnerable images) That is at odds with using a generic Turing complete language, as figuring out what the configuration is requires running code that may do all kinds of weird things (download data, delete files, etc) (that is similar to the postscript/PDF case. To figure out how many pages a postscript file has, you have to run code) Now, some will say they trust those writing config files to not do stupid things and only use that power when it is absolutely required, but that’s not an opinion shared by all.
notfed 3y ago
I think NixOS/Nix is a good example of where we need one. Nix is used for a massive amount of configuration which needs to be functional and turing-complete. Nix is pretty nice and compact as a configuration language, but the "programming language" aspect of Nix, IMO is ugly and unreadable, and could have been designed better.
DrBazza 3y ago
Don't repeat yourself / single-point-of-truth. Config files can be full of repetition, from values, to sections/objects. Defining a value or object once, dramatically reduces search+replace errors for example.
- rco8786 3y ago
  Is that mutually exclusive with using your normal programming language as a config language?
  - Jtsummers 3y ago
    What do you do when your normal programming language stops being your normal programming language (you add a language, switch languages), but you want to preserve details of the configuration? Translate it all into your new language(s)? How do you keep these things synced up? You extract the details into a configuration language/serialization format and then deserialize in every language you work with. Embedding your configuration in your application language only works well as long as you have one application language. (Or perhaps you use Lua or TCL which are both almost trivially embedded into every other language.)
    - danenania 3y ago
      You can compile your general purpose language DSL to json and then I think this isn’t too much of a problem. A bigger issue imo is packaging. General purpose languages aren’t typed well-designed for producing single, understandable standalone files like config languages are. Like I think a simple config DSL in TypeScript could potentially be a perfect way to solve this problem, except that no one wants to lug around a package.json, package-lock.json, and node_modules directory just to write a bit of config. Yet the ability to bring in npm modules—especially those containing relevant types—is where a lot of the attraction of using TypeScript comes from.
      - Jtsummers 3y ago
        So turn the general purpose language into a non-general purpose language by creating a compiler for a subset of it. That might be what we'd call a configuration language then since it's no longer actually the original language (in full). Like JSON and others.
ncallaway 3y ago
> Why do we need special configuration languages? Why not just use existing languages? Sure, or you could turn around and ask: Why do we need languages at all? Surely we can express everything with assembly? Or, shouldn’t one language work for all cases? Why not just use C everywhere? And the answer becomes obvious: some languages are better at certain tasks than others! Then, a “configuration language” is just going to be a programming language that is best suited to generating configuration. The same way that C or Rust have different use cases than C# and Java.
- civilized 3y ago
  Okay, then in your terms my question would be "why is the best language for configuration not an existing language?"
  - ncallaway 3y ago
    I think the fact that most configuration languages tend towards being declarative, while many popular languages are imperative would tend towards an answer. But, like, the answer is because "people had an itch that they couldn't scratch".
lamontcg 3y ago
People keep hacking up YAML with templating to produce a configuration language which is a bad solution (see for example https://leebriggs.co.uk/blog/2019/02/07/why-are-we-templating-yaml https://leebriggs.co.uk/blog/2019/02/07/why-are-we-templatin...). The problem is that the people who are using templated YAML don't really want to learn a full-blown language since the vast majority of them never, ever want to write a compiler or a highly concurrent HTTP server. They definitely don't care about functional programming or want to know what a monad is or want to have to learn about first class functions and closures or async/await. They just want a pretty trivial language that you can learn in a weekend or two. The kind of necessary complexity that you might 'add' to such a language before anything else is making it side-effect free so that you can't alter the running state of the system in the language and the purpose is to render a YAML/JSON/TOML config to feed as declarative config into tools like terraform. That focus is much different from a programming language which gives you the ability to execve() arbitrary commands and comes with concurrency primitives that templated-YAML users just don't care about at all. When you look at it from the perspective of someone who already knows a computer language well it doesn't seem useful, but the point is to address the users who don't know any computer languages particularly well. And 24 years ago I thought we needed to just use general purpose computer languages and turn system administrators into programmers (or fire the ones that wouldn't learn) but that has clearly never happened, and we're retaining the less-technical-than-SWE roles who manage infrastructure, although its now DevOps people managing k8s with YAML. If just using a general purpose language met the social needs that we have then it would have happened already. The existence of templated YAML and its overwhelming success proves that there's a need that has to be met. I'm still skeptical that any of these configuration management languages is thinking clearly about what need is driving templated YAML though, but its nice that there's an explosion of them so that hopefully sooner or later one of them will really stick and become popular.
catlifeonmars 3y ago
There’s no smooth gradient between data format and Turing complete programming language. Let’s say I want environment conditional logic (such as looking up a map of AMIs to AWS regions), but don’t want to allow arbitrary code execution, or I want to embed the parser inside the service that accepts the config. Those are the niches that HCLs fill. A programming language that allows you to enable language constructs individually might be interesting (imagine being able to easily turn on list comprehensions but not allow imports or to disable mutation).
- patrickmay 3y ago
  > There’s no smooth gradient between data format and Turing complete programming language. S-expressions beg to differ.
  - catlifeonmars 3y ago
    Funny you should mention that, I implemented an S-expression based language spelled using JSON for a dynamic configuration use case. It was actually not terrible for a day or two of work, but it was a far cry from a general purpose production grade configuration language due to lack of tooling, intellisense, and a small stdlib. Also, many people found the syntax foreign and confusing; not a deal breaker but just another hurdle. FWIW I agree with you, but driving adoption would be hard.

raphinou 3y ago

I'm guessing here, but maybe you see these configuration languages as templates? However a configuration language is much more than that. For example, jsonnet lets you combine objects and override deeply nested fields [1]. I'm not sure how practical it would be to achieve the same result with a general purpose language. 1: https://jsonnet.org/learning/tutorial.html#oo https://jsonnet.org/learning/tutorial.html#oo

aleksiy123 3y ago

I think because there are some desirable traits for config languages that don't exist in general purpose languages. Not exhaustive list but generally: Usually constrained to reduce complexity and try to eliminate the need for testing config. Interopable between many different programming languages. Readable by programmers working in different languages. I think this usually makes config languages favour declarative over imperative which usually eliminates most general purpose languages. Another topic is why do we use configuration at all and what is the difference between code and config.

Someone 3y ago
It also is desirable that you can programmatically process configuration files (example: go through all your docker files to determine which ones use vulnerable images) That is at odds with using a generic Turing complete language, as figuring out what the configuration is requires running code that may do all kinds of weird things (download data, delete files, etc) (that is similar to the postscript/PDF case. To figure out how many pages a postscript file has, you have to run code) Now, some will say they trust those writing config files to not do stupid things and only use that power when it is absolutely required, but that’s not an opinion shared by all.

notfed 3y ago

I think NixOS/Nix is a good example of where we need one. Nix is used for a massive amount of configuration which needs to be functional and turing-complete. Nix is pretty nice and compact as a configuration language, but the "programming language" aspect of Nix, IMO is ugly and unreadable, and could have been designed better.

DrBazza 3y ago

Don't repeat yourself / single-point-of-truth. Config files can be full of repetition, from values, to sections/objects. Defining a value or object once, dramatically reduces search+replace errors for example.

rco8786 3y ago
Is that mutually exclusive with using your normal programming language as a config language?
- Jtsummers 3y ago
  What do you do when your normal programming language stops being your normal programming language (you add a language, switch languages), but you want to preserve details of the configuration? Translate it all into your new language(s)? How do you keep these things synced up? You extract the details into a configuration language/serialization format and then deserialize in every language you work with. Embedding your configuration in your application language only works well as long as you have one application language. (Or perhaps you use Lua or TCL which are both almost trivially embedded into every other language.)
  - danenania 3y ago
    You can compile your general purpose language DSL to json and then I think this isn’t too much of a problem. A bigger issue imo is packaging. General purpose languages aren’t typed well-designed for producing single, understandable standalone files like config languages are. Like I think a simple config DSL in TypeScript could potentially be a perfect way to solve this problem, except that no one wants to lug around a package.json, package-lock.json, and node_modules directory just to write a bit of config. Yet the ability to bring in npm modules—especially those containing relevant types—is where a lot of the attraction of using TypeScript comes from.
    - Jtsummers 3y ago
      So turn the general purpose language into a non-general purpose language by creating a compiler for a subset of it. That might be what we'd call a configuration language then since it's no longer actually the original language (in full). Like JSON and others.

ncallaway 3y ago

> Why do we need special configuration languages? Why not just use existing languages? Sure, or you could turn around and ask: Why do we need languages at all? Surely we can express everything with assembly? Or, shouldn’t one language work for all cases? Why not just use C everywhere? And the answer becomes obvious: some languages are better at certain tasks than others! Then, a “configuration language” is just going to be a programming language that is best suited to generating configuration. The same way that C or Rust have different use cases than C# and Java.

civilized 3y ago
Okay, then in your terms my question would be "why is the best language for configuration not an existing language?"
- ncallaway 3y ago
  I think the fact that most configuration languages tend towards being declarative, while many popular languages are imperative would tend towards an answer. But, like, the answer is because "people had an itch that they couldn't scratch".

lamontcg 3y ago

People keep hacking up YAML with templating to produce a configuration language which is a bad solution (see for example https://leebriggs.co.uk/blog/2019/02/07/why-are-we-templating-yaml https://leebriggs.co.uk/blog/2019/02/07/why-are-we-templatin...). The problem is that the people who are using templated YAML don't really want to learn a full-blown language since the vast majority of them never, ever want to write a compiler or a highly concurrent HTTP server. They definitely don't care about functional programming or want to know what a monad is or want to have to learn about first class functions and closures or async/await. They just want a pretty trivial language that you can learn in a weekend or two. The kind of necessary complexity that you might 'add' to such a language before anything else is making it side-effect free so that you can't alter the running state of the system in the language and the purpose is to render a YAML/JSON/TOML config to feed as declarative config into tools like terraform. That focus is much different from a programming language which gives you the ability to execve() arbitrary commands and comes with concurrency primitives that templated-YAML users just don't care about at all. When you look at it from the perspective of someone who already knows a computer language well it doesn't seem useful, but the point is to address the users who don't know any computer languages particularly well. And 24 years ago I thought we needed to just use general purpose computer languages and turn system administrators into programmers (or fire the ones that wouldn't learn) but that has clearly never happened, and we're retaining the less-technical-than-SWE roles who manage infrastructure, although its now DevOps people managing k8s with YAML. If just using a general purpose language met the social needs that we have then it would have happened already. The existence of templated YAML and its overwhelming success proves that there's a need that has to be met. I'm still skeptical that any of these configuration management languages is thinking clearly about what need is driving templated YAML though, but its nice that there's an explosion of them so that hopefully sooner or later one of them will really stick and become popular.

catlifeonmars 3y ago

There’s no smooth gradient between data format and Turing complete programming language. Let’s say I want environment conditional logic (such as looking up a map of AMIs to AWS regions), but don’t want to allow arbitrary code execution, or I want to embed the parser inside the service that accepts the config. Those are the niches that HCLs fill. A programming language that allows you to enable language constructs individually might be interesting (imagine being able to easily turn on list comprehensions but not allow imports or to disable mutation).

patrickmay 3y ago
> There’s no smooth gradient between data format and Turing complete programming language. S-expressions beg to differ.
- catlifeonmars 3y ago
  Funny you should mention that, I implemented an S-expression based language spelled using JSON for a dynamic configuration use case. It was actually not terrible for a day or two of work, but it was a far cry from a general purpose production grade configuration language due to lack of tooling, intellisense, and a small stdlib. Also, many people found the syntax foreign and confusing; not a deal breaker but just another hurdle. FWIW I agree with you, but driving adoption would be hard.