4 ms·

The configuration rabbit hole, for practically any system that becomes widely applied or through large amounts of feature iterations. With provisioning systems

The configuration rabbit hole, for practically any system that becomes widely applied or through large amounts of feature iterations. With provisioning systems you rapidly walk the ladder because they are basically starting about five steps down on this already: You start with an INI/yaml/json Then you might have several overlaid INIs Eventually: https://docs.spring.io/spring-boot/docs/2.1.13.RELEASE/reference/html/boot-features-external-config.html https://docs.spring.io/spring-boot/docs/2.1.13.RELEASE/refer... Go ahead and make a java crack or something similar. Every line of that hierarchy is bathed in the blood of IT workers in the real world. Wait! Not even close to being done. Templating, expansions, "classes", embedded expressions: now we have computation folks! I don't actually know the computational power of HCL, but eventually you end up with a Turing Complete configuration input that the poster wants. And I'm actually leaving a lot of things out. So many systems go through this, picking their own path. Kind of like workflow engines, almost all major systems will end up with a workflow engine in it (and the builds are workflow anyway). But there aren't well-conserved implementations of these overarching meta-patterns, so everyone bespokes it or picks from a mind numbingly vast array of pseudo-matching options. Good luck!

dsheets 3y ago
The article did not advocate for a TC config language. I think Ruud would probably have been happy with a total primitive recursive language.
- ruuda 3y ago
  To me the distinction is not important. It is possible to write programs that take too long to execute in non-Turing complete languages, and when a program hangs, you just terminate it. I think what people really want is for their configuration to remain simple, but not being Turing complete does not guarantee that. For RCL I added a limit on the number of evaluation steps, because without it some programs trap the fuzzer.
  - dsheets 3y ago
    We agree. You don’t want universal computation in your configuration. Your “gas” approach is the right one and very weak computationally. I further posit that you don’t even want to offer gas fuelled TC and everyone would be happy with a provably total primitive recursive language fuelled by gas. Why? Accidents happen. I’d rather have my configuration tool tell me statically that some construct isn’t provably terminating than wait to find out later when someone imports my config snippet library and uses it in an unexpected way.
stevekemp 3y ago
It's scary to think how many times I've had to evolve configuration options as systems developed and the user-base changed. As you say, I'd start out with INI files, then move on to JSON, after that it would come a hack to allow the JSON to be generated by executing a shell/ruby/perl script. (i.e. "--config=!xx" would execute XX and parse the output, instead of reading a file). Later still we started embedding lua, or similar, to allow things to be templated and "dynamic" on a per-host basis. I was always fond of the Apache-style configuration system, and I guess HCL is close to that, but there aren't any great universal solutions unless you go all-in with scripting, and then you end up with emacs!
qznc 3y ago
I defined myself 5 levels: Level 1 is just values in a file. The Linux kernel uses that. Level 2 is a list of values, e.g. ini files. Level 3 allows nesting. JSON, XML, and YAML are here. Level 4 allows computation but limited. Dhall and Starlark are here. Level 5 is a Turing-complete language. Python, Javascript, etc. RCL seems to be level 5, so I'm not sure if there is really an advantage compared to Python.
- mst 3y ago
  I think the goal is to have a language that feels like it was designed for level 3/4 but allows you to break out to level 5 as an escape hatch. Given how many configuration languages eventually end up with a half-assed level 5 (Greenspun comes to mind) it strikes me as, at least, an experiment worth performing.
gary_0 3y ago
I wonder if anyone would take me seriously if I suggested WASM as a configuration language. When run, the WASM would be given imports that provide a small API for manipulating the config DOM, and importing and executing further WASM files. Then you can use whatever high-level config language you want (or, say, a subset of Rust) and compile it to WASM. Existing JSON/YAML/etc could be compiled to WASM that simply builds the corresponding DOM. Funny joke?
- coderedart 3y ago
  using wasm goes into crazy territory imo. It would probably be bigger, and opaque, with no proper errors (unless you compile debug info which would make it much bigger). I think we already have an almost perfect language i.e. Lua Everyone knows it (or can learn it easily). tiny runtime to embed. sandboxed by default. garbage collected. Only feature missing is static typing.
- viktree 3y ago
  I think this is what the zellij terminal multiplexer uses for extensions. It's worth mentioning that in the opposite direction, if you wanted to have an interpreted config option with fewer compile steps, there's also the option to use lua — a language for which the compiler itself is quite small and portable See, [1] : https://zellij.dev https://zellij.dev
marcosdumay 3y ago
Hum, most systems do not go down that hole at all. Most software stop at most at hierarchical parameters. For the ones that do go down the hole, you are almost always better on separating them into components, to minimize the ones that need complex configuration, and accelerating the journey of those small pieces. (But yeah, it would be nice to settle on a good workflow description language that isn't a makefile.) Now, it seems that everybody that is pushing those high-complexity languages wants an infrastructure description language, and just keeps calling it by "configuration language". Since those two problems are completely different, insisting on using the wrong name is quite harmful to the goal.

dsheets 3y ago

The article did not advocate for a TC config language. I think Ruud would probably have been happy with a total primitive recursive language.

ruuda 3y ago
To me the distinction is not important. It is possible to write programs that take too long to execute in non-Turing complete languages, and when a program hangs, you just terminate it. I think what people really want is for their configuration to remain simple, but not being Turing complete does not guarantee that. For RCL I added a limit on the number of evaluation steps, because without it some programs trap the fuzzer.
- dsheets 3y ago
  We agree. You don’t want universal computation in your configuration. Your “gas” approach is the right one and very weak computationally. I further posit that you don’t even want to offer gas fuelled TC and everyone would be happy with a provably total primitive recursive language fuelled by gas. Why? Accidents happen. I’d rather have my configuration tool tell me statically that some construct isn’t provably terminating than wait to find out later when someone imports my config snippet library and uses it in an unexpected way.

stevekemp 3y ago

It's scary to think how many times I've had to evolve configuration options as systems developed and the user-base changed. As you say, I'd start out with INI files, then move on to JSON, after that it would come a hack to allow the JSON to be generated by executing a shell/ruby/perl script. (i.e. "--config=!xx" would execute XX and parse the output, instead of reading a file). Later still we started embedding lua, or similar, to allow things to be templated and "dynamic" on a per-host basis. I was always fond of the Apache-style configuration system, and I guess HCL is close to that, but there aren't any great universal solutions unless you go all-in with scripting, and then you end up with emacs!

qznc 3y ago

I defined myself 5 levels: Level 1 is just values in a file. The Linux kernel uses that. Level 2 is a list of values, e.g. ini files. Level 3 allows nesting. JSON, XML, and YAML are here. Level 4 allows computation but limited. Dhall and Starlark are here. Level 5 is a Turing-complete language. Python, Javascript, etc. RCL seems to be level 5, so I'm not sure if there is really an advantage compared to Python.

mst 3y ago
I think the goal is to have a language that feels like it was designed for level 3/4 but allows you to break out to level 5 as an escape hatch. Given how many configuration languages eventually end up with a half-assed level 5 (Greenspun comes to mind) it strikes me as, at least, an experiment worth performing.

gary_0 3y ago

I wonder if anyone would take me seriously if I suggested WASM as a configuration language. When run, the WASM would be given imports that provide a small API for manipulating the config DOM, and importing and executing further WASM files. Then you can use whatever high-level config language you want (or, say, a subset of Rust) and compile it to WASM. Existing JSON/YAML/etc could be compiled to WASM that simply builds the corresponding DOM. Funny joke?

coderedart 3y ago
using wasm goes into crazy territory imo. It would probably be bigger, and opaque, with no proper errors (unless you compile debug info which would make it much bigger). I think we already have an almost perfect language i.e. Lua Everyone knows it (or can learn it easily). tiny runtime to embed. sandboxed by default. garbage collected. Only feature missing is static typing.
viktree 3y ago
I think this is what the zellij terminal multiplexer uses for extensions. It's worth mentioning that in the opposite direction, if you wanted to have an interpreted config option with fewer compile steps, there's also the option to use lua — a language for which the compiler itself is quite small and portable See, [1] : https://zellij.dev https://zellij.dev

marcosdumay 3y ago

Hum, most systems do not go down that hole at all. Most software stop at most at hierarchical parameters. For the ones that do go down the hole, you are almost always better on separating them into components, to minimize the ones that need complex configuration, and accelerating the journey of those small pieces. (But yeah, it would be nice to settle on a good workflow description language that isn't a makefile.) Now, it seems that everybody that is pushing those high-complexity languages wants an infrastructure description language, and just keeps calling it by "configuration language". Since those two problems are completely different, insisting on using the wrong name is quite harmful to the goal.