6 ms·
A reasonable configuration language
- AtlasBarfed 3y agoThe configuration rabbit hole, for practically any system that becomes widely applied or through large amounts of feature iterations. With provisioning systems you rapidly walk the ladder because they are basically starting about five steps down on this already: You start with an INI/yaml/json Then you might have several overlaid INIs Eventually: https://docs.spring.io/spring-boot/docs/2.1.13.RELEASE/reference/html/boot-features-external-config.html https://docs.spring.io/spring-boot/docs/2.1.13.RELEASE/refer... Go ahead and make a java crack or something similar. Every line of that hierarchy is bathed in the blood of IT workers in the real world. Wait! Not even close to being done. Templating, expansions, "classes", embedded expressions: now we have computation folks! I don't actually know the computational power of HCL, but eventually you end up with a Turing Complete configuration input that the poster wants. And I'm actually leaving a lot of things out. So many systems go through this, picking their own path. Kind of like workflow engines, almost all major systems will end up with a workflow engine in it (and the builds are workflow anyway). But there aren't well-conserved implementations of these overarching meta-patterns, so everyone bespokes it or picks from a mind numbingly vast array of pseudo-matching options. Good luck!
- dsheets 3y agoThe article did not advocate for a TC config language. I think Ruud would probably have been happy with a total primitive recursive language.
- ruuda 3y agoTo me the distinction is not important. It is possible to write programs that take too long to execute in non-Turing complete languages, and when a program hangs, you just terminate it. I think what people really want is for their configuration to remain simple, but not being Turing complete does not guarantee that. For RCL I added a limit on the number of evaluation steps, because without it some programs trap the fuzzer.
- dsheets 3y agoWe agree. You don’t want universal computation in your configuration. Your “gas” approach is the right one and very weak computationally. I further posit that you don’t even want to offer gas fuelled TC and everyone would be happy with a provably total primitive recursive language fuelled by gas. Why? Accidents happen. I’d rather have my configuration tool tell me statically that some construct isn’t provably terminating than wait to find out later when someone imports my config snippet library and uses it in an unexpected way.
- stevekemp 3y agoIt's scary to think how many times I've had to evolve configuration options as systems developed and the user-base changed. As you say, I'd start out with INI files, then move on to JSON, after that it would come a hack to allow the JSON to be generated by executing a shell/ruby/perl script. (i.e. "--config=!xx" would execute XX and parse the output, instead of reading a file). Later still we started embedding lua, or similar, to allow things to be templated and "dynamic" on a per-host basis. I was always fond of the Apache-style configuration system, and I guess HCL is close to that, but there aren't any great universal solutions unless you go all-in with scripting, and then you end up with emacs!
- qznc 3y agoI defined myself 5 levels: Level 1 is just values in a file. The Linux kernel uses that. Level 2 is a list of values, e.g. ini files. Level 3 allows nesting. JSON, XML, and YAML are here. Level 4 allows computation but limited. Dhall and Starlark are here. Level 5 is a Turing-complete language. Python, Javascript, etc. RCL seems to be level 5, so I'm not sure if there is really an advantage compared to Python.
- mst 3y agoI think the goal is to have a language that feels like it was designed for level 3/4 but allows you to break out to level 5 as an escape hatch. Given how many configuration languages eventually end up with a half-assed level 5 (Greenspun comes to mind) it strikes me as, at least, an experiment worth performing.
- gary_0 3y agoI wonder if anyone would take me seriously if I suggested WASM as a configuration language. When run, the WASM would be given imports that provide a small API for manipulating the config DOM, and importing and executing further WASM files. Then you can use whatever high-level config language you want (or, say, a subset of Rust) and compile it to WASM. Existing JSON/YAML/etc could be compiled to WASM that simply builds the corresponding DOM. Funny joke?
- coderedart 3y agousing wasm goes into crazy territory imo. It would probably be bigger, and opaque, with no proper errors (unless you compile debug info which would make it much bigger). I think we already have an almost perfect language i.e. Lua Everyone knows it (or can learn it easily). tiny runtime to embed. sandboxed by default. garbage collected. Only feature missing is static typing.
- viktree 3y agoI think this is what the zellij terminal multiplexer uses for extensions. It's worth mentioning that in the opposite direction, if you wanted to have an interpreted config option with fewer compile steps, there's also the option to use lua — a language for which the compiler itself is quite small and portable See, [1] : https://zellij.dev https://zellij.dev
- marcosdumay 3y agoHum, most systems do not go down that hole at all. Most software stop at most at hierarchical parameters. For the ones that do go down the hole, you are almost always better on separating them into components, to minimize the ones that need complex configuration, and accelerating the journey of those small pieces. (But yeah, it would be nice to settle on a good workflow description language that isn't a makefile.) Now, it seems that everybody that is pushing those high-complexity languages wants an infrastructure description language, and just keeps calling it by "configuration language". Since those two problems are completely different, insisting on using the wrong name is quite harmful to the goal.
- macintux 3y agoA reasonable configuration language blog post. Explains the motivation, provides some meaningful examples and a happy accident, and concludes with an overview of prominent alternatives. Worth a read, and worth trying out sometime, if nothing else because I too don't use jq often enough to remember more complex usages.
- raphinou 3y agojsonnet[1] and kapitan[2] are the tools I currently use. Their learning curve is not optimal (and I tried to contribute to smoothen it with a jsonnet course[3] and a 'get started wit kapitan' blog post[4]), but once used to it it's hard to do without, and their combination makes them even more useful (esp. if you deploy K8s). In Ruud's case, Jsonnet might have been worth looking at as Hashicorp tools can be configured with json in addition to HCL. But that would have been less fun I guess ;-) I hope for Ruud it finds its niche, there's quite some competition in this field! 1: https://jsonnet.org/ https://jsonnet.org/ 2: https://kapitan.dev/ https://kapitan.dev/ 3: referal link: https://www.udemy.com/course/jsonnet-from-scratch/?referralCode=EFA565D6914866D5FADE https://www.udemy.com/course/jsonnet-from-scratch/?referralC... 4: https://www.yvesdennels.com/posts/starting-with-kapitan/ https://www.yvesdennels.com/posts/starting-with-kapitan/
- js2 3y agoThe author mentions Jsonnet in the appendix[^1]: > I never properly evaluated Jsonnet, but probably I should. Superficially it looks like one of the more mature formats, and in many ways it looks similar to RCL. Its has a page comparing itself against other configuration languages. [^1]: https://ruudvanasseldonk.com/2024/a-reasonable-configuration-language#appendix-a-non-exhaustive-list-of-configuration-languages https://ruudvanasseldonk.com/2024/a-reasonable-configuration...
- raphinou 3y agoI should have made it clearer, but I wrote my comment in reaction to his mention that he "never properly evaluated Jsonnet".
- senorsmile 3y agoKapitan sounds interesting. It definitely looks difficult to figure out. Even your blog post only gives me a hint, leaving me with lots of questions. I am going to o dig deeper.
- alledm 3y ago
- nynx 3y agoThis looks very restrained. Well done!
- lolinder 3y agoReading through this I'm reminded of another article from about a month ago, An app can be a home-cooked meal [0]. Whenever someone starts on a new language or framework, people are quick to ask "why" and bemoan the proliferation of languages and frameworks. I think this article is a good illustration of the "why". What the author understands that so many don't is that a language can be a home-cooked meal. A programming language is nothing more or less than a tool for use by a programmer. Compilers have a lot of mystique about them because of all the crazy optimizations that production-grade compilers put in, but fundamentally a compiler is just a pipeline that transforms a data structure from a format that the human would prefer to interact with into a format that a specific machine can work with. rcl is a home-cooked meal kind of configuration language. It was never intended to serve a wide audience, it was intended to solve a specific human's pain points and help that specific human with their task. Its value lies in that it doesn't need to try to be anything else. [0] An app can be a home-cooked meal (1051 points, 288 comments) https://news.ycombinator.com/item?id=38877423 https://news.ycombinator.com/item?id=38877423
- mst 3y agoI'm working on something that may end up being rcl-like. My intention is to make it something that can update existing configuration files just as well as -be- a configuration file, such that if somebody finds it useful, they can use it without their colleagues needing to be aware and with the same commit diffs as if they'd done the same work by hand. How well that will work in practice is still an open question, though a bunch of my experiments in that direction seem to have worked out nicely so far. But so long as it helps -me- make changes to a repository that are good, so long as I can ensure nobody else working on that repository needs to care that I used it to help, I figure it's worth the attempt. (also, hey, I'm having fun :)
- stakhanov 3y ago> a language can be a home-cooked meal [...] a tool for use by a programmer [...] a pipeline that transforms a data structure from a format that the human would prefer to interact with into a format that a specific machine can work with. You're neglecting the second function that a language serves, which is as a communication device to communicate a program from one human being to another, and in that second function, a language is utterly useless if it lacks critical mass in terms of how many people "speak" it. This runs completely counter to the "home-cooked meal" model. You run into problems, even if you only have one human being trying to communicate the program to their future self, because there's a tendency to forget a language you don't use regularly, so you might cook up a very brilliant language to solve a problem that you have right now, but that you don't have sufficiently frequently. So you don't practice the language regularly, and then you come back 10 years later and want to change something about the program and you might find yourself in trouble.
- civilized 3y agoI have seen many of these posts now and I have a simple question. Why do we need special configuration languages? Why not just use existing languages? I understand why data formats like JSON and YAML are valuable, and I understand why it would be valuable to use a programming language to automate the generation of such formatted data. But the niche of the configuration language remains mysterious to me.
- raphinou 3y agoI'm guessing here, but maybe you see these configuration languages as templates? However a configuration language is much more than that. For example, jsonnet lets you combine objects and override deeply nested fields [1]. I'm not sure how practical it would be to achieve the same result with a general purpose language. 1: https://jsonnet.org/learning/tutorial.html#oo https://jsonnet.org/learning/tutorial.html#oo
- aleksiy123 3y agoI think because there are some desirable traits for config languages that don't exist in general purpose languages. Not exhaustive list but generally: Usually constrained to reduce complexity and try to eliminate the need for testing config. Interopable between many different programming languages. Readable by programmers working in different languages. I think this usually makes config languages favour declarative over imperative which usually eliminates most general purpose languages. Another topic is why do we use configuration at all and what is the difference between code and config.
- Someone 3y agoIt also is desirable that you can programmatically process configuration files (example: go through all your docker files to determine which ones use vulnerable images) That is at odds with using a generic Turing complete language, as figuring out what the configuration is requires running code that may do all kinds of weird things (download data, delete files, etc) (that is similar to the postscript/PDF case. To figure out how many pages a postscript file has, you have to run code) Now, some will say they trust those writing config files to not do stupid things and only use that power when it is absolutely required, but that’s not an opinion shared by all.
- hnlmorg 3y ago> It’s 2024, so RCL has some features that you might expect from a “modern” language: trailing commas I have the opposite opinion here. Commas should be like semi-colons: only required if you want multiple statements on a line. Eg fruit = [ “apples” “oranges” “bananas” ] No commas yet still extremely explicit
- mcphage 3y agoThat’s not really the opposite opinion… the opposite is not allowing trailing commas (like JSON) so if you remove the last item from a list, you need to adjust the line above as well. And when you add something to a list, you need to know if something will come after it, even though you don’t necessarily know. Commas as separator is pretty much orthogonal, and… well, better overall.
- hnlmorg 3y agoI got what they were saying. My point is that if you have a line feed then a comma is an entirely redundant piece of syntax. Commas should only be needed if you are placing multiple statements on a single line. Just like how several programming languages treat semi-colons (eg Go, bash, JavaScript, etc). So mandating them is the opposite of that modern languages should be insisting upon. In Murex (my own language) both commas and semi-colons can be dropped if the statement or expression is terminated by a line feed. list = %[ chair table “bed-side table” ] YAML, for all of its warts, also behaves similarly too. It’s a much better way to handle lists and objects because the comma only exists for the same reason the semi-colon does in C-like syntax: as a parser hint. But if you’ve got a new line then that hint is completely redundant.
- OJFord 3y agoIt's only redundant in languages that make it redundant, like you mentioned C at the end - it's the new line that's redundant there (or only for readability), it's the semicolon with the 'end' semantics.
- zer00eyz 3y agoThe problem isnt "configuration" Yaml, JSON, xml, text files... all work great for configuration. But the assumption is that your configuring a piece of software on an already existing system. A config file as means of setting up a system, installing software, and establishing how its going to run is exceedingly stupid. Write your app so it can run on bare metal, install with apt, yum or your tool(s) of choice for your org. Build it so it scales diagonally, and works against spot instances where you can (because sometimes more or less cores are cheaper). Dont even get me started on the nonsense of everyone and their ideas for "secrets management", it makes me miss LDAP.
- Kinrany 3y agoAre you suggesting that all configuration should be dynamic and multi-tenant (for the lack of a better word), not happen at startup?
- zer00eyz 3y agoScripting a config file. Templates for config. We did this already, xml, XSD, XSLT, DOM... It sucked. Config should be easy and flat and simple. IF it isnt that source it from code, or from a service... Most of the nonsense of config madness is a byproduct of, or hidden in, containers. If we were writing installable software much of that would go away...
- MrDarcy 3y agoWhat’s this magical config service do? Take data in from a request and return transformed data back in response to configure the caller? What is the essential difference between this config service and existing config languages?
- Jtsummers 3y ago> Write your app so it can run on bare metal, install with apt, yum or your tool(s) of choice for your org. If it's running on bare metal, why are you using apt or yum? Those are parts of an OS you wouldn't have if you're running your program without an OS.
- habitue 3y agoI know the standard response is "why another one?" but honestly, his diagnosis of the existing options is pretty good. He even uses the best examples of thoughtfully designed config languages: cue, dhall and nix, and ... yeah I pretty much agree with his qualms about those languages. What is a config language really? It's a language that doesn't allow side effects and evaluates to json.
- skybrian 3y agoSure, it doesn't allow side effects, but what can it depend on? If a file defines a language that downloads a dependency from a URL, does that count? How about if it's automatically cached? Once you start getting dependencies from others, it seems like a configuration language could drift into package management.
- ianburrell 3y agoDownloading anything has side effects and can't be allowed by side effect free language. It is pretty rare that config language allow downloads. Downloading dependencies isn't usually done by config languages. Config languages generate a config, JSON or YAML, from source. The config file can be used by engine that applies the config including downloading dependencies.
- skybrian 3y agoXML isn't rare and it has URL's baked in. But yes, it's a bad idea.
- anileated 3y agoThinking inside the program, downloading dependencies certainly doesn’t need to have side effects. It may add new failure modes, that’s all. If you step outside then sure, it has side effects, but so does simply running the program (any program), in 100% of the cases.
- GoblinSlayer 3y agoDo you really want github actions have no side effects?
- paulddraper 3y ago> I was struggling with that day was to define six cloud storage buckets in Terraform...The kind of thing you’d do with a two-line nested loop in any general-purpose language I understand this is just an example, but FYI the modern solution is to use CDKTF rather than HCL for Terraform. That allows you to choose your favorite general purpose lang: Python, TypeScript, Go, Java, C#.
- OJFord 3y agoI use Terraform because of HCL, the absolute best thing about it is the declarative config; if someone insists on using their favourite general purpose language: 1) they're wrong; 2) there's plenty of other options for that and I'm not interested in fighting in Terraform's corner knowing they won't use it for the most fundamental reason it's good.
- paulddraper 3y agoSo, your reply to Ruud is "you're holding it wrong" or "it actually fine you just think it isn't"?
- OJFord 3y agoNo, principally because although he writes: > Aside from Nix, Python, and HCL, which I’ve already discussed extensively, He doesn't seem to have much more to say about the latter than that. 'My reply', bluntly, is something like 'maybe it has shortcomings, you haven't addressed them; its most fundamental key advantage is not addressed at all, about it or any other language, and is not a feature of your new one'.
- rirze 3y agoYou do know all that CDKTF does is compile programming language code to JSON (1:1 with Terraform's HCL)? CDKTF is not accessing the Terraform state directly, it's allowing you to express your configuration that you would normally write in Terraform in a programatic way.
- deleted 3y ago[deleted]
- deathanatos 3y agoI quite like what has been come up with here. In particular, I understand exactly how HCL would drive someone down this path, as it is infuriating to try to get that language to compute what you want computed. I think at the end of the day, k8s YAML is focused on the datatype, i.e., what would be the result of your `rcl evaluate`. I do think this would be a much saner path than what Helm provides, though, and Helm's textual templating, as opposed to understanding values and building the actual datastructures, and then converting that to a format like JSON or YAML … is the wrong path. (I.e., I can see RCL being a possible replacement for Helm.) > The language is a superset of json. I'm going to introduce what I think is pretty much a universal law: languages claiming to be supersets of other languages are not supersets. In the case of JSON, there's basically one counter-example that breaks all alleged supersets: "\ud83d\udca9" And if we try it: » printf '"\\ud83d\\udca9"' | cargo r -- evaluate Finished dev [unoptimized + debuginfo] target(s) in 0.01s Running `target/debug/rcl evaluate` stdin:1:2 ╷ 1 │ "\ud83d\udca9" ╵ ^~~~~~ Error: Invalid escape sequence: not a Unicode scalar value. Help: For code points beyond U+FFFF, use '\u{...}' instead of a surrogate pair. Now! This to me is not a bug in RCL: this particular facet of JSON is utter crazy town, and I would strongly encourage you to not adopt it. (Since down this road lies madness, like unpaired surrogates. JSON's grammar & standard is sloppy here, and JSON/JS's syntax of using the UTF-16 encoding, and not just the scalar value … it's the part of JavaScript that is just what JS is, but is the part that we shouldn't be copying.) It is much saner to just have a flag/way to indicate "my input is JSON" and then to just parse it via an actual JSON parser. Then let RCL evolve on its own merits. If there's a lot of happy overlap, and most JSON documents are blissfully polyglots with RCL, that's fine too / a happy little accident. (But the option is important if you want to apply it somewhere programmatically, where the inputs are JSON.) YAML breaks the same way, too: it too is a "superset" of JSON that isn't.
- lolinder 3y agoHave you ever run into a case where this distinction matters in practice? If it's a superset of all JSON that actually exists in the wild, that feels functionally the same as being a superset of JSON, even if the spec technically allows edge cases it can't handle.
- 3y ago
- eclectic29 3y agoWow! Another configuration language post. 2nd day in a row. And that too on the front page. It's 2024 and the software industry has not solved the problem of configuration yet. Umpteen solutions every year - languages, formats, etc - all for just doing configuration. Just wow! It amazes me. It would be so much better if all the great minds focused on real issues than keep spinning on the hamster wheel of config file formats and languages.
- mst 3y agoFor templating/generating YAML it's worth a look at https://yglu.io/ https://yglu.io/ and ingy's latest piece of madness, https://yamlscript.org/ https://yamlscript.org/ I make no claim that anybody who does look will like either of them, but I do claim they're worth a look even if it turns out that you don't.
- notnmeyer 3y agoI feel like just reading about these took years off my life.
- heads 3y agoConfiguration languages are an interface to the API presented by the software you are using. There can be a hard boundary between the two — a C codebase for a web server and the XML-like file that configured it — or there can be an imperceptible transition between the two where the configuration is just another module in the code base. As one of the authors working in a 2000+ module Python codebase, nothing gives me greater pleasure than to drive one part of the codebase by creating another module. Nothing gives me greater sadness than to be forced to interact with something through a YAML config file, command line flags, or launching GitLab pipelines. All three of those boundaries interrupt the powerful electromagnetic force fields that pervade the system: type checking, linting, and symbol finding (IDE integration). In time, I vow to destroy every last one of these non-code boundaries. Another unwanted-boundary demon on the exorcism list is polyrepos. There once was a time when the world worked like this but instead of Python it was C. It wasn’t as rich as the dynamic world we have today — I’m certain I don’t want to go back to configuring my softest by recompiling it — but it did have a lot of the advantages of working in one language environment across all the things.
- somewhereoutth 3y agoThis, but with Javascript. Perhaps the 'electromagnetic force fields' are less powerful, but the array and object literals are perfect for configuration.
- sanderjd 3y agoDiff'rent strokes for diff'rent folks (and diff'rent use cases)... For me, the panacea is to be able to easily do both. I want to be able to use an application as a library from a repl or script by "configuring" it via dataclasses. And I want to be able to run the same application from a cli in different configurations in different production environments by checking in sets of config files that I know are more constrained in what they can do than a generic python runtime. There's no fundamental reason it is hard to expose both of these interfaces without repeating yourself a bunch!
- et1337 3y agoInteresting, I’m in the middle of a huge project to move us AWAY from compiling a separate binary for each environment. In our case, I don’t see how hardcoding config values can play nice with infrastructure as code tools, or the need to offer an on-premises version of the app.
- frizlab 3y agoI’m curious to know what this guy would think about pkl.
- deepsun 3y ago> The kind of thing you’d do with a two-line nested loop ... > that is was far simpler to just copy-paste the config six times. And that's the right way for infrastructure and build configs. You do want over-verbose, no-logic code in the configs. Unlike app code, you rarely need to change it, and readability is even more important. 6 duplicates is not too many yet for infra code. In terms of copy-paste tolerance, I'd rank from least to most: app code, tests code, infra configs, and build pipeline configs.
- acjohnson55 3y agoI agree with you in a lot of situations but I think there are situations where the intention you want to communicate is "these 6 situations are the same, except for this variation, because X". If you duplicate, it can be hard to capture in a way that is as robust. And then you risk logical errors in your configuration, which can be very damaging.
- paholg 3y agoAnd yet, sometimes you do want to reuse code in infra configs and build pipelines. I have definitely wanted to reuse code in hcl, to the point that I wrote a Ruby script to generate it because that was easier than trying to do it in hcl. Clearly I'm not the only one. Gitlab even sees the use enough to support syntax that is invalid yaml: https://docs.gitlab.com/ee/ci/yaml/yaml_optimization.html#reference-tags https://docs.gitlab.com/ee/ci/yaml/yaml_optimization.html#re... This, of course, causes yaml-language-server to complain. What I wouldn't give to have a language where I could, say "goto definition" for things like this rather than get LSP errors.
- ljaothasdfs 3y agoGreat start! I'd like this even more if you dropped the python-style f-strings for a JavaScript or HCL flavor instead. You noted at the bottom that RCL type system may take the path of TypeScript. This is brilliant idea because structural type systems match perfectly with a configuration language.
- timka 3y agoIMO Nickel looks better in this regard as it allows you to use typing only when it helps. Plus, it has contracts that complement the type system.
- vivzkestrel 3y agohow is this different from the other one posted yesterday https://news.ycombinator.com/item?id=39232976 https://news.ycombinator.com/item?id=39232976
- AndyKluger 3y agoI want to showcase one way to do the example jq/non-jq query using yamlpath: yaml-get -p '.tags[has_child(amd)][parent()].name' machines.json That can replace rcl query --output=raw machines.json '[ for m in input: if m.get("tags", []).contains("amd"): m.name ]'
- PinkSheep 3y agoAlthough I'm late to the party, I'm surprised by only 4 mentions of "Lua" in all of comments. At its basis, a valid Lua file can consist of one table. Then the data description will be very similar to how people operate with JSON files. At any point the user is able to write code or use other standard library functions... which is pretty limited, but at the same time it's easy to sandbox plain Lua away from "dangerous" functions such as os.exec or the debug library. There are only a few caveats: - Lua uses a 64-bit IEEE float format with some truncated precision - The standard interpreter doesn't handle very huge tables for parsing - Treating .lua as code, not data, you'd likely prepend "return " to your table to just receive the table object from the config file - Unkeyed arrays start from 1, though storing at index 0 is possible - Cascading of data definitions is possible, like fall-through from one semi-filled table to the default table. Use the __index metamethod Personally I find Lua's table syntax vastly more readable than JSON. Although the language naturally has comments, the parser ignores them, if that's something you want. But the same workaround works: just store the comment in some keyname_comment value. Apart from sandboxing you'd only need to limit the script execution time to stop attempts like "while true; end" PS: Lua started as a configuration language. Wiki: > Lua's predecessors were the data-description/configuration languages SOL (Simple Object Language) and DEL (data-entry language).