21 ms·

Configs suck? Try a real programming language

• q3k 7y ago
  As the article mentions, there's 'real' programming languages made for configuration, that solve at least some of the issues outlined, like Dhall, Cue, Jsonnet. After using both approaches a number of times (a general purpose language vs. one of the three), I'm imploring anyone trying to give this a shot to _not_ use a general purpose language. For instance: I'm mostly familiar with Jsonnet, which has guarantees that make it much easier to use than a Real Language: no arbitrary file loads (paths must be static), no side effects, no ambient state (like env vars or Internet). Relative imports make it very easy to drop in the language somewhere in a repo and not have to worry about venvs/PYTHONPATH/system libraries... The interpreter being a single binary and a thin C library also simplifies integration quite a bit, compared to interfacing with full-blown interpreter/compiler installations. The downside is, of course, that you must learn a new language. But we're all pretty good at this (unless you're one of those people obsessed with using JS/npm everywhere for some inexplicable reason). Also Jsonnet still has the pedigree of being a superset of JSON, which is painful (ie. magical int/float number type). But the alternatives (Dhall, Cue) fix that, and also provide a vastly better type system. I've been using Jsonnet in production for a couple of years (after leaving Google where I Saw The Light with BCL), and couldn't be happier. The infrastructure of the Warsaw Hackerspace (a production k8s cluster) is all brought up using Nix and Jsonnet and is open for all to inspect [1]. And yes. Let's please stop using JSON and YAML for everything. Even Python is better than yet another YAML templating tool. [1] - https://gerrit.hackerspace.pl/plugins/gitiles/hscloud/+/refs/heads/master/cluster https://gerrit.hackerspace.pl/plugins/gitiles/hscloud/+/refs...
  • lvncelot 7y ago
    That is really cool - as chance would have it, I've just spent this weekend converting all my personal infra stuff to Jsonnet and I'm having a blast! This is some really nice inspiration.
  • desert_boi 7y ago
    BCL is something that's new to me. Anyone know what it stands for?
    • q3k 7y ago
      Imagine Jsonnet, but internal to Google, old and quirky. It's not public, but the name has been leaked for a while now. Edit: here's a thesis that works with a related language, GCL, with code samples: https://pure.tue.nl/ws/portalfiles/portal/46927079/638953-1.pdf https://pure.tue.nl/ws/portalfiles/portal/46927079/638953-1....
      • oddthink 7y ago
        "Old and quirky" is an understatement. It's a god-awful mess, with semantics so murky that people end up just copying patterns and hoping that it'll do what they want it to do. And it can't be killed, because every bit of those implementation-defined semantics are used by someone.
        • q3k 7y ago
          up.up.up.up
    • V-eHGsd_ 7y ago
      borg config language, IIRC.
    • fooker 7y ago
      Borg Control Language
    • jbotz 7y ago
      Borg Control Language, as others have said, Google's configuration language for "Borg", the precursor of Kubernetes. Both Jsonnet and Cue are descendants of BCL, developed by people who were previously involved in BCL development. Jsonnet is a direct descendant of BCL, "BCL done right". Cue is an indirect descendant of BCL, you might say "BCL rethought".
  • octopoc 7y ago
    Very interesting, thanks for the links! I have a different approach. When I'm deciding about a new technology that we could add to our tech stack, I ask myself if it's really worth it. Adding a new tech is very expensive because it adds to the list of things that the developers have to be good at. I would avoid adding a configuration language and prefer using the same general purpose language that the rest of the software is written in. At work, we have a C# ASP.NET Core application we build. - We use Pulumi's C# API to deploy to the cloud. This lets us avoid yet another YAML based config format, and it lets us edit the deployment in a familiar language with a pretty simple API in our favorite IDE. (Link: https://www.pulumi.com/ https://www.pulumi.com/) - We're going to move to Nuke instead of having Azure Pipelines YAML files. One less YAML file format to remember. (Link: https://www.nuke.build/ https://www.nuke.build/) - We're going to do benchmarks with BenchmarkDotNet, to keep our code in C#. (Link: https://benchmarkdotnet.org/ https://benchmarkdotnet.org/) - We're going to be adding Spark-based analytics, and I'm undecided about whether we should use Python or the C#. Python is so ubiquitous in analytics and there are so many libraries that it might be worth adding another language to our system. (Link: https://dotnet.microsoft.com/apps/data/spark https://dotnet.microsoft.com/apps/data/spark)
    • q3k 7y ago
      Yes. I don't like that it a) uses imperative, general purpose programming languages b) requires keeping an external statefile for deploying prod. EDIT: parent originally asked 'we use Pulumi, have you heard of it?'
      • octopoc 7y ago
        Yeah that is a pretty big downside to Pulumi. That's something it shares with Terraform.
        • lima 7y ago
          Large parts of Pulumi actually use Terraform providers.
      • reilly3000 7y ago
  • malkia 7y ago
    Same here, but stuck with json/yaml, etc. What are you thoughts on starlark for configuration language? Imperative, but with lots of functional/declarative bits - like frozen vars, no recursion, sandboxing, execution in parallel... I've also looked at jsonnet, even contributed one or two patches to speed it up in C++, but still felt the evaluation time was higher (the "go" version is faster than C++, but they (last I've checked) used slightly different evaluation rules, also C++ would get slower in bigger app due to the overuse of dynamic_cast (at least that's what I've found)). Yeah, I'm also trying to find the magical bcl/gcl - but what was so useful (when I worked at google) was the diff tool (was it written by an intern, it was amazingly useful) - e.g. your presubmit would tell you - how the end (evaluated) result would look like, as one change in one place might have quite the ripple effect.
    • q3k 7y ago
      > What are you thoughts on starlark for configuration language? I haven't tried that yet. It might work. I mean, I've used starlark as _a_ configuration language (notably, in Bazel, or as a little prolog-standin for defining complex rules), but never really built an entire infrastructure on top of it. I would have to look into how it works my usecase (can't lie that jsonnet having an existing third-party kubernetes library was a big seller for me, even if I end up extending it). I do prefer, however, functional/lazily evaluated/declarative languages for configuration - while starlark has a very imperative taste to it. > [...] but still felt the evaluation time was higher [...] Yeah, that can be painful, but has not fully bitten me - my configs rarely get large enough to notice, and even if they do there is a large bottleneck somewhere else (ie. actually applying the config). Definitely also have it on my backlog to look into sending some patches for that. > was the diff tool There exists something like this, kind of, for when you're using jsonnet to emit k8s resources: `kubecfg diff`. But that's for diffing against prod, not an evaluation against a previous VCS tip. Definitely a tool that should exist, though :). I do think Dhall is a bit further down the road of having this sort of tooling actually exist upstream.
    • lima 7y ago
      skycfg is doing this: https://skycfg.fun https://skycfg.fun I'm not convinced - the ergonomics aren't great, it does not support CRDs, is mostly untyped, and you end up unnecessarily encapsulating everything with dozens of function parameters since you cannot easily merge nested data. Starlark is great for configuring an imperative tool like Bazel which is only declarative at the top level, but is impractical for pure configuration data. Having the full power of a general purpose language isn't great for configuration - it quickly leads to lots of complexity at the wrong layer. Cue has a radically different approach[1] that works very well in practice. The author of Cue, mplv, wrote the first version of bcl/gcl and Cue is the spiritual successor of sorts :-) It's still early, but many people (including me) already use it in production. Their Slack is very friendly and helpful! [1]: https://cuelang.org/docs/usecases/configuration/ https://cuelang.org/docs/usecases/configuration/
  • karlicoss 7y ago
    (author here). Thanks for the links, I'll add them!
  • int_19h 7y ago
    > no arbitrary file loads (paths must be static), no side effects, no ambient state (like env vars or Internet) This all sounds like it should be a part of the sandbox, but it's not clear why it actually requires a whole new language. Many existing runtimes can be sandboxed in a similar manner. So why not just sandbox e.g. JS for JSON?
• parhamn 7y ago
  There are cool things somewhere in the middle now too, e.g Cue [1], which is really cool. I was recently thinking that its a shame there isn't a cross language proto-lite kind of thing (I'd settle for json with structures and pointers) that has a great devx. The multi step transform that configs are right now (from the serialization representation to the code one) could be eliminated. IMO interop is one the most important things for configs, so I'd be hesitant to use a thing that needs a preinstalled runtime. [1] https://cuelang.org/ https://cuelang.org/
• throw_m239339 7y ago
  Ctrl+F search XML, no result. I know the majority of developers hate XML config files, yet... I try to make my apps depend on config files and manifests as little as possible, but DSL vs fully featured scripting languages has always been a big conundrum in software development. Is python a solution? the problem is that at some point you might be tempted to add json/yaml config files to your python config scripts...
  • toolslive 7y ago
    about 5 minutes after you start using xml for configuration, you want to reference a bit of global information (environment variable, port, ....) and add variable substitution ${foo} as an extra feature.
• klysm 7y ago
  Glad to see Dhall (https://dhall-lang.org/ https://dhall-lang.org/) mentioned here. As we seem to be heading towards immutable-'everything' though, sometimes I wonder how valuable dynamic (in the sense of picked up at start) configuration is for backend services. It seems preferable to build everything, config included, into one single artifact. Potentially this gives more optimization opportunities as a lot branches could be completely eliminated (feature flags, etc.). No doubt the simplest way to build your config directly into your code is to just.. have it be code. A potentially super annoying trade-off is that it's difficult to test minor configuration tweaks, and the annoyingness scales non-linearly with your build time.
  • q3k 7y ago
    And even more annoying if you would rather not restart your services unless absolutely necessary (ie. you're handling long-lived TCP connections that cannot be handed off gracefully). Not to mention that some kind of configuration tweaks (ie. ACL changes, quota changes, etc) might happen at a vastly different cadence than software rollouts.
    • jchw 7y ago
      > if you would rather not restart your services unless absolutely necessary (ie. you're handling long-lived TCP connections that cannot be handed off gracefully). If you have no way to hand off your TCP connections gracefully, that is probably not a great position to be in because computers are unreliable and sometimes you need security updates. One approach I tried experimentally for a TCP server (but unfortunately did not have the opportunity to try in production) was to have a small server acting as the frontend that handled sessions and called into the real implementation which was essentially stateless. This small server could theoretically support self updating without dropping connections, if need be. (The main reason I was doing this was so that the rest of the stack could be dynamically-scaling based on Kubernetes, while this part remained relatively static.) I think anytime you have forced statefulness, it’s worth isolating as much as possible so it doesn’t constrain the design of the whole system. (Even better, of course, is to just have a protocol where clients can gracefully handle draining and reconnect without side effects.) > ACL changes, quota changes, etc At a certain level I think these are runtime data and not configs. Aforementioned service also dealt with ACLs and configs and we were serving it through another service that persisted to a database layer. One cool thing about doing it as a service is that you can manage and scale it like any other service.
• conroy 7y ago
  Bazel uses a subset of Python for describing build rules Bazel's configuration language is called Starlark[0] (used to be Skylark). It's not a strict subset of Python. It has implementations in Java[1] and Go[2]. I haven't had a chance to use it yet, but it seems very useful as a general-purpose scripting language, especially for embedding into tools. [0] https://docs.bazel.build/versions/master/skylark/language.html https://docs.bazel.build/versions/master/skylark/language.ht... [1] https://github.com/bazelbuild/bazel/tree/master/src/main/java/com/google/devtools/starlark https://github.com/bazelbuild/bazel/tree/master/src/main/jav... [2] https://github.com/google/starlark-go https://github.com/google/starlark-go
  • busterarm 7y ago
    Bazel is convenient but also a full-time pain in my ass.
    • q3k 7y ago
      As someone already put it, 'Bazel is the worst system except all the others'. It has some warts (Python integration, non-hermetic-by-default rules, starlark structure javaesque complexity ...), but I still haven't far a good replacement for my use: a monorepo build system that has to touch a ton of different languages and can be run in CI without having to prebuild tons of fragile builder images.
    • simfoo 7y ago
      It absolutely is, especially if you do C++ with it. So many bugs and missing features in the native rules and skylark api, it's not even close to a 1.0
  • closeparen 7y ago
    I've had great success with the Go Starlark interpreter. Very easy to get started with. I map some of my internal structs to its dictionary type and let CLI callers filter my collections by providing predicates as Starlark expressions. The mapping to and from Go types is a bit obtuse and could probably be smoothed over by reflection or an intermediate representation like JSON in many cases. It's not just data though: you can also send functions in both directions. Embedding an interpreter for filtering vs. trying to compile all possible filtering needs into the binary greatly improved my program's power and shrunk the volume of application code. I recommend it. Don't be alarmed by the Bazel connection: it's much easier to use and understand (and better documented) than Bazel itself. Although I am a lot better at Bazel now for having taken the time to understand Starlark. All that said: if you find yourself needing to support logic and behavior in your configuration language, you may not actually have a good enough abstraction to justify a config-driven system. Take a hard look at whether it might be cleaner to write plain old code, or APIs for other people's code. A significant number of internal platforms at my company are unnecessarily obtuse because they try to put everything in boxes like this vs. being legitimately composable/programmable through ordinary software engineering tools and processes.
• zer 7y ago
  I use schemy [0] for non trivial config tasks in .NET projects. LISPs in general are well suited for this. Very small implementations are easily done and data is code, it’s pretty great. Of course, if one has already access to a Python runtime that is probably fine as well. [0] https://github.com/microsoft/schemy https://github.com/microsoft/schemy
  • Koshkin 7y ago
    Lisps are great as embedded scripting and configuration languages! Lisp is XML done right.
• The_rationalist 7y ago
  Gradle should be cited: It allow to configure build/automation rules with either groovy or Kotlin script
  • karlicoss 7y ago
    author here! Thanks, I've actually had it in my prompts but forgot to add. Will amend!
• stared 7y ago
  I cannot disagree more. Configs are wonderful, as they follow the "rule of least power" with putting settings in a place without logic operations, variables, etc. Yes, they are not general. Still, having one dedicated place for all constant switches is much better than constants being out in a dozen (or hundreds) of places all over code.
  • q3k 7y ago
    But that's the point of this. You can have a single high-level config that emits the same high-level settings onto multiple config 'facades'. For instance, a single jsonnet/python program will let you tweak a list of domains, and that will in turn create a facade (or generate a file) for nginx (to configure vhosts) and for a letsencrypt tool (to acquire TLS certs). Less repeating across files, less chance of making a mistake, and since this is a programming language, you can have a single codebase that performs facade generation, without having to go through a CM system.
    • throw_m239339 7y ago
      But then your config files in Python also need tests. Are they still a config file then or config scripts? How do you limit what they can and can do? Easy to do with JSON or XML, harder with Python.
      • q3k 7y ago
        This is why I'm not a huge fan of using Python for this (see my other comment) - and would much rather recommend Dhall/Cue/Jsonnet. And you can (and definitely should) write tests for these, or even just assertions for generated facades. It's much better than the alternative, which is just testing in production/qa/canary.
        • jevgeni 7y ago
          I cannot imagine how handing over a code base with configs that themselves require testing can be smooth.
• SomeHacker44 7y ago
  Clojure and Common Lisp (among other Lisps, I'm sure) are great at handling programmable configuration files. I do not think I would use this author's suggestion though.
• lokar 7y ago
  I have long argued that teams should write configs in one of the languages they already know and are using in the system. This has many obvious benefits, including testing! A key aspect is for the owner of the "config system" (the stuff that takes the output of each config program and applies it) to standardize the API for config generators. Inputs, outputs, runtime env, etc. Then let teams integrate with that API however they want.
• 11235813213455 7y ago
  What about jsonc or JSON5?
  • Snelius 7y ago
    Looks like author does not know what is it :)
• zevv 7y ago
  This is how Lua got started - its predecessors started out as data description or configuration languages. The users felt it would be useful to have some forms of flow control, which led to the birth of Lua. The whole story can be found here: https://www.lua.org/history.html https://www.lua.org/history.html
  • Rochus 7y ago
    It's still very suitable for configurations. The statements can be used e.g. to configure rules.
  • stevekemp 7y ago
    Indeed, I wrote a console-based mail-client inspired by mutt but with Lua for the configuration file. The Lua code made it easy to configure things such as the path to sendmail, and your email address. But also writing functions to filter, colour, and display messages was a joy. mutt is awesome, but the configuration language it uses is a mess of history. No loops and conditionals being somewhat limiting. People end up writing "profiles" by essentially reading the configuration values from a different init-file. Horrid.
• jevgeni 7y ago
  Under no circumstance would I allow my projects to be configured by a full-blown programming language. There is no way to ensure that there are no regressions under different configurations. The amount of complexity introduced just for configuration is insane. If you feel the need to put code in your runtime config, that means your design sucks.
  • q3k 7y ago
    But configuration is a complex task! At least from my experience. Limiting yourself to solving it with mediocre tools (for instance language that disallow writing tests or even comments!) is a footgun, where you disregard good engineering tools for stringly types, duplicated configuration.
    • jevgeni 7y ago
      We might have different target audiences of our work, but in my line configs should not require tests. Because their task is to adjust the functioning of an application. If they all of a sudden require tests, that means (at least to me) that it's not part of the configuration anymore, but a part of the application proper and should be treated as such.
      • q3k 7y ago
        For me the target is mostly verbose YAML configuration of existing software (which I can now emit from an in-memory object instead of templating and having to remember all the YAML quirks) and k8s manifests. Especially when I have to stamp the same piece of software multiple times in multiple deployments (environments/regions/etc). For instance: I just brought up a new Factorio game server yesterday, just by adding a single line to my jsonnet configuration machinery: https://gerrit.hackerspace.pl/c/hscloud/+/246 https://gerrit.hackerspace.pl/c/hscloud/+/246 . This in turn brought up a bunch of new k8s resources, including some persistent volumes, a deployment, and a loadbalancer service, after running a single `kubecfg update` command.
        • 7y ago
• cft 7y ago
  Many real programming languages are compiled. Does it mean I have to recompile the binary every time I change config?
  • jbreckmckye 7y ago
    Many such 'real' languages have support for some kind of interpreted mode. For example if you set a path to Stack as a script shebang, you can write interpreted-mode shell scripts in Haskell.
    • cft 7y ago
      What is the support for the "interpreted mode" in C, C++ or Rust?
  • cozzyd 7y ago
    dlopen? :)
  • sparkie 7y ago
    The author mentions XMonad, which does precisely that. However, you can update XMonad without "restarting" the program. When you update the config, which is just a haskell program - the config is recompiled and launched. The state of your current session is then passed over to the new config process, and the old one config process is discarded.
• z3t4 7y ago
  People often forget that formats such as JSON and XML is not meant to be hand written. It's a serialization format. It's meant to be parsed, while still human readable. It's very nice to have them human readable to make it easier to debug, and make quick changes. But you should probably use a tool to edit them.
• cryptica 7y ago
  I think not containing any logic is a useful feature for config files. Logic adds complexity and potential for bugs and vulnerabilities - I actually like that JSON forces you to use simple primitives as config variables. I hate it when config files start containing function definitions; you can't pass those around to different processes. I'm generally against the idea of complex configs. Configs should be simple. Sometimes that requires extra planning or thinking from developers. Every time I've allowed logic in config files, I've regretted it. The main annoyance about JSON is lack of commenting ability. This is a point I agree with but not a big deal IMO, it's still better than all other alternatives. JSON is ideal in terms of being both machine-readable and human-readable. Some people would argue that YAML is more human-readable, but it's definitely less machine-readable. With YAML, there are too many situations where some random code somewhere will remove all the new lines and tabs (for whatever reason) and mess everything up. JSON is resilient to machine sanitization. JSON is simple, robust and readable.
• cozzyd 7y ago
  I occasionally do this for python-only projects or for Makefiles, but (especially in the latter case) it's quite fragile. I usually use libconfig for configuration, which is better than json and yaml and has implementations in many languages (although I'm not aware of any javascript implementation).
• jillesvangurp 7y ago
  These days, using a statically typed language for configuration makes a lot of sense. You get stronger typing and with type inference not necessarily a lot more verbosity. Kotlin and kotlin script seems to work reasonably well for Gradle. Gradle unfortunately has a bit of groovy legacy, which makes it a bit convoluted for some things. But Kotlin DSLs are quite nice for configuring things and it removes a lot of ambiguity when your editor can tell you a property doesn't exist or that a list instead of a string is expected. or help you autocomplete things that are allowed. Basically the ultimate in configuration languages is something that has a minimal syntax but provides enough structure for tooling to help you validate things, provide autocomplete and other help, etc. There are now several languages emerging that have both static typing and some level of support for internal DSLs. Ruby sort of pioneered a lot of configuration DSLs back in the day but since it was dynamically typed, tool support for these DSLs was a lot harder. With languages like Kotlin, Swift, C#, Rust, etc. you get enough expressiveness that you can do similar internal DSLs but with the advantage of more robust tooling support.
  • mr_tristan 7y ago
    This is a good example of a structured approach, with the caveat that there are multiple phases. With gradle, there's three big phases: initialization, configuration, then execution. (See https://docs.gradle.org/current/userguide/build_lifecycle.html#sec:build_phases https://docs.gradle.org/current/userguide/build_lifecycle.ht...). I'd think you'd need to do something similar with any structured configuration DSL - have distinct phases where you may "init, plan, apply", etc. This, it's a pretty complex problem to have. But that complexity may be worth it as part of a "developer productivity engineering" effort, where you're really looking to use data to minimize the time between a configuration change, and the time it's applied in a runtime environment. Doing that is a hard problem.
  • involans 7y ago
    Agreed, but the best combination is type-safe and non-turing complete, ideally with restricted effects. [Dhall](https://github.com/dhall-lang/dhall-lang https://github.com/dhall-lang/dhall-lang) gets this so right that I'm increasingly shocked that people never seem to have heard of it.
• option 7y ago
  I came to a very similar conclusion with a big IF. If your project isn't concerned about security, then yes, Python is a way to go.
• adamnemecek 7y ago
  There was a time when I thought you want a declarative language for your build system. After having used Rust, which allows you to use Rust in your build scripts I think that that is the way forward. You can say generate game assets in your build script, access network, use third party crates etc etc. Declarative languages are a work around for bad APIs.
• marpstar 7y ago
  I did this with a server-side TypeScript project I built last year. The biggest advantage I noticed (and it's mentioned in the article): validation (i.e. type-safety). Specifically, the project uses TypeORM and it was nice to be able to import the types for the DB connection options to make sure I had all of the necessary properties. I was using ts-node to run server-side without compilation, which made importing an `app.config.ts` file quite elegant, IMO.
• ryukafalz 7y ago
  One of my favorite projects that does this is Guix, which uses Guile for its configuration. See this page for an example: https://guix.gnu.org/manual/en/html_node/Using-the-Configuration-System.html https://guix.gnu.org/manual/en/html_node/Using-the-Configura... It's a nice balance, I think - most of the time you can treat it as a normal config file, but if you need to drop into some programmatic stuff you can. I've done this in my own dotfiles to add custom package definitions before contributing them upstream, for example: https://github.com/jfrederickson/dotfiles/blob/master/guix/guix/manifest.scm https://github.com/jfrederickson/dotfiles/blob/master/guix/g...
• zaphar 7y ago
  Using json or yaml for your languages config is nice for the application but it's not great for for the developer especially if they want to reuse values or settings across multiple applications. The solution then is to use a real programming language to generate that config. This is why languages like Dhall, JSonnet, and Cuelang exist. It's why I built my own toy language UCG to explore the space. But you could just as easily use python, javascript, or some other language too.
  • Koshkin 7y ago
    > json or yaml... Dhall, JSonnet, and Cuelang... UCG... python, javascript God, what are we being punished for?
    • imtringued 7y ago
      If you think that's punishment then you haven't suffered enough. I've seen enough custom written config formats in my life that choosing a widely supported format like YAML or JSON is always the right choice. YAML may have warts but you can learn them in 10 minutes and you're done. Those custom configuration formats don't even have a name or documented syntax.
• thesuperbigfrog 7y ago
  Not a popular answer, but XML is mature, widely supported by many programming languages, and addresses all of the frustration criteria mentioned in the article: - doesn't have comments XML uses the same <!-- stuff here --> SGML-style comments that HTML does - bits of configs can't be reused XML can include other XML via XInclude: https://www.w3.org/TR/xinclude/ https://www.w3.org/TR/xinclude/ - can't contain any logic You can technically put logic in XML using XSLT (https://www.w3.org/TR/2017/REC-xslt-30-20170608/ https://www.w3.org/TR/2017/REC-xslt-30-20170608/), but I would ask if one SHOULD put logic into your configuration. I personally would recommend against it if at all possible. Keep configuration as declarative as possible and put the logic in your program. It's better to have a configuration logic module or use dependency injection than put logic in your config. - Programming language constructs are reinvented from scratch Again, XSLT could be used, but should your configuration really need to be Turing-complete? Granted, XML is complicated and not without criticisms, but it is mature, standardized, and well-supported. Python has great XML support: https://docs.python.org/3/library/xml.html https://docs.python.org/3/library/xml.html - can't be validated XML can be validated against XML Schema (https://www.w3.org/XML/Schema https://www.w3.org/XML/Schema), Relax NG (https://relaxng.org/ https://relaxng.org/), and possibly other standards. - implicit conversions and portability issues XML can be used portably between different platforms and programming languages depending on how the data is represented. XML Schema defines a way to describe data types that is flexible and interoperabile: https://www.w3.org/TR/xmlschema-2/ https://www.w3.org/TR/xmlschema-2/ Again, I know it's not the popular answer, but I was surprised that it was not even mentioned in the article.
  • q3k 7y ago
    The main reason I'm not using XML seriously is that it only really has a single native, scalar type: string. Sure, you can schema validate away a large part of this problem, but it's still somewhat of a wart.
  • dirtydroog 7y ago
    XML needs to die
  • mdaniel 7y ago
    Merely for your consideration, I suspect a lot of people have opposition to XML because it is noisy when edited by a human: <httpd> <baseDir>/opt/server</baseDir> </httpd> Of that snippet, "</baseDir>" and "</httpd>" are characters of my life that I will never get back, since they're just parser niceities and not _value_ Moving up the enlightenment chain: {"httpd": {"baseDir": "/opt/server"}} or its brace-and-quoteless friend: httpd: baseDir: /opt/server place a lot more emphasis on the payload and less on the packaging
• SanchoPanda 7y ago
  A large number of configs are created/used by people that don't know any programming languages, it seems rough to exclude all those people.
  • q3k 7y ago
    It seems worse to me to treat them like they can't learn a programming language.
    • SanchoPanda 7y ago
      Not forcing them to is different from assuming they can't.
      • DonHopkins 7y ago
        Not forcing them to learn a programming language, but instead forcing them to learn a half-assed quirky ham-strung config language that's just as complex as a programming language's syntax, and as hard to learn as a programming language, but isn't useful for anything else, or widely supported by any other tools, or well documented, or well designed, certainly isn't doing them any favors, or respecting their time and effort. Case in point, PHP's "Smarty" templating engine. Because lord forbid people learn to use PHP as a templating language (which is ALREADY IS), when instead they can learn to use an ad hoc, informally-specified, bug-ridden, slow implementation of half of PHP, which is implemented in PHP, and dynamically generates PHP code. https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule >Greenspun's tenth rule of programming is an aphorism in computer programming and especially programming language circles that states: >Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp. Your templating engine sucks and everything you have ever written is spaghetti code (yes, you) http://www.workingsoftware.com.au/page/Your_templating_engine_sucks_and_everything_you_have_ever_written_is_spaghetti_code_yes_you http://www.workingsoftware.com.au/page/Your_templating_engin... >This is where folks will start to get technical about why their code isn't spaghetti code. >Oh no! Not me because I use (Django|Rails|Haskell|Node.js|Symfony|CakePHP|Tornado|Plone) and the superior templating language (Django Template|ERB|Mustache|PHP|TAL|Smarty) doesn't allow business logic in templates and enforces a separation of concerns >-You, the reader, December 2011 >The problem is that in every single one of these circumstances, the templating engine has 2 primary responsibilities: >The fragmentification and inclusion of common interface assets (to avoid duplication and maintaining the same code in multiple files) - eg. headers, footers, sidebars and the like >The output of dynamic content >The output of dynamic content requires logic (if/else, loops being about the bare minimum). >The creators of templating languages have, to varying degress, availed themselves of a false dichotomy, namely that there exists "presentation" logic and "business" logic, the former of which belongs in templates and the latter of which has absolutely no place in templates and must be banished immediately. >I say false dichotomy because this is an arbitrary separation - a choice made by the person who created the templating engine and is completely subjective. Yet another reason smarty sucks https://www.sitepoint.com/community/t/yet-another-reason-smarty-sucks/6982 https://www.sitepoint.com/community/t/yet-another-reason-sma... >Which brings me right back to my chief criticism of smarty which to this day it’s adherents can’t answer – what the Hell is the point? [...] >I hated smarty from the day I tried it and I was always wondering why would anyone use it since you’re forced to learn some weird syntax that’s so similar to php - so why not use php from the start? https://news.ycombinator.com/item?id=4605887 https://news.ycombinator.com/item?id=4605887 >PHP is itself a template engine. Bolting a template engine on top of that makes no sense whatsoever. https://news.ycombinator.com/item?id=20736493 https://news.ycombinator.com/item?id=20736493 >+1 for Turing complete programming languages instead of half-assed config languages. https://news.ycombinator.com/item?id=20735231 https://news.ycombinator.com/item?id=20735231 >I was suspicious of YAML from day one, when they announced "Yet Another Markup Language (YAML) 1.0", because it obviously WASN'T a markup language. Who did they think they were fooling?
• AlphaSite 7y ago
  How about something like HCL?
• WalterBright 7y ago
  This reminds me of makefiles. Most programmers wouldn't dream of checking in code with no comments. But makefiles with byzantine code in it, no comments whatsoever, nothing even saying what the makefile is for, are routine. The makefiles degenerate into such an awful mess that then people look for alternatives to make. The sensible solution is to treat a makefile like code - document it and use sensible coding practices like organizing it instead of writing spaghetti soup.
  • Koshkin 7y ago
    Indeed; I've always wondered what is it that's keeping us from having a saner alternative to make in the form of a simple yet more powerful interpreted language with all needed procedures and functions built in.
    • smilliken 7y ago
      You're describing Nix: https://nixos.org/nix/ https://nixos.org/nix/ It fills the same niche as make, but is a pure functional programming language, reproducible, and has a large package repository. It's pretty much my favorite piece of software because it allows me to depend on third party packages with minimal risk of breaking the build.
• strictnein 7y ago
  Still don't get the hang up about comments with JSON. I'd find it strange to encounter a config file with lots of internal documentation. They should be small with well named, descriptive settings. These are config files. A comment is something that gets ignored, this "comment" will be ignored by whatever ingests this JSON. If not, there are other problems. If you have to write a treatise here, there are other problems. { "comment": "This is a comment", "dev": { ... }, "stage": { ... }, "prod": { ... } } Oh no, that line doesn't start with a # or //, whatever will we do!
  • karlicoss 7y ago
    - sometimes the schema is strict (and you might not be able to control it) - you might not be certain if you break something with such a comment (i.e. you have to double check that all the "comment" fields are filtered) - you're not geting highlighting from your text editor, it's easier to miss a comment - you can't have a comment for a list item Basically, you can leave the comments, but you're really discouraged to do so. Most people would just give up and not bother.
    • worble 7y ago
      I think the real loss here is that with a # or // the person reading the file immediately knows this is a comment. This line won't be read by the interpretor, and it's purely here for me to read it, so I probably should do. The following however { "comment": "Some comment", ... } Suddenly I'm not sure if this is meant for me, or if it's actually a configurable state without going into the program itself and checking if it uses it.
      • Ididntdothis 7y ago
        Putting a comment into the payload is just a terrible hack. this shouldn't be necessary,
  • Ididntdothis 7y ago
    Your "comment" is part of the data so it's not a real comment. I don't understand the resistance against adding comments to JOSN. This seems pretty silly to me.
• jstewartmobile 7y ago
  Turing completeness in config files is a proven bad idea. If it is truly needed, use/write a config generator instead. We have so many tools in the build chain now that config generation can be easily automated into the deployment process. See m4, BPF, PDF, etc...
  • KMag 7y ago
    There are plenty of examples where eschewing Turing-completeness has resulted in safer systems that are easier to reason about. I haven't experienced cases where I've needed Turing-complete configuration, and regard Turing-complete configuration as a code smell. However, people at least feel they sometimes need Turing-completeness. When they do, I think we can all agree that ideally they'd not need to add another language to their project in order to do so. I think the ideal case would be a declarative Turing-incomplete configuration language with an escape hatch to a Turing-complete superset, similar to Rust's unsafe blocks. Bonus points for forcing impurity to be similarly confined without scaring people with monads. If done well, with both an AoT compiler generating highly optimized native code and an interpreter for loading configuration, one might even be able to bring the "configuration should be Turing-incomplete" and "configuration should be in the same language as the application" folks under the same big tent. In an ideal world, the AoT compiler could optionally partially evaluate your program with respect to a configuration file, basically conditional compilation on steroids, with guarantees that the conditional compilation hasn't altered program behavior. Static analysis would be easier, both for developer tools and the compiler's optimizer. A quick code search would help you focus on the gnarly bits of code. A code reviewer would take pause before approving some code with an "unsafe impure loopy {}" block (assuming "loopy" is the keyword for the Turing-complete superset).
• sly010 7y ago
  setuptools doesn't deserve to be mentioned in the same capacity as bazel/nix/dhall. Having the entire python language (specifically the ability to inspect the local system) available BEFORE you declare your package (as a side effect!) means you cannot reliably learn anything about a package without having an entire operating system with python and all package dependencies pre-installed. It can lead to scenarios where you cannot automatically query a packages dependencies without having those dependencies already pre-installed. Or when installing packages in different order yields totally different results. On the data <-> code spectrum, configuration should be closer to data than to code, perhaps a pure function that takes a high level config to a lower level config.
• m0zg 7y ago
  Seems like the config language would benefit from being functional. I've actually sort of seen an example of this: in GCL/BCL (Google/Borg configuration language). I used to joke that it's essentially a FP language with all the good features removed. All the alternatives I have seen outside Google are much worse, however. See here for more details: https://pure.tue.nl/ws/portalfiles/portal/46927079/638953-1.pdf https://pure.tue.nl/ws/portalfiles/portal/46927079/638953-1.... It ticks off most of the author's boxes, but it's so woefully underspecified and so horrifyingly complex, that Google has been trying to replace it with something else for the past decade, unsuccessfully (at least as of a few years ago, don't know about now). But you get to provide external parameters, compute values, use some rudimentary logic, inherit configs, reuse configs in larger configs, and so on and so forth. You can also spend a couple of weeks trying to understand the config structure of something like an ads backend. You do get some niceties though: even fairly large services, consisting of dozens of different backends/mixers/frontends can be brought up/upgraded/shut down/reconfigured with a single CLI invocation, soup to nuts, including things like monitoring and load balancing.
• thethethethe 7y ago
  I write a ton of config in a bespoke python config language and FAANG and there are definitely downsides to having logic in your configs. Config is supposed to be explicit and when you start iterating over lists and parsing conditionals, those layers of abstraction can make reading a simple config quite cumbersome. Additionally, each team will often write their own libraries which diverge from the standard practices furthering the stratification. That being said, these config can get very complex and I can’t imagine expressing them in something even relatively friendly like yaml
• rot25 7y ago
  This reminds me of Proxy Auto Configs (https://en.wikipedia.org/wiki/Proxy_auto-config https://en.wikipedia.org/wiki/Proxy_auto-config). PAC files are Proxy configs that are programmed in JavaScript. To do this devices usually embed a JavaScript runtime in the operating system to parse proxy files. This introduces a lot more attack surface than a standard config file would and has resulted in remote code execution vulnerabilities in android and windows https://android.googlesource.com/platform/external/chromium-libpac/+/af8de44fa4f7c228d812e8db86d4d16585ed1050 https://android.googlesource.com/platform/external/chromium-... https://googleprojectzero.blogspot.com/2017/12/apacolypse-now-exploiting-windows-10-in_18.html https://googleprojectzero.blogspot.com/2017/12/apacolypse-no...
• mapgrep 7y ago
  This approach seems to have served emacs quite well; the config is just some elisp. It’s weird more programmers don’t copy this approach, given how many of them use emacs. Of course, you start out just cargo culting emacs config, so many may not even realize emacs is configured via code. You can also get pretty far with setq functions, essentially just assigning values to variables. But learning to grok and code elisp is when you start to really see emacs’ power.
• gaogao 7y ago
  At work, we generally do configs via Python that builds to json. It's a good best of both worlds imo, as it avoids a bunch of the downsides in the article.
• sudosteph 7y ago
  Related: This is the same reason I've always preferred doing server config management with Chef (which uses ruby for everything) to Puppet (which has it's own DSL). Ansible, which uses YAML config with Python behind the scene is sort of a middle ground, but often wins out because it's simpler to learn and operate than a full Chef setup. I still find myself missing Chef when I have to write stuff for ansible though. To me, from best experiences to worst: Real language (ruby, Python) > Yaml/Toml > ini config > JSON > DSLs
• s_y_n_t_a_x 7y ago
  Lua is perfect for this. Simple syntax, one of it's design goals was to be a config language.
• ronyfadel 7y ago
  The neat thing about config files is that they’re easily serializable and transferrable over the network, the file system etc.. A programming language can’t do that (or if it could via eval, it would be a bad idea)
• yongjik 7y ago
  This just sounds like solving a problem at the wrong level and making it worse. If your config is so complicated that it warrants a full-blown language, the logic should go into the main program that reads the config and decides what to do, not left inside config. Having a programmable config is how you end up with horrible interdependencies where function f() does foo, and you can't figure out why it's not blowing up, until you realize the only caller is dynamically configured in staging_env_4.json which is emitted by config_generator.py which always ensures the precondition for foo, but only if it's also generating prod_env.json at the same time, but it's OK because we always do that anyway, and you're not sure if you need another cup of coffee or whiskey.
  • digitaltrees 7y ago
    Definitely this. Configs reflect the architecture, of there is a lot of complexity that means the architecture is overly complex and can likely be cleaned up.
  • umvi 7y ago
    What if it's config for CI (i.e. Jenkins)? In that case your "config" needs to support branching ("build this if master else build that").
    • kakwa_ 7y ago
      CI systems tends to be more about defining a workflow than setting simple parameters, specially if they also handle CD. For that reason, something looking more like a programing language feels more appropriate to me. I kind of like the Jenkinsfile syntax for this use case as a CI workflow is not a static definition defined by a configuration file (yaml, json or ini) but the syntax is also not a fully "open" programing language like Python which would authorize too many potential hacks. A specialized DSL in this case seems more appropriate and being the correct balance between flexibility and complexity, at least for most needs.
    • yongjik 7y ago
      I don't know Jenkins, but my take is that "DSL is good, dynamic full-language config is bad." If you need a fully programmable environment, then you need a fully programmable environment. The important thing is that the DSL is a well-defined contract between what you specify and what the underlying software does, so if you say "If X then do Y", then that's exactly what it does. You should be able to follow the DSL line-by-line and debug it. (You've probably noticed that I'm not a fan of declarative languages a la Makefile: you can never figure out where things "start" and "end".) A "full language config" (like Python) is much worse than DSL: usually there's no clear semantics about when any snippet is executed, yet it's powerful enough to bring in the whole outside environment, which is what's going to happen in a team with 5+ people.
• indygreg2 7y ago
  I converted PyOxidizer's configuration files from TOML to Starlark because I found it effectively impossible to express complex primitives in a static configuration file and the static nature was constraining end-user utility. A common solution to this problem is to invent some kind of templating or pre-evaluation of your static config file. But I find these solutions quickly externalize a lot of complexity and are frustrating because it is often difficult to debug their evaluation. At the point you want to do programming-like things in a config file, you might as well use a "real" programming language. Yes, it is complex in its own way. But if your target audience is programmers, I think it is an easy decision to justify. I'm extremely happy with Starlark and PyOxidizer's configuration files are vastly more powerful than the TOML ones were. https://pyoxidizer.readthedocs.io/en/stable/config.html https://pyoxidizer.readthedocs.io/en/stable/config.html
  • mixmastamyk 7y ago
    > I found it effectively impossible to express complex primitives in a static configuration file I don't quite understand this part. Any examples?
    • indygreg2 7y ago
      Compare https://pyoxidizer.readthedocs.io/en/v0.4.0/config.html#file-processing-semantics https://pyoxidizer.readthedocs.io/en/v0.4.0/config.html#file... to https://pyoxidizer.readthedocs.io/en/stable/config.html#file-processing-semantics https://pyoxidizer.readthedocs.io/en/stable/config.html#file.... In PyOxidizer's case, I wanted to create virtual pipelines of actions to perform. In TOML, we could create sections to express each stage in a logical pipeline. But if you wanted to share stages between pipelines, you were out of luck. With Starlark, you can define a stage as a function and have multiple functions reference it because is "just" calling a function. I suppose I could have defined names for stages and made this work in TOML. So let's use a slightly more complicated example. PyOxidizer config files need to allow filtering of resources. Essentially, call fn(x) to determine whether something is relevant. In the TOML world, we had to define explicit stages that applied filtering semantics: there were config primitives dedicated to applying filtering logic. PyOxidizer's config files had to expose primitives that could perform filtering logic desired by end-users. By contrast, Starlark exposes an iterable of objects and config files can examine attributes of each object and deploy their own logic for determining whether said object is relevant. This is far more powerful, as config files can define their own filtering rules in a programming language without being constrained by what the TOML-based config syntax supports.
• maelito 7y ago
  What about writing YAML, parsing it in your preferred programming language, and document what's expected ? We're doing it with thousands of lines of YAML, it's working great.
  • Koshkin 7y ago
    There is something wrong with the idea to use a markup language for configurations. Sure, the idea is extremely popular, and there are reasons for that, but a homoiconic scripting language (a Lisp or, to an extent, Javascript/JSON) would serve the purpose much better.
    • imtringued 7y ago
      YAML isn't a markup language and that's why the name is now "YAML ain't markup language".
• akyu 7y ago
  At one of my old jobs, we used XML for config files, but users could write C# in the XML which would get dynamically parsed and run at runtime. Needless to say it was an absolute nightmare.
• preommr 7y ago
  My 2¢: 90% of the time config files are simple enough for something like json (+ comments) The other 10% of the time where you need some level of logic to modify some setting value, there's no substitute for something that people are already familiar with. Just let people have their 3 line js function instead of a 2-3 other dependencies that add more convoluted logic to the build system. Trying to create a config specific language with more advanced features/logic is an absolute mess where you end up learning a second language, people are unhappy with how certain features aren't implemented, and others are unhappy with how too many features exist, and it's just all round chaos. tl;dr There should be no inbetween: Either you have a simple key/value config file that someone can learn in 5 minutes, or you let people use a language they're already familiar with.
• choward 7y ago
  > your program crashes because of something that would be trivial to catch with any simple type system Then goes on to recommend python. What?
  • XelNika 7y ago
    Sure, Python is dynamically typed, but it supports type hints and avoids implicit type conversions. I'm sure that quote is aimed squarely at YAML, it is an absolute nightmare by comparison.
• verdverm 7y ago
  Check out https://cuelang.org https://cuelang.org Created by people who have dealt with this problem at scale for many years. Picked it up a month ago, absolutely amazing.
• stepanhruda 7y ago
  Imo this article points out legitimate problems, but downplays the downsides. The real solution is to combine both: 1. Use a general purpose language (or ideally a very dumb subset like Starlark) to generate the configs. You get all the reusability, type safety, comments etc. 2. Have #1 output a deterministic config like json/yaml/etc, so you can always refer to what the config actually looks like without needing to debug the code in your head. Check that in and whoever uses the config only uses the static output, they shouldn’t care how it came to life. This way a config is easy to reason about both on the author and consumer side.
• cfv 7y ago
  If your config files contain anything that could not be expressed as an OS variable, that's a maintenance nightmare right there. Eventually some user somewhere is going to jam entire mini programs in there and then complain when his castle of cards implodes. Consider, if possible, the chance that if your program needs a subprogram for configuring itself, then maybe a bootstrap event hook or some tap into the bootstrap process like a plugin or something is a better fit for what you are trying to do. Actual managed extension points instead of ad-hoc ones will make your program much more useful in the long run.
• atrettel 7y ago
  I think that the security issue needs more attention than the author gives it. Config files are often shared and not rigorously checked, especially if they are very long. Arbitrary code execution is a real security risk that should not be minimized. For example, years ago I was a frequent user of the chemical kinetics code called Cantera. It calculates the dynamics of combustion reactions, with the big application being for jet engines. One of the files that it needs to load is a mechanism file (called the CTI file). This contains all of the information about the gas properties and chemical reactions. Different situations might require different mechanisms (propane mechanism versus JP8 mechanism). Anyhow, Cantera's mechanism file format is literally a Python script. See the link below for the most commonly used mechanism file that comes with Cantera: https://github.com/Cantera/cantera/blob/master/data/inputs/gri30.cti https://github.com/Cantera/cantera/blob/master/data/inputs/g... This file is 2000 lines long, and many mechanism files are even longer. I told my colleagues that it is possible to execute arbitrary Python code using the files but I was unable to convince them that it was a security risk. I think that these kind of config files are a big security risk for engineering firms, because they make it much easier to conduct industrial espionage. All that a bad actor has to do is put a few lines in one and get an engineer to run it once. Then they could steal designs, analyses, business plans, financial data, and many other things. It's a serious threat that should not be minimized.
• henrik_w 7y ago
  This reminded me of the Configuration Complexity Clock: https://mikehadlow.blogspot.com/2012/05/configuration-complexity-clock.html https://mikehadlow.blogspot.com/2012/05/configuration-comple... Hacker News discussion here: https://news.ycombinator.com/item?id=14298715 https://news.ycombinator.com/item?id=14298715
• jscholes 7y ago
  > exec(Path('config.py').read_text(), config) I'm guessing that this is in place of just importing the config module, because of this line: > You can even import the very package you're configuring. Which sounds like a recipe for circular import disaster.
• IshKebab 7y ago
  Have fun writing a GUI to edit your config file...
• nick_ 7y ago
  A "real" programming language? Ruby, Python, JavaScript? This is dynamically-typed language propaganda. Consistently, dynamic programming language communities insist on their superiority while raising "difficult" issues solved decades ago by most enterprise-grade languages/frameworks. It's like watching adolescents go through their growth phases. Try a language that doesn't suck.
  • Koshkin 7y ago
    If you think about it, configuration is a form of scripting.
• crazypython 7y ago
  Arras uses a JavaScript file that generates JSON.
• renewiltord 7y ago
  Interesting. This is the opposite of modern design defunctionalization. I think the arguments made for that apply to why this is not a good idea. The advantage of declarative configuration is that it provides a sync barrier to the human and a safe entry-point. I imagine in a pure language without global runtime state you could use this method but in the more mainstream languages it is likely to trip you up. I will refrain.
• rq1 7y ago
  I wouldn’t call Python a “real language” to begin with. Troll aside, nix is mentioned and IMHO is the perfect tool for the task, but the author throws it away in few lines because it’s an “overkill”.
• exabrial 7y ago
  Here's a philosophical question... when does one put configuration in files vs a GUI? When changes in the GUI are made, do you update the config files? This always bothered me. Even databases don't get this rightish. Take MySQL, you can SET GLOBAL TIMEZONE after the database has started, or you can set it in my.conf. To me, have multiple places do things leads to confusion and head-desking.
• exabrial 7y ago
  I really think configuration files should have a schema of some sort, something that an editor can read and know the complete set of valid options for, and highlight invalid options. Something aiken to an XSD for XML (but not XML, sucks to edit with command lines tools).
• S_A_P 7y ago
  My gut is telling me NO NO NO NO NO NO! But like most things development I’m sure the answer is maybe. Some of the time. For a specific set of problems. In certain cases. With certain languages.
• ebg13 7y ago
  I do this all the time in projects at work and it's extremely useful. Most importantly it pleasantly separates core logic from configuration-specific niggling detail logic (now, for instance, you can generate that thousand entry list of similar things in a few concise lines of code without a bespoke preprocessor pipeline). But the best way to sell it to people who don't understand or who aren't yet on board is to name it properly, which the article fails to do. What this is describing is no longer configuration. This is now a modular plugin architecture that only in the most basic usage case implements a configuration interface.
• oweiler 7y ago
  Groovy solves this really nice https://mrhaki.blogspot.com/2009/10/groovy-goodness-using-configslurper.html?m=1 https://mrhaki.blogspot.com/2009/10/groovy-goodness-using-co...
• indymike 7y ago
  Saltstack is an interesting compromise... states can be described in a serialization format or in Python. I used Saltstack a lot at my last company and really liked it because for most cases, you could just use YAML... but when that didn't work, you could bring in a "real" programming language (python).
• anderspitman 7y ago
  My wishlist for a perfect declarative language (configs being one primary use) is something like: * Not turing complete. * Imports/exports * Comments * Variables * Statically typed (amongst other benefits, allows you to use it for RPC, like protobuf) I've started implementing such a language 2 or 3 times, but JSON is just barely good enough that I haven't been able to justify the effort to flesh it out. Taking JSON and adding types (inline in the grammar; schemas aren't good enough) and comments would get us 80% of the way there.
  • karlicoss 7y ago
    Looks like Dhall would be a perfect fit for you. > Dhall is a programmable configuration language that you can think of as: JSON + functions + types + imports https://dhall-lang.org https://dhall-lang.org
    • pron 7y ago
      It's hard to take Dhall seriously because it seems like its authors don't take it seriously, and work on it for entertainment. They went through the trouble of creating bindings for five (!!) different languages, and the most popular of them, by a pretty wide margin, is Ruby. I assume Fortran, Delphi, and Idris are next on their list.
      • Gabriel439 7y ago
        Dhall now has bindings to Rust and Go and Java/Python bindings are currently in progress We also do take things seriously, including: * Creating a language server (do any other configuration languages have this?) * Soliciting donations to fund high priority work (https://opencollective.com/dhall https://opencollective.com/dhall) * Working on a book (https://github.com/Gabriel439/dhall-manual https://github.com/Gabriel439/dhall-manual) * Maintaining a formal language semantics (https://github.com/dhall-lang/dhall-lang/tree/master/standard https://github.com/dhall-lang/dhall-lang/tree/master/standar...) If you think we're still missing something please let us know as we are responsive to user feedback
• ehosca 7y ago
  https://github.com/lightbend/config/blob/master/HOCON.md#hocon-human-optimized-config-object-notation https://github.com/lightbend/config/blob/master/HOCON.md#hoc...
• twomoretime 7y ago
  My takeaway from the comments is that software development is not monolithic and what's appropriate for your application may be anything from a couple lines in a text file to literal python.
• mjw1007 7y ago
  One variant of this is to use Python syntax but parse it with the `ast` module rather than evaluating it. This avoids the security problems and the risk that someone will start to write programs inside the configuration file, but gives you the reasonably nice and powerful syntax. It turns out that teenagers in my country have often done some Python in school, so it's becoming one of the friendlier file formats for non-programmers.
  • karlicoss 7y ago
    (author here). The ast suggestion sounds cool, never came to my mind. Thanks, I'll try it sometime!
• phab 7y ago
  > termination checking > Anyone knows examples of conservative static analysis tools that check for termination in general purpose languages? As this is provably impossible, this "solution" introduces exactly one of the problems the author was trying to avoid: > can't be validated
  • karlicoss 7y ago
    (author here) That's why 'conservative'. I.e. it's allowed to reject a valid, terminating program, but if it does pass the check, your program is guaranteed to terminate. This is something that's possible, the only question is the tradeoff between the subset of the syntax and how complicated is the static analysis.
    • phab 7y ago
      But if it's allowed to reject a valid terminating program it's allowed to reject arbitrary, otherwise valid, configuration programs. In other words, you can no longer trust the output of the validator - the weakened model significantly reduces the utility of the validator.
      • karlicoss 7y ago
        That's how most analysis tools work. Not necessarily 'dynamic' languages even, e.g. - Clang (depending on warnings level) may reject a valid program -- doesn't make it less useful, you just suppress the check for the offending line and carry on - Rust borrow checker may be seemingly picky and reject a perfectly valid program form your viewpoint. Does it make it less useful? I wouldn't say so.
        • phab 7y ago
          Right, but if you've introduced the ability to ignore the validator then you've traded away the guarantee that your config program will be safe to execute. My point is that the validator can't give you the safety property that is claimed as a defence against one of the inherent issues with this approach.
• smsm42 7y ago
  > I would argue that when you can't define temporary variables, helper functions, substitute strings or concatenate lists, it's a bit fucked up. I must admit that's where I stopped taking it seriously. If you need helper functions and temp variables, you do need a good Turing-complete language, but we're no longer talking about config files. Maybe about config system, but not a config file. And confusing the two means missing the point why config files exist at all. I mean not everybody has to buy into code vs. data division, but if you miss entirely why it exists maybe it not thought through enough.
  • KMag 7y ago
    I agree with you that helper functions within configuration are a code smell, but I find (immutable) temporary variables helpful. It's pretty common to want named constants to avoid copy-pasting configuration and to make consistent modification easier. You could have an inclusion mechanism and put those constant parts in their own configuration file, but then you've effectively renamed "my_constant" to "include('my_constant.cfg')". Immutable temporary variables are just non-globally scoped constants.
• mdale 7y ago
  We use json schema to counter a lot of the concerns listed around type checks, validation, back references to structures etc. More importantly this helps us have tight validation around configuration ecosystem that defined experimentation and server side overrides. See config delta blog post: https://medium.com/crunchyroll/introducing-crunchyroll-config-delta-90b212e61a15 https://medium.com/crunchyroll/introducing-crunchyroll-confi...
• robbyoconnor 7y ago
  uhm... toml and yaml are fine. You could also use JSON or environment vars...
• warent 7y ago
  Unless we're talking about some specific cases of dependency injection, adding logic to your configurations sounds like a living hell. Can you imagine the exponential explosion of complexity that you'd have to deal with? How do you even _test_ something like this? If you program requires configurations so dynamic that it needs to be its own program, there's something very smelly about your architecture. Next thing we'll need is a config config to config your config, etc. Infinite regression into a very stupid place.
  • klysm 7y ago
    Configs usually have a decent amount of logic hid in the implementation for defaults and other implicit things. Having a language like Dhall puts a hard limit on the possible complexity of the configuration while simultaneously allowing you to make everything explicit without being redundant.
• lonelappde 7y ago
  Config is code. Replace "configuration" with "scripting layer", and you'll have a much better perspective on "config" "vs" "code". Especially if your "code" is also a scripting language or open source.
• jayd16 7y ago
  Don't we have configs so we can change programs without programming?
• michaelfeathers 7y ago
  Configuration files are globals and they are lousy for the same reason globals are lousy when they are in a program. If you use a programming language, you had least have a way to scope.
• cageface 7y ago
  Smug lisp weenies, as lisp fans sometimes call themselves , can be a bit annoying at times but as I see all these new config file formats, data dsls, hybrid scripts like svelte etc emerge you do have to admit that none of these are necessary in a language with a programmable syntax like lisp.
• mattbillenstein 7y ago
  I do something a bit different - salt pillar or ansible vault contains my "settings" for the entire backend, so as a step in my deploy, I just write out all the settings as a config.json so it's accessible from any application programming language. Then I'll have a python module config.py that reads that and injects those settings with perhaps a small amount of logic into that namespace on import. So you where ever you might need a setting you can just "import config; config.foo" to get a specific config value. For small systems, I even do service discovery this way - where does the db live? config.postgres.host/port or something yada yada.
  • mattbillenstein 7y ago
    Note one kinda neat thing about json is you can use jq from scripts to get values out of it as well. `psql -h $(jq -r '.services.postgres.host' config.json)`
• mlthoughts2018 7y ago
  I’m surprised to see nobody brought up the 12 Factor App principle about config, https://12factor.net/config https://12factor.net/config Anything that's changing from one run of your application to another is config. Static settings that don’t change from one run to another are not config, they are just static pieces of data and it does not matter how you store them other than that you ensure it meets the operating need of your application. Configs, however, which can change from one run to the next, are different. They need to be factored out of code completely and only addressed by inspection of the runtime environment. Whatever tool you use to ensure they are injected to the environment is also a totally uninteresting decision as long as it meets the operating needs of your deployment system. That’s it. 1. store static constants and data items however you want as long as your program meets its operating requirements 2. config is not the same as static constants or static data, config can change from one run of the application to another. 3. factor config completely out of the code so it is solely referenced as part of the environment 4. store external configs however you want so long as the operating constraints of the deployment system are met. Within items 1 and 4, debate over relative merits of different tools is almost always useless bikeshedding unless it boils down to a real operating constraint of either the app itself or the deployment system. For example, a system that puts secret access tokens (config, not static settings) into the environment by storing them in plaintext environment variables might violate a security operating constraint of the deploy system and so a different system that manages encrypted secrets injected into the environment safely could be the winner for real operating reasons. Meanwhile, whether to store “staging” vs “production” database connections in Python / YAML / JSON / TOML / etc. because of comments / whitespace / use of builtin library “extra” code / whatever is just pure bikeshedding waste of time.
• arvindrajnaidu 7y ago
  Agree. Config is hard-coding. So code it.
• mixmastamyk 7y ago
  I've read a lot of "one true way" in this discussion. Do we need programmable configuration? Depends on the requirements really. Django seems to work ok with Python configuration. I haven't seen found much logic happen in the projects I've used. But the user is the developer not an end-user. So security is not an issue from this angle. I'm of the opinion that a simple config format like .ini is best for the end-user of a small application, while a validated schema (with code if needed) is best for a large one. To that end I'm currently experimenting with tconf to bring together those approaches under one package: https://github.com/mixmastamyk/tconf https://github.com/mixmastamyk/tconf
• greggman3 7y ago
  I somewhat agree and so does much of the JavaScript world. eslint or maybe jslint started off with json and later ended up letting you make a JavaScript file. As a simple example of something you might want to do, read the version from the package.json and generate a banner or an installation version. Grunt (not sure who still uses Grunt but I do) is JavaScript based. And yes, sharing parts is something I commonly do as in define common options and re-use them in dev, production, or minimized, un-minimized. WebPack takes a JavaScript file For most build tasks I've had some place where I needed to use code to make the build better but it can go both ways. The worst experience I've ever had is Scons (python). No one I know ever really understood how it worked and so every programmers wrote wrong and hard to understand code trying to insert their special needs into the build.
• drdaeman 7y ago
  I disagree. Those configs would be unmanageable. You would no longer be able to set a value, interfaces like `git config` or your-favorite-preferences-GUI-pane wouldn't be possible. And, generally speaking, I believe it's a bad and non-user-friendly thing. However, it makes perfect sense to use a programming language, if your program is programmable and is meant to be programmable. Like a shell's rc files. It's also probably okay to use programs as configs, if everything related to the program is built decoratively, so all the configuration files are generated (from some higher-order configuration) and not meant to be ever manipulated, only replaced with the newly generated versions. Programmatically generating programs is simple, programmatically manipulating programs is not. It is also fine if you explicitly want to restrict an ability to manipulate the configuration or create management interfaces, requiring human programming (whenever the tool is meant to be programmable or not). Feels like a weird idea, but I can see this being considered as a trade-off.
• phissenschaft 7y ago
  https://queue.acm.org/detail.cfm?id=2898444 https://queue.acm.org/detail.cfm?id=2898444 """ ... embrace the inevitability of programmatic configuration, and maintain a clean separation between computation and data. The language to represent the data should be a simple, data-only format such as JSON or YAML, and programmatic modification of this data should be done in a real programming language, where there are well-understood semantics, as well as good tooling ... """
• yellowapple 7y ago
  > This is considered as a positive by many, but I would argue that when you can't define temporary variables, helper functions, substitute strings or concatenate lists, it's a bit fucked up. What's "a bit fucked up" is expecting the configuration file to be able to support this when this can and should be done by the application itself. If you feel the need to do these things in the configuration file itself, then the application is providing insufficient abstractions. Like sure, there's definitely use in an application dynamically loading its own potentially-user-supplied code to modify its behavior, but most people don't call that code "configuration files"; we call that code "extensions" or "modules" or "plugins" or "scripts" or somesuch.
• PopeDotNinja 7y ago
  I'm gonna call this thermonuclear programming... Exploding your program with another program.
• kevin_thibedeau 7y ago
  This is an area where Tcl can shine. Sandboxed sub-interpreters can be stripped of all unwanted commands so that you are no longer Turing complete. You are left with simple configurations that just set variables. If more power is needed you let some control structures back in or select custom commands to do what a dumb config file can't.
  • bjoli 7y ago
    Yup. I just did a similar thing in guile (using sandboxed environments). as a scheme weenie I feel like I am looking at a world of people re-discovering things I have taken for granted since... forever? I went from trying to embed python in a C application to embedding TCL and was blown away how easy it was, and that it supported threads. this was back in 2003 I remember trying it with lua, but the parallelism story wasn't really great until at least 5.1. TCL did it right from 8.4 and things hasn't really changed since, except that Lua catched up.
• Pmop 7y ago
  I used to develop game engines when I was starting to learn programming. First C then C++. I knew JSON and there's an excelent header-only library for C++. However, I quickly learned from the indie gamedev community that parsing y=x always solves most of your problems, and it's quite trivial to do it, specially if you're using Python. Certainly better than using a full blown programming language for conf files.
• thiht 7y ago
  I hate when config is done using a real programming language. In the JS world, Grunt sucked, Gulp sucked, Webpack sucks, Parcel came with zero-config (ie. 3 lines of JSON max) and it rocks. Maybe it's me, but I love it when my conf is static and just easy to understand.
• kire456 7y ago
  I am quite turned off by the following passage near the top of this page: > bits of configs can't be reused. For example, while YAML, in theory, supports reusing/including bits of the config (they call it anchors), some software like Github Actions doesn't support it. I use YAML configuration for Home Assistant all the time, and the includes work just fine. This article is off to a really weak start by saying "config files can not have X, because well they can just fine but arbitrary software Y doesn't support it". Especially damning because the proposed solution is to embed a whole other programming language within your program! If you can change the code of the consuming program, then just add support for imports, instead of adding a Python dependency.
• peanut-walrus 7y ago
  As a sysadmin, my main requirements for configs are: * Human-readable and writable, including on systems where you have only very basic text editing tools * Easy to template (for example mass deployments via Ansible or other such tool) * Easy to use with grep/sed/etc The format I've found that is easiest to work with is what sysctl and OpenWRT use for their configs. You can have complex hierarchies, but every line stands on its own. This means you don't have to be careful about where in the config a particular line is, if it is in the proper block, etc. Also getting information on a particular sub-item is as easy as running a single grep command.
• socialdemocrat 7y ago
  I think Python is a terrible language for config files. It requires a full python installation. Lua or a simple LISP are better choices as they can trivially be embedded in every program at very low cost in terms of added code. It further avoid juggling different Python versions.
  • acqq 7y ago
    Exactly. And once one has something like that embedded, one can use that flexibility for many kinds of adaptations and scripting of the main program.
• Kwantuum 7y ago
  You can add eslint to the things that allow scripts as config files, just like webpack. I found this very useful because I just started a full-stack javascript project, and I need most of the linting config to be the same (tab-width, syntax, etc.) but then node and browsers don't have the same syntax for imports, don't have the same globals, etc. so I want to reuse a part of the config. And because you can write the config in js, it's trivial to import the result of the common config file (which is also used for isomorphic code, which should have no client/server specific code) and just add or remove what I need from that to return a different config for linting client and server files.
• ThePhysicist 7y ago
  Ansible might be a valid example of a system that should have been built using a programming language instead of a data language. Due to the template language mixed with interpretation it can be quite difficult to understand the source of a particular error, and the limitations of Jinja make it quite difficult to express some things that would be straightforward to write in Python. That said you can be quite productive with Ansible and the config is very easy to read and simple to write. I think the original Ansible author now tries a Python based approach with his new tool (opsmops), so we'll see how that works out!
• sramsay 7y ago
  My users consistently forget a comma in a JSON config file, and then open an issue because "the program doesn't work." This is indeed a problem, but something tells me the solution isn't "have them use Python."
  • jfkebwjsbx 7y ago
    Your users should not be writing JSON or Python to begin with, so using Python is fine.
    • sramsay 7y ago
      So no configuration files at all?
• bschwindHN 7y ago
  I'm happy with json5 for now, it addresses my complaints with json and is supported by the main tools I use.
• nickjj 7y ago
  I'm surprised the author didn't mention Suckless tools https://tools.suckless.org/ https://tools.suckless.org/. Their philosophy is to use minimal software and all configuration is done in a single programming language file. But it's also written in C which means you need to recompile the app to change config values. They typically provide a bunch of patch files that you can optionally bake into your custom binary, or you can write your own modifications too.
  • zabil 7y ago
    > you need to recompile the app to change config values. This works for a user using one of the suckless tools like dwm or surf. However, for server side applications there maybe multiple configurations and it may not be feasible to compile the whole app for every environment or deployment.
• blain_the_train 7y ago
  In terms of how this should be done, clojure and edn shine here and prove the point rather strongly that solutions like ymal are unnecessary at best and deeply limiting at worst. Yaml is the poster boy for easy but not simply.
• bonebutter 7y ago
  I hope I'm not to late to the party but I'm building a project doing just this. It's called Anyfig and allows you to create your configs during runtime in Python for pure-Python projects. Check it out :) https://github.com/OlofHarrysson/anyfig https://github.com/OlofHarrysson/anyfig
• otikik 7y ago
  Lua is great for this.