7 ms·

> This is not structured data. This is programming masquerading as configuration I find myself expressing this same opinion to people on a frequent basis. See

> This is not structured data. This is programming masquerading as configuration I find myself expressing this same opinion to people on a frequent basis. See ansible for another big example. Ansible has a try/catch equivalent in yaml [0]! Unfortunately, YAML is more or less a lowest common denominator for these sorts of interactions - being usable from any language is a really big boon. Would love to see the world adopt a trivial lisp or similar for this sort of thing, but that feels a ways off. Helm 3 introducing lua, for example, is a big step forward. Pretty much exactly what I've been envisioning for making k8s deployments less crummy, so I hope they end up with a good product. [1] Helm 2 is using Go-templated YAML, and it's terribly unergonomic and not particularly maintainable. Until you run templating, it's not even valid yaml, so text editors are terribly useless in aiding you with linting. [0]: https://docs.ansible.com/ansible/latest/user_guide/playbooks_blocks.html#error-handling https://docs.ansible.com/ansible/latest/user_guide/playbooks... [1]: https://sweetcode.io/a-first-look-at-the-helm-3-plan/ https://sweetcode.io/a-first-look-at-the-helm-3-plan/

joshlemer 8y ago
What about https://dhall-lang.org/ https://dhall-lang.org/ ?
- bpicolo 8y ago
  Dhall does indeed look like it starts to address this problem - haven't seen it before. I'd have to read deeper to decide whether I like it or not, though. I may try to convert some existing complex YAML I'm using into it to form an opinion.
  - Leace 8y ago
    Dhall is nice that it can natively support configuration migrations via total functions. For example Config1 can be migrated to Config2 by applying a function that always completes. It's also possible to constrain that function so it must accept Config1 and return Config2. Dhall supports functions from URLs and it's good to pin them using SHA hashes of their internal representation.
- q3k 8y ago
  Also https://jsonnet.org/ https://jsonnet.org/ . Both of them are basically BCL/GCL but with some semantic/scoping horrors removed.
  - jopsen 8y ago
    I was involved with json-e a while ago: https://taskcluster.github.io/json-e/ https://taskcluster.github.io/json-e/ Which to me seemed more intuitive. But I'm also pretty excited about the UI-as-code features happening in Dart: https://medium.com/dartlang/making-dart-a-better-language-for-ui-f1ccaf9f546c https://medium.com/dartlang/making-dart-a-better-language-fo... Mostly I prefer YAML w. json-e because it's declarative and expressive enough for those few cases where you needed fancy stuff.. But the new Dart features coming up, makes me think I could write config as code -- we'll see :) (Disclaimer: I work at Google)
- crdoconnor 8y ago
  Everything dhall advertises itself as useful for is basically as a replacement for situations where YAML has been abused or where the separation of concerns between code and config was improperly cut (this is largely how YAML files get to be mind numbingly repetitive). It does this by creating something not quite as powerful as a real programming language (sometimes a feature; probably not here though) while still complicated enough to effectively be unreadable to an average Joe.
- kevin_thibedeau 8y ago
  You can make Tcl as non-Turing complete as you wish. Stripped of most or all builtin commands, it becomes a flexible configuration language. You can add back power in small controlled doses as it becomes necessary. You can also make it more palatable to the cool kids by adding infix assignment and expression evaluation with a few simple helper procs.
  - yellowapple 8y ago
    This was a big part of what motivated me to create OTPCL [1]; using full-blown Erlang or Elixir for configuration files feels dangerous and heavy-handed, and I want to be able to readily define DSLs for configuration files instead to offer better control/safety/ergonomics. Plus, I happen to like Tcl ;) Imagine being able to go from a foo.app like this: {application,otpcl, [{description,"Open Telecom Platform Command Language"}, {vsn,"0.1.0"}, {modules,[otpcl,otpcl_env,otpcl_eval,otpcl_init,otpcl_parse, otpcl_shell,otpcl_stdlib,otpcl_stdmeta]}, {registered,[]}, {applications,[kernel,stdlib]}, {licenses,["ISC"]}, {links,[{"Bitbucket","https://bitbucket.org/YellowApple/otpcl"}, {"GitHub","https://github.com/YellowApple/otpcl"}]}]}. to something more like a foo.app.otpcl: application otpcl { description "Open Telecom Platform Command Language" vsn 0.1.0 modules otpcl otpcl_env otpcl_eval otpcl_init otpcl_parse \ otpcl_shell otpcl_stdlib otpcl_stdmeta registered applications kernel stdlib licenses ISC links { Bitbucket "https://bitbucket.org/YellowApple/otpcl" GitHub "https://github.com/YellowApple/otpcl" } } I dunno about you, but I like the latter a lot better :) It ain't a 1:1 representation, but it'd be straightforward to define each of those parameters as commands that emit the corresponding Erlang forms that make up an actual app spec. While OTPCL's by no means ready for primetime yet, I'm hoping it'll eventually be good enough to fill that niche in the Erlang/OTP ecosystem the same way Tcl was meant to fill that niche for software in general. [1]: https://otpcl.github.io https://otpcl.github.io / https://github.com/otpcl/otpcl https://github.com/otpcl/otpcl
  - eridius 8y ago
    There's a reason MacPorts[1] uses Tcl for its package definitions. The packages for the most part look just like structured data, but it's just a Tcl DSL that allows them to e.g. write hooks in Tcl instead of having to embed a shell script[2]. [1]: https://www.macports.org https://www.macports.org [2]: https://guide.macports.org/#development.examples https://guide.macports.org/#development.examples
vmchale 8y ago
> being usable from any language is a really big boon. Personally, I'm not particularly happy with the idea that I would use something inferior (like YAML) because various mainstream languages are bad and don't make more sophisticated languages possible.
deleted 8y ago
[deleted]
weberc2 8y ago
> Helm 3 introducing lua Oh, this is cool! I've actually been thinking for a while that things like Helm or CloudFormation templates or other infra-as-code things would be better served by something like Starlark, but Lua probably suits as well.
- simtel20 8y ago
  Have you looked at the work being done by stripe on skycfg? https://github.com/stripe/skycfg https://github.com/stripe/skycfg has the advantage of preserving protobuf message types as they're passed, so you get type checking.
  - weberc2 8y ago
    I'm aware of it, but I haven't looked much at it. I'm a little confused about its scope--is it for building out protobuf specs and Kubernetes config files? Or does it somehow use protobuf specs to build out Kubernetes config files?
    - simtel20 8y ago
      It's the latter. You import protobuf specs (compiled into golang). You build a runtime that imports the specs and they're available in the python interpreter.
guitarbill 8y ago
Ansible deserves some credit at least. It tried to avoid the pitfalls Chef and Puppet fell into, which is having a DSL that allows arbitrary logic, i.e. basically just a Ruby script. These become horrible to trace and debug. Ansible only allowed conditionals or loop constructs in some well-defined places, which actually does help when trying to comprehend Ansible playbooks. (Although one might still argue a strict DSL would offer the best of both worlds.)
- bmurphy1976 8y ago
  I feel like there is a large gap between shell scripts and Ansible. If we draw a line, you end up with something like this, with each option bringing more power, but also complexity with it. Bash/Make => ? => Ansible => Chef/Puppet/Salt => CFEngine There's a big jump in complexity from Bash/Make to Ansible. This is a place where Python or Ruby would make a lot of sense but without all the dsl/yaml/declarative nonsense. A small library of reusable functions would fit in well here, written in a language that isn't fundamentally broken (bash).
  - weberc2 8y ago
    It's unfortunate that "shell scripts" represent the imperative solution. Shell scripting is loaded with incidental complexity and footguns that other scripting languages (like Starlark) lack. The latter would be entirely appropriate for this sort of advanced configuration.
- eropple 8y ago
  From my perspective--as somebody who slings this stuff a lot--what you describe as a "pitfall" is what I describe as "how to make things work." I don't find that Chef cookbooks are all that onerous to debug at all. Cleanly-written Ruby (and you can write Chef cookbooks in very clean Ruby, it really isn't that big of an ask) isn't hard to trace and in the extreme case--and this has happened twice in my career--I just install Pry and drop a breakpoint into the Chef cookbook, then proceed to poke around, isolate the problem, and move on with my day. This, contrasted with Ansible's flinging around of Jinja templates and trying to turn YAML into a bad programming language, is a relief, and when I was a consultant I charged a premium for Ansible first because it frustrated me but second because it was generally harder for me to find out what was actually going on due to the verbosity, the commonplace copy-pasting, the relatively poor method of inventorying, and the "strap in, it's gonna get bad" anytime somebody decided to break out of the YAML mines and go write their own module. There remains a very awkward sysadmin/developer divide in the devops world, and I think that criticisms like the one you describe generally tend to hail from the former. As a developer whose output on occasion happens to be configured systems (when it isn't web pages, etc.), I'd rather a predictable programming environment over a "configuration language" every day (and it's one of the reasons I really wish that Noah's attempt to turn cookbooks into gems paid off; having them be a completely separate artifact from a Ruby gem is just silly).
75dvtwin 8y ago
Another potential option, is Lightbend Config https://github.com/lightbend/config https://github.com/lightbend/config It adds includes/inheritance, substitutions, unit of measure parity , comments , and few other things -- on top of JSON. does not have if-logic, loops, or arbitrary functions, however. On plus side, the library is available in a number of languages, easily embeddable. And there is an IntelliJ plugin for it, to color the syntax of the config files.
- debug-desperado 8y ago
  HOCON is a very nice configuration format that should be preferred to JSON and YAML - it really deserves more adoption. I always reach for Lightbend Config on non-Spring projects.
- dtech 8y ago
  It has a pretty nice format, but I've run into it's limitations more than once. Array indexing is not possible in the config or library, and nested-environmental variable substitution is not possible (e.g. $ACCESS_KEY_{$ENV_NAME})
lawn 8y ago
> This is programming masquerading as configuration On a slight side-note this is something that's been bothering me with markdown and other markup languages. They're very good for short comments but when you're writing a blog post, or god forbid an article series or a book, sometimes having access to a programming language is fantastic. It's why I like a markup like Pollen use. Also X-expressions fits very well to model a document.

joshlemer 8y ago

What about https://dhall-lang.org/ https://dhall-lang.org/ ?

bpicolo 8y ago
Dhall does indeed look like it starts to address this problem - haven't seen it before. I'd have to read deeper to decide whether I like it or not, though. I may try to convert some existing complex YAML I'm using into it to form an opinion.
- Leace 8y ago
  Dhall is nice that it can natively support configuration migrations via total functions. For example Config1 can be migrated to Config2 by applying a function that always completes. It's also possible to constrain that function so it must accept Config1 and return Config2. Dhall supports functions from URLs and it's good to pin them using SHA hashes of their internal representation.
q3k 8y ago
Also https://jsonnet.org/ https://jsonnet.org/ . Both of them are basically BCL/GCL but with some semantic/scoping horrors removed.
- jopsen 8y ago
  I was involved with json-e a while ago: https://taskcluster.github.io/json-e/ https://taskcluster.github.io/json-e/ Which to me seemed more intuitive. But I'm also pretty excited about the UI-as-code features happening in Dart: https://medium.com/dartlang/making-dart-a-better-language-for-ui-f1ccaf9f546c https://medium.com/dartlang/making-dart-a-better-language-fo... Mostly I prefer YAML w. json-e because it's declarative and expressive enough for those few cases where you needed fancy stuff.. But the new Dart features coming up, makes me think I could write config as code -- we'll see :) (Disclaimer: I work at Google)
crdoconnor 8y ago
Everything dhall advertises itself as useful for is basically as a replacement for situations where YAML has been abused or where the separation of concerns between code and config was improperly cut (this is largely how YAML files get to be mind numbingly repetitive). It does this by creating something not quite as powerful as a real programming language (sometimes a feature; probably not here though) while still complicated enough to effectively be unreadable to an average Joe.
kevin_thibedeau 8y ago
You can make Tcl as non-Turing complete as you wish. Stripped of most or all builtin commands, it becomes a flexible configuration language. You can add back power in small controlled doses as it becomes necessary. You can also make it more palatable to the cool kids by adding infix assignment and expression evaluation with a few simple helper procs.
- yellowapple 8y ago
  This was a big part of what motivated me to create OTPCL [1]; using full-blown Erlang or Elixir for configuration files feels dangerous and heavy-handed, and I want to be able to readily define DSLs for configuration files instead to offer better control/safety/ergonomics. Plus, I happen to like Tcl ;) Imagine being able to go from a foo.app like this: {application,otpcl, [{description,"Open Telecom Platform Command Language"}, {vsn,"0.1.0"}, {modules,[otpcl,otpcl_env,otpcl_eval,otpcl_init,otpcl_parse, otpcl_shell,otpcl_stdlib,otpcl_stdmeta]}, {registered,[]}, {applications,[kernel,stdlib]}, {licenses,["ISC"]}, {links,[{"Bitbucket","https://bitbucket.org/YellowApple/otpcl"}, {"GitHub","https://github.com/YellowApple/otpcl"}]}]}. to something more like a foo.app.otpcl: application otpcl { description "Open Telecom Platform Command Language" vsn 0.1.0 modules otpcl otpcl_env otpcl_eval otpcl_init otpcl_parse \ otpcl_shell otpcl_stdlib otpcl_stdmeta registered applications kernel stdlib licenses ISC links { Bitbucket "https://bitbucket.org/YellowApple/otpcl" GitHub "https://github.com/YellowApple/otpcl" } } I dunno about you, but I like the latter a lot better :) It ain't a 1:1 representation, but it'd be straightforward to define each of those parameters as commands that emit the corresponding Erlang forms that make up an actual app spec. While OTPCL's by no means ready for primetime yet, I'm hoping it'll eventually be good enough to fill that niche in the Erlang/OTP ecosystem the same way Tcl was meant to fill that niche for software in general. [1]: https://otpcl.github.io https://otpcl.github.io / https://github.com/otpcl/otpcl https://github.com/otpcl/otpcl
- eridius 8y ago
  There's a reason MacPorts[1] uses Tcl for its package definitions. The packages for the most part look just like structured data, but it's just a Tcl DSL that allows them to e.g. write hooks in Tcl instead of having to embed a shell script[2]. [1]: https://www.macports.org https://www.macports.org [2]: https://guide.macports.org/#development.examples https://guide.macports.org/#development.examples

vmchale 8y ago

> being usable from any language is a really big boon. Personally, I'm not particularly happy with the idea that I would use something inferior (like YAML) because various mainstream languages are bad and don't make more sophisticated languages possible.

deleted 8y ago

[deleted]

weberc2 8y ago

> Helm 3 introducing lua Oh, this is cool! I've actually been thinking for a while that things like Helm or CloudFormation templates or other infra-as-code things would be better served by something like Starlark, but Lua probably suits as well.

simtel20 8y ago
Have you looked at the work being done by stripe on skycfg? https://github.com/stripe/skycfg https://github.com/stripe/skycfg has the advantage of preserving protobuf message types as they're passed, so you get type checking.
- weberc2 8y ago
  I'm aware of it, but I haven't looked much at it. I'm a little confused about its scope--is it for building out protobuf specs and Kubernetes config files? Or does it somehow use protobuf specs to build out Kubernetes config files?
  - simtel20 8y ago
    It's the latter. You import protobuf specs (compiled into golang). You build a runtime that imports the specs and they're available in the python interpreter.

guitarbill 8y ago

Ansible deserves some credit at least. It tried to avoid the pitfalls Chef and Puppet fell into, which is having a DSL that allows arbitrary logic, i.e. basically just a Ruby script. These become horrible to trace and debug. Ansible only allowed conditionals or loop constructs in some well-defined places, which actually does help when trying to comprehend Ansible playbooks. (Although one might still argue a strict DSL would offer the best of both worlds.)

bmurphy1976 8y ago
I feel like there is a large gap between shell scripts and Ansible. If we draw a line, you end up with something like this, with each option bringing more power, but also complexity with it. Bash/Make => ? => Ansible => Chef/Puppet/Salt => CFEngine There's a big jump in complexity from Bash/Make to Ansible. This is a place where Python or Ruby would make a lot of sense but without all the dsl/yaml/declarative nonsense. A small library of reusable functions would fit in well here, written in a language that isn't fundamentally broken (bash).
- weberc2 8y ago
  It's unfortunate that "shell scripts" represent the imperative solution. Shell scripting is loaded with incidental complexity and footguns that other scripting languages (like Starlark) lack. The latter would be entirely appropriate for this sort of advanced configuration.
eropple 8y ago
From my perspective--as somebody who slings this stuff a lot--what you describe as a "pitfall" is what I describe as "how to make things work." I don't find that Chef cookbooks are all that onerous to debug at all. Cleanly-written Ruby (and you can write Chef cookbooks in very clean Ruby, it really isn't that big of an ask) isn't hard to trace and in the extreme case--and this has happened twice in my career--I just install Pry and drop a breakpoint into the Chef cookbook, then proceed to poke around, isolate the problem, and move on with my day. This, contrasted with Ansible's flinging around of Jinja templates and trying to turn YAML into a bad programming language, is a relief, and when I was a consultant I charged a premium for Ansible first because it frustrated me but second because it was generally harder for me to find out what was actually going on due to the verbosity, the commonplace copy-pasting, the relatively poor method of inventorying, and the "strap in, it's gonna get bad" anytime somebody decided to break out of the YAML mines and go write their own module. There remains a very awkward sysadmin/developer divide in the devops world, and I think that criticisms like the one you describe generally tend to hail from the former. As a developer whose output on occasion happens to be configured systems (when it isn't web pages, etc.), I'd rather a predictable programming environment over a "configuration language" every day (and it's one of the reasons I really wish that Noah's attempt to turn cookbooks into gems paid off; having them be a completely separate artifact from a Ruby gem is just silly).

75dvtwin 8y ago

Another potential option, is Lightbend Config https://github.com/lightbend/config https://github.com/lightbend/config It adds includes/inheritance, substitutions, unit of measure parity , comments , and few other things -- on top of JSON. does not have if-logic, loops, or arbitrary functions, however. On plus side, the library is available in a number of languages, easily embeddable. And there is an IntelliJ plugin for it, to color the syntax of the config files.

debug-desperado 8y ago
HOCON is a very nice configuration format that should be preferred to JSON and YAML - it really deserves more adoption. I always reach for Lightbend Config on non-Spring projects.
dtech 8y ago
It has a pretty nice format, but I've run into it's limitations more than once. Array indexing is not possible in the config or library, and nested-environmental variable substitution is not possible (e.g. $ACCESS_KEY_{$ENV_NAME})

lawn 8y ago

> This is programming masquerading as configuration On a slight side-note this is something that's been bothering me with markdown and other markup languages. They're very good for short comments but when you're writing a blog post, or god forbid an article series or a book, sometimes having access to a programming language is fantastic. It's why I like a markup like Pollen use. Also X-expressions fits very well to model a document.