5 ms·

Based on everyone's feedback about YAML (we didn't expect this much), we'll probably reconsider this!

gen220 4y ago
Take a look at Apache Aurora [1] if you want some inspiration on how to mold python into a config language. I used this system for a few years and mainly agree with the person you're replying to – having a proper programming language to define config is a very nice feature that I miss. [1]: https://aurora.apache.org/ https://aurora.apache.org/
- mileswjohnson 4y ago
  Thanks will take a look at.
- malkia 4y ago
  one of the benefits of starlark (unlike python): "Starlark is suitable for use in highly parallel applications. An application may invoke the Starlark interpreter concurrently from many threads, without the possibility of a data race, because shared data structures become immutable due to freezing." from https://github.com/bazelbuild/starlark/blob/master/spec.md https://github.com/bazelbuild/starlark/blob/master/spec.md - it's not python, you can't do recursion (!) and it's more limited (you can't read a file in bazel, and parse it, you have to make this operation into the graph somehow)
- oblio 4y ago
  What Python badly needs is Lua-like sandboxing/allowlisting. You get an interpreter with access to 0 functions by default and can add stuff on top. Maybe Python already has it, I'm not aware of it.
jcalder 4y ago
Glad to hear it! To synthesis my comment down (because it's easier once I've written it once, poorly): The complexity you see in Bazel/Buck/Pants build files may seem like a result of their decision to use a programming language. That's a red herring. That complexity is fundamental to the problems people need to solve to build software. If you remove the ability to solve it in your build system the complexity will move to other systems. Wrappers, CI, whatever. The complexity is just the problem space rather than a result of bad tooling.
- codethief 4y ago
  +1 Glad someone else brought up this topic and phrased it way better than I ever could have! This other comment further down puts it very well, too: https://news.ycombinator.com/item?id=34892724 https://news.ycombinator.com/item?id=34892724 > If you remove the ability to solve it in your build system the complexity will move to other systems. Adding to this: I think one of the main limitations of purely declarative (e.g. YAML-based) configurations is that a declarative DSL (as opposed to an imperative DSL) means you (only) get to use config options the inventors of the config language have thought of and implemented (i.e. "defined"). You cannot put into words things that require vocabulary that doesn't exist in your language. Meanwhile, an imperative language gives you the ability to define on your own what you would like the machine to do. As an example, consider Gitlab CI configs (YAML-based) or Ansible playbooks. In both cases, once you hit the boundaries of the DSL spec (which, in my experience, typically happens on day one) you can't do anything other than 1) give up or 2) leave the realms of the DSL and as you say, > move [the complexity] to other systems
  - hakre 4y ago
    Well thought and I can imagine this can be further differentiated. E.g. what is considered boundaries or limits in a declarative way also work as guardrails. So while it could be possible to hit actual limits on day one, there is often more than only two options to go on. Declarative for me often appears easier to find the correct one or to correctly know that that system is not designed to do that. The other differentiation I think is worth on the following part, also which I'd liked a lot btw: > move [the complexity] to other systems which stemmed from: > If you remove the ability to solve it in your build system the complexity will move to other systems. I think it is even _more_ beneficial to consider that this does not start at the build system: If you don't have and don't add the ability at development (make it a property of the project), the complexity will continue to _increase_ each time you move it into other systems after it (build, version control, CI, ... etc.).
nsm 4y ago
There is also a Rust implementation of Starlark as a starting point https://github.com/facebookexperimental/starlark-rust https://github.com/facebookexperimental/starlark-rust To add to everyone else, please don't use YAML. Starlark is great _precisely_ because it is a readable, well known (nearly Python) language that is limited at the same time (no unbounded for loops, no way to do non-deterministic things like get the current time or `random()`).
crubier 4y ago
Yeah, 100%, use a scripting language. Maybe JS? You could use deno core https://deno.com/blog/roll-your-own-javascript-runtime https://deno.com/blog/roll-your-own-javascript-runtime

Take a look at Apache Aurora [1] if you want some inspiration on how to mold python into a config language. I used this system for a few years and mainly agree with the person you're replying to – having a proper programming language to define config is a very nice feature that I miss. [1]: https://aurora.apache.org/ https://aurora.apache.org/

mileswjohnson 4y ago
Thanks will take a look at.
malkia 4y ago
one of the benefits of starlark (unlike python): "Starlark is suitable for use in highly parallel applications. An application may invoke the Starlark interpreter concurrently from many threads, without the possibility of a data race, because shared data structures become immutable due to freezing." from https://github.com/bazelbuild/starlark/blob/master/spec.md https://github.com/bazelbuild/starlark/blob/master/spec.md - it's not python, you can't do recursion (!) and it's more limited (you can't read a file in bazel, and parse it, you have to make this operation into the graph somehow)
oblio 4y ago
What Python badly needs is Lua-like sandboxing/allowlisting. You get an interpreter with access to 0 functions by default and can add stuff on top. Maybe Python already has it, I'm not aware of it.

jcalder 4y ago

Glad to hear it! To synthesis my comment down (because it's easier once I've written it once, poorly): The complexity you see in Bazel/Buck/Pants build files may seem like a result of their decision to use a programming language. That's a red herring. That complexity is fundamental to the problems people need to solve to build software. If you remove the ability to solve it in your build system the complexity will move to other systems. Wrappers, CI, whatever. The complexity is just the problem space rather than a result of bad tooling.

codethief 4y ago
+1 Glad someone else brought up this topic and phrased it way better than I ever could have! This other comment further down puts it very well, too: https://news.ycombinator.com/item?id=34892724 https://news.ycombinator.com/item?id=34892724 > If you remove the ability to solve it in your build system the complexity will move to other systems. Adding to this: I think one of the main limitations of purely declarative (e.g. YAML-based) configurations is that a declarative DSL (as opposed to an imperative DSL) means you (only) get to use config options the inventors of the config language have thought of and implemented (i.e. "defined"). You cannot put into words things that require vocabulary that doesn't exist in your language. Meanwhile, an imperative language gives you the ability to define on your own what you would like the machine to do. As an example, consider Gitlab CI configs (YAML-based) or Ansible playbooks. In both cases, once you hit the boundaries of the DSL spec (which, in my experience, typically happens on day one) you can't do anything other than 1) give up or 2) leave the realms of the DSL and as you say, > move [the complexity] to other systems
- hakre 4y ago
  Well thought and I can imagine this can be further differentiated. E.g. what is considered boundaries or limits in a declarative way also work as guardrails. So while it could be possible to hit actual limits on day one, there is often more than only two options to go on. Declarative for me often appears easier to find the correct one or to correctly know that that system is not designed to do that. The other differentiation I think is worth on the following part, also which I'd liked a lot btw: > move [the complexity] to other systems which stemmed from: > If you remove the ability to solve it in your build system the complexity will move to other systems. I think it is even _more_ beneficial to consider that this does not start at the build system: If you don't have and don't add the ability at development (make it a property of the project), the complexity will continue to _increase_ each time you move it into other systems after it (build, version control, CI, ... etc.).

nsm 4y ago

There is also a Rust implementation of Starlark as a starting point https://github.com/facebookexperimental/starlark-rust https://github.com/facebookexperimental/starlark-rust To add to everyone else, please don't use YAML. Starlark is great _precisely_ because it is a readable, well known (nearly Python) language that is limited at the same time (no unbounded for loops, no way to do non-deterministic things like get the current time or `random()`).

crubier 4y ago

Yeah, 100%, use a scripting language. Maybe JS? You could use deno core https://deno.com/blog/roll-your-own-javascript-runtime https://deno.com/blog/roll-your-own-javascript-runtime