5 ms·

Honest question, why not just use… JavaScript? I get that this is scoped to only templating json, but the syntax is already diverging pretty heavily from actual

Honest question, why not just use… JavaScript? I get that this is scoped to only templating json, but the syntax is already diverging pretty heavily from actual JavaScript, while also trying to pretend to be similar? You’re using operators and a self/this keyword and things, but I can only assume they obey different rules. Could just be config.js. (Or just generate JSON in your app programmatically, it’s a pretty simple well-defined syntax and there’s already a hash-to-json library for whatever platform you’re using)

gjulianm 4y ago
I honestly... don't know. I have been using Jsonnet to programatically generate Grafana dashboards, because AFAIK the only official library to generate them is written in Jsonnet [1]. In my experience, it works, but it doesn't really give me any distinct advantage and instead it gives me some headaches. Maybe for things that look almost like JSON it'd be helpful, but the moment you start dealing with more complex generations you start finding lack of typing, lack of IDE support, lack of easy debugging, etc, fairly problematic. For example, something I really dislike is that due to how the expressions are evaluated, the only way to add debug/trace statements is to use them to "transform" a value you're going to use, if you don't use the result of the trace in the final output, the trace does not appear. Also, I really dislike the error messages. Again, due to the lazy evaluation design, when you mess up something the error message might appear deep in some unrelated call stack and with a jarring lack of context. Debugging it is a real pain. 1: https://github.com/grafana/grafonnet-lib https://github.com/grafana/grafonnet-lib
- ithkuil 4y ago
  > debugging is a real pain yes! As always, it's mostly a tooling issue. Surely, some aspects of the language like lazy evaluation, make it even harder, but fundamentally there is no reason we cannot significantly improve on the overall experience. And this is not just about debugging when things go wrong. It's also hard to navigate the codebase and understand where are the templates that you care about. Things can happen at multiple levels (that's the feature!) and so it can be quite hard to figure out which files you need to touch if you want to change something in the output. I'm working on a tool to answer that question: "if I wanted to change this field in the generated out put, what are the places in the input files that contribute to produce this value" $ ursonnet testdata/child.jsonnet '$.deployment.spec.template.spec.containers[0].resources.limits.cpu' testdata/common.libsonnet:27 testdata/base.jsonnet:5 testdata/common.libsonnet:23 testdata/common.libsonnet:22 testdata/config.libsonnet:5 The tool lives in https://github.com/mkmik/ursonnet https://github.com/mkmik/ursonnet . I got the basics working but it doesn't work on my larger codebases due to some bug I didn't have yet time hunting down. Having some interest/feedback/help from the community would help making this a reality.
- thinkmassive 4y ago
  Grafana have been working on a replacement for grafonnet but unfortunately haven’t shared much detail. It sounds like they completely rewrote the dashboard spec using CUE, but there will be converters available at release. In some of the discussions where the community is trying to figure out how to future-proof current dashboard definitions, Grafana Labs has also recommended this Python tool (unofficially I suppose) by Weaveworks, called grafanalib: https://github.com/weaveworks/grafanalib https://github.com/weaveworks/grafanalib
  - verdverm 4y ago
    You may be thinking of the https://github.com/grafana/thema https://github.com/grafana/thema library, which is the extracted version from some other projects / internal tooling.
    - thinkmassive 4y ago
      Here’s the discussion, “Roadmap: As-code,” that I’ve been tracking: https://github.com/grafana/grafana/discussions/39593 https://github.com/grafana/grafana/discussions/39593 Thema is mentioned there, as “the presumed successor to grafonnet,” but that hasn’t officially been confirmed and it sounds like bigger changes might be underway. Discussions are also happening elsewhere, like their Slack, but that link is the most complete overview I’ve found.
      - verdverm 4y ago
        Thema is still under heavy development for sure. I chat with Sam about it on the CUE slack from time to time. I have a related effort I am working on for data model evolution, tracking, and transformation generations as part of https://github.com/hofstadter-io/hof https://github.com/hofstadter-io/hof
mike_hearn 4y ago
Yes that's a common problem with config languages. They're torn between competing priorities: 1. Fast to parse with a small engine, good error messages, safe to evaluate. 2. Powerful, can express config with arbitrary logic. In Conveyor we try an alternative approach. The config is HOCON, which is a superset of JSON syntax designed for human readability/writability/convenience first and foremost, so it's got a very nice and clean feel to it. You can see an example here: https://github.com/hydraulic-software/github-desktop/blob/conveyorize/conveyor.conf https://github.com/hydraulic-software/github-desktop/blob/co... It can be parsed with a normal-sized config library and the errors you get are reasonable. But then what if you hit the limits of what it can express? We added support for "hashbang includes": include "#!script.js" You can embed arbitrary commands in the config which are executed when found unless the app is running in untrusted mode. The script is expected to produce more config on stdout, which is then included. This lets you encode only the minimal needed logic using a full programming language, whilst the rest stays declarative.
- wruza 4y ago
  In case you want a fresh opinion, every time I stumble upon a config like this, I wish it was just a .json, .js, .py, heck even .pl or .php file rather than that. Why. It’s unclear what’s the syntactic structure of it at a glance. Yes, in stricter languages there’s more of /[()[\]{}"'`,;]/, but also I know for sure what’s a literal, what’s an identifier, what’s a key, what’s a number or a date. The whole structure and ordering is obvious. Same issue I have with nginx, terraform and other ad-hoc half-languages half-formats half-templates. The worst part is that you have to hack them once a year, but it never “sinks in” for good even if you read the docs.
  - mike_hearn 4y ago
    Thanks. HOCON is a superset of JSON so you can write JSON if you like. There's also a spec: https://conveyor.hydraulic.dev/7.2/configs/hocon-spec/ https://conveyor.hydraulic.dev/7.2/configs/hocon-spec/ You can also convert HOCON to JSON and back. I find it a lot easier to work with configs when you can get rid of superfluous syntax, can use comments, substitutions, include files etc. But you don't have to use those.
wruza 4y ago
Agreed. Tomorrow they’ll realize that templating often requires string capitalization and other operations and will start reinventing Math, String.prototype and exception handling, if not yet. Nevermind, https://jsonnet.org/ref/stdlib.html https://jsonnet.org/ref/stdlib.html It’s just a new language with a new runtime, for some reason marketed as a json templater. Literally any language with “json” module is at least equivalent to it, but more familiar to developers.
GoblinSlayer 4y ago
>Or just generate JSON in your app programmatically Or make an application run on a normal config.
phailhaus 4y ago
Scoping to templating json lets the library enforce a really strict contract that you don't get with javascript. For example, * Accessing a missing field is totally fine in JavaScript and returns undefined; this is an error in Jsonnet. * JSonnet guarantees no side-effects, not true of JS. * When you run a jsonnet file, you have a consistent CLI that handles things like external vars, whereas you'd have to write your own wrapper to do this in JS. * JSonnet lets you export to multiple different config formats, whereas JS only support JSON really. Again, you'd have to write a wrapper to do this. * JavaScript would let you return any valid JS object, your wrapper would need to validate. Like yes, you could write your own JS library to mimic a lot of these behaviors, but the constraints are what make the tool useful. I do wish it had better typing though, so you could more easily inspect the structure of the resulting object.
- graypegg 4y ago
  If the constraints are what makes it special, why is it just not very constrained? If you really wanted something totally focused on having multiple configs sharing some base config template, then you wouldn’t be able to do anything other than overwrite specific keys entirely. (json deep merge) Jsonnet seems to let you concat strings, do some math using it’s standard lib (?!), reference other parts of the object etc. I also don’t think “object matches type description” is a niche that JS is lacking, there’s tons of libraries out there to assert that. You’re just trading footguns for different footguns IMO.
  - phailhaus 4y ago
    Yep, it's all about them tradeoffs. JS gives you absolute freedom, JSonnet doesn't. JS is easier to debug, JSonnet isn't because it's very declarative.
- spion 4y ago
  1. can be fixed with typescript 2. can be fixed with Deno - https://deno.land/manual@v1.31.3/basics/permissions https://deno.land/manual@v1.31.3/basics/permissions 3, 4, 5. Why not have that wrapper to solve those? (AFAIK you do need the wrappers for jsonnet too)
mschaef 4y ago
> Honest question, why not just use… JavaScript? I get that this is scoped to only templating json, but the syntax is already diverging pretty heavily from actual JavaScript, while also trying to pretend to be similar? Jsonnet's stated aim is to be a superset of JSON rather than something similar to JavaScript. Having used Jsonnet for a while, it's nice in that it makes it relatively easy to take existing JSON and incrementally turn it into templates. You wind up with the ability to create some nice abstractions for elements of JSON. The close coupling between the syntax of the emitted output and the Jsonnet script itself makes it far easier to write a correct template than when using something like gotmpl to create Yaml in a Helm chart. Whether or not this is worth another language is a judgement call, but it's not that hard to learn or work with, so I've tended to find Jsonnet a nice tool to have around.
lovehashbrowns 4y ago
Used it previously to template some Terraform module configurations. The way the flow would work would be YAML config -> Jsonnet + defaults -> Terraform + Spinnaker We wanted to avoid having to inject a more complex piece to the puzzle. It worked okay at first, but I knew from the get-go that wasn't going to hold for very long and eventually there was a Python app that took the YAML config and the Jsonnet+defaults and mashed them together before sending the info to TF + Spinnaker.
rssoconnor 4y ago
jsonnet has lazy evaluation. This allows you to create mutual references between different parts of your configuration without having to manually topologically sort the evaluation order yourself.
bboreham 4y ago
One take on that idea: https://github.com/jkcfg/jK https://github.com/jkcfg/jK

gjulianm 4y ago

I honestly... don't know. I have been using Jsonnet to programatically generate Grafana dashboards, because AFAIK the only official library to generate them is written in Jsonnet [1]. In my experience, it works, but it doesn't really give me any distinct advantage and instead it gives me some headaches. Maybe for things that look almost like JSON it'd be helpful, but the moment you start dealing with more complex generations you start finding lack of typing, lack of IDE support, lack of easy debugging, etc, fairly problematic. For example, something I really dislike is that due to how the expressions are evaluated, the only way to add debug/trace statements is to use them to "transform" a value you're going to use, if you don't use the result of the trace in the final output, the trace does not appear. Also, I really dislike the error messages. Again, due to the lazy evaluation design, when you mess up something the error message might appear deep in some unrelated call stack and with a jarring lack of context. Debugging it is a real pain. 1: https://github.com/grafana/grafonnet-lib https://github.com/grafana/grafonnet-lib

ithkuil 4y ago
> debugging is a real pain yes! As always, it's mostly a tooling issue. Surely, some aspects of the language like lazy evaluation, make it even harder, but fundamentally there is no reason we cannot significantly improve on the overall experience. And this is not just about debugging when things go wrong. It's also hard to navigate the codebase and understand where are the templates that you care about. Things can happen at multiple levels (that's the feature!) and so it can be quite hard to figure out which files you need to touch if you want to change something in the output. I'm working on a tool to answer that question: "if I wanted to change this field in the generated out put, what are the places in the input files that contribute to produce this value" $ ursonnet testdata/child.jsonnet '$.deployment.spec.template.spec.containers[0].resources.limits.cpu' testdata/common.libsonnet:27 testdata/base.jsonnet:5 testdata/common.libsonnet:23 testdata/common.libsonnet:22 testdata/config.libsonnet:5 The tool lives in https://github.com/mkmik/ursonnet https://github.com/mkmik/ursonnet . I got the basics working but it doesn't work on my larger codebases due to some bug I didn't have yet time hunting down. Having some interest/feedback/help from the community would help making this a reality.
thinkmassive 4y ago
Grafana have been working on a replacement for grafonnet but unfortunately haven’t shared much detail. It sounds like they completely rewrote the dashboard spec using CUE, but there will be converters available at release. In some of the discussions where the community is trying to figure out how to future-proof current dashboard definitions, Grafana Labs has also recommended this Python tool (unofficially I suppose) by Weaveworks, called grafanalib: https://github.com/weaveworks/grafanalib https://github.com/weaveworks/grafanalib
- verdverm 4y ago
  You may be thinking of the https://github.com/grafana/thema https://github.com/grafana/thema library, which is the extracted version from some other projects / internal tooling.
  - thinkmassive 4y ago
    Here’s the discussion, “Roadmap: As-code,” that I’ve been tracking: https://github.com/grafana/grafana/discussions/39593 https://github.com/grafana/grafana/discussions/39593 Thema is mentioned there, as “the presumed successor to grafonnet,” but that hasn’t officially been confirmed and it sounds like bigger changes might be underway. Discussions are also happening elsewhere, like their Slack, but that link is the most complete overview I’ve found.
    - verdverm 4y ago
      Thema is still under heavy development for sure. I chat with Sam about it on the CUE slack from time to time. I have a related effort I am working on for data model evolution, tracking, and transformation generations as part of https://github.com/hofstadter-io/hof https://github.com/hofstadter-io/hof

mike_hearn 4y ago

Yes that's a common problem with config languages. They're torn between competing priorities: 1. Fast to parse with a small engine, good error messages, safe to evaluate. 2. Powerful, can express config with arbitrary logic. In Conveyor we try an alternative approach. The config is HOCON, which is a superset of JSON syntax designed for human readability/writability/convenience first and foremost, so it's got a very nice and clean feel to it. You can see an example here: https://github.com/hydraulic-software/github-desktop/blob/conveyorize/conveyor.conf https://github.com/hydraulic-software/github-desktop/blob/co... It can be parsed with a normal-sized config library and the errors you get are reasonable. But then what if you hit the limits of what it can express? We added support for "hashbang includes": include "#!script.js" You can embed arbitrary commands in the config which are executed when found unless the app is running in untrusted mode. The script is expected to produce more config on stdout, which is then included. This lets you encode only the minimal needed logic using a full programming language, whilst the rest stays declarative.

wruza 4y ago
In case you want a fresh opinion, every time I stumble upon a config like this, I wish it was just a .json, .js, .py, heck even .pl or .php file rather than that. Why. It’s unclear what’s the syntactic structure of it at a glance. Yes, in stricter languages there’s more of /[()[\]{}"'`,;]/, but also I know for sure what’s a literal, what’s an identifier, what’s a key, what’s a number or a date. The whole structure and ordering is obvious. Same issue I have with nginx, terraform and other ad-hoc half-languages half-formats half-templates. The worst part is that you have to hack them once a year, but it never “sinks in” for good even if you read the docs.
- mike_hearn 4y ago
  Thanks. HOCON is a superset of JSON so you can write JSON if you like. There's also a spec: https://conveyor.hydraulic.dev/7.2/configs/hocon-spec/ https://conveyor.hydraulic.dev/7.2/configs/hocon-spec/ You can also convert HOCON to JSON and back. I find it a lot easier to work with configs when you can get rid of superfluous syntax, can use comments, substitutions, include files etc. But you don't have to use those.

wruza 4y ago

Agreed. Tomorrow they’ll realize that templating often requires string capitalization and other operations and will start reinventing Math, String.prototype and exception handling, if not yet. Nevermind, https://jsonnet.org/ref/stdlib.html https://jsonnet.org/ref/stdlib.html It’s just a new language with a new runtime, for some reason marketed as a json templater. Literally any language with “json” module is at least equivalent to it, but more familiar to developers.

GoblinSlayer 4y ago

>Or just generate JSON in your app programmatically Or make an application run on a normal config.

phailhaus 4y ago

Scoping to templating json lets the library enforce a really strict contract that you don't get with javascript. For example, * Accessing a missing field is totally fine in JavaScript and returns undefined; this is an error in Jsonnet. * JSonnet guarantees no side-effects, not true of JS. * When you run a jsonnet file, you have a consistent CLI that handles things like external vars, whereas you'd have to write your own wrapper to do this in JS. * JSonnet lets you export to multiple different config formats, whereas JS only support JSON really. Again, you'd have to write a wrapper to do this. * JavaScript would let you return any valid JS object, your wrapper would need to validate. Like yes, you could write your own JS library to mimic a lot of these behaviors, but the constraints are what make the tool useful. I do wish it had better typing though, so you could more easily inspect the structure of the resulting object.

graypegg 4y ago
If the constraints are what makes it special, why is it just not very constrained? If you really wanted something totally focused on having multiple configs sharing some base config template, then you wouldn’t be able to do anything other than overwrite specific keys entirely. (json deep merge) Jsonnet seems to let you concat strings, do some math using it’s standard lib (?!), reference other parts of the object etc. I also don’t think “object matches type description” is a niche that JS is lacking, there’s tons of libraries out there to assert that. You’re just trading footguns for different footguns IMO.
- phailhaus 4y ago
  Yep, it's all about them tradeoffs. JS gives you absolute freedom, JSonnet doesn't. JS is easier to debug, JSonnet isn't because it's very declarative.
spion 4y ago
1. can be fixed with typescript 2. can be fixed with Deno - https://deno.land/manual@v1.31.3/basics/permissions https://deno.land/manual@v1.31.3/basics/permissions 3, 4, 5. Why not have that wrapper to solve those? (AFAIK you do need the wrappers for jsonnet too)

mschaef 4y ago

> Honest question, why not just use… JavaScript? I get that this is scoped to only templating json, but the syntax is already diverging pretty heavily from actual JavaScript, while also trying to pretend to be similar? Jsonnet's stated aim is to be a superset of JSON rather than something similar to JavaScript. Having used Jsonnet for a while, it's nice in that it makes it relatively easy to take existing JSON and incrementally turn it into templates. You wind up with the ability to create some nice abstractions for elements of JSON. The close coupling between the syntax of the emitted output and the Jsonnet script itself makes it far easier to write a correct template than when using something like gotmpl to create Yaml in a Helm chart. Whether or not this is worth another language is a judgement call, but it's not that hard to learn or work with, so I've tended to find Jsonnet a nice tool to have around.

lovehashbrowns 4y ago

Used it previously to template some Terraform module configurations. The way the flow would work would be YAML config -> Jsonnet + defaults -> Terraform + Spinnaker We wanted to avoid having to inject a more complex piece to the puzzle. It worked okay at first, but I knew from the get-go that wasn't going to hold for very long and eventually there was a Python app that took the YAML config and the Jsonnet+defaults and mashed them together before sending the info to TF + Spinnaker.

rssoconnor 4y ago

jsonnet has lazy evaluation. This allows you to create mutual references between different parts of your configuration without having to manually topologically sort the evaluation order yourself.

bboreham 4y ago

One take on that idea: https://github.com/jkcfg/jK https://github.com/jkcfg/jK