10 ms·

Jsonnet – A data templating language

201 points by fhrow4484 8y ago

nikolay 7y ago
Jsonnet has been around for a while, but it looks like stale. Its stdlib is limited and nonstandard and the whole lazy evaluation is something new developers struggle with. I've been using it at work for 2 years and it's sad that it hasn't moved much. I think it lost a huge opportunity if it had more active and more open for innovation development. I'm sorry to say that, but the lead is busy with stuff and is highly opinionated. Also, Jsonnet is slow. We have a few hundred Jsonnet templates and it takes a very long time to compile, which is a pity given it's written in C (now being slowly and painfully rewritten in Go for whatever reasons).
haxterstockman 8y ago
No thanks I'll just write JavaScript
- moltar 8y ago
  Exactly my thoughts. I don’t see how it’s better than JS or TypeScript. I can render any object into JSON. This is in fact how I manage Kubernetes manifests. I use this package: https://www.npmjs.com/package/kubernetes-models https://www.npmjs.com/package/kubernetes-models Then I write normal TypeScript code to create Kubernetes objects/resources. Then I render it to JSON and feed to kubectl. The beauty of this solution is that it’s OOP and DRY.
  - atombender 8y ago
    You're assuming that all code can be trusted. Jsonnet is much easier to make secure than JS, since it can be statically guaranteed to be safe and side-effect-free (e.g. can't access the file system). This permits evaluating arbitrary untrusted data, which can be a boon to systems like CI servers, databases, or even Kubernetes (think Helm charts).
    - CraftThatBlock 8y ago
      He is producing JSON, just as you said. Not sure you understood him. He has a TS script that outputs JSON, which he is using as config.
      - atombender 8y ago
        The script to generate that JSON is code. Jsonnet also generates JSON. So his script is filling the same role as Jsonnet would. The point stands. He might trust his code, but I was highlighting reasons why this is not true everywhere, and indeed the ability to evaluate untrusted code can open up previously unavailable avenues. For example, storing server-side Jsonnet that can be safely evaluated by a multitenant backend.
- rlander 8y ago
  This should be the first thing on the website: “why use this and not plain Js”. I read everything and I still don’t understand why.
arthurz 8y ago
We have just had an article hitting nearly the top on not to use JSON for config https://arp242.net/weblog/json_as_configuration_files-_please_dont https://arp242.net/weblog/json_as_configuration_files-_pleas...
- fhrow4484 8y ago
  That's the reason I posted this, since this is kind of using, without the drawbacks (of not having comments etc) I wonder if folks had had experience using it for kubernetes configs like https://jsonnet.org/articles/kubernetes.html https://jsonnet.org/articles/kubernetes.html
  - weberc2 8y ago
    Pretty sure stripe built skycfg for this very purpose, but it uses starlark for the language, which is quite a lot nicer IMHO.
- paulddraper 8y ago
  Presumably, the author of that would be happy with Jsonnet, which isn't JSON.
- Noumenon72 8y ago
  I was not convinced by that article but it raised my awareness to the point I was happy to click on this one.
randomcoder1000 8y ago
This looks like a public version of gcl/borgcfg... But with JavaScript mixed in (so even worse, if that was possible)
- atombender 8y ago
  Jsonnet does not have JavaScript, though. It's JavaScript-like, but not nearly as complex (e.g. no prototypes, classes, etc.).
  - randomcoder1000 8y ago
    Yeah, I guess this doesn't use JavaScript expressions so it wouldn't have the weird stringiness and truthiness rules JS has, which is what I was worried about. Still, there are a lot of other issues with BCL.. I think I would rather see a piccolo style solution done with Skylark instead of BCL+json. From reading the website I see it actually was inspired by gcl/bcl for kubecfg, so hopefully they used some lessons learned!
manigandham 8y ago
Considering how easy it is to run Javascript anywhere, even small inline statements interpreted dynamically, I can't think of a single reason to use this instead.
c_r_w 8y ago
Some devs from my company just recently published https://jkcfg.github.io/ https://jkcfg.github.io/, building on lessons learned from many years of configuring Kubernetes. Since we are discussing JS for config management.
- Ameo 8y ago
  That logo is so cool. Extremely simple but elegant.
- hanniabu 8y ago
  Heads up for your friends, the home page is not very legible on mobile and the menu doesn't work (when you click the icon it disappears and leaves no options).
- geezerjay 8y ago
  Love the concept, but the name is far from being the best. There's already a relatively popular tool for JSON processing called jq, whose name in some languages already reads like "jk". https://stedolan.github.io/jq/ https://stedolan.github.io/jq/
  - c_r_w 7y ago
    I had that question and also the "just kidding" question when I heard the name. What can you do, naming things is hard.
mafro 8y ago
Why would I use this over, say, Jinja? Do I really want imperative code in my config files? And if so, why not just use vanilla JS?
- weberc2 8y ago
  Jinja is a text templating language. A valid ninja template may not produce valid JSON/YAML/etc. Sometimes you need more power than a config language gives you, specifically when you want reuse. Helm uses something like jinja, but the template files are often complex and messy (and they are looking into supporting Lua to deal with these problems). CloudFormation built function semantics on top of JSON/YAML twice: once in the form of templates and again in the form of its intrinsic functions. Skycfg uses the starlark language to output YAML, and this is the best approach since it has the right semantics and uses a familiar language (starlark is stripped down Python).
  - fiatjaf 8y ago
    Why not jq, then?
    - danielhlockard 8y ago
      jq is a json parsing tool, no?
      - roganartu 8y ago
        A very powerful one. You could in theory stream multiple files into jq and have it output effectively arbitrary JSON based on their contents since it supports conditionals. I don't think it would be very pretty since it's not designed for that use case, but the jq language is certainly powerful enough to allow it.
      - fiatjaf 8y ago
        It can be used to generate JSON using functions, input data, conditional expressions and anything you want.
deleted 8y ago
[deleted]
antoineMoPa 8y ago
Configuration is nice, but good defaults are better, and this looks like an invitation to make super complex configurations.
cwp 8y ago
This is great. I've been looking for something like this and have been considering nix, Starlark and Dhall. This seems to hit a sweet spot between power and familiarity.
- snthpy 8y ago
  I agree, Jsonnet looks really nice. One thought I had though is that I wish they didn't use the % operator for string interpolation though and rather the more recent f-string syntax. I wonder whether it will be possible to add that or if there is a reason to prefer one over the other in this context?
self_awareness 8y ago
The name might come from a 'sonnet', but all I can see is a 'JSON parsing library written for Microsoft .NET'.
mullikine 8y ago
It's a nice tool to generate json, but then so is jq. I hope to start using it. jsonnet /dev/stdin <<< '{x: 1, y: self.x} + {x: 10}' {"x":10,"y":10}
rococode 8y ago
AllenNLP uses Jsonnet for config files, for anyone ondering what one potential use case is. For example, here's a config for NER with ELMo: https://github.com/allenai/allennlp/blob/master/training_config/ner_elmo.jsonnet https://github.com/allenai/allennlp/blob/master/training_con... I guess the only really jsonnet-specific thing here is std.extVar() call, but generally speaking it's just a more flexible JSON file that doesn't cost much more to use (brainpower or physical resources). I use it to mix and match configs while debugging sometimes as you can do things like if statements and adding dicts together. It's also nice for modularization, you can do things like having a function that returns a dict, so if your config needs two slightly different datareaders you can just pull out all the common parts and call the function twice instead of having two copies of the entire datareader config (with only minor differences).
falaki 8y ago
https://databricks.com/blog/2018/10/12/writing-a-faster-jsonnet-compiler.html https://databricks.com/blog/2018/10/12/writing-a-faster-json...
badtuple 8y ago
There's a real need for JSON templating languages not targeting developers. Tools like Zapier, Customer.io's webhooks, etc. allow integration with arbitrary APIs and have become a standard part of many marketing/sales stacks. These apps use things like Jinja, Mustache, or Liquid which are great for text but not JSON templates. Curly braces as primary delimiters, not allowing trailing commas on final elements, and other features of JSON make generating it a real hassle. Jsonnet is too complex to be that tool, but I'm glad people are working in the space.
- nfoz 8y ago
  What is it about "curly braces as primary delimiters" that you think is a hassle? What would you suggest instead?
  - rmetzler 8y ago
    Curly braces on US Layout keyboards are fine, they just need to be balanced. On keyboards with German layout curly braces are really hard to type and on Mac keyboards with German layout they are not even marked. So I don’t think any programmer can work with such a layout, yet many people do exactly that.
  - badtuple 8y ago
    It is easy to get confused whether a brace is part of the json or part of the mustache/handlebars/etc. { "hello": {"{{place}}":{{{#items}}"{{thing}}":{{n}},{{/items}}}}, "urls": [{{#items}}"{{url}}",{{/items}}] } It's unambigious and can be reasoned through, but it's certainly hard to tell what structure the template is building at a glance. This is not a complex or contrived example. Syntax highlighting, linters, proper indentation, etc. all help...but when talking about non-programmers who aren't used to this it's an incredible hurdle. People are also lazy and error prone, so you can't rely on indentation to make things clear, especially since it's not always clear how you should indent nesting of the template vs nesting of the json datastructure you're trying to build. Most modern templating languages have a config option for changing the delimeter. Setting it to something like (( or << might help differentiate things a bit, but then you're throwing away easy use of examples, documentation, stack overflow questions, and so much more that already exists for that language. That documentation and copy-pastable community help is not something to throw away lightly if you're designing a tool for this audience. It's also worth pointing out that the above template generates subtly incorrect json with trailing commas. In pure mustache it's not possible to get rid of that trailing comma. You have to either change your data (add a "last":true key to the last element of any arrays), or have some kind of post-processing step. All of the existing solutions are workable, they just aren't ideal.
jpgvm 8y ago
All the people suggesting to use JS instead of Jsonnet are completely missing the point. Jsonnet is a functional lazy evaluated language that supports powerful referential concepts. Unfortunately the website sucks at getting this across. JS on the other hand is an imperative programming language. If you say "use JS" you may as well say "use Python", the 2 are equivalent in this context and neither are in the same class as Jsonnet in terms of robustness and simplicity. I didn't get it until I used it but after using it I don't think I would go back to test templating when it comes to dealing with JSON/things that are equivalent. i.e I would use Jsonnet to create JSON to convert to YAML rather than templating YAML directly. Same goes for generating objects directly in JS/Python/jq - yes it can be done, yes sometimes you can write a nice DSL but in most cases Jsonnet is going to allow you to write said DSL faster using functions and it's going to be simpler and less prone to errors.
- kosinus 8y ago
  We tried jsonnet for several months to generate config, but eventually abandoned it because the templates got too hairy. Plus rendering is slow. We ended making the effort to simplify config of all our app’s components to just read from one large yaml file. Far happier with this solution.
  - jkaplowitz 8y ago
    I think databricks recently released an alternative jsonnet compiler/renderer written in Scala that's faster than the standard Google one (including a workaround for the slowness of JVM startup/shutdown) but compatible with it: https://databricks.com/blog/2018/10/12/writing-a-faster-jsonnet-compiler.html https://databricks.com/blog/2018/10/12/writing-a-faster-json... It was discussed on HN though I don't have the link handy. But sure, if you are okay with a single yaml file, that's certainly simpler.
- mruts 8y ago
  JS is a lot closer to Scheme and Lisps than an imperative language. In fact, it used to have a lispy syntax before Brenden Eich was forced to change it to make it more approachable.
yegle 8y ago
Finally a public version of GCL/Borgcfg :-) For people who doesn't know, GCL (Generic/Google Config Language) is a language that uses dynamic scope. You can define a `template` object with a variable defined as `external`. This way you can create similar objects by providing a concrete value for those `external`s at the time of instantiating. The following paper was referenced in Google's borg paper and gives a good overview of the syntax. https://research.tue.nl/en/studentTheses/gcl-viewer https://research.tue.nl/en/studentTheses/gcl-viewer
- ithkuil 8y ago
  (shameless plug) The missing piece to turn it into borgcfg: https://github.com/bitnami/kubecfg https://github.com/bitnami/kubecfg
- akhilcacharya 8y ago
  Maybe it’s because I’m not a top 1% intelligence that I don’t get this but under what circumstances would configurations become complex enough to where this becomes necessary?
  - snlacks 8y ago
    I don't think it's an intelligence thing just what problems one's come across. Generating configs for various deployments where resources change? Generating configs for developer set up? If you start having a lot of environmental variables or private keys it might be more manageable to get them from another secure source.
  - awinder 8y ago
    IMO this would mostly be useful if you had an architecture that evolved over time where you started with flat files for configuration and later moved to a centralized configuration system, many of which have concepts of references. Duplicating configuration or tapping into some of the advanced functions in here isn’t super helpful when you’re purely dealing with flat files, but this becomes helpful if you’re combining it with some synced external state because you can use this to “smooth over” differences in data shape.
wryun 8y ago
Still plugging json-e over jsonnet, since it's valid JSON itself: https://taskcluster.github.io/json-e/ https://taskcluster.github.io/json-e/
okket 8y ago
Previous discussion from 9 months ago: https://news.ycombinator.com/item?id=17363871 https://news.ycombinator.com/item?id=17363871 (55 comments)
zeliard 8y ago
There's also another competing json templating project from google in the works. Doesn't seem to be production ready yet, but it's open sourced already - https://github.com/cuelang/cue https://github.com/cuelang/cue
arianvanp 8y ago
Also check out https://dhall-lang.org https://dhall-lang.org It's GCL but with a strong type system and termination guarantees. I personally like the syntax a bit more than jsonnet. It's a bit less noisy. I'm also a bit biased because I work on https://github.com/dhall-lang/dhall-kubernetes https://github.com/dhall-lang/dhall-kubernetes . We're currently working on a 2.0 release so the current state of the repo is a bit in turmoil. Programmable configuration language like dhall an jsonet are very powerful and I like them way more than for example helm charts, where templating happens on the string level instead of at the data level.
- Trufa 8y ago
  Just my two cents but I would go against the first interaction with the user being find the bug, I didn't didn't find it and missed the point, granted I didn't take much time and I'm on mobile, but I guess that's the reality for many people if not most :)
  - cyphar 8y ago
    The bug is that your SSH keys are in /home/bill/.ssh/. But I agree the first page really don't help sell the idea.
  - nitheshm 8y ago
    The typo is in the name bill in the value of publicKey. I assume this is a way to introduce the user to definitions that help you avoid these kind of mistakes. I think it would be more suitable as a use case description rather than an exercise for the user.
- malkia 8y ago
  Can dhall be compiled for Windows?
  - arianvanp 8y ago
    I don't see why not. Though I haven't tried it myself.
- 8y ago
peterwwillis 8y ago
I have never used a language/tool like this at scale and enjoyed it. For example, embedding a bash script in a templating language. This is such an incredibly crap idea I don't know how it keeps getting perpetuated. Use of this pattern works as follows: user A writes the script, tests it, shoves it into template. user B modifies the template, commits it, it later breaks silently because it was not tested, because it's in a template language, so you can't lint it and it has no tests written for it (because how did you know it would even get generated based on your input/function, let alone all the places it would be used and how, because templates). user C finds the hidden breakage after tearing out hair, removes the script from template, and writes it as a normal script that takes options and writes a test for good measure. (If you try to use normal dev patterns around the templates, like generate-template-then-commit-to-vcs, relying on and testing the generated versions, your tests have to be reworked whenever you regenerate, and later the generator gradually begins to break, so you just modify the copy in VCS rather than using the template) A separate problem is that templating languages that give you advanced functionality lead you to want to use that functionality, and the more you do, the harder it is to maintain over time. It may not be readily apparent that trying to make Ansible configs work declaratively requires an annoying amount of complexity, or writing entire applications that cannot be easily reused and calling them "modules" or "plugins". (Whatever you are trying to template/generate, the only way to know for sure you're doing it validly is to have a function that actually parses and outputs the format according to a spec, because just spitting out a template and then externally validating it becomes very difficult to debug, because you don't know why the template is spitting out the output the way it is. So your templating language eventually becomes a compiler) Since this is a new language, you also have to re-implement everything you normally have for a language, like a plug-in for your IDE, linter, debugger, code analyzer, port the standard libraries to different systems, etc. And of course all new devs have to learn this thing and all its quirks before they can start doing work. An entire ecosystem eventually needs to exist to support its continued use, and if you have problems, you hope this ecosystem continues to flourish so you can get support for this unsupported language tool thing, or you'll be hiring internal devs to do it. Maybe the problem is these start out as tools for developers, and only when they get used for operations do the anti-patterns surface. I've gone through it several times and it always ends up a PITA.
- 8y ago
netingle 8y ago
We’ve been using jsonnet to combine Prometheus alerts and Grafana dashboards into little packages called monitoring Mixins: https://github.com/kubernetes-monitoring/kubernetes-mixin https://github.com/kubernetes-monitoring/kubernetes-mixin A bunch more exist for things like Prometheus, consul, etc, vault, cluster, ceph, memcached etc We’ve also built a super simple package manager for this called jsonnet-bundler (jb for short). And of course we use jsonnet (with the ksonnet lib) at Grafana Labs for all our Kubernetes jobs.
davidgould 8y ago
The Lua language was originally a data templateing language [0], and is still very good for that, but like all such things, it grew. Contrary to Greenspun's Tenth Rule [1], Lua's implementation of Lisp is not at all ad-hoc, informally-specified, bug-ridden, or slow. [0] https://www.lua.org/history.html https://www.lua.org/history.html [1] https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule
h1d 8y ago
When do you need to use a complex configuration system like this? Aren't they supposed to be predefined values?
- perlgeek 8y ago
  https://gocd.io/ https://gocd.io/ allows you store CI/CD pipeline configuration in a git repository. If you have many different components, you really want to factor out common things into templates (like build, deploy), with the option to disable some stages (some components might not have end-to-end tests, for example), plug in some values etc. I've long been on the lookout for a template system for data structures for this use case, so I'll take a look at some of the suggestions in this thread.
gfiorav 8y ago
I needed a templated language recently (couldn’t do with Mustache methods). Looked at jsonnet but it looked quite abandoned. Ended up finding Json-E [0]. Pretty powerful, and give you the option to pass in methods to the context as to expose them. No support for Async methods, but I wrapped around it with some Promises and did the trick. [0] https://github.com/taskcluster/json-e https://github.com/taskcluster/json-e
sheeshkebab 8y ago
Ehh... this feels ugly and dirty (not that js code for generating similar json files is typically any better)
mshahi8210 8y ago
Lua is also good for same purpose.
bboreham 8y ago
I always want to know what the debugging experience is with any new language. Being side-effect free is cool and all but if I can’t actually figure out why it did A and not B then I’m going to get frustrated.
beders 8y ago
If your configuration files are so hard to write that you need to use a templating language that is even harder to write, maybe you need to re-think what a configuration is.
paultopia 8y ago
I would be interested to hear what heavy lispers think of this. The whole mixing code and data together feels very lispy, albeit with a heavier syntax here... is this sort of a step off from s-expressions, or does it occupy a fundamentally different place?
mafro 8y ago
If there's anyone here who has used both dhall and jsonnet in anger, can you comment on your experiences? https://news.ycombinator.com/item?id=17529722 https://news.ycombinator.com/item?id=17529722

201 points by fhrow4484 8y ago

Jsonnet has been around for a while, but it looks like stale. Its stdlib is limited and nonstandard and the whole lazy evaluation is something new developers struggle with. I've been using it at work for 2 years and it's sad that it hasn't moved much. I think it lost a huge opportunity if it had more active and more open for innovation development. I'm sorry to say that, but the lead is busy with stuff and is highly opinionated. Also, Jsonnet is slow. We have a few hundred Jsonnet templates and it takes a very long time to compile, which is a pity given it's written in C (now being slowly and painfully rewritten in Go for whatever reasons).

haxterstockman 8y ago

No thanks I'll just write JavaScript

moltar 8y ago
Exactly my thoughts. I don’t see how it’s better than JS or TypeScript. I can render any object into JSON. This is in fact how I manage Kubernetes manifests. I use this package: https://www.npmjs.com/package/kubernetes-models https://www.npmjs.com/package/kubernetes-models Then I write normal TypeScript code to create Kubernetes objects/resources. Then I render it to JSON and feed to kubectl. The beauty of this solution is that it’s OOP and DRY.
- atombender 8y ago
  You're assuming that all code can be trusted. Jsonnet is much easier to make secure than JS, since it can be statically guaranteed to be safe and side-effect-free (e.g. can't access the file system). This permits evaluating arbitrary untrusted data, which can be a boon to systems like CI servers, databases, or even Kubernetes (think Helm charts).
  - CraftThatBlock 8y ago
    He is producing JSON, just as you said. Not sure you understood him. He has a TS script that outputs JSON, which he is using as config.
    - atombender 8y ago
      The script to generate that JSON is code. Jsonnet also generates JSON. So his script is filling the same role as Jsonnet would. The point stands. He might trust his code, but I was highlighting reasons why this is not true everywhere, and indeed the ability to evaluate untrusted code can open up previously unavailable avenues. For example, storing server-side Jsonnet that can be safely evaluated by a multitenant backend.
rlander 8y ago
This should be the first thing on the website: “why use this and not plain Js”. I read everything and I still don’t understand why.

arthurz 8y ago

We have just had an article hitting nearly the top on not to use JSON for config https://arp242.net/weblog/json_as_configuration_files-_please_dont https://arp242.net/weblog/json_as_configuration_files-_pleas...

fhrow4484 8y ago
That's the reason I posted this, since this is kind of using, without the drawbacks (of not having comments etc) I wonder if folks had had experience using it for kubernetes configs like https://jsonnet.org/articles/kubernetes.html https://jsonnet.org/articles/kubernetes.html
- weberc2 8y ago
  Pretty sure stripe built skycfg for this very purpose, but it uses starlark for the language, which is quite a lot nicer IMHO.
paulddraper 8y ago
Presumably, the author of that would be happy with Jsonnet, which isn't JSON.
Noumenon72 8y ago
I was not convinced by that article but it raised my awareness to the point I was happy to click on this one.

randomcoder1000 8y ago

This looks like a public version of gcl/borgcfg... But with JavaScript mixed in (so even worse, if that was possible)

atombender 8y ago
Jsonnet does not have JavaScript, though. It's JavaScript-like, but not nearly as complex (e.g. no prototypes, classes, etc.).
- randomcoder1000 8y ago
  Yeah, I guess this doesn't use JavaScript expressions so it wouldn't have the weird stringiness and truthiness rules JS has, which is what I was worried about. Still, there are a lot of other issues with BCL.. I think I would rather see a piccolo style solution done with Skylark instead of BCL+json. From reading the website I see it actually was inspired by gcl/bcl for kubecfg, so hopefully they used some lessons learned!

manigandham 8y ago

Considering how easy it is to run Javascript anywhere, even small inline statements interpreted dynamically, I can't think of a single reason to use this instead.

c_r_w 8y ago

Some devs from my company just recently published https://jkcfg.github.io/ https://jkcfg.github.io/, building on lessons learned from many years of configuring Kubernetes. Since we are discussing JS for config management.

Ameo 8y ago
That logo is so cool. Extremely simple but elegant.
hanniabu 8y ago
Heads up for your friends, the home page is not very legible on mobile and the menu doesn't work (when you click the icon it disappears and leaves no options).
geezerjay 8y ago
Love the concept, but the name is far from being the best. There's already a relatively popular tool for JSON processing called jq, whose name in some languages already reads like "jk". https://stedolan.github.io/jq/ https://stedolan.github.io/jq/
- c_r_w 7y ago
  I had that question and also the "just kidding" question when I heard the name. What can you do, naming things is hard.

mafro 8y ago

Why would I use this over, say, Jinja? Do I really want imperative code in my config files? And if so, why not just use vanilla JS?

weberc2 8y ago
Jinja is a text templating language. A valid ninja template may not produce valid JSON/YAML/etc. Sometimes you need more power than a config language gives you, specifically when you want reuse. Helm uses something like jinja, but the template files are often complex and messy (and they are looking into supporting Lua to deal with these problems). CloudFormation built function semantics on top of JSON/YAML twice: once in the form of templates and again in the form of its intrinsic functions. Skycfg uses the starlark language to output YAML, and this is the best approach since it has the right semantics and uses a familiar language (starlark is stripped down Python).
- fiatjaf 8y ago
  Why not jq, then?
  - danielhlockard 8y ago
    jq is a json parsing tool, no?
    - roganartu 8y ago
      A very powerful one. You could in theory stream multiple files into jq and have it output effectively arbitrary JSON based on their contents since it supports conditionals. I don't think it would be very pretty since it's not designed for that use case, but the jq language is certainly powerful enough to allow it.
    - fiatjaf 8y ago
      It can be used to generate JSON using functions, input data, conditional expressions and anything you want.

deleted 8y ago

[deleted]

antoineMoPa 8y ago

Configuration is nice, but good defaults are better, and this looks like an invitation to make super complex configurations.

cwp 8y ago

This is great. I've been looking for something like this and have been considering nix, Starlark and Dhall. This seems to hit a sweet spot between power and familiarity.

snthpy 8y ago
I agree, Jsonnet looks really nice. One thought I had though is that I wish they didn't use the % operator for string interpolation though and rather the more recent f-string syntax. I wonder whether it will be possible to add that or if there is a reason to prefer one over the other in this context?

self_awareness 8y ago

The name might come from a 'sonnet', but all I can see is a 'JSON parsing library written for Microsoft .NET'.

mullikine 8y ago

It's a nice tool to generate json, but then so is jq. I hope to start using it. jsonnet /dev/stdin <<< '{x: 1, y: self.x} + {x: 10}' {"x":10,"y":10}

rococode 8y ago

AllenNLP uses Jsonnet for config files, for anyone ondering what one potential use case is. For example, here's a config for NER with ELMo: https://github.com/allenai/allennlp/blob/master/training_config/ner_elmo.jsonnet https://github.com/allenai/allennlp/blob/master/training_con... I guess the only really jsonnet-specific thing here is std.extVar() call, but generally speaking it's just a more flexible JSON file that doesn't cost much more to use (brainpower or physical resources). I use it to mix and match configs while debugging sometimes as you can do things like if statements and adding dicts together. It's also nice for modularization, you can do things like having a function that returns a dict, so if your config needs two slightly different datareaders you can just pull out all the common parts and call the function twice instead of having two copies of the entire datareader config (with only minor differences).

falaki 8y ago

https://databricks.com/blog/2018/10/12/writing-a-faster-jsonnet-compiler.html https://databricks.com/blog/2018/10/12/writing-a-faster-json...

badtuple 8y ago

There's a real need for JSON templating languages not targeting developers. Tools like Zapier, Customer.io's webhooks, etc. allow integration with arbitrary APIs and have become a standard part of many marketing/sales stacks. These apps use things like Jinja, Mustache, or Liquid which are great for text but not JSON templates. Curly braces as primary delimiters, not allowing trailing commas on final elements, and other features of JSON make generating it a real hassle. Jsonnet is too complex to be that tool, but I'm glad people are working in the space.

nfoz 8y ago
What is it about "curly braces as primary delimiters" that you think is a hassle? What would you suggest instead?
- rmetzler 8y ago
  Curly braces on US Layout keyboards are fine, they just need to be balanced. On keyboards with German layout curly braces are really hard to type and on Mac keyboards with German layout they are not even marked. So I don’t think any programmer can work with such a layout, yet many people do exactly that.
- badtuple 8y ago
  It is easy to get confused whether a brace is part of the json or part of the mustache/handlebars/etc. { "hello": {"{{place}}":{{{#items}}"{{thing}}":{{n}},{{/items}}}}, "urls": [{{#items}}"{{url}}",{{/items}}] } It's unambigious and can be reasoned through, but it's certainly hard to tell what structure the template is building at a glance. This is not a complex or contrived example. Syntax highlighting, linters, proper indentation, etc. all help...but when talking about non-programmers who aren't used to this it's an incredible hurdle. People are also lazy and error prone, so you can't rely on indentation to make things clear, especially since it's not always clear how you should indent nesting of the template vs nesting of the json datastructure you're trying to build. Most modern templating languages have a config option for changing the delimeter. Setting it to something like (( or << might help differentiate things a bit, but then you're throwing away easy use of examples, documentation, stack overflow questions, and so much more that already exists for that language. That documentation and copy-pastable community help is not something to throw away lightly if you're designing a tool for this audience. It's also worth pointing out that the above template generates subtly incorrect json with trailing commas. In pure mustache it's not possible to get rid of that trailing comma. You have to either change your data (add a "last":true key to the last element of any arrays), or have some kind of post-processing step. All of the existing solutions are workable, they just aren't ideal.

jpgvm 8y ago

All the people suggesting to use JS instead of Jsonnet are completely missing the point. Jsonnet is a functional lazy evaluated language that supports powerful referential concepts. Unfortunately the website sucks at getting this across. JS on the other hand is an imperative programming language. If you say "use JS" you may as well say "use Python", the 2 are equivalent in this context and neither are in the same class as Jsonnet in terms of robustness and simplicity. I didn't get it until I used it but after using it I don't think I would go back to test templating when it comes to dealing with JSON/things that are equivalent. i.e I would use Jsonnet to create JSON to convert to YAML rather than templating YAML directly. Same goes for generating objects directly in JS/Python/jq - yes it can be done, yes sometimes you can write a nice DSL but in most cases Jsonnet is going to allow you to write said DSL faster using functions and it's going to be simpler and less prone to errors.

kosinus 8y ago
We tried jsonnet for several months to generate config, but eventually abandoned it because the templates got too hairy. Plus rendering is slow. We ended making the effort to simplify config of all our app’s components to just read from one large yaml file. Far happier with this solution.
- jkaplowitz 8y ago
  I think databricks recently released an alternative jsonnet compiler/renderer written in Scala that's faster than the standard Google one (including a workaround for the slowness of JVM startup/shutdown) but compatible with it: https://databricks.com/blog/2018/10/12/writing-a-faster-jsonnet-compiler.html https://databricks.com/blog/2018/10/12/writing-a-faster-json... It was discussed on HN though I don't have the link handy. But sure, if you are okay with a single yaml file, that's certainly simpler.
mruts 8y ago
JS is a lot closer to Scheme and Lisps than an imperative language. In fact, it used to have a lispy syntax before Brenden Eich was forced to change it to make it more approachable.

yegle 8y ago

Finally a public version of GCL/Borgcfg :-) For people who doesn't know, GCL (Generic/Google Config Language) is a language that uses dynamic scope. You can define a `template` object with a variable defined as `external`. This way you can create similar objects by providing a concrete value for those `external`s at the time of instantiating. The following paper was referenced in Google's borg paper and gives a good overview of the syntax. https://research.tue.nl/en/studentTheses/gcl-viewer https://research.tue.nl/en/studentTheses/gcl-viewer

ithkuil 8y ago
(shameless plug) The missing piece to turn it into borgcfg: https://github.com/bitnami/kubecfg https://github.com/bitnami/kubecfg
akhilcacharya 8y ago
Maybe it’s because I’m not a top 1% intelligence that I don’t get this but under what circumstances would configurations become complex enough to where this becomes necessary?
- snlacks 8y ago
  I don't think it's an intelligence thing just what problems one's come across. Generating configs for various deployments where resources change? Generating configs for developer set up? If you start having a lot of environmental variables or private keys it might be more manageable to get them from another secure source.
- awinder 8y ago
  IMO this would mostly be useful if you had an architecture that evolved over time where you started with flat files for configuration and later moved to a centralized configuration system, many of which have concepts of references. Duplicating configuration or tapping into some of the advanced functions in here isn’t super helpful when you’re purely dealing with flat files, but this becomes helpful if you’re combining it with some synced external state because you can use this to “smooth over” differences in data shape.

wryun 8y ago

Still plugging json-e over jsonnet, since it's valid JSON itself: https://taskcluster.github.io/json-e/ https://taskcluster.github.io/json-e/

okket 8y ago

Previous discussion from 9 months ago: https://news.ycombinator.com/item?id=17363871 https://news.ycombinator.com/item?id=17363871 (55 comments)

zeliard 8y ago

There's also another competing json templating project from google in the works. Doesn't seem to be production ready yet, but it's open sourced already - https://github.com/cuelang/cue https://github.com/cuelang/cue

arianvanp 8y ago

Also check out https://dhall-lang.org https://dhall-lang.org It's GCL but with a strong type system and termination guarantees. I personally like the syntax a bit more than jsonnet. It's a bit less noisy. I'm also a bit biased because I work on https://github.com/dhall-lang/dhall-kubernetes https://github.com/dhall-lang/dhall-kubernetes . We're currently working on a 2.0 release so the current state of the repo is a bit in turmoil. Programmable configuration language like dhall an jsonet are very powerful and I like them way more than for example helm charts, where templating happens on the string level instead of at the data level.

Trufa 8y ago
Just my two cents but I would go against the first interaction with the user being find the bug, I didn't didn't find it and missed the point, granted I didn't take much time and I'm on mobile, but I guess that's the reality for many people if not most :)
- cyphar 8y ago
  The bug is that your SSH keys are in /home/bill/.ssh/. But I agree the first page really don't help sell the idea.
- nitheshm 8y ago
  The typo is in the name bill in the value of publicKey. I assume this is a way to introduce the user to definitions that help you avoid these kind of mistakes. I think it would be more suitable as a use case description rather than an exercise for the user.
malkia 8y ago
Can dhall be compiled for Windows?
- arianvanp 8y ago
  I don't see why not. Though I haven't tried it myself.
8y ago

peterwwillis 8y ago

I have never used a language/tool like this at scale and enjoyed it. For example, embedding a bash script in a templating language. This is such an incredibly crap idea I don't know how it keeps getting perpetuated. Use of this pattern works as follows: user A writes the script, tests it, shoves it into template. user B modifies the template, commits it, it later breaks silently because it was not tested, because it's in a template language, so you can't lint it and it has no tests written for it (because how did you know it would even get generated based on your input/function, let alone all the places it would be used and how, because templates). user C finds the hidden breakage after tearing out hair, removes the script from template, and writes it as a normal script that takes options and writes a test for good measure. (If you try to use normal dev patterns around the templates, like generate-template-then-commit-to-vcs, relying on and testing the generated versions, your tests have to be reworked whenever you regenerate, and later the generator gradually begins to break, so you just modify the copy in VCS rather than using the template) A separate problem is that templating languages that give you advanced functionality lead you to want to use that functionality, and the more you do, the harder it is to maintain over time. It may not be readily apparent that trying to make Ansible configs work declaratively requires an annoying amount of complexity, or writing entire applications that cannot be easily reused and calling them "modules" or "plugins". (Whatever you are trying to template/generate, the only way to know for sure you're doing it validly is to have a function that actually parses and outputs the format according to a spec, because just spitting out a template and then externally validating it becomes very difficult to debug, because you don't know why the template is spitting out the output the way it is. So your templating language eventually becomes a compiler) Since this is a new language, you also have to re-implement everything you normally have for a language, like a plug-in for your IDE, linter, debugger, code analyzer, port the standard libraries to different systems, etc. And of course all new devs have to learn this thing and all its quirks before they can start doing work. An entire ecosystem eventually needs to exist to support its continued use, and if you have problems, you hope this ecosystem continues to flourish so you can get support for this unsupported language tool thing, or you'll be hiring internal devs to do it. Maybe the problem is these start out as tools for developers, and only when they get used for operations do the anti-patterns surface. I've gone through it several times and it always ends up a PITA.

8y ago

netingle 8y ago

We’ve been using jsonnet to combine Prometheus alerts and Grafana dashboards into little packages called monitoring Mixins: https://github.com/kubernetes-monitoring/kubernetes-mixin https://github.com/kubernetes-monitoring/kubernetes-mixin A bunch more exist for things like Prometheus, consul, etc, vault, cluster, ceph, memcached etc We’ve also built a super simple package manager for this called jsonnet-bundler (jb for short). And of course we use jsonnet (with the ksonnet lib) at Grafana Labs for all our Kubernetes jobs.

davidgould 8y ago

The Lua language was originally a data templateing language [0], and is still very good for that, but like all such things, it grew. Contrary to Greenspun's Tenth Rule [1], Lua's implementation of Lisp is not at all ad-hoc, informally-specified, bug-ridden, or slow. [0] https://www.lua.org/history.html https://www.lua.org/history.html [1] https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule

h1d 8y ago

When do you need to use a complex configuration system like this? Aren't they supposed to be predefined values?

perlgeek 8y ago
https://gocd.io/ https://gocd.io/ allows you store CI/CD pipeline configuration in a git repository. If you have many different components, you really want to factor out common things into templates (like build, deploy), with the option to disable some stages (some components might not have end-to-end tests, for example), plug in some values etc. I've long been on the lookout for a template system for data structures for this use case, so I'll take a look at some of the suggestions in this thread.

gfiorav 8y ago

I needed a templated language recently (couldn’t do with Mustache methods). Looked at jsonnet but it looked quite abandoned. Ended up finding Json-E [0]. Pretty powerful, and give you the option to pass in methods to the context as to expose them. No support for Async methods, but I wrapped around it with some Promises and did the trick. [0] https://github.com/taskcluster/json-e https://github.com/taskcluster/json-e

sheeshkebab 8y ago

Ehh... this feels ugly and dirty (not that js code for generating similar json files is typically any better)

mshahi8210 8y ago

Lua is also good for same purpose.

bboreham 8y ago

I always want to know what the debugging experience is with any new language. Being side-effect free is cool and all but if I can’t actually figure out why it did A and not B then I’m going to get frustrated.

beders 8y ago

If your configuration files are so hard to write that you need to use a templating language that is even harder to write, maybe you need to re-think what a configuration is.

paultopia 8y ago

I would be interested to hear what heavy lispers think of this. The whole mixing code and data together feels very lispy, albeit with a heavier syntax here... is this sort of a step off from s-expressions, or does it occupy a fundamentally different place?

mafro 8y ago

If there's anyone here who has used both dhall and jsonnet in anger, can you comment on your experiences? https://news.ycombinator.com/item?id=17529722 https://news.ycombinator.com/item?id=17529722