10 ms·

From my experience, while YAML itself is something one can learn to live with, the true horror starts when people start using text template engines to generate

From my experience, while YAML itself is something one can learn to live with, the true horror starts when people start using text template engines to generate YAML. Like it's done in Helm charts, for example, https://github.com/helm/charts/blob/master/stable/grafana/templates/deployment.yaml https://github.com/helm/charts/blob/master/stable/grafana/te... Aren't these "indent" filters beautiful?

regecks 7y ago
Why would they have chosen to use template/text to generate YAML? That seems insane. Surely using an encoder on an object/structure hierarchy (like people do with encoding/json) is the way to go? On the other hand, the quality of the yaml libraries in Go wasn't great, last time I had to choose a configuration file format.
- dharmab 7y ago
  A lot of people working with YAML have an ops background and aren't familiar with basic data structures.
  - dev_dull 7y ago
    personally I’d prefer a templatized yaml file over an over-engineered, snowflake DSL created by a “real programmer” and not an ops person.
    - celim307 7y ago
      Ok, those aren't the only two choices though.
  - kevml 7y ago
    That’s disingenuous. Most “ops” folks I work with despise templating files but it’s the easiest way to parameterize things, especially when providing ways for “devs” to do “ops”. Yes, we may not have a cleaner way of deploying k8s Deployment configs to different clusters but the desire to templatize YAML is easy for everyone to understand. The decision to abstract or templatize is one rooted in time and cost, not ability to understand data structures.
- Noumenon72 7y ago
  What is "an encoder"? Like a function that takes the same variables as the template would but does some work itself generating things?
  - matt_kantor 7y ago
    An encoder is anything that serializes some data. Think `JSON.stringify()`.
    - sk5t 7y ago
      YMMV, I believe most folks would call that "serialization," reserving "encoding" for turning a notionally written-down-ish representation into bytes; e.g. string -> utf8 bytes, or float -> IEEE-754 bytes.
      - regecks 7y ago
        Right, for some reason in Go (which Helm is written in), the standard library calls them encoders/decoders with marshal/unmarshal as the operations. Serialize is definitely the more common term generally.
      - matt_kantor 7y ago
        I've seen it used even more generally for any `A -> B` (or `A -> Option<B>`) where B could be faithfully decoded back to A. In my head, serialization is a special case of encoding when B is some type of string. In this case it's YAML. I could probably have worded my previous comment more precisely.
- wmil 7y ago
  It probably starts with an existing YAML config file that you only need to pass one or two variables to. Then things get out of hand.
PopeDotNinja 7y ago
> the true horror starts when people start using text template engines to generate YAML I just had a shiver recalling a Kubernetes wrapper wrapper wapper wrapper at a former job. I think there were at least two layers of mystical YAML generation hell. I couldn't stop it, and it tanked much joy in my work. It was a factor in me moving on.
- api 7y ago
  Kubernetes works well but pretty it is not.
- msoad 7y ago
  Oh my god! I'm working on the same wrapper wrapper wrapper
  - kminehart 7y ago
    try moving to something like http://skycfg.fun http://skycfg.fun
  - PopeDotNinja 7y ago
    I called ours The Yamlith. Name yours!
    - DonHopkins 7y ago
      What was the straw that broke the YAML's back?
      - codeduck 7y ago
        get out.
    - BigJono 7y ago
      The Yamdenburg would be apt since it gives you a pretty good indication of how that's going to end.
sethammons 7y ago
k8s and helm is where I learned to dislike yaml. I now want a compiled and type safe language that generates whatever config a system needs. I'm pretty much thinking I want Go as a pre-config where I can set variables, loops, and conditionals and that my editor can help with auto-complete. Maybe I can "import github.com/$org/helmconfig" and in the end write one or more files for config.
- felixschl 7y ago
  You should check out dhall-lang
- halfmatthalfcat 7y ago
  Helm 3 is moving to Lua, that may be better or worse.
  - option_greek 7y ago
    Sounds like another short sighted decision. Why don't they support an intermediatory representation that many languages can support. Even yaml would be fine if other languages can generate it. If they had to absolutely use something why not something more main stream and popular like Python. Helm asks too much for the functionality it provides.
    - silon42 7y ago
      -1 for Turing complete config languages.
      - vardump 7y ago
        Any reason why you think they're bad? Sounds enticing to me to be able to have a bit of logic in configuration file.
        bmn__ 7y ago
        https://news.ycombinator.com/item?id=17186338 https://news.ycombinator.com/item?id=17186338
dastx 7y ago
At my old place we developed a small tool that wraps CloudFormation with a templating language (jinja2). This was actually great as it CloudFormation is extremely verbose and often unnecessarily complex. Templating it out and adding custom functions to jinja2 made the cfn templates much easier to understand. I think it all depends. Most of the time I would agree that you shouldn't template yaml, but sometimes, it's the lesser of two evils.
- zwkrt 7y ago
  Templating CFN is really good practice once you hit a certain scale. If you have 5 DDB tables deployed to multiple regions, and on each of them you want to specify keys, attributes, throughput, and usage alarms, at a minimum. That’s already 30-40 values that need to be specified, depending on table schemas. Add EC2, auto scaling, networking, load balancer, and SQS/SNS—now untemplated cloud formation is really unpleasant to work with. Some of the values like DDB table attributes are common across all regions, other values like tags are common across all infra in the same region. Some values are a scalar multiple of others, or interpolated from multiple sources. For example, a DDB capacity alarm for a given region is a conjunction of the table name (defined at the application level), a scalar multiple of the table capacity (defined at the regional deployment level), and severity (owned by those that will be on-call). To add insult to injury, a stack can only have 60 parameters, which you butt up against quickly if you try to naively parameterize your deployment. Given all these gripes, auto-generating CFN templates was easiest for me. I used a hierarchical config (global > application > region > resource) so the deployment params could be easily manipulated, maintained, and where “exceptions to the rule” would be obvious instead of hidden in a bunch of CFN yaml. To generate CFN templates I used ERB instead of jinja, but to similar effect. A side benefit of this is side-stepping additional vendor lock-in in the form of the weird and archaic CFN operators for math, string concatenation, etc. I don’t have a problem learning them, but it’s one of those things that one person learns, then everyone who comes after them has to re-learn. My shop already uses ruby, so templating in the same language is a no-brainer.
- auslander 7y ago
  > This was actually great as it CloudFormation is extremely verbose and often unnecessarily complex I think its opposite, the most lean way to deploy AWS resources. Did you wrote it yourself, in text editor? I was doing it for 5 years now. You can omit values if you're fine with defaults, you only state what needs to be different. Other tip is use Export and ImportValue to link stacks. I kept on using JSON, even after all my buddies jumped on YAML. JSON is just more reliable, harder to miss syntax errors, and can be made readable by not using linters and keep long lines that belong on one line. Also, the brackets are exactly what they are in Python :) > wraps CloudFormation with a templating language (jinja2) Not sure it it is a good idea. Everyone's use case is different, though. A well written CFN template is like a rubber stamp, just change the Parameters. The template itself doesn't need to change.
xxxpupugo 7y ago
Shoot, it will be a hell to test such monster, say you have a single typo somewhere, lol
cppforlife 7y ago
agreed, text templating of yaml (or any structured content) does not make sense. too much context (actual config structure) is lost if plain text is used. i've collaborated on ytt (https://get-ytt.io https://get-ytt.io) - yaml templating tool. it works directly with yaml structure to bind templating directives. for example setting a value is associated with a specific yaml node so that you dont have to do any manual indenting etc. like you would with plain text templating. defining functions that return yaml structures becomes very easy as well. common problems such as improperly escaped values are gone. i'm also experimenting with a "strict" mode [1] that raises error for questionable yaml features, for example, using NO to mean false. i think that yaml is here to stay (at least for some time) and it's worth investing in making tools that make dealing with yaml and its common uses (templating) easier. [1] https://github.com/k14s/ytt/blob/master/docs/strict.md https://github.com/k14s/ytt/blob/master/docs/strict.md
DonHopkins 7y ago
I developed Yet Another JSON Templating Language, whose main virtue was that it was extremely simple to use and implement, and it could be easily implemented in JavaScript or any other languages supporting JSON. We had joy, we had fun, we had seasons in the sun, but as I added more and more features and syntax to cover specific requirements and uncommon edge cases, I realized I was on an inevitable death-march towards my cute little program becoming sufficiently complicated to trigger Greenspun's tenth rule. https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule There is no need for Yet Another JSON Templating Language, because JavaScript is the ultimate JSON templating language. Why, it even supports comments and trailing commas! Just use the real thing to generate JSON, instead of trying to build yet another ad-hoc, informally-specified, bug-ridden, slow implementation of half of JavaScript.
ljackman 7y ago
Some templating languages such as Jsonnet[0] add built-in templating and just enough programmability to cover basic operations like templating and iteration. I originally felt it was overly complex, but after seeing some of the Go text/template and Ansible Jinja examples in the wild, it actually seems like a good idea. Perhaps we should more strongly distinguish between “basic” data definition formats and ones that need to be templated. JSON5 for the former and Jsonnet for the latter, for example.

regecks 7y ago

Why would they have chosen to use template/text to generate YAML? That seems insane. Surely using an encoder on an object/structure hierarchy (like people do with encoding/json) is the way to go? On the other hand, the quality of the yaml libraries in Go wasn't great, last time I had to choose a configuration file format.

dharmab 7y ago
A lot of people working with YAML have an ops background and aren't familiar with basic data structures.
- dev_dull 7y ago
  personally I’d prefer a templatized yaml file over an over-engineered, snowflake DSL created by a “real programmer” and not an ops person.
  - celim307 7y ago
    Ok, those aren't the only two choices though.
- kevml 7y ago
  That’s disingenuous. Most “ops” folks I work with despise templating files but it’s the easiest way to parameterize things, especially when providing ways for “devs” to do “ops”. Yes, we may not have a cleaner way of deploying k8s Deployment configs to different clusters but the desire to templatize YAML is easy for everyone to understand. The decision to abstract or templatize is one rooted in time and cost, not ability to understand data structures.
Noumenon72 7y ago
What is "an encoder"? Like a function that takes the same variables as the template would but does some work itself generating things?
- matt_kantor 7y ago
  An encoder is anything that serializes some data. Think `JSON.stringify()`.
  - sk5t 7y ago
    YMMV, I believe most folks would call that "serialization," reserving "encoding" for turning a notionally written-down-ish representation into bytes; e.g. string -> utf8 bytes, or float -> IEEE-754 bytes.
    - regecks 7y ago
      Right, for some reason in Go (which Helm is written in), the standard library calls them encoders/decoders with marshal/unmarshal as the operations. Serialize is definitely the more common term generally.
    - matt_kantor 7y ago
      I've seen it used even more generally for any `A -> B` (or `A -> Option<B>`) where B could be faithfully decoded back to A. In my head, serialization is a special case of encoding when B is some type of string. In this case it's YAML. I could probably have worded my previous comment more precisely.
wmil 7y ago
It probably starts with an existing YAML config file that you only need to pass one or two variables to. Then things get out of hand.

PopeDotNinja 7y ago

> the true horror starts when people start using text template engines to generate YAML I just had a shiver recalling a Kubernetes wrapper wrapper wapper wrapper at a former job. I think there were at least two layers of mystical YAML generation hell. I couldn't stop it, and it tanked much joy in my work. It was a factor in me moving on.

api 7y ago
Kubernetes works well but pretty it is not.
msoad 7y ago
Oh my god! I'm working on the same wrapper wrapper wrapper
- kminehart 7y ago
  try moving to something like http://skycfg.fun http://skycfg.fun
- PopeDotNinja 7y ago
  I called ours The Yamlith. Name yours!
  - DonHopkins 7y ago
    What was the straw that broke the YAML's back?
    - codeduck 7y ago
      get out.
  - BigJono 7y ago
    The Yamdenburg would be apt since it gives you a pretty good indication of how that's going to end.

sethammons 7y ago

k8s and helm is where I learned to dislike yaml. I now want a compiled and type safe language that generates whatever config a system needs. I'm pretty much thinking I want Go as a pre-config where I can set variables, loops, and conditionals and that my editor can help with auto-complete. Maybe I can "import github.com/$org/helmconfig" and in the end write one or more files for config.

felixschl 7y ago
You should check out dhall-lang
halfmatthalfcat 7y ago
Helm 3 is moving to Lua, that may be better or worse.
- option_greek 7y ago
  Sounds like another short sighted decision. Why don't they support an intermediatory representation that many languages can support. Even yaml would be fine if other languages can generate it. If they had to absolutely use something why not something more main stream and popular like Python. Helm asks too much for the functionality it provides.
  - silon42 7y ago
    -1 for Turing complete config languages.
    - vardump 7y ago
      Any reason why you think they're bad? Sounds enticing to me to be able to have a bit of logic in configuration file.
      - bmn__ 7y ago
        https://news.ycombinator.com/item?id=17186338 https://news.ycombinator.com/item?id=17186338

dastx 7y ago

At my old place we developed a small tool that wraps CloudFormation with a templating language (jinja2). This was actually great as it CloudFormation is extremely verbose and often unnecessarily complex. Templating it out and adding custom functions to jinja2 made the cfn templates much easier to understand. I think it all depends. Most of the time I would agree that you shouldn't template yaml, but sometimes, it's the lesser of two evils.

zwkrt 7y ago
Templating CFN is really good practice once you hit a certain scale. If you have 5 DDB tables deployed to multiple regions, and on each of them you want to specify keys, attributes, throughput, and usage alarms, at a minimum. That’s already 30-40 values that need to be specified, depending on table schemas. Add EC2, auto scaling, networking, load balancer, and SQS/SNS—now untemplated cloud formation is really unpleasant to work with. Some of the values like DDB table attributes are common across all regions, other values like tags are common across all infra in the same region. Some values are a scalar multiple of others, or interpolated from multiple sources. For example, a DDB capacity alarm for a given region is a conjunction of the table name (defined at the application level), a scalar multiple of the table capacity (defined at the regional deployment level), and severity (owned by those that will be on-call). To add insult to injury, a stack can only have 60 parameters, which you butt up against quickly if you try to naively parameterize your deployment. Given all these gripes, auto-generating CFN templates was easiest for me. I used a hierarchical config (global > application > region > resource) so the deployment params could be easily manipulated, maintained, and where “exceptions to the rule” would be obvious instead of hidden in a bunch of CFN yaml. To generate CFN templates I used ERB instead of jinja, but to similar effect. A side benefit of this is side-stepping additional vendor lock-in in the form of the weird and archaic CFN operators for math, string concatenation, etc. I don’t have a problem learning them, but it’s one of those things that one person learns, then everyone who comes after them has to re-learn. My shop already uses ruby, so templating in the same language is a no-brainer.
auslander 7y ago
> This was actually great as it CloudFormation is extremely verbose and often unnecessarily complex I think its opposite, the most lean way to deploy AWS resources. Did you wrote it yourself, in text editor? I was doing it for 5 years now. You can omit values if you're fine with defaults, you only state what needs to be different. Other tip is use Export and ImportValue to link stacks. I kept on using JSON, even after all my buddies jumped on YAML. JSON is just more reliable, harder to miss syntax errors, and can be made readable by not using linters and keep long lines that belong on one line. Also, the brackets are exactly what they are in Python :) > wraps CloudFormation with a templating language (jinja2) Not sure it it is a good idea. Everyone's use case is different, though. A well written CFN template is like a rubber stamp, just change the Parameters. The template itself doesn't need to change.

xxxpupugo 7y ago

Shoot, it will be a hell to test such monster, say you have a single typo somewhere, lol

cppforlife 7y ago

agreed, text templating of yaml (or any structured content) does not make sense. too much context (actual config structure) is lost if plain text is used. i've collaborated on ytt (https://get-ytt.io https://get-ytt.io) - yaml templating tool. it works directly with yaml structure to bind templating directives. for example setting a value is associated with a specific yaml node so that you dont have to do any manual indenting etc. like you would with plain text templating. defining functions that return yaml structures becomes very easy as well. common problems such as improperly escaped values are gone. i'm also experimenting with a "strict" mode [1] that raises error for questionable yaml features, for example, using NO to mean false. i think that yaml is here to stay (at least for some time) and it's worth investing in making tools that make dealing with yaml and its common uses (templating) easier. [1] https://github.com/k14s/ytt/blob/master/docs/strict.md https://github.com/k14s/ytt/blob/master/docs/strict.md

DonHopkins 7y ago

I developed Yet Another JSON Templating Language, whose main virtue was that it was extremely simple to use and implement, and it could be easily implemented in JavaScript or any other languages supporting JSON. We had joy, we had fun, we had seasons in the sun, but as I added more and more features and syntax to cover specific requirements and uncommon edge cases, I realized I was on an inevitable death-march towards my cute little program becoming sufficiently complicated to trigger Greenspun's tenth rule. https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule There is no need for Yet Another JSON Templating Language, because JavaScript is the ultimate JSON templating language. Why, it even supports comments and trailing commas! Just use the real thing to generate JSON, instead of trying to build yet another ad-hoc, informally-specified, bug-ridden, slow implementation of half of JavaScript.

ljackman 7y ago

Some templating languages such as Jsonnet[0] add built-in templating and just enough programmability to cover basic operations like templating and iteration. I originally felt it was overly complex, but after seeing some of the Go text/template and Ansible Jinja examples in the wild, it actually seems like a good idea. Perhaps we should more strongly distinguish between “basic” data definition formats and ones that need to be templated. JSON5 for the former and Jsonnet for the latter, for example.