27 ms·

Why are we templating YAML?

• MrStonedOne 8y ago
  The real question is why are we using yaml at all?
  • quickthrower2 8y ago
    Baffles me. I don't like any language or file format where whitespace matters. Even Haskell bothers me in this regard. To me white space shouldn't add cognitive load. I want to look at the symbols not the formatting of the antisymbols to understand what is going on. Write JSON and use your editor tools to format it with nice indentation, and you are sweet! That said Yaml makes an excellent format for reading, but not for writing.
    • amanzi 8y ago
      White space matters with Python but I rarely run into issues with indentation in Python. But YAML on the other hand, I have nothing but nightmares.
      • kccqzy 8y ago
        I have the same experience. I just think think it is possible to design easy-to-write indentation-sensitive formats but YAML is not. For example is always baffles me that a: - b - c has a list inside an object but the list is not further indented. There's in fact a hierarchy relationship but absolutely no indentation.
        • Rapzid 8y ago
          You can indent it though, but the hyphen already means a list item. That is a preferred syntax for many that are used to YAML though.
          • kccqzy 8y ago
            That's kind of my point. Indentation is optional, so many don't indent, making the resulting document harder to understand.
          • emmelaich 8y ago
            FWIW, I think yaml / ansible linters and style checkers typically require indentation.
          • XorNot 8y ago
            The hyphen is associated with the a though. In what's posted, a is a list, not a map - the type of the parent is being declared by that hyphen.
        • mikepurvis 8y ago
          That's at your option, though. It's perfectly valid yaml to indent the list as many spaces as you want.
      • setr 8y ago
        I think its because python has very strict and very simple indentation rules (a nested block must be indented, and the file requires consistent indentation. Is there anything else?) YAML gives you options, or varies necessity somewhat arbitrarily on the structures, which is (marginally) good for reading, but a lot of headache in writing
        • amanzi 8y ago
          Also because the longer the YAML file, the bigger your indentation gets. It gets exponentially worse as you go.
          • setr 8y ago
            In python that’s been a non-issue, but I’m not sure YAML can reset indentation as python does with functions
    • wolph 8y ago
      I personally love Python where whitespace matters a lot too. But I regularly find myself fighting with the whitespace in yaml for some reason. To me yaml (especially with templating) seems like a very contrived way of avoiding having to program, while still effectively programming. I much prefer json, or if more intelligence is required, actual javascript objects. The only big downside of json where yaml does shine is the support for comments within your files.
    • crdoconnor 8y ago
      I never quite understood this attitude. Not treating whitespace as meaningful means it's necessary to have a longer, syntactically noisier file. That's fine if you're primarily concerned with computers exchanging data (JSON) but where readability and writability matters, extra syntactic weight is a headache. I can't imagine writing JSON by hand but I write plenty of YAML by hand every day without issues.
      • quickthrower2 8y ago
        Because deleting a tab could entirely change the meaning of the file, breaking everything, with no compile time errors. IDE tooling or linting might go somewhat to make me more comfortable with YAML.
    • emmelaich 8y ago
      > I don't like any language or file format where whitespace matters. .... To me white space shouldn't add cognitive load. 1. there is no language where whitespace does not matter 2. the very purpose of indentation is to ease cognitive load. Now, you may not want it to be inflexible or mandatory but your argument needs elucidating.
      • quickthrower2 8y ago
        Ok: I don't like any language of file format where the regex s/\s+/\s/g would change the meaning of the program except in parts of the program within string delimiters, except for the parts of string delimiters which contain literal code e.g. ins ES6 `Hello ${world}`.
        • emmelaich 8y ago
          OK, but I'd restrict the set even more to exclude voluntary indentation. I mean, you'd want that caught by style checkers anyway, even if the language didn't require it. So we're left with the cases where you'd like compact code / one-liners but are forced to use multiple lines for language syntax reasons. I'd say that that's a pretty small set of cases and long way from the phrase "I don't like whitespace sensitive languages" which you (and many others!) use. I conceded it can be annoying in Python and I do wish there was an escape route sometimes.
        • andybak 8y ago
          So you prefer languages way that regex only appears to change the meaning? Non-semantic white space can be deceptive. Semantic whitespace isn't.
    • h1d 8y ago
      Experienced programmers use decent editors to indent their code anyway but at least you don't see codes with indentation going left and right at random locations with Python.
  • nightfly 8y ago
    I've taken the strategy of emitting JSON, but accepting YAML input (maybe a restricted subset in cases where it's untrusted data). YAML can function as a super-set of JSON, so you can have comments in hand-written/modified data this way while emitting a simpler to parse data format.
  • mixmastamyk 8y ago
    It's a lot easier for end-users to read and write by hand. If that is not a requirement, a simpler machine-friendly format is likely better.
    • amanzi 8y ago
      Writing YAML is not easy for end-users. The indentation--especially in large files--is really difficult to keep track of, and some really basic stuff like when to quote strings is not consistent at all.
      • mixmastamyk 8y ago
        Nope, indentation is not a problem with indentation guides. A misplaced comma in a JSON file however kills it, gets me every time I have to write it by hand. Also, it is true YAML has too many features, though I find they are typically ignored or disabled.
        • krapp 8y ago
          A misplaced comma is also not a problem in an editor with syntax highlighting, but between the two, indentation could take longer to fix.
          • mixmastamyk 8y ago
            Syntax highlighting doesn't count commas. The problem is JSON doesn't allow trailing commas, so adding or reordering frequently results in a difficult to diagnose error.
            • krapp 8y ago
              Finding out if a set ends in a comma or not still makes it easier to debug than YAML.
              • mixmastamyk 8y ago
  • IshKebab 8y ago
    I think it is partly momentum - at this point most CI systems use YAML in spite of its insanity - even Azure Pipelines! And partly it's because the syntax for multiline string literals is very minimal, which is kind of a nice feature for CI since you tend to have a lot of them. It's still insane though. TOML is much more reasonable.
  • Rapzid 8y ago
    The best thing that ever happened to Cloud Formation was the new YAML syntax. Writing YAML is easy with a good editor like VSCode. Install the YAML code outline extension for sugar; works great for OpenAPI specs. YAML flow style offers some good options for keeping the file compact.
    • SamWhited 8y ago
      I'm all for good editor support, but you shouldn't need it to write a simple document. Complicated editors should be supporting tools at best, because you won't always be in a position to use one. The things we write should be simple enough to be written and understood by hand with minimal mistakes (then we can make it even easier in some cases using nice editors and tooling).
      • shirakawasuna 8y ago
        I don't think I've ever used anything under than vanilla vim for editing YAML files... with next to zero issues. The problem for me is that the multiple ways to do the same thing result in something that's pretty opaque to clear specifications that aren't just very simple examples / structures.
        • SamWhited 8y ago
          Me too, except it's just always wrong the first time… drives me nuts. Maybe it's just me.
  • atsjie 8y ago
    Anyone working with cloudformation would wonder why we ended up writing json?
  • jszymborski 8y ago
    I really like TOML as a compromise between "looks pretty" and "parses well". That being said, I find it funny that the author is crying afoul, saying that YAML isn't as well suited to be templated as JSON is, when XML had schemas sorted out ages ago. People ran away from XML, saying it was this verbose, ugly, bandwidth hungry (back when we were sending XML over AJAX) behemoth (which it totally is)... but I think when your usecase is complicated enough to worry about templating, you should take a hard look at XML and ask whether it might suit your better.
  • deathanatos 8y ago
    Easier to read and write than JSON. JSON requires constant quoting, can't support multiline strings, has no comments, has no/little typing (e.g., no datetime type). It's not good if a human needs to encode data. For configs, I think TOML beats YAML hands down; I think YAML's spot is at encoding data structures that humans need to read/write. I do agree that YAML, the spec, is fairly complicated. But YAML, as used in most projects, by most people, is not, and can be picked up fairly quickly. It is easier to visually read as it removes much of the clutter that would exist in the comparable JSON. It isn't typically necessary to know the entirety of the YAML spec to be useful with YAML, and most of the parts you won't know will get introduced by an obvious-looking sigil, which can be used to figure out what you're dealing with. When I've actually sat with folks struggling with YAML, it's almost always in configuration tools, and it's also always around the templating bits. Ansible, in particular, has a bizarre templating: it happens after YAML parsing, which is not the mental model most people use when approaching it. I've also found that most of the people I've spoken to intertwine YAML and Ansible's templating functions, thinking they're one in the same. I do not think Ansible makes good use of YAML: I would rather write task files in an actual programming language, since they are — at their core — a program. (The tasks do have some metadata attached to them, but the core task itself is a program. A function in some real language can get metadata attached to it in a number of ways, and that would be a better solution.)
    • guitarbill 8y ago
      It's worth giving Ansible some credit here. Alternatives like Chef, can and do contain arbitrary Ruby logic which can make Chef "cookbooks" horrible to figure out and debug. So while Ansible may not be great, other solutions also have drawbacks, and it isn't quite as easy as do <x>.
      • rbankston 8y ago
        I've worked with both. The simple items are nicer in ansible but when there are a lot of custom roles and functions ansible isn't much better than chef from what I've seen in practice.
    • pytester 8y ago
  • pavon 8y ago
    Because json is poorly suited for config files, and it is very convenient to have config files that closely mirror apis, like kubernetes does. I've seen too many teams who try to use json for config end up developing yet another custom superset of json to allow comments or multiline strings or such. If you are going to use a superset of json might as well use an existing one like yaml or json5. That said I agree with all the criticisms of templating yaml. We have to do the same (with helm and other tools), and I have pushed hard to adopt conventions that we only use flow-style and not block-style to avoid all the white space problems when splicing together chucks of yaml. And on the plus side we get trailing commas and other such niceties which don't exist in json and make it harder template.
• CryoLogic 8y ago
  I think they are missing the real selling point of JSON. It's basically interoperable with JavaScript objects. That means you write it, send it, store it, operate on it, etc. with little or no modification. The author says "converting between the two is trivial" which may be true, but the developer overhead is less trivial. And it will always be JSON in the client - JS doesn't support YAML objects.
  • altmind 8y ago
    Except that you cannot encode pretty valid Double.NaN or Date(), which is showstopper for many. I remember mongodb started with json, but switched to in-house bson pretty early because of the json limitations.
    • svnpenn 8y ago
      new Date().toJSON()
      • altmind 8y ago
        When you parse back the result of this, you are not getting Date back.
    • CryoLogic 8y ago
      Is there anything wrong with just using Date.now() to get a UNIX timestamp to store and then converting it for presentation on the client?
  • deleted 8y ago
    [deleted]
  • ergothus 8y ago
    > real selling point of JSON Is that really the selling point? I interact with numerous services that speak JSON...and most of them aren't written in JS. YAML and JSON both have to be parsed to be used, even by JS. Otherwise it's just a string. To me the real selling point in JSON is the dead-simplicity for humans to read and CHANGE. YAML is human-READABLE, but frankly I often screw up changing it because the formatting is a little too magical and I'm a little too unfamiliar. JSON is downright picky and obnoxious...but that makes it really easy to make a change. Screw up the quotes? It'll complain about the quote. Dangling comma? It will complain. Did I cut-and-paste from an HTML display that screwed up my whitespace by compressing everything down to a single space? Nothing cares.
• the8472 8y ago
  Helm reminds me of the 90s level of webtech. I.e. php cgi files mixing html, logic and includes.
  • jrockway 8y ago
    I believe people are back to doing that for web stuff. Open up any modern web stuff and it looks like "const widget = <div>foo</div>". Not making this up! https://reactjs.org/docs/introducing-jsx.html https://reactjs.org/docs/introducing-jsx.html
    • dmix 8y ago
      The difference, at least for Vue single file template components is that it's 3 separate areas in one file: - <template /> (Templated HTML) - <script /> (JS OR Typescript) - <style /> (CSS OR SASS/etc) Whereas in old PHP files you could mix it in anywhere and the files were a big mess. Including inline SQL into your view templates which is hardly a good separation of concerns. While a Vue component can be separated into separate files, at least as one it all represents one isolated piece of the interface. https://flaviocopes.com/vue-single-file-components/ https://flaviocopes.com/vue-single-file-components/
    • h1d 8y ago
      So, are you saying those PHP programmers who mix HTML right into code wasn't wrong after all?
    • AlphaSite 8y ago
      I’m not sure, but isn’t the paradigm flipped, markup in code rather than the reverse.
• rhacker 8y ago
  LOL we are doing the same with k8s. We deploy to any environment, where each environment is a different k8s namespace. We even have a namespace for each developer. The variables are things like the image name (the tag is effectively a git commit id, or sometimes different for not-committed yet stuff). Beyond that just about nothing really needs to be a variable, but we do have different RAM amounts. We have a couple ways that we template this out, but mostly we literally just do this in bash: sed -e "s/\$CI_COMMIT_SHA/$CI_COMMIT_SHA/" kube-deploy.template.yaml | kubectl -n $ENV apply -f - (Where CI_COMMIT_SHA comes from gitlab) ENV comes from our gitlab CI file. That all being said, the extent of our k8s integration is lots of stuff like that. We could write a JS file that creates a JSON k8s template, but honestly, that would be more work and more learning than we had to for what we're doing. Why would we do more just because we want to avoid templating in a YAML file?
• taypo 8y ago
  because people are trying to use configuration files for the things that programming languages are made for. it's masturbation.
• aetherlord 8y ago
  I didn't see this mentioned anywhere else, so another alternative (that I've seen and really like conceptually, but haven't used so far) to all this wildness with YAML and JSON -> https://github.com/dhall-lang/dhall-lang https://github.com/dhall-lang/dhall-lang, and for kubernetes specifically -> https://github.com/dhall-lang/dhall-kubernetes https://github.com/dhall-lang/dhall-kubernetes
  • svnpenn 8y ago
    god those examples are ugly - commas at the beginning of a line? mismatched brace styles?
    • necubi 8y ago
      It's a haskell thing. The main advantage is that each line is independent. You can comment out a line or add a line at the end without modifying anything else.
      • basil-rash 8y ago
        Each line is not independent: you cannot comment out the first line. A better approach is to allow trailing commas. (I suppose you could allow leading commas, does Haskell support this?)
        • nh2 8y ago
          Unfortunately no. Allowing trailing commas, like Python does, would be really great. Unfortunately trailing commas already mean something: (a,b,) is a function that still takes 1 argument to make a triple. It's called "TupleSections".
    • chriswarbo 8y ago
      Not sure what you mean by "mismatched brace styles". The convention of putting separators like commas at the start of the following line rather than the end of the preceding line is common in Haskell, which Dhall is built with. The advantages are: - All of the separators are in the same column, along with the opening and closing characters. This makes it trivial to check if we've missed a separator. - Appending new lines to the end will not affect previous lines (i.e. we don't need to go and add a comma). This avoids making mistakes and polluting diffs. Unfortunately the error-prone diff pollution we avoid at the last line instead occurs at the first line. It's still less error-prone than trailing commas, since we can look in the separator column and either spot that it's empty, or that it contains two opening braces (depending on whether we inserted or copy/pasted).
• fredsted 8y ago
  Helm charts are great, and accomplishes the task, but things get really weird and complex if you're not careful. Oh man, the indentation and _helper.tpl nightmares. It's not obvious at all. And there's still a lot of repetition for each chart; writing a deployment or whatever is so verbose in K8s. On the other hand, Ansible uses yaml and there it works great. I feel like Ansible uses yaml in a way that's easier to understand and the way it was meant to be written. With Ansible, you're writing configuration, not templates of configuration. I don't think a layer on top of Ansible like Helm is for K8s wouldn't make sense.
• nickjj 8y ago
  Who knows. It makes me throw up a little in my mouth every time I see hundreds of lines of YAML to configure something like Traefik with Kubernetes. The worst is when people say they prefer that because "I don't have to write a config file for my backend". That's true but instead now you have extremely verbose configuration mixed in with other verbose configuration. But in YAML's defense I think it's more of a problem with the tools that use it more so than YAML itself. Ansible is a great example of how amazing YAML can be to manage complex configuration in a concise way.
  • gh02t 8y ago
    I agree that simple YAML can be nice as a quick and clean tool, but Ansible is an example of everything wrong with how YAML is used. Layering program flow constructs like loops, variables, templates, references etc. are exactly the sort of abuses that make YAML feel awkward.
• BossingAround 8y ago
  I know I'm in a minority, but I really dislike YAML... I recently did a lot of Ansible and boy, at the beginning, I was just struggling a lot. Syntactic whitespace kills me. I don't like it in Python either, but for some reason, when I write Python, it's a lot easier. Maybe YAML is just a bit more complex (and Python has better IDE support..?)
  • nerdponx 8y ago
    The simple parts of YAML are great for simple applications. The complicated parts of YAML are not great for complicated applications.
  • bluedino 8y ago
    >> I really dislike YAML I wasn’t aware anyone liked it
    • SamWhited 8y ago
      Tools keep using it even where it's the wrong tool for the job, so someone must like it, surely? I'm sure it will be just like XML where it's the trendy thing for a while in the early days, then everyone stops and hates it for a while. Except XML at least has a handful of applications where it's the right tool for the job (it has a nice streaming mode), YAML doesn't even have that.
      • hazz99 8y ago
        YAML is great for simple human-editable configuration files. Its very easy to write, and can be picked up quite quickly. Opting for YAML over XML/JSON/whatever doesn't make me a tool. It made life much easier for myself & my colleagues.
        • kjeetgill 8y ago
          I think they meant tools like docker or npm and stuff. Not calling people tools.
        • 8y ago
• SamWhited 8y ago
  As others in this thread have said: I ask this question all the time, except s/templating/using/. YAML is insanely over complicated; it's as bad or worse than XML for config files, and it doesn't even have the nice streaming mode. Not to mention that it's a bit of a security nightmare (seriously, who put pointers into the YAML spec?). And, on a more subjective note, YAML is just confusing: between all the significant whitespace and the random single character symbols that no one ever remembers what they do, I never get a YAML document right on the first try. Templating it really does add a whole new level of headache too.
  • macspoofing 8y ago
    >it's as bad or worse than XML for config files XML works very well for config files. It's schema-optional (but is there), well-specified, human-readable, has plethora of supporting technologies (making things like templating easy), and is well supported by every language. At the very least it is way better than JSON.
    • Macha 8y ago
      It's missing one important part for config files. It's tedious to write by hand.
      • titanix2 8y ago
        XML is not tedious at all with the right tooling. For example a tool like Visual Studio IntelliSense proposes only elements and attributes valid in the context, automatically close tag, format the file and complete opening tags too so it makes editing XML file a breath.
        • MrStonedOne 8y ago
          The tooling bar for config files is set firmly at notepad.exe If your config file requires more tooling than that you fucked up.
• Animats 8y ago
  Next, inheritance for JSON! Look how well it worked for CSS.
• miohtama 8y ago
  If you have been around long enough you still remember the world that was excited about XML and templating it using XSLT. As a hindsight it was a horrible world. Even though YAML is not optimal, it is a human friendly compromise between too verbose XML and machine only JSON. It lacks native templating, leading to funny constructs e.g. with Ansible files. However human kind has made progress and will make progress further, so it is just a matter of time until someone comes up with sane "native templated YAML" and all projects will adopt it.
  • chriswarbo 8y ago
    > If you have been around long enough you still remember the world that was excited about XML and templating it using XSLT. As a hindsight it was a horrible world. I actually really like the idea behind XSLT: machine-friendly, human-tolerable, structured data + declarative rules for turning that data into a display, or a report, or whatever else. The execution was horrible though: incredibly verbose, lots of overcomplication due to XML weirdness/asymmetries (e.g. attributes vs elements vs text, namespaces, ...); mixtures of different languages hidden inside each other (e.g. XPath hidden in attributes); etc. I would really like to see what this could look like if done in a more minimalist, lispy fashion (normal code-is-data stuff in Lisp is similar, but I think term-rewriting is a more appropriate evaluation mechanism for such rules)
    • pytester 8y ago
      XSLT was one in a litany of domain specific languages (ant, apache rewrite rules, latex macros, etc.) that evolved towards turing completeness because that's what the problem space demanded. In most if not all of these cases an existing and well designed turing complete programming language would likely have better served them.
      • astine 8y ago
        I don't see a reason that a DSL can't be Turing complete and still a better option than an existing language. If you look at old-school make files, they are little more than shell script with top level rules that you can invoke from the command line. You could theoretically just use shell script but the make scripts still simplify the task quite a bit.
• illumin8 8y ago
  His main argument is that you need to template differently for different environments (dev/stage/prod) and cloud regions (us-west/us-east/emea/apac). This is actually a solved problem, and you shouldn't be doing it in your YAML/JSON templates. You should be using an external parameter store to do this, and using a single template for everything. See https://aws.amazon.com/blogs/compute/query-for-the-latest-amazon-linux-ami-ids-using-aws-systems-manager-parameter-store/ https://aws.amazon.com/blogs/compute/query-for-the-latest-am... This is a simple use case: I want to deploy the latest AMI (Amazon Machine Image) in any region, so I always get the latest patched Linux base image to run my application on. I don't (and shouldn't) want to update my YAML/JSON every time a new image is published. So, why are people having to go to these crazy templating macro lengths? Just store the changing bits in an external config/parameter store like etcd and let your infrastructure as code templates remain unchanged.
  • jaxxstorm 8y ago
    > It's a solved problem For who, exactly?
    • illumin8 8y ago
      For anyone using EC2 Parameter Store, a free service in AWS: > Parameters: > AMI: > Type: AWS::SSM::Parameter::Value<String> > Default: /aws/service/ecs/optimized-ami/amazon-linux/recommended/image_id Gives you the latest, fully patched Linux OS in any of 19 regions that you launch it in. Free K/V store that works really well for apps and custom parameters.
      • jaxxstorm 8y ago
        Well, that’s great for anyone using the EC2 parameter store, but isn’t really helpful for all those situations where you’re not. “This is not a problem for my specific set of use cases around AWS” is not a “solved problem”
  • jrockway 8y ago
    Based on the appearance of Helm, this is probably referring to Kubernetes, which really does have per-environment things that can't be represented in any form other than namespaces. For example, you have an app running (that's a deployment in k8speak). You want network traffic to get to this app, so you set up a load balancer. The load balancer config needs to know the name of your app (or more precisely, a selector for "pods" that are created by the deployment), which will change between environments, so now there is that common variable that has to be updated in both places. That's a simple example but is why people are templating their k8s configs. Yes, you could work around it by giving every environment its own namespace and using the same set of objects in every environment (differing only by namespace, which should probably be in the file... but since you can't edit the files, you can pass it in to kubectl probably), but there are other cases where even that doesn't work. You do need some way of saying "this is the base configuration and this is what we change for staging and production". Helm is a way to do that, and a popular one, but it's pretty ugly. Hence this article.
• joeduffy 8y ago
  My belief is that we've been slowly building up to using general purpose languages, one small step at a time, throughout the infrastructure as code, DevOps, and SRE journeys this past 10 years. INI files, XML, JSON, and YAML aren't sufficiently expressive -- lacking for loops, conditionals, variable references, and any sort of abstraction -- so, of course, we add templates to it. But as the author (IMHO rightfully) points out, we just end up with a funky, poor approximation of a language. I think this approach is a byproduct of thinking about infrastructure and configuration -- and the cloud generally -- as an "afterthought," not a core part of an application's infrastructure. Containers, Kubernetes, serverless, and more hosted services all change this, and Chef, Puppet, and others laid the groundwork to think differently about what the future looks like. More developers today than ever before need to think about how to build and configure cloud software. We started the Pulumi project to solve this very problem, so I'm admittedly biased, and I hope you forgive the plug -- I only mention it here because I think it contributes to the discussion. Our approach is to simply use general purpose languages like TypeScript, Python, and Go, while still having infrastructure as code. An important thing to realize is that infrastructure as code is based on the idea of a goal state. Using a full blown language to generate that goal state generally doesn't threaten the repeatability, determinism, or robustness of the solution, provided you've got an engine handling state management, diffing, resource CRUD, and so on. We've been able to apply this universally across AWS, Azure, GCP, and Kubernetes, often mixing their configuration in the same program. Again, I'm biased and want to admit that, however if you're sick of YAML, it's definitely worth checking out. We'd love your feedback: - Project website: https://pulumi.io/ https://pulumi.io/ - All open source on GitHub: https://github.com/pulumi/pulumi https://github.com/pulumi/pulumi - Example of abstractions: https://blog.pulumi.com/the-fastest-path-to-deploying-kubernetes-on-aws-with-eks-and-pulumi https://blog.pulumi.com/the-fastest-path-to-deploying-kubern... - Example of serverless as event handlers: https://blog.pulumi.com/lambdas-as-lambdas-the-magic-of-simple-serverless-functions https://blog.pulumi.com/lambdas-as-lambdas-the-magic-of-simp... Pulumi may not be the solution for everyone, but I'm fairly optimistic that this is where we're all heading. Joe
  • Waterluvian 8y ago
    In ROS we have these XML launch files that are just awful. They have enough features to be a really bad programming language for configuring and launching (often conditionally) numerous robot software nodes. In ROS2 the launchfile can now just be a Python script. Very much learned all this the hard way and the solution was to just support Python. I think it's brilliant.
• epage 8y ago
  There have been times I've wanted to templatize my configuration but I don't want to do it with text-based templates but templates within the configuration files syntax (be it yaml, toml, or something else). Not sure what this is called, I've been calling it "structural templating". So far the only things close to this are - Azure pipeline's syntax: https://docs.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops https://docs.microsoft.com/en-us/azure/devops/pipelines/proc... - Something called Jasonette: https://docs.jasonette.com/templates/ https://docs.jasonette.com/templates/ - Something called Jsonnet: https://jsonnet.org/ https://jsonnet.org/ Azure Pipeline's approach I think is closest to what I've been looking for. Anything else in this space?
  • Edmond 8y ago
    we're working on this: https://www.youtube.com/watch?v=4yepPOznakk https://www.youtube.com/watch?v=4yepPOznakk http://codesolvent.com/config-node/ http://codesolvent.com/config-node/
  • wryun 8y ago
    I prefer https://github.com/taskcluster/json-e https://github.com/taskcluster/json-e Has the advantage/disadvantage that it's still valid json/yaml I wrote a scary command line wrapper for it: https://wryun.github.io/rjsone/ https://wryun.github.io/rjsone/ Has libs for Python, Go, and JS, and there's a bazel interface.
    • igor47 8y ago
      plug for json-e i found it much less painful-looking than jsonnet
    • epage 8y ago
      This looks really nice. I'll have to give this a try at some point to see how I feel about it vs Azure Pipelines. From my quick look, this looks more general purpose at the cost of more verbosity.
• deleted 8y ago
  [deleted]
• desc 8y ago
  My approach to this these days is: * all configuration, without exception, is XML. * all configuration may be generated from any other format imaginable, but it's sure as fuck going into the Big Main Godlike Application as XML. Separation, interfaces, etc. Disclaimer: I work in .NET almost exclusively. The .NET configuration APIs generally work, as long as you only ever use them for reading; treating config as something the application itself can fiddle with is a fast route to madness.
• jrockway 8y ago
  I saw this title and immediately knew the article would be about Helm. I don't think anyone wants to use Helm. People use it for a set-and-forget thing that they don't care about (who cares that it's called impressive-leopard-kubernetes-dashboard, after all.) kustomize is much more sane for your own stuff: https://github.com/kubernetes-sigs/kustomize https://github.com/kubernetes-sigs/kustomize It is actually a little bit too magical for my taste, but I continue to use it because it hasn't done anything stupid. I have one file that maps logical names to images in a container repository. If I create a service called "foo" pointing to selector.app.label="foo" in the base, then in production it's called foo-prd and the label magically updates to foo-prd for the selector. It actually understands what it's generating, and while they might have taken it a little bit too far, it's far better than just dumb text replacement.
  • alexk 8y ago
    > I don't think anyone wants to use Helm. Here is why everyone should use Helm: Helm 2.0 introduced package as a first-class concept for Kubernetes and created the standard to distribute applications, thanks to Helm thousands of people could discover and collaborate on cloud-native deployments of the open source software https://github.com/helm/charts/tree/master/stable https://github.com/helm/charts/tree/master/stable published and managed by organizations and contributors all over the world. Helm 3.0 keeps innovating, it adopts the most forward-thinking approach to package management and Kubernetes config management by using higher level domain specific language based on Lua to create expressive package management system: https://sweetcode.io/a-first-look-at-the-helm-3-plan/ https://sweetcode.io/a-first-look-at-the-helm-3-plan/ Helm is also backed by CNCF[1] and is the best choice so far for organizations to create a reproducible CI/CD pipeline in a Kubernetes cluster. [1] https://www.cncf.io/blog/2018/06/01/cncf-to-host-helm/ https://www.cncf.io/blog/2018/06/01/cncf-to-host-helm/
    • jhanschoo 8y ago
      I'm not very fond of helm's seemingly imperative approach, though. Am I mistaken about that?
    • the_duke 8y ago
• zaphar 8y ago
  Wholly agree with this which is why I've been experimenting with a project similar in goals to jsonnet as a side project of my own. I didn't know about jsonnet when I started or I'd probably have just used/contributed to that instead. Why template yaml when you could just generate it? Or generate json, or toml, or xml, or environment variables...
  • Aeolun 8y ago
    If you are generating it, why use it at all?
    • leg100 8y ago
      Because more often than not that is what the target system expects (k8s most notably).
• sankyo 8y ago
  In the Clojure world we use EDN (extensible data notation) which is a subset of Clojure. https://github.com/edn-format/edn https://github.com/edn-format/edn You can extend it, convert it to JSON if necessary, and it is easy to read.
• peterwwillis 8y ago
  YAML is a data stream, not a program. Please do not shove programs into data. Your data does not need to be "expressive", it just needs to provide input to a program. If your data files need to be complex, you need a program to generate them for you. I've danced the dance of ini -> json -> yaml -> weird hybrid -> embedded logic, and it ends with "program that asks for what the thing you want looks like and generates data files". Industrial software design figured this out ages ago.
  • TeMPOraL 8y ago
    And you end up with program compiled to a config anyway, so with a proper toolchain it means your real config is the program. People keep increasing complexity unintentionally precisely because they don't realize that code = data. There's no real distinction. Code is data is code. You will end up having Turing completeness somewhere, it's just a matter of choosing (or blindly selecting, like most people do) where. For a popular product, it eventually gets embedded in the configuration language, turning it into half-assed programming language (see most web-related templating). For less popular products / more enterprise'y settings, you can probably get away with embedding the Turing-complete part in your bureaucracy. That is, I can't code my config to make it do what I want, but I can pay you to get developers to write some code and export it to the config language as a keyword. There's a spectrum to this, and tradeoffs galore. But ultimately, YAML is nothing but a tree notation. Tree notation is enough to represent high-level programming languages. Lisp without parenthesis, if you might, or Python, if you squint your eyes.
    • peterwwillis 8y ago
      "Data" to me is the least complex input from a human, whereas "code" is more complex and necessitates a lot more work to make sure it's correct and bug-free. If you embed "code" in "data", you made your thing way more complex and subject to software design patterns. But in software operation, we already have to contend with highly complex systems, so we want to remove as much time and effort and complexity as possible from the instrumentation. To put it another way: if you had to run a nuclear reactor, do you want to instrument it by constantly writing new code, or turning a dial? I'd rather turn a dial. That means I have to develop the code for that dial ahead of time, but in the end, actually using it will be safer.
• niftich 8y ago
  File-based configs are a troublesome abstraction: they package unrelated concerns into a rigid document whose form must take a particular, application-dependent shape, and the assembly and disassembly of that document essentially becomes an API where key-value pairs are mixed with complex glue code. The application has to do this internally, but anyone who's generating their configs are also doing parts of this externally. Templates try to bandage over that by drilling down the abstraction to key-value pairs themselves. And imperative constructs that sneak into templating languages are an artifact of wanting to gain expressiveness without losing the benefits of declarative form -- but really, the two are at odds. YAML is a red herring -- we had the same headaches with XML a decade prior. The problem is always that there's relationships among the data (or even multiple instances of the config) that we care about, but that the structure of a single config file at rest cannot model. Databases -- let's say, an SQL one -- are actually among the better solutions, because they allow the universe of config items to live in structured places without overspecifying the exact form the data must take when serialized into a file. Then, data can be normalized where it makes sense to avoid repetition and introduce propagation. An SQL database gives all the tools needed to accomplish this, using mostly declarative code. Databases in a KV sense are often used for configuration, and SQLite's rise has increased richly structured configs that are specified at a higher level than what's typically done with other serialization formats, but the full approach has not caught on outside big enterprise systems and complex applications. Which is a shame, because it's hardly more complex than the current awkward pairing of a full serializer and a templating engine.
  • afiori 8y ago
    SQL as a configuration format is not that bad of an idea
• kyberias 8y ago
  Why is it so hard to write the meat of the argument in the first paragraph? So much setup...
• jeremysalwen 8y ago
  When I joined Google, I realized that it was kinda surprising I hadn't seen something like GCL in the outside world (not to say it doesn't exist, just it's ubiquitous enough for me to hear about it). This seems to be an example of that hole being filled.
• spacesuitman2 8y ago
  A consistent observation has been that all config languages tend to become turing complete over time.
• TeMPOraL 8y ago
  Wait, what? I feel this article is missing the bigger problem - one that for some reason just cannot die. The problem is that of gluing strings together. YAML is not an unstructured text file, it's a tree notation. Whatever "templating" or "generation" mechanism you want to use, it needs to respect the tree nature of the language it operates on. It needs to respect semantics. Gluing strings together is literally what causes SQL Injection to exist. It caused countless of defacements on the web, and countless of broken websites. I would think we've learned our lessons, but for some reason, I see these template languages still alive and kicking.
  • hinkley 8y ago
    It’s DRY run amok. People don’t want to see the same bit of anything in two places, forgetting that code will be read, so they remove a “redundancy” but create duplicated effort for everyone, every time they have to decipher the thing later. I also don’t think we have a workable definition of “configuration”. 70% of the config at my work is hard coded service discovery. If we moved the service discovery anywhere else (say, consul, kubernetes, hell - Docker swarm), we’d need far fewer sets of config than we have deployment environments. When there are only two or three you don’t need templating. How often do you really have the same service deployed twice in prod and legitimately want it to work differently? I can count the scenarios I know on one hand and none of them have occurred for me in almost ten years, except read replicas and that shouldn’t be more than a few lines of config.
  • ekimekim 8y ago
    The article goes on to talk about Jsonnet, which takes the exact approach you describe - it generates JSON by aiming to be a "templated JSON" where the templating involves generating semantic objects, not strings. Here's an example (adapted from some real-world code) where I specify the k8s cpu limit in one place, and then look up that info in several other places to avoid needing to change multiple values later: { local container = self, requests: {cpu: 5.5, memory: "2G"}, limits: container.requests + {memory: "4G"}, environment: [ { name: "NUM_THREADS", value: std.toString(std.ceil(container.requests.cpu)), }, ], } Note how I can patch the container.requests object with an alternate memory limit, and how I can calculate an expression for the NUM_THREADS value in order to automatically set it to ceil() of the requested cpu. (edited for nicer formatting of the code)
• fiddlerwoaroof 8y ago
  Dhall-lang ( https://dhall-lang.org https://dhall-lang.org ) is another, somewhat interesting, attempt to solve this program: it comes with a non-Turing complete programming language, so you can bring some abstraction to your configuration files without having to worry about things like infinite loops.
• ssspaju 8y ago
  I'd rephrase the question: "Why are we templating YAML with text-based templating tools?" I and @akx wrote a templating tool called Emrichen that's specifically designed for producing YAML and JSON from YAML templates: https://github.com/con2/emrichen https://github.com/con2/emrichen In contrast to other template systems, Emrichen templates are not just "based on YAML", they _are_ YAML. YAML tags like "!Var varname" are used to perform things like variable substitution, loops etc. Variables can be of any JSON type, not just strings, and the template is evaluated top–down.
• l0b0 8y ago
  YAML is the bastard offspring of XML. A bunch of ways to write semantically identical stuff is bullshit. Let data files be data files, and build them using languages actually suited for the job. JSON is plenty complex for 105% of the use cases of YAML, with much fewer downsides.
• zoobab 8y ago
  I use j2cli to template my YAMLs, pretty powerful to template stuff without using ansible. You just need to right environment variables. And you can put if logic if you want.
• shusson 8y ago
  I've recently settled on typescript object literals for creating complex config files. Adv: - It's still quite declarative. - supports expressions. - supports merging objects using the spread `...` operator. This enables breaking up large configs files into smaller files. - supports type constraints. - serialises to JSON easily.
  • tobyhinloopen 8y ago
    Same here... just using the same language as the rest application, usually a file with constants or a class/struct/module/whatever. the only reason config files in another language might be required is when you require configuring the application after it is compiled to native binary code. Obviously this doesn't apply to all these applications written in scripting languages. I do wonder how you create type-safe config files though. I currently have a `development.ts` and a `production.ts`, where the `production.ts` is only loaded when `node_env === production`. `production.ts` contains blank/default values and the file is overwritten on the server with production secrets.
    • shusson 8y ago
      > I do wonder how you create type-safe config files though The types help prevent erroneous config declarations. e.g ``` enum LogLevel { Info = "info", Debug = "debug" } const MyConfig = { logOptions: { level: LogLevel.Debug } }; ``` And since we serialise `MyConfig`, the config has to be transpiled by Typescript.
• gjvc 8y ago
  We cannot now be far off some ansible [1]-like tool using LISP s-expressions for its configuration language. [1] or any of that ilk
• equalunique 8y ago
  Maybe we should be exploring using Dhall instead of YAML. >One of the clearest signals I’ve gotten from users is that Dhall is “the YAML killer”, for the following reasons: >Dhall solves many of the problems that pervade enterprise YAML configuration, including excessive repetition and templating errors >Dhall still provides many of the good parts of YAML, such as multi-line strings and comments, except with a sane standard >Dhall can be converted to YAML using a tiny statically linked executable, which provides a smooth migration path for “brownfield” deployment. Source: http://www.haskellforall.com/2019/01/dhall-year-in-review-2018-2019.html?m=1 http://www.haskellforall.com/2019/01/dhall-year-in-review-20...
• fasteo 8y ago
  Off-topic: The whole discussion is about deploy-time configuration management, but our problem is more about run-time configuration management. We have done the classical memcached+database custom solution, but I was wondering if there is any accepted library/tool to change application run-time behavior. We have tried consul KV store [1], but does not quite fit in our environment. My ideal solution would be a webapp with some text editor (think codemirror). Changes in this text file would push the configuration data to a running application. [1] https://learn.hashicorp.com/consul/getting-started/kv https://learn.hashicorp.com/consul/getting-started/kv
• mschaef 8y ago
  Reading the commentary on whitespace, my thoughts immediately jumped to the C preprocessor. Even though it's built into the language, it has the same sorts of problems as a template engine has with generating YAML: the preprocessor just wasn't enough aware of the syntactic structure of the language to make it easy to generate anything of significant complexity. I'm not proud of this (and like to think I could come up with something better these days), but this code was a bit of a nightmare for that reason: https://github.com/mschaef/vcsh/blob/master/vm/number.c#L256 https://github.com/mschaef/vcsh/blob/master/vm/number.c#L256 Lisp macros do better, but they have the problem that the macros (and their potentially unusual evaluation rules) can easily just blend in with ordinary function calls.
• alxarch 8y ago
  Over the last week I created a tool processing YAML and JSON files using Jsonnet. It's called [ycat](https://github.com/alxarch/ycat https://github.com/alxarch/ycat) and is inspired by `jq` but uses Jsonnet for processing. It can also be used just as `cat` to concatenate JSON/YAML files. It's still young but very usefull, especially for handling complex kubernetes configurations.
• louiskottmann 8y ago
  I feel like Ansible generating YAML with Jinja templates is really a sweet-spot, with idempotency and reusability. I find it pushes me to write plain YAML files for variables and defaults (Ansible), while allowing strong templating of generated files (Jinja) and letting the result be readable (YAML). By readable I mean minimum programming bloat (code spread on many lines just to write a for loop) and minimum extra syntax that clutters the screen (brackets and quotes). It also lets me write very little custom code (aside from variables, obviously). If I had to use a more "powerful" YAML-like replacement, it would mix all these into files, written differently by people with different styles, and it would have bloat all over the place. The main issue I have with helm is that values.yml is not templatable by default so you have to generate it if you want reusability. YAML does one thing and does it well, it's readable and bloat-free. Maybe we need more tools like "kubectl explain" to know the syntax though.
• nikolay 8y ago
  Jsonnet is nice, but not actively developed, unfortunately. I highly recommend you to support UCG [0], which is written in Rust (Jsonnet has two implementations, which is one source of the slowness). [0]: https://github.com/zaphar/ucg https://github.com/zaphar/ucg