9 ms·

> Everyone hates YAML. Everyone writes a lot of YAML. I don’t know a single engineer (ops or not) who enjoys writing YAML, yet it is utterly unavoidable. I’ve

by ljm 7y ago

> Everyone hates YAML. Everyone writes a lot of YAML. I don’t know a single engineer (ops or not) who enjoys writing YAML, yet it is utterly unavoidable. I’ve lost count of the amount of bugs and broken deploys that have happened because of YAML, or because of a type error caused by it.

cbanek 7y ago
What people really hate is having to write complicated configuration. No matter what format you put it in, it is still complicated configuration. It's hard if not impossible to test, there's only one right way to do it, and it is likely interconnected with other complicated configuration. Whatever format it happens to be written in (now YAML is the apparently trendy way) it is guilty by association.
- raverbashing 7y ago
  True, though YAML is the most "human readable/writable" of the usual suspects (YAML/JSON/XML)
  - Kwpolska 7y ago
    I’ve yet to see an editor that works well with YAML, yet JSON and XML are simple. Add comments and trailing commas to JSON and it would be perfect to write. And XML isn’t that evil, really.
  - DonHopkins 7y ago
    I'd say that spreadsheets are vastly more readable/writable/editable/maintainable than YAML or JSON or XML (i.e. no punctuation and quoting nightmares), and they're easy to learn and use, so orders of magnitude more people know how to use them proficiently, plus tools to edit spreadsheets are free and widely available (i.e. Google Sheets), and they support real time multi user collaboration, version control, commenting, formatting, formulas, scripting, import/export, etc. They're much more compact for repetitive data, but they can also handle unstructured and tree structured data, too. To illustrate that, here's something I developed and wrote about a while ago, and have used regularly with great success to collaborate with non-technical people who are comfortable with spreadsheets (but whose heads would explode if I asked them to read or write JSON, YAML or XML): Representing and Editing JSON with Spreadsheets I’ve been developing a convenient way of representing and editing JSON in spreadsheets, that I’m very happy with, and would love to share! https://medium.com/@donhopkins/representing-and-editing-json-with-spreadsheets-8f8832db5239 https://medium.com/@donhopkins/representing-and-editing-json... Here is the question I’m trying to answer: How can you conveniently and compactly represent, view and edit JSON in spreadsheets, using the grid instead of so much punctuation? My goal is to be able to easily edit JSON data in any spreadsheet, conveniently copy and paste grids of JSON around as TSV files (the format that Google Sheets puts on your clipboard), and efficiently export and import those spreadsheets as JSON. So I’ve come up with a simple format and convenient conventions for representing and editing JSON in spreadsheets, without any sigils, tabs, quoting, escaping or trailing comma problems, but with comments, rich formatting, formulas, and leveraging the full power of the spreadsheet. It’s especially powerful with Google Sheets, since it can run JavaScript code to export, import and validate JSON, provide colorized syntax highlighting, error feedback, interactive wizard dialogs, and integrations with other services. Then other apps and services can easily retrieve those live spreadsheets as TSV files, which are super-easy to parse into 2D arrays of strings to convert to JSON.
    - tlarkworthy 7y ago
      That is super cool, please don't over complicate it with utility features. I have been considering a project to manage a kubernetes cluster via Google spreadsheet. Google docs have great features relating to user authentication and permissions. The project would needs to visualize the JSON state representation for the k8s cluster... your project is ideal. e.g. calling another google service with the JSON using a token minted BY THE USER CURRENTLY USING THE SHEET
      - DonHopkins 7y ago
        Thanks for the encouragement! I agree, I'd like to keep it from becoming complicated. My hope is to keep it simple and refine it into a clean well defined core syntax that's easy to implement in any language, with an optional extension mechanism (for defining new types and layouts), without falling into the trap of markdown's or yaml's almost-the-same-but-slightly-different dialects. (I wrote more about that at the end of the article, if you made it that far.) The spreadsheet itself brings a lot of power to the table. (Pun not intended!) There are some cool things you can do using spreadsheet expressions, like make random values that change every time you download the CSV sheet, which is great for testing. But expressions have their limitations: they can't add new rows and columns and structures, for example. However, named ranges are useful for pointing to data elsewhere in other sheets, and you can easily change their number of rows and columns. For convenience and expressivity, I've defined ways of including other named sheets and named ranges by reference, and 2d arrays of uniformly typed values, and also define compact tables of identical nested JSON object/array structures by using declarative headers (one object per row, which I described in the article, but it's not so simple, and needs more examples and documentation).
        tlarkworthy 7y ago
        yeah my eye brows are fairly raised at the thought of embedding a templating language in it. For production use of a spreadsheet, I imagine pulling the source code out of the spreadsheet using https://github.com/google/clasp https://github.com/google/clasp and synchronising with a repository using Terraform. At which point Terraform has a weak templating engine already, but its generally enough for building reusable infra. Additional features can be provided within the spreadsheet using reusable libraries. One pain point with embedding functional dataprocessing in a spreadsheet for JSON data, is a decent way of writing tree expressions, to which I would turn to the de facto JSON tooling jq for inspiration. if you want to take this further, I am up for building some infra for continuous deployment spreadhseets through terraform. tom <dot> larkworthy <at> futurice.com But I would not embed stuff inline with the JSON. I would have a pure sheet dedicated to stuff going in, and a compute sheet for stuff join out. And the definition for stuff going out should basically be a JQ expression, that can "shell out" to sheets expressions https://github.com/sloanlance/jq/issues/1 https://github.com/sloanlance/jq/issues/1
  - xpe 7y ago
    TOML is a worthy contender. It is my favorite simple but powerful-enough data language. Here is a side-by-side comparison of data in TOML versus YAML: https://gist.github.com/oconnor663/9aeb4ed56394cb013a20 https://gist.github.com/oconnor663/9aeb4ed56394cb013a20 And some comments that resonate with me: The yaml spec is overly complex and parsing it properly is a nightmare. I rather prefer TOML because of it's simplicity. Unless one really need the gazillion extra features which yaml provides (which one probably doesn't), I'd say sticking with TOML seems to be the saner choice. I've recently kind of changed my mind on unquoted strings. They're nice when you're editing config files by hand, but they run into parsing issues in simple cases like when the string looks like an int, or of course when the string contains quotation marks itself.
  - IshKebab 7y ago
    I disagree. Yes it is human readable if you just want to read the words (like your would with Markdown), but with a configuration file you want to understand the structure. YAML makes that quite confusing IMO. It seems like a random array of dashes and indentation. JSON is much more human readable in that respect because the structure is explicit so there's no ambiguity. I'd say TOML is somewhere in-between. But both are vastly preferable to YAML because they don't have Javascript-style type insanity.
- DonHopkins 7y ago
  Pure YAML will never DRY. https://en.wikipedia.org/wiki/Don%27t_repeat_yourself https://en.wikipedia.org/wiki/Don%27t_repeat_yourself If you write procedural configurations with a real Turing Complete, but not Turing-Tarpit language like Python or JavaScript, then you don't need to repeat yourself ridiculous numbers of times, and manage thousands of lines of hand written eye-sore almost-but-not-quite-entirely-unlike-tea YAML. Plus you can implement validation, warnings and error checking, and support multiple input and output formats. But so many DevOps dogmatically go out of their way to avoid writing any code, even when the alternative means writing a hell of a lot more brittle unmaintainable YAML where you have to carefully check every line for correctness, and make sure that you're actually repeating the same thing in every location required without any typos, and don't let any of those repetitions slip through the cracks when you make changes. With real Turing complete code, macros, and templates, you can accomplish the same task as with pure un-DRY YAML data, much more easily and efficiently, with orders of magnitudes fewer lines of code, that you can actually understand, maintain, validate, and test, so you can be sure it actually works without meticulously checking every line of output by hand. It's better to combine DRY combinations of different non-Turing-complete data formats like CSV, YAML, JSON, INI, XML, using the most appropriate formats depending on the type of data. Spreadsheets are much better than JSON or YAML or XML for many different kinds of data with repeating structure (so you don't repeat key names), and JSON and YAML are much better for irregular tree-shaped data (since you're not restricted to rigid structures), and XML for tree structured data and documents including text. You end up needing different formats as input as well as output. So you need a real Turing Complete language to read them all in, combine and validate them, and shit out the various other formats required by your tools and environment.
  - cbanek 7y ago
    > But so many DevOps go out of their way to avoid writing code Well, this is the real problem, is that DevOps people should be writing code in my opinion, especially code to automate deployments and handle configuration. But many times it's just a new job label for people with the same ol' ops and sysadmin skillset who don't want to write code. It doesn't help that when they make unmaintainable piles of configuration that nobody understands, it typically adds to their job security.
- ken 7y ago
  I've used a lot of JSON. It's OK. No comments, not many data types. But the spec is only a couple pages, and I've never been in doubt about how something should be escaped, or parsed. I could probably write a bare-bones parser in an afternoon, if I needed to. I've tried to work with YAML a few times. The tree structure and extra data types are great. Everything else is a huge pain. There's at least 3 versions of the spec, and the latest one is nearly 100 pages. The parts I need are always in some "extension", so there's even more that I need to support. It has a system for serializing native Objects, so you have to be careful with untrusted data because there are some interesting security issues. It's so complex, I have trouble knowing what to quote, or how. It's not feasible to write your own parser in any reasonable amount of time. Worst of all, every parsing library is slightly different, so (not unlike SOAP) you kind of have to know that it's going to be parsed with (say) PyYAML. Complicated configuration is indeed a problem in any format, but YAML makes even simple things complex. From the beginning, I really wanted to like YAML. Unfortunately, I think their goals (human-readable text, language-agnostic, rich data types, efficient, extensible, easy to implement, easy to use) are impossible. You simply can't achieve all of them at once.
- davnicwil 7y ago
  I'm launching a CI service [0] which instead of using YAML configs to run builds on a third party platform, will let you run the builds yourself on your own machines so you can just use a script or whatever you want to do your builds/deploys. I share your frustration and was motivated by it to build this. Why should I spend ages writing up everything as config files when I have a script that already works, is easy to change and debug, and can handle any custom thing I need? I think config files to describe devops processes are a good approach for huge companies with huge teams, lots of churn etc. The approach perhaps has simplicity & stability benefits - works for everyone everywhere without understanding any detail, changes are a bit easier to track, etc. But for small teams wanting control, speed and the flexibility of just writing code to do what you want it can often be an inefficient approach. At least in my experience. You should check Box CI out. Launching very soon! [0] https://boxci.dev https://boxci.dev
weberc2 7y ago
Yeah, what I want is JSON with comments and nice multi-line string support. I don't like how much syntactic magic YAML does (I don't need or want the country code for Norway to be parsed as a boolean False value). I still don't know what the exclamation point does (e.g., !Ref). And clearly YAML is the wrong tool for infra-as-code since CloudFormation has to build a macro system, conditionality, referencing, and a couple different systems for defining and calling functions (templates being one and their implicit functions being another). We also see tools like Troposphere and CDK which are effectively different ways to generate CloudFormation YAML via programming languages (or more precisely programming languages that were designed for humans). And it's not just limitations inherent to CloudFormation--Helm has long had templates for generating YAML, but those also weren't sufficiently powerful/expressive/ergonomic so Helm3 is supporting Lua as well. And as I understand it, Terraform is constantly adding more and more powerful features into HCL. So what's the solution? It's pretty simple--we should keep the YAML around, but it should be the intermediate representation (IR), not the human interface. The human interface should be something like a functional language[^1] (or an imperative language that is written in a functional style) that evaluates to that YAML IR layer. The IR is then passed to something like Kubernetes or Terraform or CloudFormation which understand it, but it's not the human interface. As for the high-level language, something like [Starlark][0] would work well. It's purpose-built for being an evaluated configuration language. However, I would argue that a static type system (at least an optional static type system) is important--it's easy enough to imagine someone extending Starlark with type annotations and building a static type checker (which is much easier for Starlark since it's a subset of Python which is intended to be amenable to static analysis). This, I think, is the proper direction for infrastructure-as-code tooling. [^1]: Functional in that it is declarative instead of imperative--not necessarily that the syntax should be as hard to read as OCaml or Haskell. Also, while YAML is also declarative, it doesn't have a notion of evaluation or variables. [0]: https://docs.bazel.build/versions/master/skylark/language.html https://docs.bazel.build/versions/master/skylark/language.ht...
- chris11 7y ago
  TeamCity actually lets you set job configurations with kotlin in source control. The config is actually a kotlin dsl, so not only can you import settings, you can also have TeamCity generate projects, jobs, and settings from your code.
woile 7y ago
I was wondering if anyone has started using cue[1] and has an opinion about it, seems like a super interesting project aiming to solve this [1] https://cuelang.org/ https://cuelang.org/
- mbo 7y ago
  https://dhall-lang.org/ https://dhall-lang.org/ too
perlgeek 7y ago
Enjoying it might be an overstatement, but I prefer YAML over JSON, XML, INI files, bash scripts and most other config mechanisms. Maybe the only comparable thing that comes to mind that I prefer is jsonnet.
nightfly 7y ago
Part of my job is introducing people to Ansible. For the people not intimately familiar with YAML, it's the biggest obstacle.
eropple 7y ago
I’ve done “devops“ at both large and small organizations. I don’t write YAML. I use Pulumi and TypeScript for system provisioning; it’s all syntactically and type-checked as I write it, and reusable besides. I refuse to deal with the maintainability disaster that is Ansible (I used to use Chef but now pretty much everything can be handled via Fargate) and I don’t have a need for Kubernetes (except in my home lab where I use it with Pulumi’s providers). The one place I could write YAML is AWS CodeBuild buildspecs. I write them in JSON, when they’re not being assembled on the fly from Pulumi. You can do it too—just pick better tools for it.
- throwaway122379 7y ago
  Interesting never heard of Pulumi, thanks You might be interested in https://www.jetbrains.com/teamcity/ https://www.jetbrains.com/teamcity/ It has a Kotlin strongly typed DSL for pipelines, i absolutely despite Jenkins and Groovy, so many bugs causing huge issues due to weak typing and lack of testing :(
  - eropple 7y ago
    I’ve used Teamcity. It’s totally fine. I don’t want to host anything, though, and I want them inside my VPC. We’ll use AWS until we can’t stand it anymore and them we’ll fire up a Jenkins (just because there’s more internal expertise).
- fock 7y ago
  Seeing it mentioned here twice, I went and checked out their website/github. What I see is lots of 20-line examples setting up a docker-container/very basic vm/aws-lambda. While typescript might have better "testability" I'm not sure, where this goes if you just copy/execute bash-scripts or inject javascript code into aws-lambda. To me it seems like it just reinvents the "classic" sys-admin but instead of writing/copying/executing bash-scripts, dev(op)s are now churning out bash-scripts wrapped in tested wrapper code - well... In contrast ansible has a lot of declarative building blocks (iirc all are unit-tested!), which either fail or do the specified job. And yeah, the specified job might be copying and templating a config-file for some oldschool service (and you might be shocked: people still use these!), which is inherently not really testable (except by integration tests. or maybe you write a test to get the file and to check whether you got to type the name right twice) - I'm not sure how pulumi can help you with that? And yes, I would love a concise, descriptive DSL (compare for example spack, which does this nicely) over/as an alternative to the loops-crafted-on-YAML-mess of ansible but I take the latter any day over some "we-can-call-cloud-providers-apis-and-kubernetes-but-why-would-you-copy-files?"-stuff like pulumi.
iamaelephant 7y ago
I have hardly written a line of YAML in my career. It's hardly unavoidable, no one is forcing you to use bad tools. Especially considering that Kubernetes is massively over-engineered for most systems that aren't themselves over-engineered.
FlorianRappl 7y ago
To this day I am baffled why choose YAML? Personally, I think its more error prone, less flexible, and harder to read than JSON. Not to say JSON is a perfect format, but it sure feels better than YAML.
- charrondev 7y ago
  Reason number 1: aliases and anchors. Reason number 2: allowing comments. JSON is a terrible config format, and an ok data interchange format.
  - wpietri 7y ago
    Agreed. It amazes me that we got out of the XML-for-everything era with a lot of people thinking the problem was XML, and not the for-everything part. JSON-for-everything is just as maddening to me.
- probably_wrong 7y ago
  Comments was the reason for me to move from JSON to YAML in config files. I remember perfectly fine what every option does, but adding comments to each option is a must when it comes to sharing my code with anyone else.
  - FlorianRappl 7y ago
    If its a config file I'm not sure why you'll need comments. Comments / documentation should be in the system you'll do the config for. If your set options are that quirky / need documentation; why not explain them in a README? But maybe I misunderstood. Can you give an example of a comment that makes sense / is required in a configuration JSON?
    - probably_wrong 7y ago
      > If your set options are that quirky / need documentation; why not explain them in a README? Mostly because I don't know whether the person customizing my code will read the README, but they'll definitely see the comments I wrote right before the configuration option itself. With comments and some extra whitespace, they can go through the file line by line, read what a specific option does, configure it properly, and move on to the next one. No back and forth to the README required.
linuxftw 7y ago
We probably haven't met, but I think YAML is great. If you're doing a lot of work in python and ansible, then you just start using YAML everywhere. I actually struggled with YAML at first when starting with Ansible because I didn't know it was YAML, and I didn't understand lists vs dictionaries in YAML (seems obvious in hindsight, but just didn't click for whatever reason). > I’ve lost count of the amount of bugs and broken deploys that have happened because of YAML, or because of a type error caused by it. YAML can be linted. Also, golang might be helpful to read i and deserialize your YAML files against a known-type so you can catch these type errors. This should probably be done during CI when you create a pull request or merge request. This obviously requires doing things on your end to make it all fit together.
c3534l 7y ago
It's a hell of a lot better than JSON and I like that everything uses it.
BLanen 7y ago
I find it interesting that it's always called "infrastructure as code" but then everything is basically "infrastructure as config files" where any control flow is application-specific constructs for a language not built for the task. Why not just read python, like how webpack uses javascript as its "config".
xpe 7y ago
I recommend taking a look at TOML [1] as an alternative to YAML for many situations. Objectives: TOML aims to be a minimal configuration file format that's easy to read due to obvious semantics. TOML is designed to map unambiguously to a hash table. TOML should be easy to parse into data structures in a wide variety of languages. A reddit thread titled "YAML vs TOML" has some merit [2]. YAML downsides: - Implicit typing causes surprise type changes. (e.g. put 3 where you previously had a string and it will magically turn into an int). - A bunch of nasty "hidden features" like node anchors and references that make it look unclear (although to be fair a lot of people don't use this). TOML downsides: - Noisier syntax (especially with multiline strings). - The way arrays/tables are done is confusing, especially arrays of tables. Rust uses TOML to configure its build system, Cargo [3]. [1]: https://github.com/toml-lang/toml https://github.com/toml-lang/toml [2]: https://www.reddit.com/r/devops/comments/6f82nu/yaml_vs_toml/ https://www.reddit.com/r/devops/comments/6f82nu/yaml_vs_toml... [3]: https://doc.rust-lang.org/cargo/getting-started/first-steps.html https://doc.rust-lang.org/cargo/getting-started/first-steps....

by ljm 7y ago

cbanek 7y ago

What people really hate is having to write complicated configuration. No matter what format you put it in, it is still complicated configuration. It's hard if not impossible to test, there's only one right way to do it, and it is likely interconnected with other complicated configuration. Whatever format it happens to be written in (now YAML is the apparently trendy way) it is guilty by association.

raverbashing 7y ago
True, though YAML is the most "human readable/writable" of the usual suspects (YAML/JSON/XML)
- Kwpolska 7y ago
  I’ve yet to see an editor that works well with YAML, yet JSON and XML are simple. Add comments and trailing commas to JSON and it would be perfect to write. And XML isn’t that evil, really.
- DonHopkins 7y ago
  I'd say that spreadsheets are vastly more readable/writable/editable/maintainable than YAML or JSON or XML (i.e. no punctuation and quoting nightmares), and they're easy to learn and use, so orders of magnitude more people know how to use them proficiently, plus tools to edit spreadsheets are free and widely available (i.e. Google Sheets), and they support real time multi user collaboration, version control, commenting, formatting, formulas, scripting, import/export, etc. They're much more compact for repetitive data, but they can also handle unstructured and tree structured data, too. To illustrate that, here's something I developed and wrote about a while ago, and have used regularly with great success to collaborate with non-technical people who are comfortable with spreadsheets (but whose heads would explode if I asked them to read or write JSON, YAML or XML): Representing and Editing JSON with Spreadsheets I’ve been developing a convenient way of representing and editing JSON in spreadsheets, that I’m very happy with, and would love to share! https://medium.com/@donhopkins/representing-and-editing-json-with-spreadsheets-8f8832db5239 https://medium.com/@donhopkins/representing-and-editing-json... Here is the question I’m trying to answer: How can you conveniently and compactly represent, view and edit JSON in spreadsheets, using the grid instead of so much punctuation? My goal is to be able to easily edit JSON data in any spreadsheet, conveniently copy and paste grids of JSON around as TSV files (the format that Google Sheets puts on your clipboard), and efficiently export and import those spreadsheets as JSON. So I’ve come up with a simple format and convenient conventions for representing and editing JSON in spreadsheets, without any sigils, tabs, quoting, escaping or trailing comma problems, but with comments, rich formatting, formulas, and leveraging the full power of the spreadsheet. It’s especially powerful with Google Sheets, since it can run JavaScript code to export, import and validate JSON, provide colorized syntax highlighting, error feedback, interactive wizard dialogs, and integrations with other services. Then other apps and services can easily retrieve those live spreadsheets as TSV files, which are super-easy to parse into 2D arrays of strings to convert to JSON.
  - tlarkworthy 7y ago
    That is super cool, please don't over complicate it with utility features. I have been considering a project to manage a kubernetes cluster via Google spreadsheet. Google docs have great features relating to user authentication and permissions. The project would needs to visualize the JSON state representation for the k8s cluster... your project is ideal. e.g. calling another google service with the JSON using a token minted BY THE USER CURRENTLY USING THE SHEET
    - DonHopkins 7y ago
      Thanks for the encouragement! I agree, I'd like to keep it from becoming complicated. My hope is to keep it simple and refine it into a clean well defined core syntax that's easy to implement in any language, with an optional extension mechanism (for defining new types and layouts), without falling into the trap of markdown's or yaml's almost-the-same-but-slightly-different dialects. (I wrote more about that at the end of the article, if you made it that far.) The spreadsheet itself brings a lot of power to the table. (Pun not intended!) There are some cool things you can do using spreadsheet expressions, like make random values that change every time you download the CSV sheet, which is great for testing. But expressions have their limitations: they can't add new rows and columns and structures, for example. However, named ranges are useful for pointing to data elsewhere in other sheets, and you can easily change their number of rows and columns. For convenience and expressivity, I've defined ways of including other named sheets and named ranges by reference, and 2d arrays of uniformly typed values, and also define compact tables of identical nested JSON object/array structures by using declarative headers (one object per row, which I described in the article, but it's not so simple, and needs more examples and documentation).
      - tlarkworthy 7y ago
        yeah my eye brows are fairly raised at the thought of embedding a templating language in it. For production use of a spreadsheet, I imagine pulling the source code out of the spreadsheet using https://github.com/google/clasp https://github.com/google/clasp and synchronising with a repository using Terraform. At which point Terraform has a weak templating engine already, but its generally enough for building reusable infra. Additional features can be provided within the spreadsheet using reusable libraries. One pain point with embedding functional dataprocessing in a spreadsheet for JSON data, is a decent way of writing tree expressions, to which I would turn to the de facto JSON tooling jq for inspiration. if you want to take this further, I am up for building some infra for continuous deployment spreadhseets through terraform. tom <dot> larkworthy <at> futurice.com But I would not embed stuff inline with the JSON. I would have a pure sheet dedicated to stuff going in, and a compute sheet for stuff join out. And the definition for stuff going out should basically be a JQ expression, that can "shell out" to sheets expressions https://github.com/sloanlance/jq/issues/1 https://github.com/sloanlance/jq/issues/1
- xpe 7y ago
  TOML is a worthy contender. It is my favorite simple but powerful-enough data language. Here is a side-by-side comparison of data in TOML versus YAML: https://gist.github.com/oconnor663/9aeb4ed56394cb013a20 https://gist.github.com/oconnor663/9aeb4ed56394cb013a20 And some comments that resonate with me: The yaml spec is overly complex and parsing it properly is a nightmare. I rather prefer TOML because of it's simplicity. Unless one really need the gazillion extra features which yaml provides (which one probably doesn't), I'd say sticking with TOML seems to be the saner choice. I've recently kind of changed my mind on unquoted strings. They're nice when you're editing config files by hand, but they run into parsing issues in simple cases like when the string looks like an int, or of course when the string contains quotation marks itself.
- IshKebab 7y ago
  I disagree. Yes it is human readable if you just want to read the words (like your would with Markdown), but with a configuration file you want to understand the structure. YAML makes that quite confusing IMO. It seems like a random array of dashes and indentation. JSON is much more human readable in that respect because the structure is explicit so there's no ambiguity. I'd say TOML is somewhere in-between. But both are vastly preferable to YAML because they don't have Javascript-style type insanity.
DonHopkins 7y ago
Pure YAML will never DRY. https://en.wikipedia.org/wiki/Don%27t_repeat_yourself https://en.wikipedia.org/wiki/Don%27t_repeat_yourself If you write procedural configurations with a real Turing Complete, but not Turing-Tarpit language like Python or JavaScript, then you don't need to repeat yourself ridiculous numbers of times, and manage thousands of lines of hand written eye-sore almost-but-not-quite-entirely-unlike-tea YAML. Plus you can implement validation, warnings and error checking, and support multiple input and output formats. But so many DevOps dogmatically go out of their way to avoid writing any code, even when the alternative means writing a hell of a lot more brittle unmaintainable YAML where you have to carefully check every line for correctness, and make sure that you're actually repeating the same thing in every location required without any typos, and don't let any of those repetitions slip through the cracks when you make changes. With real Turing complete code, macros, and templates, you can accomplish the same task as with pure un-DRY YAML data, much more easily and efficiently, with orders of magnitudes fewer lines of code, that you can actually understand, maintain, validate, and test, so you can be sure it actually works without meticulously checking every line of output by hand. It's better to combine DRY combinations of different non-Turing-complete data formats like CSV, YAML, JSON, INI, XML, using the most appropriate formats depending on the type of data. Spreadsheets are much better than JSON or YAML or XML for many different kinds of data with repeating structure (so you don't repeat key names), and JSON and YAML are much better for irregular tree-shaped data (since you're not restricted to rigid structures), and XML for tree structured data and documents including text. You end up needing different formats as input as well as output. So you need a real Turing Complete language to read them all in, combine and validate them, and shit out the various other formats required by your tools and environment.
- cbanek 7y ago
  > But so many DevOps go out of their way to avoid writing code Well, this is the real problem, is that DevOps people should be writing code in my opinion, especially code to automate deployments and handle configuration. But many times it's just a new job label for people with the same ol' ops and sysadmin skillset who don't want to write code. It doesn't help that when they make unmaintainable piles of configuration that nobody understands, it typically adds to their job security.
ken 7y ago
I've used a lot of JSON. It's OK. No comments, not many data types. But the spec is only a couple pages, and I've never been in doubt about how something should be escaped, or parsed. I could probably write a bare-bones parser in an afternoon, if I needed to. I've tried to work with YAML a few times. The tree structure and extra data types are great. Everything else is a huge pain. There's at least 3 versions of the spec, and the latest one is nearly 100 pages. The parts I need are always in some "extension", so there's even more that I need to support. It has a system for serializing native Objects, so you have to be careful with untrusted data because there are some interesting security issues. It's so complex, I have trouble knowing what to quote, or how. It's not feasible to write your own parser in any reasonable amount of time. Worst of all, every parsing library is slightly different, so (not unlike SOAP) you kind of have to know that it's going to be parsed with (say) PyYAML. Complicated configuration is indeed a problem in any format, but YAML makes even simple things complex. From the beginning, I really wanted to like YAML. Unfortunately, I think their goals (human-readable text, language-agnostic, rich data types, efficient, extensible, easy to implement, easy to use) are impossible. You simply can't achieve all of them at once.
davnicwil 7y ago
I'm launching a CI service [0] which instead of using YAML configs to run builds on a third party platform, will let you run the builds yourself on your own machines so you can just use a script or whatever you want to do your builds/deploys. I share your frustration and was motivated by it to build this. Why should I spend ages writing up everything as config files when I have a script that already works, is easy to change and debug, and can handle any custom thing I need? I think config files to describe devops processes are a good approach for huge companies with huge teams, lots of churn etc. The approach perhaps has simplicity & stability benefits - works for everyone everywhere without understanding any detail, changes are a bit easier to track, etc. But for small teams wanting control, speed and the flexibility of just writing code to do what you want it can often be an inefficient approach. At least in my experience. You should check Box CI out. Launching very soon! [0] https://boxci.dev https://boxci.dev

weberc2 7y ago

Yeah, what I want is JSON with comments and nice multi-line string support. I don't like how much syntactic magic YAML does (I don't need or want the country code for Norway to be parsed as a boolean False value). I still don't know what the exclamation point does (e.g., !Ref). And clearly YAML is the wrong tool for infra-as-code since CloudFormation has to build a macro system, conditionality, referencing, and a couple different systems for defining and calling functions (templates being one and their implicit functions being another). We also see tools like Troposphere and CDK which are effectively different ways to generate CloudFormation YAML via programming languages (or more precisely programming languages that were designed for humans). And it's not just limitations inherent to CloudFormation--Helm has long had templates for generating YAML, but those also weren't sufficiently powerful/expressive/ergonomic so Helm3 is supporting Lua as well. And as I understand it, Terraform is constantly adding more and more powerful features into HCL. So what's the solution? It's pretty simple--we should keep the YAML around, but it should be the intermediate representation (IR), not the human interface. The human interface should be something like a functional language[^1] (or an imperative language that is written in a functional style) that evaluates to that YAML IR layer. The IR is then passed to something like Kubernetes or Terraform or CloudFormation which understand it, but it's not the human interface. As for the high-level language, something like [Starlark][0] would work well. It's purpose-built for being an evaluated configuration language. However, I would argue that a static type system (at least an optional static type system) is important--it's easy enough to imagine someone extending Starlark with type annotations and building a static type checker (which is much easier for Starlark since it's a subset of Python which is intended to be amenable to static analysis). This, I think, is the proper direction for infrastructure-as-code tooling. [^1]: Functional in that it is declarative instead of imperative--not necessarily that the syntax should be as hard to read as OCaml or Haskell. Also, while YAML is also declarative, it doesn't have a notion of evaluation or variables. [0]: https://docs.bazel.build/versions/master/skylark/language.html https://docs.bazel.build/versions/master/skylark/language.ht...

chris11 7y ago
TeamCity actually lets you set job configurations with kotlin in source control. The config is actually a kotlin dsl, so not only can you import settings, you can also have TeamCity generate projects, jobs, and settings from your code.

woile 7y ago

I was wondering if anyone has started using cue[1] and has an opinion about it, seems like a super interesting project aiming to solve this [1] https://cuelang.org/ https://cuelang.org/

mbo 7y ago
https://dhall-lang.org/ https://dhall-lang.org/ too

perlgeek 7y ago

Enjoying it might be an overstatement, but I prefer YAML over JSON, XML, INI files, bash scripts and most other config mechanisms. Maybe the only comparable thing that comes to mind that I prefer is jsonnet.

nightfly 7y ago

Part of my job is introducing people to Ansible. For the people not intimately familiar with YAML, it's the biggest obstacle.

eropple 7y ago

I’ve done “devops“ at both large and small organizations. I don’t write YAML. I use Pulumi and TypeScript for system provisioning; it’s all syntactically and type-checked as I write it, and reusable besides. I refuse to deal with the maintainability disaster that is Ansible (I used to use Chef but now pretty much everything can be handled via Fargate) and I don’t have a need for Kubernetes (except in my home lab where I use it with Pulumi’s providers). The one place I could write YAML is AWS CodeBuild buildspecs. I write them in JSON, when they’re not being assembled on the fly from Pulumi. You can do it too—just pick better tools for it.

throwaway122379 7y ago
Interesting never heard of Pulumi, thanks You might be interested in https://www.jetbrains.com/teamcity/ https://www.jetbrains.com/teamcity/ It has a Kotlin strongly typed DSL for pipelines, i absolutely despite Jenkins and Groovy, so many bugs causing huge issues due to weak typing and lack of testing :(
- eropple 7y ago
  I’ve used Teamcity. It’s totally fine. I don’t want to host anything, though, and I want them inside my VPC. We’ll use AWS until we can’t stand it anymore and them we’ll fire up a Jenkins (just because there’s more internal expertise).
fock 7y ago
Seeing it mentioned here twice, I went and checked out their website/github. What I see is lots of 20-line examples setting up a docker-container/very basic vm/aws-lambda. While typescript might have better "testability" I'm not sure, where this goes if you just copy/execute bash-scripts or inject javascript code into aws-lambda. To me it seems like it just reinvents the "classic" sys-admin but instead of writing/copying/executing bash-scripts, dev(op)s are now churning out bash-scripts wrapped in tested wrapper code - well... In contrast ansible has a lot of declarative building blocks (iirc all are unit-tested!), which either fail or do the specified job. And yeah, the specified job might be copying and templating a config-file for some oldschool service (and you might be shocked: people still use these!), which is inherently not really testable (except by integration tests. or maybe you write a test to get the file and to check whether you got to type the name right twice) - I'm not sure how pulumi can help you with that? And yes, I would love a concise, descriptive DSL (compare for example spack, which does this nicely) over/as an alternative to the loops-crafted-on-YAML-mess of ansible but I take the latter any day over some "we-can-call-cloud-providers-apis-and-kubernetes-but-why-would-you-copy-files?"-stuff like pulumi.

iamaelephant 7y ago

I have hardly written a line of YAML in my career. It's hardly unavoidable, no one is forcing you to use bad tools. Especially considering that Kubernetes is massively over-engineered for most systems that aren't themselves over-engineered.

FlorianRappl 7y ago

To this day I am baffled why choose YAML? Personally, I think its more error prone, less flexible, and harder to read than JSON. Not to say JSON is a perfect format, but it sure feels better than YAML.

charrondev 7y ago
Reason number 1: aliases and anchors. Reason number 2: allowing comments. JSON is a terrible config format, and an ok data interchange format.
- wpietri 7y ago
  Agreed. It amazes me that we got out of the XML-for-everything era with a lot of people thinking the problem was XML, and not the for-everything part. JSON-for-everything is just as maddening to me.
probably_wrong 7y ago
Comments was the reason for me to move from JSON to YAML in config files. I remember perfectly fine what every option does, but adding comments to each option is a must when it comes to sharing my code with anyone else.
- FlorianRappl 7y ago
  If its a config file I'm not sure why you'll need comments. Comments / documentation should be in the system you'll do the config for. If your set options are that quirky / need documentation; why not explain them in a README? But maybe I misunderstood. Can you give an example of a comment that makes sense / is required in a configuration JSON?
  - probably_wrong 7y ago
    > If your set options are that quirky / need documentation; why not explain them in a README? Mostly because I don't know whether the person customizing my code will read the README, but they'll definitely see the comments I wrote right before the configuration option itself. With comments and some extra whitespace, they can go through the file line by line, read what a specific option does, configure it properly, and move on to the next one. No back and forth to the README required.

linuxftw 7y ago

We probably haven't met, but I think YAML is great. If you're doing a lot of work in python and ansible, then you just start using YAML everywhere. I actually struggled with YAML at first when starting with Ansible because I didn't know it was YAML, and I didn't understand lists vs dictionaries in YAML (seems obvious in hindsight, but just didn't click for whatever reason). > I’ve lost count of the amount of bugs and broken deploys that have happened because of YAML, or because of a type error caused by it. YAML can be linted. Also, golang might be helpful to read i and deserialize your YAML files against a known-type so you can catch these type errors. This should probably be done during CI when you create a pull request or merge request. This obviously requires doing things on your end to make it all fit together.

c3534l 7y ago

It's a hell of a lot better than JSON and I like that everything uses it.

BLanen 7y ago

I find it interesting that it's always called "infrastructure as code" but then everything is basically "infrastructure as config files" where any control flow is application-specific constructs for a language not built for the task. Why not just read python, like how webpack uses javascript as its "config".

xpe 7y ago

I recommend taking a look at TOML [1] as an alternative to YAML for many situations. Objectives: TOML aims to be a minimal configuration file format that's easy to read due to obvious semantics. TOML is designed to map unambiguously to a hash table. TOML should be easy to parse into data structures in a wide variety of languages. A reddit thread titled "YAML vs TOML" has some merit [2]. YAML downsides: - Implicit typing causes surprise type changes. (e.g. put 3 where you previously had a string and it will magically turn into an int). - A bunch of nasty "hidden features" like node anchors and references that make it look unclear (although to be fair a lot of people don't use this). TOML downsides: - Noisier syntax (especially with multiline strings). - The way arrays/tables are done is confusing, especially arrays of tables. Rust uses TOML to configure its build system, Cargo [3]. [1]: https://github.com/toml-lang/toml https://github.com/toml-lang/toml [2]: https://www.reddit.com/r/devops/comments/6f82nu/yaml_vs_toml/ https://www.reddit.com/r/devops/comments/6f82nu/yaml_vs_toml... [3]: https://doc.rust-lang.org/cargo/getting-started/first-steps.html https://doc.rust-lang.org/cargo/getting-started/first-steps....