50 ms·
The Yaml document from hell
- werdnapk 4y agoAlmost all of the issues mentioned would be solved by using quoted strings.
- miohtama 4y agoWhat are the practical modern alternatives for YAML use cased? TOML, others?
- brazzy 4y agoRead the article?
- progx 4y agoTOML is good for simple plain structures. As soon as you have more nested structures, you can also use json.
- CipherThrowaway 4y agoFor simple use cases like key-value style config you can use JSON or TOML without too many issues. For IaC the alternative is general purpose programming languages using declarative infrastructure APIs. E.g. TS, Go, Python with CDK, CDKTF, Pulumi. For CI are stuck with YAML for now.
- ainar-g 4y agoPeople often forget about the environment, and I feel like a lot of the simpler services and applications could be configured through it easily. For anything that requires more complex structures, JSON5 (https://json5.org/ https://json5.org/) has been gaining traction. It's basically JSON that allows a couple of more things, including comments, trailing commas, and even unquoted keys, if you're into that.
- rr888 4y agoThis for this it indeed looks very much better than json or yaml.
- RedNifre 4y agoedn Extensible Data Notation, it's to Clojure what json is to Javascript.
- AndyKluger 4y agoAs another top-level comment mentions, one alternative is NestedText, where everything is a string unless the ingesting code says otherwise, and there is no escaping needed at all.
- tgv 4y agoTLDR: unexpected conversions to other types in YAML 1.1. A larger JSON document is hell to write and read, though.
- deleted 4y ago[deleted]
- ancieque 4y agoJSON is arguably much worse for usage in Git. Have fun matching braces in the default diff tools out in the wild. YAML is the much better alternative with similar semantics to JSON. The numeric issues still hold true though.
- politelemon 4y agoIsn't TOML just the .INI files of yore? At least it looks very similar.
- CodesInChaos 4y agoThe feature set of toml is similar to json, but with ini like syntax. Most significantly ini-files have a fixed two level structure (section -> key -> value) while toml supports arbitrary nested arrays and dictionaries.
- Freak_NL 4y agoAnd don't forget one of the more understated benefits of TOML over INI: > A TOML file must be a valid UTF-8 encoded Unicode document.
- sampo 4y agoIs there a tool that parses yaml and outputs json, so you could check if the yaml you wrote parses as you intent?
- ancieque 4y agoI guess this can easily be scripted with https://pypi.org/project/yq/ https://pypi.org/project/yq/ Internally this seems to be a wrapper around jq :)
- chriswarbo 4y agoI use https://kislyuk.github.io/yq https://kislyuk.github.io/yq (it also includes and 'xq' command, which converts XML to JSON)
- jwilk 4y agogojq (https://github.com/itchyny/gojq https://github.com/itchyny/gojq) and fq (https://github.com/wader/fq https://github.com/wader/fq) can read YAML and output JSON. But that won't nescessarily help, because other parsers can intepreret that YAML differently.
- avereveard 4y ago"Parsing json is easy" https://seriot.ch/projects/parsing_json.html https://seriot.ch/projects/parsing_json.html https://raw.githubusercontent.com/nst/JSONTestSuite/master/results/pruned_results.png https://raw.githubusercontent.com/nst/JSONTestSuite/master/r...
- netgusto 4y agoThis is not claimed by the article afaict. It's stated though that json spec/syntax is simple, and that it's simple as a language; my interpretation is that this means "simple/understandable/predictable for humans".
- CipherThrowaway 4y agoI was at a place 7-8 years ago with a hand-rolled JSON parser that had been written across a few afternoons by one guy without much ceremony. It was untouched after that aside from some renaming and other minor refactors. As usual with these things, the decision to hand-roll was made by one very opinionated dev who got his way because the business depended on him. When that article was published it started an argument among the team. The guy ended up implementing the test suite to defend his decision. To my surprise the parser passed all tests apart from 3 or 4 unicodey-ones where it was more permissive than it should have been. There were plenty of production bugs at this place but none of them were due to the parser. IMO parsing JSON really is simple. If you read through the article you'll see that most of the test cases are dead simple and hardly landmines.
- shp0ngle 4y agoJSON is a subset of YAML. (you can just dump JSON in your YAML document and it’s a valid YAML.) So parsing YAML will be strictly harder, as it needs to include a JSON parser.
- jefftk 4y agoIf language A is a subset of language B it doesn't follow that parsing B is strictly harder. This is not guaranteed. Consider two languages: 1. Some number of "a" followed by the same number of "b". For example, aaabbb but not aabbb. 2. Any number of "a" followed by any number of "b". For example, aabbb but not aba. In this case (1) is a subset of (2), but parsing (2) is easier than parsing (1) since in (1) you have to verify that the numbers match (which, in technical terms, means its not regular).
- edent 4y agoThis is why I invented KYLI - https://shkspr.mobi/blog/2017/03/kyli-because-it-is-superior-to-json/ https://shkspr.mobi/blog/2017/03/kyli-because-it-is-superior... People shouldn't be hand-editing these sorts of configuration documents. 21st century tooling would prevent so many of these "foot-guns".
- progx 4y agoThis one liners lead to unreadable configuration. With tools, the format is not important. But we did not talk about tools, we talk about that humans can easy read and write a configuration.
- Firadeoclus 4y agoA text editor is a tool. Text files are only "human readable"/"human editable" in the sense that such tools are ubiquitous whereas structured data editors are not. Which is a real shame. Entering, manipulating and validating small amounts of structured data is such a fundamental task when using computers that it deserves dedicated tools, and the sheer amount of issues that only exist because such tools are not ubiquitous is absolutely mindboggling.
- bjarneh 4y ago> There were two changes to it in 2005 (the removal of comments, and the addition of scientific notation for numbers) They actually had comments in JSON but removed them in 2005. That is odd and a bit annoying :-)
- jjgreen 4y agoWhat we need is some kind of extensible markup language ...
- andybak 4y agoFunny but not quite. Nobody wants extensibility and "document markup" isn't the same as "configuration specification"
- dig1 4y ago> is that for a long time it was the only viable configuration format Actually, this is not the case. We had INI format for simple stuff and XML (protected with entire schema) for complex things many years ago, which worked. Yet, we wanted something readable (like INI), but able to express complex types (XML). I don't think Toml is a viable replacement - for me, it has an INI-level of simplicity with even worst hacks to have nested structures. But, give it time and you'll have another YAML. But yes, YAML is confusing for edge cases (true/false, quoting), but I'm going to find a powerful replacement that is not XML. Maybe EDN can jump in, but for anything more complex, I'd rather have a Lisp interpreter and structures around than any of mentioned and upcoming formats. Thinking that JSON is a suitable replacement, imagine writing Ansible or k8s stuff in it; that would be fun ;)
- CipherThrowaway 4y ago> Thinking that JSON is a suitable replacement, imagine writing Ansible or k8s stuff in it; that would be fun ;) Writing in YAML doesn't feel much better. YMMV but I've been on teams using Pulumi for k8s and the developer experience has been significantly better. I can automate, type check, lint, click through to definitions the same way I do with other typescript. Pulumi is a young product with many rough edges but it's already been a game changer for me.
- ilyt 4y agoIt's very simple actually. ... just fucking don't, generate config in your configuration management tool of choice and then serialize it to YAML. You get all of the advantages (nice to read)and none of the disadvantages (need editor or it is PITA)
- cocochanel 4y agoI checked the sample Kubernetes project in Java [0] and it looks hideous. YAML looks much better in comparison. [0] https://www.pulumi.com/docs/get-started/kubernetes/review-project/ https://www.pulumi.com/docs/get-started/kubernetes/review-pr...
- 4y ago
- milliams 4y agoAlmost all those problems are solved by marking up the strings as strings, using " and ".
- progx 4y agoThen they should make quotes necessary in the YAML-specification. Little more to write, but better to validate.
- milliams 4y agoI agree. They make a rod for their own back by having implicit conversion rules which, while well defined, are not well understood. "Explicit is better than implicit" and all that.
- progx 4y agoAdvantage: things like "literal_block: |" or ">" are not necessary with a simple quote. And i really don't know, why in hell they add binary-Data to a configuration language for humans.
- IshKebab 4y agoGreat idea. I would also suggest adding some syntax to make lists and maps more obvious (it can be a bit unclear in YAML). Maybe {} for maps and [] for lists.
- anonumous 4y ago[flagged]
- yccs27 4y agoMy wish for a dream config language is this: Allow a choice between unambiguously-typed expressions (quoted strings, only true/false, decimal numbers) and explicit type annotations. So this: regions: $[string] - no - se options: ${bool} a: yes b: no (with possibly a different syntax) would be equivalent to regions: - "no" - "se" options: a: true b: false
- jwilk 4y ago> yaml is a superset of json This is oft-repeated, but incorrect claim: https://news.ycombinator.com/item?id=30052633 https://news.ycombinator.com/item?id=30052633
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- privacywiki 4y agoAnsible collection writer here, pray for me...
- chrizel 4y agoI've never met anyone who say's "I like YAML, it is great"... most people that worked with it say something like "YAML is annoying, I don't like it"... While introducing Kubernetes at our company in the last two years, we are currently in a process going more and more away from YAML with internal Helm charts to a much simpler process by just using HCL and Terraform, and defining Kuberentes resources as Terraform resources. As a software developer HCL just makes so much more sense than this YAML + Helm + Go templates hell, which feels like C preprocessor hell all over again. Other solutions like kustomize are neat, but I don't see how all of these YAML workarounds should be better than something like HCL with Terraform. HCL feels like a real declarative programming language (with real conditions, variables, a module system and useful built-in functions). YAML feels like another more complex JSON and other tools like Helm or Kustomize try to work around the weaknesses of YAML with some kind of templating system. YAML looks nice to read in simple demos and in small files, but is just not adequate in the real world (in my personal opinion - I know that YAML is used by a lot of people in production as of today).
- xwolfi 4y agoYup exactly my experience as well, again a stupid idea to try and make a "configuration" language out of nested key value pairs that end up needing fancy interpreters allowing more and more semantic into the keys and values to start doing what a simple program could have done in half the time... I ve worked in 4 companies over a period of 10 years, each had exactly this problem, with yml, json, xml, properties file (you dont want to see business logic conditionals in a properties text file, where the keys shapes command an interpreter to behave dynamically...) The only times I saw a team do it well was a php backend of all things where the lead said they d program all their variations in php rather than source it from configuration flat descriptors and it was amazing, clear, simple and powerful. They had to release the backend at each config change instead of releasing the config change only, but Im still unsure why exactly that's a problem: the configs are software too if we re honest with ourselves, shoe-horning them in a descriptor language isnt gonna make them flat.
- ptsneves 4y agoThe problem is mixing data with logic. I cannot imagine how maintainable a small program becomes when this concept is employed, much less a big one.
- SuperSandro2000 4y agoI saw NixOS
- djrobstep 4y agoAmazing that developers still spend so much time getting bogged down in encoding issues in 2023. Why can't we just edit some data structures directly and send send them around seamlessly? It's a deeply dysfunctional state of affairs.
- ablob 4y agoFor that, you'd need a set of commonly accepted data structures. Alternatively, you can describe and send arbitrary data structures. Although that begs the question of how to process arbitrary data structures. (JSON and XML aren't arbitrary, for that matter)
- loa_in_ 4y agoThere's a lot of alternatives out there. One of the most complete (in terms of specification) and very promising is concise encoding (https://concise-encoding.org/ https://concise-encoding.org/) which focuses on the duality of editable human compatible representation (as text) and efficient binary encoding/decoding. The biggest issue right now is lack of actual implementations. The project is open source (I'm not affiliated with it).
- smitty1e 4y agoI like to write Python to manage JSON. Keeping the data and the logic orthogonal (these effin' Fn::'s make my eyes bleed) is a win.
- FpUser 4y agoThis is why my own configs are either ini where I know 100% that it would never become complex and take more than few lines or XML for everything else.
- marc_omorain 4y agoAt CircleCI, we have had to handle many strange YAML documents. Two that come to mind: A recursive document: key: &anchor - *anchor The `~` character represents null: ~: - ~ This parses as a map of {null: [null]}
- Hendrikto 4y agoWhat were the authors trying to accomplish with these strange documents?
- tiagod 4y agoBreak their parser, probably
- marc_omorain 4y agoThe recursive anchors seemed to be user-error. Either copy+paste bugs, or an anchors places in the wrong position. The null character would appear when people try to write a string path, and use the the `~` character without quotes.
- jrmg 4y agoI do think YAML is overly complex - but there is some hyperbole in this document. - Many of these complaints are about YAML 1.1. - YAML 1.2 was released _14 years ago_. - The author makes some allusions to 1.2.2, and it requiring a team of people 14 years to make, but, from the yaml.com announcement they link to: “No normative changes from the 1.2.1 revision. YAML 1.2 has not been changed” I guess my first two comments are undercut by PyYAML using YAML 1.1 (Really?! Python’s had 24 years of the Norway problem?!)
- wyuenho 4y agoYes, the problem is PyYAML.
- IshKebab 4y agoThe article mentions the fact that YAML 1.2 is really old and the fact that it doesn't matter because YAML 1.1 is still the most commonly supported version, and the fact that it's arguably even worse because YAML 1.2 gives different parsing results to YAML 1.1! I highly recommend reading the article - it's very good.
- jwmoz 4y agoThis explains a lot! I just know my docker compose files were always a pain and unintuitive to write.
- anonumous 4y ago[flagged]
- pprotas 4y agoDid you create this account just to place an insulting comment? Be kind
- anonumous 4y agoyou should be kind to idiots? then how will they learn that they are idiots?
- unsafecast 4y agoIndeed, we're being too kind to you.
- deleted 4y ago[deleted]
- jwilk 4y ago> In Python, you can avoid this pitfall by using yaml.safe_load instead of yaml.load. It's harder to make this mistake in Python these days. yaml.load(input) has been depreacted since PyYAML 5.1 (https://github.com/yaml/pyyaml/wiki/PyYAML-yaml.load(input)-Deprecation https://github.com/yaml/pyyaml/wiki/PyYAML-yaml.load(input)-...) and finally stopped working in 6.0.
- bambax 4y agoI never could understand the hate XML got, but I'm having a bit of schadenfreude seeing what people suffer through with its replacements. An XML document with a well-thought-out domain-specific DTD would solve all these problems; instead, we have something where no sometimes means false (but not always) and 22:22 sometimes means 1342 (but not always!)... because... why not?!? All that horrible mess, it seems, because people didn't like to have to close tags.
- Jenk 4y agoHard disagree. XML is great for systems to read and write, but utterly abhorrent for humans. It's not just having to close tags. It's content vs attribute confusion, namespacing noise (which are also muddled with attributes), and there's a squint factor incurred by the density of information.
- vincnetas 4y agoi think that propper schema and autocompleate would save the day. even i would say make things easier. talking about XML
- piskov 4y agoI’ve been using this for (schema + autocomplete) for at least 18(!) years. Moreover I even got comments on xml attributes and elements, warnings if required element is missing or required count is not met. You even got visual hierarchical collapsable UI for viewing/editing that XSD schema. I repeat: 18 years of autocomplete, comments on hover, and validation for typos/count/order for XML in Visual Studio.
- Jenk 4y ago20-something years of web/app.config is the primary reason why I hate XML. Very, very closely followed by Java ("a DSL for turning XML into stacktraces") and its over complicated ecosystem of xml configurations.
- donatj 4y agoI remember the first time I played with YAML ten-ish years ago. I experimented with converting our configuration file from INI to YAML. It came with an actual noticeable increase to our app startup time in the tens of milliseconds. The YAML evangelist in my company when asked told me I should cache the results… of reading our config file… from the local filesystem. What do I store that cache as? JSON? Just cut out the middle man. Left a really bad taste in my mouth at the time I have yet to get out.
- ainar-g 4y ago“YAML evangelist” sounds terrifying, and I have never met one in real life. Most people who used YAML basically resorted to the old argument of “Well JSON doesn't have comments, so YAML it is”.
- jasode 4y ago>The literals off, no, and n, in various capitalizations (but not any capitalization!), are all false in yaml 1.1, while on, yes, and y are true. Fyi... I looked at PyYAML source code and the parser code omitted single-character 'y' in the regex. Previous comment: https://news.ycombinator.com/item?id=26681295 https://news.ycombinator.com/item?id=26681295
- jwilk 4y ago> omitted the single character options of 'y' and 'Y' but it still has 'n' and 'N' 'n'/'N' were removed already in 2006: https://github.com/yaml/pyyaml/commit/8f9b8bed40a53c28 https://github.com/yaml/pyyaml/commit/8f9b8bed40a53c28
- djcris 4y ago[flagged]
- danjc 4y agoIn the linked YouTube video talking about JSON: "People were putting instructions to the parser in comments... so I [removed] the comments." - Douglas Crockford. Let the impact of that sink in - that decision has almost certainly literally cost millions of developer hours! In fact, the article actually claims that the lack of comments in part led to YAML itself so by implication... I jest (sorta) but what a great example of an initially helpful decision having last consequences.
- wodenokoto 4y agoEvery time I’ve reached for yaml after being bitten by the Norway problem has been because I wrote comments in my json file and it failed be read.
- krapp 4y agoWow, you must really be forgetful. Maybe your productivity would improve with some post-it notes stuck next to your monitor, reminding you of the basics about how these formats work.
- chii 4y agoCrockford probably didn't imagine the idiocy of creating YAML to replace the lack of comments in JSON.
- contravariant 4y agoIf Python is valid option then lisp syntax should also be considered as a viable alternative. Heck the number of times I've seen attempts at creating a lisp in yaml, it is easier to just get it over with and make it a lisp in the first place.
- thorum 4y ago> The yaml package supports most of YAML 1.2, but preserves some behavior from 1.1 for backwards compatibility. YAML 1.1 bools (yes/no, on/off) are supported as long as they are being decoded into a typed bool value. Otherwise they behave as a string. Good solution for using YAML with strongly typed languages. No reason to leave type decisions up to a parser. I assume serde does something similar in Rust?
- vsajip 4y agoThe CFG configuration format is a text format for configuration files which is similar to, and a superset of, the JSON format. It's not new - it dates from well before its first announcement in 2008 - and has the following aims: * Allow a hierarchical configuration scheme with support for key-value mappings, lists and basic types such as strings, ints, floats, Booleans and date/times. * Support cross-references between one part of the configuration and another. * Provide a string interpolation facility to easily build up configuration values from other configuration values. * Provide the ability to compose configurations (using include and merge facilities). * Provide the ability to access real application objects safely, where supported by the platform. * Be completely declarative. It's similar to newer formats such as JSON5, HJSON, HOCON and similar but offers a number of features [0] which they don't, as indicated by the above list. It's not intended to occupy the niche where you find things like Cue, Jsonnet, Dhall and similar. It was just never especially publicised when first implemented for use in Python projects, but it now also has implementations for the JVM, .NET, Go, Rust, D, JavaScript [1], Ruby and Elixir (all BSD-3-Clause licensed) and it would be great to get feedback on the project from the HN community. [0]: https://docs.red-dove.com/cfg/intro.html https://docs.red-dove.com/cfg/intro.html - description of features and comparison with other similar systems [1]: https://docs.red-dove.com/cfg/playground.html https://docs.red-dove.com/cfg/playground.html - uses the JS implementation to create an interactive playground
- haileys 4y ago> Templating yaml is a terrible, terrible idea I've had a good time using ytt: https://carvel.dev/ytt/ https://carvel.dev/ytt/. It implements language-aware templating, which is IMO the only reasonable way to do it.
- kevincox 4y agoLanguage aware is definitely important. But honestly I've usually found it easier to use a "real programming language" to generate the structures. You can read and write as normal. Then you can just serialize them to YAML at the end. Helm is so bad. It has a `| quote` function that is almost always wrong. People use it to escape strings into YAML strings but it just adds quotes around the values. It doesn't actually escape to YAML. I had a surprising hard time convincing people that they should be using `| toJson` and ideally be doing it for every interpolation. But really working with text is just an inconvenience here. You can imagine how much better everything would work if your base template was `console.log(JSON.stringify({ /* your config here */ }))`
- harshreality 4y agoI don't get all the YAML hate. The article even mentions solutions, which are all better than adopting some nonstandard json variant or toml. - for whatever language you're using, be aware of which YAML version its YAML library supports and its defaults, and how to safe load yaml in that language. - defensively quote strings, particularly if you're on a language with an antiquated yaml library that defaults to yaml 1.1. - defensively use true/false only, and proactively convert any other booleans in your codebase to true/false. - Depending on your language, you can avoid any custom data types for any externally-distributed applications to mitigate the risks, even when it might make things more convenient. Use safe loading (most languages with yaml libs support it) to avoid loading any. The other major YAML alternatives that YAML haters recommend won't have custom data types either. Features like references and folding semantics are very convenient, and you don't even have to use them. Basic yaml enjoys better readability than json. toml is only fine if you don't have much nested data. The author's note that there are json variants that help with some json failings makes no sense. If you're going to adopt some non-standard json variant, why not just adopt yaml 1.2, make sure your language has a yaml lib that supports it, and use that? At least yaml 1.2 is standardized. It's not their fault if python's libyaml only supports yaml 1.1. It looks like pyyaml is essentially in maintenance mode and ruamel.yaml is what everyone should be using? Unfortunately nobody's gotten around to implementing safe loading natively, but that's a python problem, not a yaml 1.2 problem, and ruamel.yaml supports pure-py safe loading that's compliant with 1.2 (no integer-interpretation gotchas), which is fine in most cases where yaml is only loaded occasionally, i.e. at start-up, and performance isn't critical. Obviously YAML has historical problems, but what's better? Using another flawed or even more limited data format, inventing your own which will begin with zero adoption, or simply ensuring your environment/app uses yaml 1.2 and best practices?
- omnicognate 4y ago> Obviously YAML has historical problems, but what's better? toml
- DougBTX 4y agoOne of the major use-cases for YAML is as a host language for DSLs, essentially basic programming languages for specific use-cases. TOML isn't as good there, as these languages often need deep nesting of data structures. GitHub Actions is the obvious example.
- polyrand 4y agoI wonder why generated JSON is not used more widely, something like jsonnet[0], although I've never used it. Most systems come with Python 3 installed, which supports working with JSON[1] without installing any dependencies. So, I'm wondering if we could use Python to generate JSON data that is then consumed by other tools. This would fix the lack of comments and trailing commas in JSON, you get functions and other abstractions from Python, and you don't need to install any third-party tools (vs. something like jsonnet). [0]: https://jsonnet.org/ https://jsonnet.org/ [1]: https://docs.python.org/3/library/json.html https://docs.python.org/3/library/json.html
- reilly3000 4y agoI’m at a crossroads and evaluating whether to introduce jsonnet to my org. We are doing a massive conversion of legacy yaml manifests from a proprietary vendor format, and there are not a lot of chances like this where you can make such a change at scale. Any experience reports or advice on jsonnet would be appreciated.
- Duologic 4y agoAt Grafana Labs we're using jsonnet at scale, while being a powerful functional language it is also excellent for rendering JSON/YAML config. We have developed Tanka[0] to work with Kubernetes, for other purposes I can recommend this course[1] (authored by me). [0] https://tanka.dev/ https://tanka.dev/ [1] https://jsonnet-libs.github.io/jsonnet-training-course/ https://jsonnet-libs.github.io/jsonnet-training-course/
- reilly3000 4y agoWow I love it here. Where else can you post something and get a project leader chiming in within hours? We are at a crossroads in moving towards GitOps and a CD vendor change, and are forced into some manifest conversion. From where we are coming from Helm is the obvious choice, but I do think we can do better, especially for debugging, readability, and inheritance. ArgoCD’s support for jsonnet turned me onto it and Tanka, and now I’m going to take it for a more serious spin. I’m early days with it but I’m curious about how life is with having a compiled language for an infra development. I had the same reluctance when thinking about making the Typescript leap, but that turned out just fine. I kinda want to jsonnet all the things if it works out well, but I worry the compilation steps would make it cumbersome for some use cases.
- nathell 4y agoEDN’s my favourite config format. + Simple (like JSON) + Has comments + Extensible + Built-in sets, UUIDs & timestamps + Well-suited to configuration (cf. Aero) + Lispy (well, Clojurey) - Lispy (well, Clojurey) - Handwavey specs - Not widely supported
- npteljes 4y agoWould you please link an example, a specification, or something? I think I found something on GitHub, but it was not straightforward.
- nathell 4y agoSure! Spec: https://github.com/edn-format/edn https://github.com/edn-format/edn Example (linter config): https://github.com/clj-kondo/clj-kondo/blob/634294183a0aa2ca53f705859cec0fadcbf1cabd/.clj-kondo/config.edn https://github.com/clj-kondo/clj-kondo/blob/634294183a0aa2ca...
- wg0 4y agoYAML with templating becomes troublesome even more so when these are Django tmeplates. That's why I do not prefer ansible for automation. It becomes so much cumbersome at certain point that it is more valuable to jump into a real programming language instead of wresteling with both YAML, Django templating and python. That's where I am enthusiastic proponent of pyinfra.
- rollcat 4y agoAnsible and YAML were my primary (de)motivators to create Judo (https://github.com/rollcat/judo https://github.com/rollcat/judo). This combo is extremely frustrating: for every line in a (hypothetical) shell script that would do one thing, I needed 3-5 (sometimes many more) lines of YAML. Most people on the team who were just getting started with Ansible, would often do half of their work just shelling out. I would usually push to do things "the Ansible way", but even I had to acknowledge the mental overhead of translating back & forth. I think what finally pushed me over the edge was when we started venturing into compose & k8s, and had to mix & juggle YAML+Jinja in two entirely different contexts, each with its own quirks, bugs, gotchas and brain damage. I figured I just need a layer of glue to run shell scripts across a bunch of remote hosts (hence Judo), and otherwise resort to other tooling (like Terraform, AWS CLI, k8s CLI, etc) for problems that don't map to SSH.
- sigmonsays 4y agowhy would you use this over pssh or dsh (https://www.garbled.net/clusterit.html https://www.garbled.net/clusterit.html ) ? It looks to do exactly the same thing, only slightly less efficiently.
- rollcat 4y agoI didn't know these existed, until I read your reply. Thanks - looks interesting, I will have to investigate further. First, I've done some research into alternatives, but my primary motivation was to prove you can replace Ansible with a weekend hack that closely follows the UNIX philosophy. Here I am 6 years later still using it everyday in production, so I guess I was right. Second, dsh looks a bit antiquated - defaulting to rsh? Granted I have very little experience with commercial Unices (which seem to be the primary use case there), but... rsh?
- sandwell 4y agoAgree with the final part of this article that "programmable" configuration languages like Nix and Dhall are the way forward. I've spent a lot of time writing YAML for Ansible, Cloudformation, k8s, Helm, etc. Some of the issues this article mentions are pitfalls but once you get a bit of experience with it, you know what to look out for. I've also spent and a lot of time writing Nix expressions, which is much more "joyful" IMO. Seemingly simple features like being able to create a function to reuse the same parameterized configuration makes life much easier. Add in a layer of type safety and some integration with the 'parent' app (think replacements for CloudFormation's !GetAtt or Ansible's handlers), the ability to perform basic unit tests, then configuration becomes more like writing code which I consider a good thing.
- d12bb 4y agoWhen I first discovered YAML, I loved it's apparent simplicity. Nowadays, I sigh every time I have to use it and am very happy with the Rust world having mostly adopted TOML.
- revskill 4y agoSecret fact: Hackernews data is stored as a huge YAML file on disk.
- carrja99 4y agoWe need to go back to xml. Admit it.
- singingfish 4y agoYAML is nuts, and JSON is annoying (trailing comma limitations, lack of comment syntax no matter how annoying it is that the spec is correct about why there are no comments). Both have their place though. YAML came out of perl, and both are some confluence between awesome and horriffic (although yaml wins the horrific crown for sure). I've had a little bit to do with Ingy - the inventor of yaml, and I've worked closely with some of his collaborators. Ingy is nuts, mostly in a good way, but I wouldn't put him in charge of the architecture, I'd put him in charge of the abyss.
- reilly3000 4y ago> I wouldn't put him in charge of the architecture, I'd put him in charge of the abyss. :-) I enjoyed this bit a bit too much, now I want it on my tombstone #lifegoals
- steveBK123 4y agoThis reminds me of a certain architect at my last shop who invented a DSL on top of his Python superapp. He expected all projects to go through his superapp. The DSL was configured in YAML. The YAML was often so dense he recommended devs use Jinja to generate the YAML. This meant debug was hell, plus it wasn't always clear if what you were trying to do was even supported / if not why & what needed to be changed. This was because you were now 3 levels of abstraction away from the Python code that was actually executing. Every time a dev took on a new project they had to jump on a call with architect or right hand man to figure out if what they were trying to do was going to be possible. It escalated into the architect demanding to know a sprint in advance any task devs were trying to do, in a review session, so he could explain if it was possible or not and try to triage in his DSL..
- goodoldneon 4y agoI would quit within a month
- steveBK123 4y agoSome did :-D
- d4rkp4ttern 4y agoFor managing configs of ML experiments (where each experiment can override a base config, and "variant" configs can further override the experiment config, etc), Hydra + Yaml + OmegaConf is really nice. https://hydra.cc/ https://hydra.cc/ I admit I don't fully understand all the advanced options in Hydra, but the basic usage is already very useful. A nice guide is here: https://florianwilhelm.info/2022/01/configuration_via_yaml_and_cli_with_hydra/ https://florianwilhelm.info/2022/01/configuration_via_yaml_a...
- bbojan 4y agoI've always hated YAML, and now I'm glad there's a very nicely written article to back me up. I also notice that I have a preference to procedural as opposed to declarative specifications. E.g. I find conditionals and loops in declarative specifications quite bad.
- deafpolygon 4y agoI will stick to JSON or TOML if the choice was mine. Hell, I'd much rather deal with XML than deal with YAML.
- xg15 4y ago> Json is so obvious that Douglas Crockford claims to have discovered it — not invented. I mean, it's called "JavaScript Object Notation" for a reason. From what I know, the beginnings of it really were in the early days of AJAX: People were looking for a good way to encode complex data structures in dynamically generated script tags. Then someone stumbled over essentially a random syntax element in JavaScript that turned out to be extremely well-suited for that task. So I think talking about "discovering" JSON as a JS syntax feature which later became its own thing is quite correct here.
- adenozine 4y agoEvery day we stray farther from the light of XML. At this point, it’s just plain ridiculous.
- PurpleRamen 4y agoYAML seems like a good that gets harmed by feature-creep. All I want when using YAML is a simple human-friendly text-format for slightly structured data. But all those special features and edge cases...as if I even have the time to learn about them. Values should be only text. Outside of the handful clear specified cases of numbers and maybe true/false/none, they should not have any other interpretation on the parser. Isn't there something like a simple YAML-definition and some YAMl-schema for the powerusers?
- singularity2001 4y agoWouldn't it be easy to make those features optional and switched off by default in "YAML--"?
- kitd 4y agoHe mentions it at the end (the 'YAML subset' alternative), but the edge cases only cause issues if your keys and values begin with special characters but are intended to be simple text. I tend to defensively quote stuff (probably much more than I have to), and I rarely find these issues. I have other issues with YAML but not this.
- ilyt 4y ago> Values should be only text. Outside of the handful clear specified cases of numbers and maybe true/false/none, they should not have any other interpretation on the parser. I think simpler way would be just forcing quoting on text. That fixes the boolean and the different base numerics problem
- shp0ngle 4y agoYAML has come from Perl world. Of course it has feature creep. It’s a Perl thing.
- rqtwteye 4y agoAt least YAML has comments, which JSON doesn't. I know people will say "JSON is not meant as config format" but in reality it is used as such. In general I don't understand why there is so much churn and focus on config file formats. They can be annoying but in the end the real problem is that we have complex configurations so a different format is not going to help much. Personally I prefer YAML over XML or JSON for config because I don't have to worry about closing brackets or braces. Maybe we need a good editor to make editing them easier.
- Kaotique 4y agoWhile it is indeed total insanity it could easily be avoided by putting quotes around all string values.
- sigmonsays 4y agoi want *any* configuration format that can rewrite itself(loaded and written to disk again) with two properties we've seem to have lost when we switched away from xml. 1. keep ordering of document 2. keep comments of document Does anything like that exist?
- InsomniacL 4y ago"YAML is Human-readable" "You must count whitespace for nesting" Pick one or the other, but not both.
- jeroenhd 4y agoYou don't need to count whitespace for nesting: test: something: complicated: {even: { more: { levels: {can: { be: {achieved: {like: {this: true } } } } } } } } Whitespace nesting also doesn't seem to be a problem for Python programmers. I don't prefer it myself (especially because of the tabs vs spaces problem) but it's not good or bad.
- InsomniacL 4y agoExtra whitespace before 'something' in your example would cause an issue. > Whitespace nesting also doesn't seem to be a problem for Python programmers. It does cause problems, IDE's mitigate the issue, the same as they do for YAML. However YAML is supposed to be a human readable data-serialization language and If you need an IDE to make it human readable then that defeats the point. I can't imagine ever being convinced that having to 'count the number of invisible things to determine what something is' is human readable even if you can use other lines as reference points.
- amarshall 4y ago> The key on is common in the wild because it is used in GitHub Actions. I would be really curious to know whether GitHub Actions’ parser looks at "on" or true under the hood. I can say, at least, that using both on: and 'on' work in GHA. I usually write the latter, makes it simpler to do ad-hoc YAML.parse or whatever on the workflow files.
- jgalt212 4y agoSomeone needs to write YAML: The Good Parts If you're strict about how you use it, it's more than workable. And, yes, the Norway problem is insane.
- vbezhenar 4y agoThe way to solve YAML issues is to invent some widely used yaml pretty printer and shout at everyone not using it. There's `yq -P` but it does not implement proper normalization for plenty of mentioned issues which is a shame, really. As I see it, everyone can just start with JSON (as JSON is a subset of YAML) and pretty-print it and use the result whenever it's supposed to be used. In other words use safe subset of YAML. May be it'll be formalized some day. Yamlite.
- speed_spread 4y agoThat safe subset exists and is implemented in a number of languages. It is called strict-yaml: https://hitchdev.com/strictyaml/ https://hitchdev.com/strictyaml/
- vbezhenar 4y agoThey don't use quotes, so not safe.
- kergonath 4y agoIn which languages is it implemented? I have seen a supposedly equivalent Rust library, but could not find anything else. Is there any C library or something like that? Unfortunately, these are the easiest to use from a lot of languages because of how easy they are to wrap. (I am genuinely interested in any useful library better than libyaml).
- dmos62 4y agoI was looking for a human-friendly format to store data for a staticly generated website. Stuff like copy, lists, addresses, contact information (so that the rendering logic is kept separate). Landed on NestedText, and apart from it having few implementations, I loved it! https://nestedtext.org/en/stable/ https://nestedtext.org/en/stable/ I did feel like I missed a more popular alternative, because I can't be the only one to want a human-friendly data format.
- AndyKluger 4y agoI'm a huge fan of NestedText, especially as there is no escaping needed ever. If you ever want to use it as a pre-format to generate either TOML, JSON, or YAML, I used the official reference implementation to make a CLI converter between them and NestedText. When generating one of these formats, you can use yamlpath queries to concisely but explicitly apply supported types to data elements. - My CLI converter: https://github.com/AndydeCleyre/nestedtextto https://github.com/AndydeCleyre/nestedtextto - yamlpath info: https://github.com/wwkimball/yamlpath/wiki/Search-Expressions https://github.com/wwkimball/yamlpath/wiki/Search-Expression...
- dmos62 4y agoWow, that's really versatile! Thanks for sharing! `json2nt` seems like a great alternative to `jq .` for pretty printing.
- notinfuriated 4y agoTo me, the yaml document from hell is a series of yaml documents I helped create that replaced a bunch of database config and hard-coded constants. It helped 'unlock' the business to create newer forms of products much more quickly and correctly, but it never got moved to an actual database. So at this point, they have some yaml documents that, if expanded to include all references, end up becoming this massive fucking tangled mess. IIRC we never moved it to a database partly because we did not have time to solve for how we actually wanted references to work. The yaml document and code that reads it still work as a way of speeding up the business compared to how things used to be. But that code takes a long time to parse, has probably caused a decent level of confusion for developers who have to mess with it who aren't on that team, and has absolutely taken a lot of clock time being parsed for validation checks in unit tests. Still not sure I would have solved that problem differently, aside from perhaps moving the thing to a database immediately after the first phase of working on it.
- shp0ngle 4y agoWhat would be good is “strict yaml”, that is, yaml with all those weird cases strictly disallowed? Which would make it backwards compatible, as the cases would just make it invalid “strict yaml”? And I’m thinking it might already exist? I’m surely not the first one to think of that
- dbfa 4y agoIt exists, and it is called "JSON" :)
- messe 4y agoJust say Norway to YAML!
- elgertam 4y agoMany commends are (correctly IMO) pointing to JSON as superior to YAML given YAML's problems, but these ignore JSON's problems due to its lack of semantics. When moving JSON between systems, the JSON parser makes certain assumptions about the JSON document, such as whether the numbers represent integers or single-precision or double-precision floats. This is certainly less obnoxious than "no" being treated as a bool, but it is a problem. A friend of mine created Preserves[0] in part to resolve this problem. I'd encourage everyone to check it out. To be fair, he's also quite pro-Dhall and Nix. [0] https://news.ycombinator.com/item?id=34354614 https://news.ycombinator.com/item?id=34354614
- theamk 4y agoThe simple answer to that is to store anything that may not fit in double without loss as a string. (I don't know of JSON decoders which use floats) After all, one of the biggest upsides of JSON is its wide language/tool compatibility. And that means intentionally limiting the format to the most basic subset of types that all languages have.
- lucasfcosta 4y agoI honestly don't understand why people still write YAML considering there are alternatives like TOML available. I think attributing semantic value to indentation is a terrible idea. It gets even worse when there is so much visual ambiguity when it comes to defining values and escaping certain character sequences.
- r3trohack3r 4y ago> loading an untrusted yaml document is generally unsafe, TIL and this is... Not great? I often parse JSON as YAML. Since YAML is a superset of JSON, all valid JSON files parse as a valid YAML file too, but now you can add comments to your JSON since YAML has comments. Didn't know I was opening my process up to code execution doing this... Knowing this, YAML is dead to me as a format. This really seems unacceptable for a markup parser.
- twic 4y agoI'm not sure it's true that parsing an untrusted YAML document is generally unsafe. That has to be a property of the parser, not the format. As TFA says: > In Python, you can avoid this pitfall by using yaml.safe_load instead of yaml.load In Ruby, there are now analogous options (including - of course lol - a gem which monkeypatches the YAML module to make it safe by default) [1]: > Use the safe-yaml gem which overrides YAML.load > Require a newer version of Psych that provides a safe-load option In Java, SnakeYaml has a document about security which i think is a long way of admitting that it's not secure [2]. It looks like eo-yaml just doesn't support tags at all. [1] https://trailofbits.github.io/rubysec/yaml/index.html https://trailofbits.github.io/rubysec/yaml/index.html [2] https://bitbucket.org/snakeyaml/snakeyaml/wiki/CVE%20&%20NIST.md https://bitbucket.org/snakeyaml/snakeyaml/wiki/CVE%20&%20NIS...
- kroltan 4y agoTo be fair, YAML itself does not specify how tags should be interpreted, technically the idea of grabbing constructors from those is the library's issue. But then again, you wouldn't want to accidentally introduce code execution when you adopt a new library/language.
- Beltalowda 4y agoPretty much all parsers have some way to "safely" parse a YAML document which disables the custom-object unmarshal stuff; sometimes this is the default, sometimes not, some libraries/languages don't support this at all in the first place. If it's enabled, then it's trivial to exploit; e.g. in Python: !!python/object/apply:os.system args: ['ls /'] If you parse this with "yaml.load()" in PyYAML (which, I believe, is the most commonly used YAML library for Python) then it will execute that code. You will need to use "yaml.safe_load()". I believe the situation in Ruby is similar, and in PHP the only way to get any safe sane behaviour is to modify a php.ini setting (which at least defaults to "off" now, but there is no way to know if a line is safe just by looking at the code). In Go this entire feature isn't supported at all AFAIK, so it's always safe. Not sure about other languages from the top of my head.
- nrvn 4y agoit is an old song. JSON (as per spec) is a lightweight data-interchange format. YAML is a "human-friendly data serialization language for all programming languages" and by mere coincidence (it does not look to be by design until version 1.2.2 [0]) JSON is its subset and not vice versa! And YAML is human-friendly in two ways: - it is notoriously "human" in interpreting input, i.e. ignoring the context or misinterpreting others' words; - it helps read serialized data via indentation-based syntax. The biggest caveat is for whatever reason everyone ignores the key highlights of both JSON and YAML that are "data-interchange format" and "data serialization language". If I were to interpret these two terms I would rather say they both present great means for heterogenous programs to exchange data with YAML being a fancier brother better suited for human eyes. They were not meant to be used for configuration purposes or what not. But people started using them exactly for this purpose instead of creating a configuration format/language that would render into program-readable JSON(if needed). CUE and Dhall are perhaps the first two attempts to create specifically a configuration language with the ability to serialize in yaml/json/etc. I very much hope using YAML, JSON and even TOML for writing configurations will become an anti-pattern soon and will be replaced by something more suited for this role. [0] https://yaml.org/spec/1.2.2/#12-yaml-history https://yaml.org/spec/1.2.2/#12-yaml-history
- PaulHoule 4y agoAbout Sometimes an application will start out with a need for just a configuration format, but over time you end up with many many similar stanzas, and you would like to share parts between them, and abstract some repetition away. This tends to happen in for example Kubernetes and GitHub Actions. When the configuration language does not support abstraction, people often reach for templating, which is a bad idea for the reasons explained earlier. Proper programming languages, possibly domain-specific ones, are a better fit. Some of my favorites are Nix and Python Templating is fine with YAML if you don't do it with strings but with a system that turns one YAML document into another YAML document, that is, do your substitutions in the YAML universe. The "code up your configuration in your favorite programming language" can really work but it frequently becomes a point of frustration itself. As much as I want to like YAML it has a number of intrinsic problems like the Norway problem... But I do think people are confusing the accidental and essential complexity and because of that they flinch and reach for non-solutions to the problems of configuration rather than really thinking it through.
- dgan 4y agoIf you only need something for internal use, nothing prevents you from using s-expressions, and interpret atoms (number -> uint16_t, dates in "yyyy-mm-dd") whatever your application feels like... With this lil' trick all parsing problems are gone. Inferior Format Users will be jealous, and Inferior-Format-Parser-Writers will hate you
- TheFlyingFish 4y agoTo me, the #1 reason to use YAML rather than anything else is if you need to embed complex multi-line strings in your document. This is a practical necessity in CI systems: you always end up with at least a little bit of shell scripting to glue things together, and embedding shell scripts into anything other than YAML is an absolute nightmare. Yes, there are too many options for expressing multi-line strings (In the vast majority of cases, I really don't care whether it has a trailing newline or not). Yes, there are weird rules that don't make sense, like how in "folding" mode a newline followed by two spaces more than the current indentation level doesn't get folded. But it's still the best option for CI, which is why Github actually ditched HCL in favor of YAML after a while: https://news.ycombinator.com/item?id=20647691 https://news.ycombinator.com/item?id=20647691
- the-alchemist 4y agoSince everyone seems to throwing their favorite format into the ring (), I will too: EDN [0] * no enclosing element (i.e., can be streamed) * maps, lists, even sets * tags (like "Person". UUIDs and timestamps are built-in tags) * floating point numbers * integers * comments * UTF-8 * true booleans * no need to worry about too many or too few commas in the right or wrong place Implementations in almost every language under the sun [1]. The format is simple enough that it's easy to implement, verify, and test. No strange string interpretation craziness (see YAML and "Norway problem"), no ambiguity between FP and integers (see JSON), comments. And if your editor has rainbow parenthesis support, reading is actually a pleasant experience. [0]: https://github.com/edn-format/edn https://github.com/edn-format/edn [1]: https://github.com/edn-format/edn/wiki/Implementations https://github.com/edn-format/edn/wiki/Implementations
- deleted 4y ago[deleted]
- rsynnott 4y agoBut, of course, _all_ yaml documents are from hell.
- luisapic 4y ago[flagged]
- jackcviers3 4y agoThis is why configurations as code should be written in actual programming languages. Dhall, Nix, and Pulumi are all solutions geared towards configuration but written with fully capable, user-extensible, programming languages.
- spacechild1 4y agoDon't forget Lua! Can be easily sandboxed, meaning you have full control over which functionality is made available to the user. Also, the Lua interpreter itself fits in less than 300kB and can be easily embedded in any language that supports C extensions or FFI, so people don't even need to write their own parsers.
- chrisshroba 4y agoCall me naive or quote xkcd, but whenever I read about the shortcomings of YAML, I daydream about a new standard that's mostly a subset of YAML, that looks like this: rules: - "All strings must be quoted and are parsed identically to JSON strings" - "true and false are the only boolean values" - """multiline strings are allowed""" - """| there can even be special multiline string types to indicate things like "dedent every line and trim the beginning and end lines if empty """ # Comments are allowed! - Number syntax just like JSON better_than_yaml: true "a number with multiword key": 1e9 nesting: present: true difficult: false It has the ease of coding of yaml, maps 1:1 with json (so can easily be re-serialized to json or yaml if needed), is familiar to folks who know yaml, and supports comments. And it eliminates ambiguous parses and gotchas, references, tags/directives, and other complexities.
- synergy20 4y agoso the suggestion is: use toml for configuration, otherwise just stay with json? there is jsonc which is json+comments but I have not used.
- kabdib 4y agoI will never willingly use YAML on a project ever again. It's a bloody disaster. I would actually prefer XML over YAML.
- mixmastamyk 4y agoStrict yaml fixes this, I recommend it. Before that I disabled as many of yaml’s features in the library I was using. Look for “safe parsing.”
- rochak 4y agoYAML is the worst format I've had the displeasure of working with. Given how ubiquitous it is, I don't see it going away soon, but I am still hopeful.
- groby_b 4y agoWelcome to dynamic typing hell. As long as you ask your configuration language to infer the data type from a piece of text, these are the kinds of edge cases you get to experience. (They vary by language. Less types means less mistakes, and so does less expressive power - but without some form of type hinting, that's where you are) Personally, I like YAML for expressive simplicity at the start of a project, and then if it becomes an actual load bearing piece of software (vs a haphazard one off, or an experiment), let's add some schema validation & typing.
- pnt12 4y agoAn unquoted version 3.10 in yaml becomes 3.1. Fun stuff!
- tinglymintyfrsh 4y agoLet's not forget: JSON is also valid YAML YAML, OpenSSL, and ASN.1 are what happen when every feature in the world is added.
- slowmovintarget 4y ago"Use the EDN, Luke!"
- theshrike79 4y agoYAML isn't the best, but it's the least worst. INI doesn't let me do hierarchies and is out right out of the gate. I started my career during the "XML ALL THE THINGS" craze and feel physically ill when I need to edit XML. It's technically the best, but it can be misused in so many ways[0] that it makes it unviable. XML Schemas, DTD etc make it a breeze to handle programmatically though. JSON might be good, but it doesn't have first-class comments and no links or refers. It even kinda has a schema spec. YAML has a ton of edge cases (like in the linked post), but it's hierarchical, human editable and machine readable-ish as long as you take the edge cases into use. You can also link and refer elements to reduce repetition. [0] elements with no values, but 20 attributes, too many elements with values, but no attributes, etc etc...
- OOPMan 4y agoI've ways preferred HOCON to YAML but it never took off :-/ It's a pity, because compared to YAML it's a lot simpler from the rules perspective while being flexible enough to do some useful stuff over plain JSON Also, it isn't white space dependent, something which annoys me know end with YAML
- nickm12 4y agoFor a human-readable, schema-free data representation, I really like Amazon's Ion format (disclosure, I'm an Amazon employee). It is basically JSON + additional types (integers, timestamps, symbols, sexpressions), and support for humane features like trailing commas and comments. It also has a more efficient binary representation. I think it could have been a contender had it been released a decade earlier, but now I think the existing formats are way too entrenched. https://amazon-ion.github.io/ion-docs/index.html https://amazon-ion.github.io/ion-docs/index.html
- deterministic 4y agoI am not a fan of LISP but I use S-expressions all the time for configuration files, dumping data etc. It is simple yet infinitely flexible.
- deterministic 4y agoYaml is the complete opposite of what you want: a simple specification language. JSON is the gold standard to measure against. However one step better IMHO is S-expressions.
- rubatuga 4y agoI'm surprised nobody mentions an alternative called NestedText. It only supports string type, which puts the complexity and responsibility of typing with the application. I think it's less prone to parsing errors. https://nestedtext.org/en/stable/ https://nestedtext.org/en/stable/
- runeks 4y ago> Leaving strings unquoted can easily lead to unintentional numbers. It seems like this sums up all the issues pointed out by the article. Ie. you’re safe as long as you quote your strings.