10 ms·

Bugs in the YAML specification

102 points by networked 10y ago

jstimpfle 10y ago
This is an opportunity to plug my private project "WSL" [1] which is a clean text serialization format for relational databases. The scope is somewhat different and it's not really released (but beginning to stabilize), but I'll be happy to hear what you think. [1] http://jstimpfle.de/projects/wsl/main.html http://jstimpfle.de/projects/wsl/main.html
- nerdponx 10y ago
  This is interesting! You should make a separate thread about it.
- kiwidrew 10y ago
  Interesting -- I like the concept. But it's completely unusable for the Real World (tm) because strings cannot contain the '[' and ']' characters AND YET there is no mechanism for escape sequences. What if my data legitimately contains those two characters???
  - jstimpfle 10y ago
    Thanks for reading and the feedback! That's just the default string type. I started out with escaping but noticed it's a lot of complexity that is rarely needed (not for my own use case, which is accounting, inventory, and some web apps which don't need it). The advantage of not having escaping is easier seds and greps which don't miss the field boundaries. The important concept though is that arbitrary datatypes elements can be added by the user of an API implementation (the python library already offers that). The datatypes define their lexical syntax, like in perl6. I will also declare more "default" datatypes and might include a C-like string after enough consideration.
    - scrollaway 10y ago
      You can't create a serious serialization format that fails its primary job of serializing data. What you implemented there isn't a serialization format, it's a SQL DSL. Any time you implement a DSL, the first question should always be: "Have you considered S-Exprs?". If s-exprs do the job, then you just saved yourself having to implement a ton of stuff, including 90% of a spec.
      - jstimpfle 10y ago
        Well, the primary job of WSL isn't serializing arbitray binary data. And it doesn't fail there either, it only encourages clean syntax by default. But it can very easily be made to serialize anything. The conventional wisdom is "All problems in computer science can be solved with another level of indirection". Nevertheless, as stated I will very likely add C-style string literals to the required set of datatypes. As to "it's a SQL DSL". It's very much not. You missed the point. And S-Exprs is completely out of scope (no schema support, no nice and canonical syntax, etc.)
    - skybrian 10y ago
      This can be useful but to avoid confusion, I wouldn't call it a string type at all. Maybe it's an identifier or a symbol or a label or something like that? If you're excluding square brackets, there are probably other special characters you want to exclude too?
      - jstimpfle 10y ago
        Thanks for the feedback, and I'm glad to see other people worry about these details, too! ASCII control characters are forbidden in the entire WSL file. Then apart from [] everything is allowed. For practical applications, by far the most important requirement of strings is being able to include space characters to make a short sequence of words, like [Trinidad and Tobago]. I don't know a better word for "sequence of words" than "String". Maybe "Words", but technically it really is a string (containing an arbitrary sequence of the allowed characters). Even approaching the enforcement of more structure would be a lot of work with little returns. And you can't include a literal newline in a C string literal, and you can't even have a NUL character in the interpretation (memory layout) of it, right? I actually started out with a C-like string as default type (so named it "String") but noticed - A big problem with string culture is that "" strings use identical start and end markers. A problem which Joe Armstrong mentioned as well. - Escaping means significantly higher complexity of parsing out the interpretation from the literal, while it's not really needed for most applications. Both these problems make data unnecessarily hard to process with dirty one-off scripts. So after considering some other options I'm now with [this style] because [] are not too often needed or when needed can often be substituted with (), and are very pleasing on the eye in most fonts. In conclusion I guess it will stay "String", and other less frequently needed types will be called "CString", "Base64", and "BinaryString". Or optional parameterization will be created for "String" to declare the escaping style without needing a separate metatype.
lwis 10y ago
I've always felt that YAML was incomplete.
- TillE 10y ago
  If you want to use it for complex things, sure, there's no end of potential features. If you need some logic with your data, I like Lua. But I think most uses of YAML are basically just a much more flexible JSON.
  - AstroJetson 10y ago
    If you want to use it for complex things, sure, there's no end of potential features. If you need some logic with your data, I like Lua. Say some more about this. I look at YAML as a way to do data exchange, like using JSON/SON. How does Lua fit into this?
    - Avshalom 10y ago
      Not TillE, but I suspect the point is you can exchange data with Lua tables just as easily as JSON and because you would do it by just embedding the Lua interpreter you can also send Lua functions around with the data and/or process the data in Lua before making it accessible from the actual program.
      - justincormack 10y ago
        Lua was originally a configuration language. Its syntax is very human readable especially for just data use cases, but you can add functions etc too.
        AstroJetson 10y ago
        Thanks. I was looking at the process to process data sharing that you would use YAML or JSON for and how Lua would be used. It's never occurred to me to serialize a table and send it. I'll put that on the stack of things to research.
crdoconnor 10y ago
Despite being a nice readable language if you use it right, the spec is rife with disgusting features and nasty surprises. This was my attempt to cut out 90% of the crap, leaving the nice, readable core (which maps on to JSON pretty neatly): https://github.com/crdoconnor/strictyaml https://github.com/crdoconnor/strictyaml
- jean- 10y ago
  Very nice! Another saner alternative to YAML which I like is TOML: https://github.com/toml-lang/toml https://github.com/toml-lang/toml
  - voaie 10y ago
    TOML don't even have a clear and definitive standard/test suits. It seems unmaintained but some people are trying to push it to 1.0 no matter what.
  - crdoconnor 10y ago
    I actually went over why I'd consider this is a better format than TOML: https://github.com/crdoconnor/strictyaml/blob/master/FAQ.rst#why-not-use-toml https://github.com/crdoconnor/strictyaml/blob/master/FAQ.rst...
- akkartik 10y ago
  I love the example at https://github.com/crdoconnor/strictyaml/blob/master/FAQ.rst#what-is-wrong-with-implicit-typing https://github.com/crdoconnor/strictyaml/blob/master/FAQ.rst.... Convinced in 2 seconds that full YAML is idiotic.
  - guitarbill 10y ago
    this whole idea of strictyaml just makes so much sense to me. plus having comments is a huge bonus. why am i only hearing about this now?
    - 10y ago
jgalt212 10y ago
If you stick the language agnostic core, yaml is great. We use it for: 1. test case specifications we use for unit tests suits that must run across multiple languages. 2. DSL for business processes
ThePhysicist 10y ago
What I find more interesting is that YAML allows you to define self-referential / circular data structures: foo: &foo bar: *foo In PyYAML, this will give you a self-referntial dictionary. Powerful but pretty catastrophic if you use (naive) recursion to analyze a user-submitted data structure.
- twelvechairs 10y ago
  Feature not a bug. YAML is what people turn to who need these kind of advanced feautures which arent provided in json or similar.
  - OskarS 10y ago
    I would imagine this feature is primarily useful if you want to serialize a whole bunch of objects that all reference each other, but it makes me feel a bit icky. It feels like it breaks an intuitive assumption you have about hierarchical formats, which is that there should be no cycles. Maybe that's just my personal bias, but I feel like the relative simplicity of JSON is a strong feature in its favor. As a developer, I have very clear understanding of the data I'm reading, and with that I can more easily make safer and more stable code.
    - junke 10y ago
      Yet, a lot of hierarchical structures I see are full of "ref" and "id" attributes, which effectively encode cyclic graphs.
      - OskarS 10y ago
        That's certainly true, but at least that's not going to cause an infinite loop while wandering down the structure, it's just gonna end at a leaf "ref" or "id".
        junke 10y ago
        In my view this is more a question of simple interface vs. simple implementation. Having cross-references at the language level gives a uniform and simplified interface at the cost of more having to develop careful tools. Not saying refs/id are bad, but at least with a uniform syntax and language support, you don't have to reimplement custom cross-reference resolvers.
eyelidlessness 10y ago
I have minimal experience with YAML (100% of which configuring for CI environments because apparently that's what all the modern ones prefer). I don't really know the ins and outs of the format, and try to stick to extremely simple representations. Today I learned that what I perceived as somewhat complicated syntax is actually overwhelmingly complex. I know that my preference for s-expressions is not shared by everyone, but the more complex a syntax I encounter, the more I wonder if simpler alternatives were even considered. Genuine question: apart from inertia, and apart from recursive references (noted in comments before I wrote this), is there a use case for YAML that isn't solved by simpler and less ambiguous formats like EDN or Transit?
- jaredklewis 10y ago
  Just one data point, but I had never heard of transit or edn before today. Both look awesome. On the other hand I picked up YAML in less than a day (many years ago) and despite its complexity can't remember that ever causing me any issues. Not 100% sure if missing from your suggested formats (which do support multi line strings) but I enjoy use of yaml's folded strings.
- fibo 10y ago
  YAML was the first human readable/writable format, so better t'han XML and JSON for us humans. Also is a superset of JSON.
  - Someone 10y ago
    Human writable? I have little experience with it, but if what Amazon uses for its CodeDeploy configuration files is representative for the format, I disagree. IMO, a human writable format shouldn't need a help page like this: http://docs.aws.amazon.com/codedeploy/latest/userguide/app-spec-ref-spacing.html http://docs.aws.amazon.com/codedeploy/latest/userguide/app-s... Yes, part of that is CodeDeploy, but saying "tab characters must not be used in indentation" for me, disqualifies a standard as "human writable". CodeDeploy makes that horrendous by requiring five or so clicks to figure out what causes an error (try forgetting adding an app spec.yml file or saving a file with a BOM)
burke 10y ago
Piling on to the general theme of the rest of the comments here: I really wish there was a more popular middle-ground between YAML and JSON. People use YAML because it's the next step up from JSON if you need comments, etc., but I think most purposes would be better served by the likes of JSON5, HJSON, or TOML (for example) if only any of those were as popular as JSON and YAML. I implemented a YAML parser from spec last month. It (YAML) goes to great lengths to provide human-friendly features, trading off computer-friendliness to an fairly extreme extent. Eliminating 'plain' scalars (unquoted strings-as-values), folded multiline literals, tags, anchors/aliases, and possibly directives, as a sort of reduced yaml would make the language a lot less silly for the kinds of things a lot of people end up using it for.
- crdoconnor 10y ago
  JSON5, HJSON and TOML are not as readable or editable as YAML. They may have a slight advantage over the default spec of YAML but they don't stack up well against YAML with the features you mentioned removed. I really don't think an entirely new standard is warranted.
  - burke 10y ago
    Right. My hypothesis here is that, if there were, instead of JSON and YAML to choose from, three standards, one of which were something slightly less human-friendly but much simpler than YAML, I think it would be widely adopted. I think YAML is more widely-used than it would otherwise be, in a broader variety of domains, simply because the only popular alternative doesn't have comments. I'm not drawing from this any sort of conclusion that we should try to push one of these alternatives as viable, just lamenting the state of things.
- mst 10y ago
  ingy and I already invented that. It's called JSONY.

102 points by networked 10y ago

This is an opportunity to plug my private project "WSL" [1] which is a clean text serialization format for relational databases. The scope is somewhat different and it's not really released (but beginning to stabilize), but I'll be happy to hear what you think. [1] http://jstimpfle.de/projects/wsl/main.html http://jstimpfle.de/projects/wsl/main.html

nerdponx 10y ago
This is interesting! You should make a separate thread about it.
kiwidrew 10y ago
Interesting -- I like the concept. But it's completely unusable for the Real World (tm) because strings cannot contain the '[' and ']' characters AND YET there is no mechanism for escape sequences. What if my data legitimately contains those two characters???
- jstimpfle 10y ago
  Thanks for reading and the feedback! That's just the default string type. I started out with escaping but noticed it's a lot of complexity that is rarely needed (not for my own use case, which is accounting, inventory, and some web apps which don't need it). The advantage of not having escaping is easier seds and greps which don't miss the field boundaries. The important concept though is that arbitrary datatypes elements can be added by the user of an API implementation (the python library already offers that). The datatypes define their lexical syntax, like in perl6. I will also declare more "default" datatypes and might include a C-like string after enough consideration.
  - scrollaway 10y ago
    You can't create a serious serialization format that fails its primary job of serializing data. What you implemented there isn't a serialization format, it's a SQL DSL. Any time you implement a DSL, the first question should always be: "Have you considered S-Exprs?". If s-exprs do the job, then you just saved yourself having to implement a ton of stuff, including 90% of a spec.
    - jstimpfle 10y ago
      Well, the primary job of WSL isn't serializing arbitray binary data. And it doesn't fail there either, it only encourages clean syntax by default. But it can very easily be made to serialize anything. The conventional wisdom is "All problems in computer science can be solved with another level of indirection". Nevertheless, as stated I will very likely add C-style string literals to the required set of datatypes. As to "it's a SQL DSL". It's very much not. You missed the point. And S-Exprs is completely out of scope (no schema support, no nice and canonical syntax, etc.)
  - skybrian 10y ago
    This can be useful but to avoid confusion, I wouldn't call it a string type at all. Maybe it's an identifier or a symbol or a label or something like that? If you're excluding square brackets, there are probably other special characters you want to exclude too?
    - jstimpfle 10y ago
      Thanks for the feedback, and I'm glad to see other people worry about these details, too! ASCII control characters are forbidden in the entire WSL file. Then apart from [] everything is allowed. For practical applications, by far the most important requirement of strings is being able to include space characters to make a short sequence of words, like [Trinidad and Tobago]. I don't know a better word for "sequence of words" than "String". Maybe "Words", but technically it really is a string (containing an arbitrary sequence of the allowed characters). Even approaching the enforcement of more structure would be a lot of work with little returns. And you can't include a literal newline in a C string literal, and you can't even have a NUL character in the interpretation (memory layout) of it, right? I actually started out with a C-like string as default type (so named it "String") but noticed - A big problem with string culture is that "" strings use identical start and end markers. A problem which Joe Armstrong mentioned as well. - Escaping means significantly higher complexity of parsing out the interpretation from the literal, while it's not really needed for most applications. Both these problems make data unnecessarily hard to process with dirty one-off scripts. So after considering some other options I'm now with [this style] because [] are not too often needed or when needed can often be substituted with (), and are very pleasing on the eye in most fonts. In conclusion I guess it will stay "String", and other less frequently needed types will be called "CString", "Base64", and "BinaryString". Or optional parameterization will be created for "String" to declare the escaping style without needing a separate metatype.

lwis 10y ago

I've always felt that YAML was incomplete.

TillE 10y ago
If you want to use it for complex things, sure, there's no end of potential features. If you need some logic with your data, I like Lua. But I think most uses of YAML are basically just a much more flexible JSON.
- AstroJetson 10y ago
  If you want to use it for complex things, sure, there's no end of potential features. If you need some logic with your data, I like Lua. Say some more about this. I look at YAML as a way to do data exchange, like using JSON/SON. How does Lua fit into this?
  - Avshalom 10y ago
    Not TillE, but I suspect the point is you can exchange data with Lua tables just as easily as JSON and because you would do it by just embedding the Lua interpreter you can also send Lua functions around with the data and/or process the data in Lua before making it accessible from the actual program.
    - justincormack 10y ago
      Lua was originally a configuration language. Its syntax is very human readable especially for just data use cases, but you can add functions etc too.
      - AstroJetson 10y ago
        Thanks. I was looking at the process to process data sharing that you would use YAML or JSON for and how Lua would be used. It's never occurred to me to serialize a table and send it. I'll put that on the stack of things to research.

crdoconnor 10y ago

Despite being a nice readable language if you use it right, the spec is rife with disgusting features and nasty surprises. This was my attempt to cut out 90% of the crap, leaving the nice, readable core (which maps on to JSON pretty neatly): https://github.com/crdoconnor/strictyaml https://github.com/crdoconnor/strictyaml

jean- 10y ago
Very nice! Another saner alternative to YAML which I like is TOML: https://github.com/toml-lang/toml https://github.com/toml-lang/toml
- voaie 10y ago
  TOML don't even have a clear and definitive standard/test suits. It seems unmaintained but some people are trying to push it to 1.0 no matter what.
- crdoconnor 10y ago
  I actually went over why I'd consider this is a better format than TOML: https://github.com/crdoconnor/strictyaml/blob/master/FAQ.rst#why-not-use-toml https://github.com/crdoconnor/strictyaml/blob/master/FAQ.rst...
akkartik 10y ago
I love the example at https://github.com/crdoconnor/strictyaml/blob/master/FAQ.rst#what-is-wrong-with-implicit-typing https://github.com/crdoconnor/strictyaml/blob/master/FAQ.rst.... Convinced in 2 seconds that full YAML is idiotic.
- guitarbill 10y ago
  this whole idea of strictyaml just makes so much sense to me. plus having comments is a huge bonus. why am i only hearing about this now?
  - 10y ago

jgalt212 10y ago

If you stick the language agnostic core, yaml is great. We use it for: 1. test case specifications we use for unit tests suits that must run across multiple languages. 2. DSL for business processes

ThePhysicist 10y ago

What I find more interesting is that YAML allows you to define self-referential / circular data structures: foo: &foo bar: *foo In PyYAML, this will give you a self-referntial dictionary. Powerful but pretty catastrophic if you use (naive) recursion to analyze a user-submitted data structure.

twelvechairs 10y ago
Feature not a bug. YAML is what people turn to who need these kind of advanced feautures which arent provided in json or similar.
- OskarS 10y ago
  I would imagine this feature is primarily useful if you want to serialize a whole bunch of objects that all reference each other, but it makes me feel a bit icky. It feels like it breaks an intuitive assumption you have about hierarchical formats, which is that there should be no cycles. Maybe that's just my personal bias, but I feel like the relative simplicity of JSON is a strong feature in its favor. As a developer, I have very clear understanding of the data I'm reading, and with that I can more easily make safer and more stable code.
  - junke 10y ago
    Yet, a lot of hierarchical structures I see are full of "ref" and "id" attributes, which effectively encode cyclic graphs.
    - OskarS 10y ago
      That's certainly true, but at least that's not going to cause an infinite loop while wandering down the structure, it's just gonna end at a leaf "ref" or "id".
      - junke 10y ago
        In my view this is more a question of simple interface vs. simple implementation. Having cross-references at the language level gives a uniform and simplified interface at the cost of more having to develop careful tools. Not saying refs/id are bad, but at least with a uniform syntax and language support, you don't have to reimplement custom cross-reference resolvers.

eyelidlessness 10y ago

I have minimal experience with YAML (100% of which configuring for CI environments because apparently that's what all the modern ones prefer). I don't really know the ins and outs of the format, and try to stick to extremely simple representations. Today I learned that what I perceived as somewhat complicated syntax is actually overwhelmingly complex. I know that my preference for s-expressions is not shared by everyone, but the more complex a syntax I encounter, the more I wonder if simpler alternatives were even considered. Genuine question: apart from inertia, and apart from recursive references (noted in comments before I wrote this), is there a use case for YAML that isn't solved by simpler and less ambiguous formats like EDN or Transit?

jaredklewis 10y ago
Just one data point, but I had never heard of transit or edn before today. Both look awesome. On the other hand I picked up YAML in less than a day (many years ago) and despite its complexity can't remember that ever causing me any issues. Not 100% sure if missing from your suggested formats (which do support multi line strings) but I enjoy use of yaml's folded strings.
fibo 10y ago
YAML was the first human readable/writable format, so better t'han XML and JSON for us humans. Also is a superset of JSON.
- Someone 10y ago
  Human writable? I have little experience with it, but if what Amazon uses for its CodeDeploy configuration files is representative for the format, I disagree. IMO, a human writable format shouldn't need a help page like this: http://docs.aws.amazon.com/codedeploy/latest/userguide/app-spec-ref-spacing.html http://docs.aws.amazon.com/codedeploy/latest/userguide/app-s... Yes, part of that is CodeDeploy, but saying "tab characters must not be used in indentation" for me, disqualifies a standard as "human writable". CodeDeploy makes that horrendous by requiring five or so clicks to figure out what causes an error (try forgetting adding an app spec.yml file or saving a file with a BOM)

burke 10y ago

Piling on to the general theme of the rest of the comments here: I really wish there was a more popular middle-ground between YAML and JSON. People use YAML because it's the next step up from JSON if you need comments, etc., but I think most purposes would be better served by the likes of JSON5, HJSON, or TOML (for example) if only any of those were as popular as JSON and YAML. I implemented a YAML parser from spec last month. It (YAML) goes to great lengths to provide human-friendly features, trading off computer-friendliness to an fairly extreme extent. Eliminating 'plain' scalars (unquoted strings-as-values), folded multiline literals, tags, anchors/aliases, and possibly directives, as a sort of reduced yaml would make the language a lot less silly for the kinds of things a lot of people end up using it for.

crdoconnor 10y ago
JSON5, HJSON and TOML are not as readable or editable as YAML. They may have a slight advantage over the default spec of YAML but they don't stack up well against YAML with the features you mentioned removed. I really don't think an entirely new standard is warranted.
- burke 10y ago
  Right. My hypothesis here is that, if there were, instead of JSON and YAML to choose from, three standards, one of which were something slightly less human-friendly but much simpler than YAML, I think it would be widely adopted. I think YAML is more widely-used than it would otherwise be, in a broader variety of domains, simply because the only popular alternative doesn't have comments. I'm not drawing from this any sort of conclusion that we should try to push one of these alternatives as viable, just lamenting the state of things.
mst 10y ago
ingy and I already invented that. It's called JSONY.