11 ms·

YAML was intended to be a superset, but it isn't quite, which is about the worst case scenario. See https://metacpan.org/pod/JSON::XS#JSON-and-YAML https://meta

by jerf 5y ago

YAML was intended to be a superset, but it isn't quite, which is about the worst case scenario. See https://metacpan.org/pod/JSON::XS#JSON-and-YAML https://metacpan.org/pod/JSON::XS#JSON-and-YAML , for instance. (I am an absolutist on this matter. To be a superset, all, that's A L L valid JSON strings must also be valid YAML to be a superset. A single failure makes it not a superset. At scale, any difference will eventually occur, which is why even small deviations matter.)

remram 5y ago
For example, this valid JSON doesn't parse as YAML: { "list": [ {}, {} ] } (tested on Python) edit: whitespace didn't quite make it through HN, here: json.loads('{\n "list": [\n {},\n\t{}\n ]\n}') yaml.load ('{\n "list": [\n {},\n\t{}\n ]\n}')
- dheera 5y ago
  try: try: import orjson as json except: try: import rapidjson as json except: try: import fast_json as json except: import json foo = json.loads(string) except: try: import yaml except: # try harder import os try: assert(os.system("pip3 install yaml") == 0) except: # try even harder try: assert(os.system("sudo apt install python3-pip && pip3 install yaml") == 0) except: assert(os.system("sudo yum install python3-pip && pip3 install yaml") == 0) import yaml try: foo = yaml.loads(string) except: try: ....
  - tomjakubowski 5y ago
    Great idea. pip install --user yaml increases the chances it will work
  - imoverclocked 5y ago
    Brilliant! What license is this published under?
    - dheera 5y ago
      Free Art License
  - nrclark 5y ago
    A note to readers: it's not always a good idea to put automated software installation in a place that users don't expect it. I've seen that kind of approach cause a ton of issues the moment that the software was used in a different environment than the author expected. It's much better IMO to fail with a message about how to install the missing dependency.
    - dheera 5y ago
      This is why there should be a way to automatically install software into a sandboxed location, e.g. a virtualenv. Considering we are having software drive cars today it should be trivial and I would say even arguably expected that software should be able to autonomously "figure out" how to run itself and avoid conflicts with other software since that's a trivial task in comparison to navigating city streets.
- 0xbadcafebee 5y ago
  Python's .netrc library also hasn't supported comments correctly for like 5 years. The bug was reported, it was never fixed. If I want to use my .netrc file with Python programs, I have to remove all comments (that work with every other .netrc-using program). It's 2022 and we can't even get a plaintext configuration format from 1980 right.
  - macintux 5y ago
    > It's 2022 and we can't even get a plaintext configuration format from 1980 right. To me, it's more depressing that we've been at this for 50-60 years and still seemingly don't have an unambiguously good plaintext configuration format at all.
    - zarzavat 5y ago
      We do, it’s called TOML. The future is here it’s just not equally distributed.
      - Tesl 5y ago
        It's perfect until you do a lot of nesting..
        donio 5y ago
        ...or any nesting. TOML sucks for anything non-trivial.
      - 41b696ef1113 5y ago
        It makes me sad every time I see a newly announced tool that went for YAML instead of TOML.
      - SPBS 5y ago
        TOML sucks for list of tables simply because they intentionally crippled inline tables to only be able to occupy one line. For ideology reasons ("we don't need to add a pseudo-JSON"). Unless your table is small, it's going to look absolutely terrible being all crammed into one line. https://github.com/toml-lang/toml/issues/516 https://github.com/toml-lang/toml/issues/516 The official way to do list of tables is (look at how much duplication there is) [[main_app.general_settings.logging.handlers]] name = "default" output = "stdout" level = "info" [[main_app.general_settings.logging.handlers]] name = "stderr" output = "stderr" level = "error" [[main_app.general_settings.logging.handlers]] name = "access" output = "/var/log/access.log" level = "info" vs handlers = [ { name = "default", output = "stdout", level = "info", }, { name = "stderr", output = "stderr", level = "error", }, { name = "access", output = "/var/log/access.log", level = "info", }, ] I would still reach for TOML first if I only needed simple key-value configuration (never YAML), but for anything requiring list-of-tables I would seriously consider JSON with trailing commas instead.
- mdaniel 5y ago
  Tested on python what? I was curious to see what error that produced, figuring it would be some whitespace due to the difference between the list items, but using the yamlized python that I had lying around, it did the sane thing: PATH=$HOMEBREW_PREFIX/opt/ansible/libexec/bin:$PATH pip list | grep -i yaml python -V python <<'DOIT' from io import StringIO import yaml print(yaml.safe_load(StringIO( ''' { "list": [ {}, {} ] } '''))) DOIT produces PyYAML 6.0 Python 3.10.1 {'list': [{}, {}]}
  - remram 5y ago
    Edited with string escapes, the tab didn't make it through HN. The error from PyYaml 5.3.1: yaml.scanner.ScannerError: while scanning for the next token found character '\t' that cannot start any token in "<unicode string>", line 4, column 1
    - notreallyserio 5y ago
      If it continues to be hard to share, I suggest encoding it as a base64 string so folks can decode it into a file with exactly the right contents.
      - CameronNemo 5y ago
        This is, unwittingly, the most YAML-relevant comment in this thread.
      - jwilk 5y ago
        Not base64, but this should be easy to reproduce: $ printf '{\n\t"list": [\n\t\t{},\n\t\t{}\n\t]\n}\n' > test.json $ jq < test.json { "list": [ {}, {} ] } $ yamllint test.json test.json 2:1 error syntax error: found character '\t' that cannot start any token (syntax)
    - ak217 5y ago
      Thanks, I'm finally able to reproduce this. It would be great if instead of the histrionic message on CPAN (which amusingly accuses others of "mass hysteria"), the author would just say "YAML documents can't start with a tab while JSON documents can, making JSON not a strict subset of YAML". The YAML spec should be updated to reflect this, but I wonder if a simple practical workaround in YAML parsers (like replacing each tab at the beginning of the document with two spaces before feeding it to the tokenizer) would be sufficient in the short term.
- ak217 5y ago
  This parses fine as YAML in all the tools I've tried. Can you provide the specific versions of the libraries you're using?
im3w1l 5y ago
On the upside, if it's almost a superset then a data producer can make sure it is polyglot by sticking to the intersection of the two. C++ is not a strict superset of C, but the ability to include C headers is very valuable.
ak217 5y ago
I wasn't able to reproduce any of the issues listed on that page. Does anyone have an example?
guitarsteve 5y ago
I’ve often heard this (YAML is a superset of JSON) but never looked into the details. According to https://yaml.org/spec/1.2.2/ https://yaml.org/spec/1.2.2/, YAML 1.2 (from 2009) is a strict superset of JSON. Earlier versions were an _almost_ superset. Hence the confusion in this thread. It depends on the version…
- DHowett 5y ago
  The JSON::XS documentation linked above reports that YAML 1.2 is not a strict superset of JSON: > Addendum/2009: the YAML 1.2 spec is still incompatible with JSON The author also details their issues in, ah, getting some of the authors of the YAML specification to agree.
- kelnos 5y ago
  CPAN link provided by the parent says 1.2 still isn't a superset: > Addendum/2009: the YAML 1.2 spec is still incompatible with JSON, even though the incompatibilities have been documented (and are known to Brian) for many years and the spec makes explicit claims that YAML is a superset of JSON. It would be so easy to fix, but apparently, bullying people and corrupting userdata is so much easier.
  - terom 5y ago
    Are these documented YAML 1.2 JSON incompatibilities listed / linked to somewhere? I assume these are something related to non-ascii string encoding / escapes?
    - guelo 5y ago
      They are listed in that same CPAN link "Please note that YAML has hardcoded limits on (simple) object key lengths that JSON doesn't have and also has different and incompatible unicode character escape syntax... YAML also does not allow \/ sequences in strings"
- peterburkimsher 5y ago
  In the beginning was the SGML. Then we said it's too verbose. We named some subsets XML, HTML, XLSX. Then we said it's still too long. So we named some subsets Markdown, and YML. Then we said it's still too long, and made JSON. What's wrong with subsets? Ambiguity in naming things. https://martinfowler.com/bliki/TwoHardThings.html https://martinfowler.com/bliki/TwoHardThings.html Is JSON the same as YML? NO. Norwegian? https://news.ycombinator.com/item?id=26671136 https://news.ycombinator.com/item?id=26671136
fouc 5y ago
> YAML was intended to be a superset My impression was JSON came years after YAML, and it was somehow coincidental that YAML was almost a superset of JSON. (Shockingly wikipedia tells me they both came out within a month of each other in 2001).
ryukoposting 5y ago
They should remove the phrase "every JSON file is also a valid YAML file" from the YAML spec. 1) it isn't true, and 2) it seems like it goes against the implication made here: > This makes it easy to migrate from JSON to YAML if/when the additional features are required. If JSON interop is provided solely as a short-term solution that eases the transition to YAML, then I applaud the YAML designers for making a great choice.
captainmuon 5y ago
I'm not a fan of YAML either, but I think you should not generate YAML files if you can avoid it. All YAML you encounter should be hand-written, so this problem should not occur. I read "YAML is a superset of JSON" not as a logical statement, but as instructions to humans writing YAML. If you know JSON, you can use that syntax to write YAML. Just like, if you know JavaScript or Python (or to some extent PHP) object syntax, you can write JSON. If you get a parse error, no biggie, you Alt+Tab to the editor where you are editing the config file and correct it. It is not like you are serving this over the net to some other program.
BBC-vs-neolibs 5y ago
MaxBarraclough 5y ago
Same applies to TypeScript. It is not a superset of JavaScript, although many people think it is. https://stackoverflow.com/a/53698835/ https://stackoverflow.com/a/53698835/
- Dylan16807 5y ago
  As long as you tell the typescript compiler not to stop when it finds type problems, all JavaScript works and compiles, right? That sounds like a superset to me. Syntactically there are no problems, and the error messages are just messages.
  - MaxBarraclough 5y ago
    > As long as you tell the typescript compiler not to stop when it finds type problems, all JavaScript works and compiles, right? Does such code count as valid TypeScript though? It sounds more as if the compiler has an option to accept certain invalid programs. You could build a C++ compiler with a flag to warn, rather than error, on encountering implicit conversions that are forbidden by the C++ standard. The language the compiler is accepting would then no longer be standard C++, but a superset. (Same for all compiler-specific extensions of course.) Personally I'm inclined to agree with this StackOverflow comment. [0] It's an interesting edge-case though. [0] https://stackoverflow.com/questions/29918324/is-typescript-really-a-superset-of-javascript/53698835#comment66912671_29918421 https://stackoverflow.com/questions/29918324/is-typescript-r...
    - Dylan16807 5y ago
      It's syntactically and functionally correct, so despite the error messages I think 'valid' is a better label. > You could build a C++ compiler with a flag to warn, rather than error, on encountering implicit conversions that are forbidden by the C++ standard. The language the compiler is accepting would then no longer be standard C++, but a superset. (Same for all compiler-specific extensions of course.) The way I see it, these errors are already on par with C++ warnings. C++ won't stop you if you make a pointer null or use the wrong string as a map key.
rob_c 5y ago
congrats to all involved for sticking to their guns here. specs exist for a reason :D

by jerf 5y ago

remram 5y ago

For example, this valid JSON doesn't parse as YAML: { "list": [ {}, {} ] } (tested on Python) edit: whitespace didn't quite make it through HN, here: json.loads('{\n "list": [\n {},\n\t{}\n ]\n}') yaml.load ('{\n "list": [\n {},\n\t{}\n ]\n}')

dheera 5y ago
try: try: import orjson as json except: try: import rapidjson as json except: try: import fast_json as json except: import json foo = json.loads(string) except: try: import yaml except: # try harder import os try: assert(os.system("pip3 install yaml") == 0) except: # try even harder try: assert(os.system("sudo apt install python3-pip && pip3 install yaml") == 0) except: assert(os.system("sudo yum install python3-pip && pip3 install yaml") == 0) import yaml try: foo = yaml.loads(string) except: try: ....
- tomjakubowski 5y ago
  Great idea. pip install --user yaml increases the chances it will work
- imoverclocked 5y ago
  Brilliant! What license is this published under?
  - dheera 5y ago
    Free Art License
- nrclark 5y ago
  A note to readers: it's not always a good idea to put automated software installation in a place that users don't expect it. I've seen that kind of approach cause a ton of issues the moment that the software was used in a different environment than the author expected. It's much better IMO to fail with a message about how to install the missing dependency.
  - dheera 5y ago
    This is why there should be a way to automatically install software into a sandboxed location, e.g. a virtualenv. Considering we are having software drive cars today it should be trivial and I would say even arguably expected that software should be able to autonomously "figure out" how to run itself and avoid conflicts with other software since that's a trivial task in comparison to navigating city streets.
0xbadcafebee 5y ago
Python's .netrc library also hasn't supported comments correctly for like 5 years. The bug was reported, it was never fixed. If I want to use my .netrc file with Python programs, I have to remove all comments (that work with every other .netrc-using program). It's 2022 and we can't even get a plaintext configuration format from 1980 right.
- macintux 5y ago
  > It's 2022 and we can't even get a plaintext configuration format from 1980 right. To me, it's more depressing that we've been at this for 50-60 years and still seemingly don't have an unambiguously good plaintext configuration format at all.
  - zarzavat 5y ago
    We do, it’s called TOML. The future is here it’s just not equally distributed.
    - Tesl 5y ago
      It's perfect until you do a lot of nesting..
      - donio 5y ago
        ...or any nesting. TOML sucks for anything non-trivial.
    - 41b696ef1113 5y ago
      It makes me sad every time I see a newly announced tool that went for YAML instead of TOML.
    - SPBS 5y ago
      TOML sucks for list of tables simply because they intentionally crippled inline tables to only be able to occupy one line. For ideology reasons ("we don't need to add a pseudo-JSON"). Unless your table is small, it's going to look absolutely terrible being all crammed into one line. https://github.com/toml-lang/toml/issues/516 https://github.com/toml-lang/toml/issues/516 The official way to do list of tables is (look at how much duplication there is) [[main_app.general_settings.logging.handlers]] name = "default" output = "stdout" level = "info" [[main_app.general_settings.logging.handlers]] name = "stderr" output = "stderr" level = "error" [[main_app.general_settings.logging.handlers]] name = "access" output = "/var/log/access.log" level = "info" vs handlers = [ { name = "default", output = "stdout", level = "info", }, { name = "stderr", output = "stderr", level = "error", }, { name = "access", output = "/var/log/access.log", level = "info", }, ] I would still reach for TOML first if I only needed simple key-value configuration (never YAML), but for anything requiring list-of-tables I would seriously consider JSON with trailing commas instead.
mdaniel 5y ago
Tested on python what? I was curious to see what error that produced, figuring it would be some whitespace due to the difference between the list items, but using the yamlized python that I had lying around, it did the sane thing: PATH=$HOMEBREW_PREFIX/opt/ansible/libexec/bin:$PATH pip list | grep -i yaml python -V python <<'DOIT' from io import StringIO import yaml print(yaml.safe_load(StringIO( ''' { "list": [ {}, {} ] } '''))) DOIT produces PyYAML 6.0 Python 3.10.1 {'list': [{}, {}]}
- remram 5y ago
  Edited with string escapes, the tab didn't make it through HN. The error from PyYaml 5.3.1: yaml.scanner.ScannerError: while scanning for the next token found character '\t' that cannot start any token in "<unicode string>", line 4, column 1
  - notreallyserio 5y ago
    If it continues to be hard to share, I suggest encoding it as a base64 string so folks can decode it into a file with exactly the right contents.
    - CameronNemo 5y ago
      This is, unwittingly, the most YAML-relevant comment in this thread.
    - jwilk 5y ago
      Not base64, but this should be easy to reproduce: $ printf '{\n\t"list": [\n\t\t{},\n\t\t{}\n\t]\n}\n' > test.json $ jq < test.json { "list": [ {}, {} ] } $ yamllint test.json test.json 2:1 error syntax error: found character '\t' that cannot start any token (syntax)
  - ak217 5y ago
    Thanks, I'm finally able to reproduce this. It would be great if instead of the histrionic message on CPAN (which amusingly accuses others of "mass hysteria"), the author would just say "YAML documents can't start with a tab while JSON documents can, making JSON not a strict subset of YAML". The YAML spec should be updated to reflect this, but I wonder if a simple practical workaround in YAML parsers (like replacing each tab at the beginning of the document with two spaces before feeding it to the tokenizer) would be sufficient in the short term.
ak217 5y ago
This parses fine as YAML in all the tools I've tried. Can you provide the specific versions of the libraries you're using?

im3w1l 5y ago

On the upside, if it's almost a superset then a data producer can make sure it is polyglot by sticking to the intersection of the two. C++ is not a strict superset of C, but the ability to include C headers is very valuable.

ak217 5y ago

I wasn't able to reproduce any of the issues listed on that page. Does anyone have an example?

guitarsteve 5y ago

I’ve often heard this (YAML is a superset of JSON) but never looked into the details. According to https://yaml.org/spec/1.2.2/ https://yaml.org/spec/1.2.2/, YAML 1.2 (from 2009) is a strict superset of JSON. Earlier versions were an _almost_ superset. Hence the confusion in this thread. It depends on the version…

DHowett 5y ago
The JSON::XS documentation linked above reports that YAML 1.2 is not a strict superset of JSON: > Addendum/2009: the YAML 1.2 spec is still incompatible with JSON The author also details their issues in, ah, getting some of the authors of the YAML specification to agree.
kelnos 5y ago
CPAN link provided by the parent says 1.2 still isn't a superset: > Addendum/2009: the YAML 1.2 spec is still incompatible with JSON, even though the incompatibilities have been documented (and are known to Brian) for many years and the spec makes explicit claims that YAML is a superset of JSON. It would be so easy to fix, but apparently, bullying people and corrupting userdata is so much easier.
- terom 5y ago
  Are these documented YAML 1.2 JSON incompatibilities listed / linked to somewhere? I assume these are something related to non-ascii string encoding / escapes?
  - guelo 5y ago
    They are listed in that same CPAN link "Please note that YAML has hardcoded limits on (simple) object key lengths that JSON doesn't have and also has different and incompatible unicode character escape syntax... YAML also does not allow \/ sequences in strings"
peterburkimsher 5y ago
In the beginning was the SGML. Then we said it's too verbose. We named some subsets XML, HTML, XLSX. Then we said it's still too long. So we named some subsets Markdown, and YML. Then we said it's still too long, and made JSON. What's wrong with subsets? Ambiguity in naming things. https://martinfowler.com/bliki/TwoHardThings.html https://martinfowler.com/bliki/TwoHardThings.html Is JSON the same as YML? NO. Norwegian? https://news.ycombinator.com/item?id=26671136 https://news.ycombinator.com/item?id=26671136

fouc 5y ago

> YAML was intended to be a superset My impression was JSON came years after YAML, and it was somehow coincidental that YAML was almost a superset of JSON. (Shockingly wikipedia tells me they both came out within a month of each other in 2001).

ryukoposting 5y ago

They should remove the phrase "every JSON file is also a valid YAML file" from the YAML spec. 1) it isn't true, and 2) it seems like it goes against the implication made here: > This makes it easy to migrate from JSON to YAML if/when the additional features are required. If JSON interop is provided solely as a short-term solution that eases the transition to YAML, then I applaud the YAML designers for making a great choice.

captainmuon 5y ago

I'm not a fan of YAML either, but I think you should not generate YAML files if you can avoid it. All YAML you encounter should be hand-written, so this problem should not occur. I read "YAML is a superset of JSON" not as a logical statement, but as instructions to humans writing YAML. If you know JSON, you can use that syntax to write YAML. Just like, if you know JavaScript or Python (or to some extent PHP) object syntax, you can write JSON. If you get a parse error, no biggie, you Alt+Tab to the editor where you are editing the config file and correct it. It is not like you are serving this over the net to some other program.

BBC-vs-neolibs 5y ago

MaxBarraclough 5y ago

Same applies to TypeScript. It is not a superset of JavaScript, although many people think it is. https://stackoverflow.com/a/53698835/ https://stackoverflow.com/a/53698835/

Dylan16807 5y ago
As long as you tell the typescript compiler not to stop when it finds type problems, all JavaScript works and compiles, right? That sounds like a superset to me. Syntactically there are no problems, and the error messages are just messages.
- MaxBarraclough 5y ago
  > As long as you tell the typescript compiler not to stop when it finds type problems, all JavaScript works and compiles, right? Does such code count as valid TypeScript though? It sounds more as if the compiler has an option to accept certain invalid programs. You could build a C++ compiler with a flag to warn, rather than error, on encountering implicit conversions that are forbidden by the C++ standard. The language the compiler is accepting would then no longer be standard C++, but a superset. (Same for all compiler-specific extensions of course.) Personally I'm inclined to agree with this StackOverflow comment. [0] It's an interesting edge-case though. [0] https://stackoverflow.com/questions/29918324/is-typescript-really-a-superset-of-javascript/53698835#comment66912671_29918421 https://stackoverflow.com/questions/29918324/is-typescript-r...
  - Dylan16807 5y ago
    It's syntactically and functionally correct, so despite the error messages I think 'valid' is a better label. > You could build a C++ compiler with a flag to warn, rather than error, on encountering implicit conversions that are forbidden by the C++ standard. The language the compiler is accepting would then no longer be standard C++, but a superset. (Same for all compiler-specific extensions of course.) The way I see it, these errors are already on par with C++ warnings. C++ won't stop you if you make a pointer null or use the wrong string as a map key.

rob_c 5y ago

congrats to all involved for sticking to their guns here. specs exist for a reason :D