3 ms·

Tested on python what? I was curious to see what error that produced, figuring it would be some whitespace due to the difference between the list items, but usi

Tested on python what? I was curious to see what error that produced, figuring it would be some whitespace due to the difference between the list items, but using the yamlized python that I had lying around, it did the sane thing: PATH=$HOMEBREW_PREFIX/opt/ansible/libexec/bin:$PATH pip list | grep -i yaml python -V python <<'DOIT' from io import StringIO import yaml print(yaml.safe_load(StringIO( ''' { "list": [ {}, {} ] } '''))) DOIT produces PyYAML 6.0 Python 3.10.1 {'list': [{}, {}]}

remram 5y ago
Edited with string escapes, the tab didn't make it through HN. The error from PyYaml 5.3.1: yaml.scanner.ScannerError: while scanning for the next token found character '\t' that cannot start any token in "<unicode string>", line 4, column 1
- notreallyserio 5y ago
  If it continues to be hard to share, I suggest encoding it as a base64 string so folks can decode it into a file with exactly the right contents.
  - CameronNemo 5y ago
    This is, unwittingly, the most YAML-relevant comment in this thread.
  - jwilk 5y ago
    Not base64, but this should be easy to reproduce: $ printf '{\n\t"list": [\n\t\t{},\n\t\t{}\n\t]\n}\n' > test.json $ jq < test.json { "list": [ {}, {} ] } $ yamllint test.json test.json 2:1 error syntax error: found character '\t' that cannot start any token (syntax)
- ak217 5y ago
  Thanks, I'm finally able to reproduce this. It would be great if instead of the histrionic message on CPAN (which amusingly accuses others of "mass hysteria"), the author would just say "YAML documents can't start with a tab while JSON documents can, making JSON not a strict subset of YAML". The YAML spec should be updated to reflect this, but I wonder if a simple practical workaround in YAML parsers (like replacing each tab at the beginning of the document with two spaces before feeding it to the tokenizer) would be sufficient in the short term.
  - ingy 5y ago
    > "YAML documents can't start with a tab while JSON documents can, making JSON not a strict subset of YAML" But YAML can start with tabs. Tabs are allowed as separating whitespace in most of the spec productions but are not allowed as indentation. Even though those tabs look like indentation, the spec productions don't interpret them as such. See my comment above and esp see https://play.yaml.io/main/parser?input=CXsKCQkibGlzdCI6IFsKCQkJe30sCgkJCXt9CgkJXQoJfQo= https://play.yaml.io/main/parser?input=CXsKCQkibGlzdCI6IFsKC... Note: the YAML spec maintainers (I am one) have identified many issues with YAML which we are actively working on, but (somewhat surprisingly) we have yet to find a case where valid JSON is invalid YAML 1.2.
    - ak217 5y ago
      Thanks for the clarification. Let's fix it in PyYAML then :) Speaking of PyYAML, I recently ran into an issue where I had to heavily patch PyYAML to prevent its parse result from being susceptible to entity expansion attacks. It would be nice to at least have a PyYAML mode to completely ignore anchors and aliases (as well as tags) using simple keyword arguments. Protection against entity expansion abuse would be nice too.
hoherd 5y ago
With leading tabs it does not work. $ sed 's/\t/--->/g' break-yaml.json --->{ --->--->"list": [ --->--->--->{}, --->--->--->{} --->--->] --->} $ jq -c . break-yaml.json {"list":[{},{}]} $ yaml-to-json.py break-yaml.json ERROR: break-yaml.json could not be parsed while scanning for the next token found character '\t' that cannot start any token in "break-yaml.json", line 1, column 1 $ sed 's/\t/ /g' break-yaml.json | yaml-to-json.py {"list": [{}, {}]}
- deleted 5y ago
  [deleted]
- MrPatan 5y ago
  Tabs are not valid JSON
  - zimpenfish 5y ago
    Do you have a link for that? https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe... says: > Insignificant whitespace may be present anywhere except within a JSONNumber [forbidden] or JSONString [interpreted as part of the string] And specifically lists tab as whitespace: > The tab character (U+0009), carriage return (U+000D), line feed (U+000A), and space (U+0020) characters are the only valid whitespace characters. More specifically, expanding https://datatracker.ietf.org/doc/html/rfc8259#section-2 https://datatracker.ietf.org/doc/html/rfc8259#section-2 gives an array as (roughly) > ws %x5B ws value (ws %x2C ws value)* ws %x5D ws Where `ws` explicitly includes `%x09`. Which seems to cover this case?
  - Someone 5y ago
    The grammar in https://www.json.org/json-en.html https://www.json.org/json-en.html disagrees. It has json element element ws value ws ws ‘0009’ ws
  - tonyg 5y ago
    Per RFC 8259: ws = *( %x20 / ; Space %x09 / ; Horizontal tab %x0A / ; Line feed or New line %x0D ) ; Carriage return

remram 5y ago

Edited with string escapes, the tab didn't make it through HN. The error from PyYaml 5.3.1: yaml.scanner.ScannerError: while scanning for the next token found character '\t' that cannot start any token in "<unicode string>", line 4, column 1

notreallyserio 5y ago
If it continues to be hard to share, I suggest encoding it as a base64 string so folks can decode it into a file with exactly the right contents.
- CameronNemo 5y ago
  This is, unwittingly, the most YAML-relevant comment in this thread.
- jwilk 5y ago
  Not base64, but this should be easy to reproduce: $ printf '{\n\t"list": [\n\t\t{},\n\t\t{}\n\t]\n}\n' > test.json $ jq < test.json { "list": [ {}, {} ] } $ yamllint test.json test.json 2:1 error syntax error: found character '\t' that cannot start any token (syntax)
ak217 5y ago
Thanks, I'm finally able to reproduce this. It would be great if instead of the histrionic message on CPAN (which amusingly accuses others of "mass hysteria"), the author would just say "YAML documents can't start with a tab while JSON documents can, making JSON not a strict subset of YAML". The YAML spec should be updated to reflect this, but I wonder if a simple practical workaround in YAML parsers (like replacing each tab at the beginning of the document with two spaces before feeding it to the tokenizer) would be sufficient in the short term.
- ingy 5y ago
  > "YAML documents can't start with a tab while JSON documents can, making JSON not a strict subset of YAML" But YAML can start with tabs. Tabs are allowed as separating whitespace in most of the spec productions but are not allowed as indentation. Even though those tabs look like indentation, the spec productions don't interpret them as such. See my comment above and esp see https://play.yaml.io/main/parser?input=CXsKCQkibGlzdCI6IFsKCQkJe30sCgkJCXt9CgkJXQoJfQo= https://play.yaml.io/main/parser?input=CXsKCQkibGlzdCI6IFsKC... Note: the YAML spec maintainers (I am one) have identified many issues with YAML which we are actively working on, but (somewhat surprisingly) we have yet to find a case where valid JSON is invalid YAML 1.2.
  - ak217 5y ago
    Thanks for the clarification. Let's fix it in PyYAML then :) Speaking of PyYAML, I recently ran into an issue where I had to heavily patch PyYAML to prevent its parse result from being susceptible to entity expansion attacks. It would be nice to at least have a PyYAML mode to completely ignore anchors and aliases (as well as tags) using simple keyword arguments. Protection against entity expansion abuse would be nice too.

hoherd 5y ago

With leading tabs it does not work. $ sed 's/\t/--->/g' break-yaml.json --->{ --->--->"list": [ --->--->--->{}, --->--->--->{} --->--->] --->} $ jq -c . break-yaml.json {"list":[{},{}]} $ yaml-to-json.py break-yaml.json ERROR: break-yaml.json could not be parsed while scanning for the next token found character '\t' that cannot start any token in "break-yaml.json", line 1, column 1 $ sed 's/\t/ /g' break-yaml.json | yaml-to-json.py {"list": [{}, {}]}

deleted 5y ago
[deleted]
MrPatan 5y ago
Tabs are not valid JSON
- zimpenfish 5y ago
  Do you have a link for that? https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe... says: > Insignificant whitespace may be present anywhere except within a JSONNumber [forbidden] or JSONString [interpreted as part of the string] And specifically lists tab as whitespace: > The tab character (U+0009), carriage return (U+000D), line feed (U+000A), and space (U+0020) characters are the only valid whitespace characters. More specifically, expanding https://datatracker.ietf.org/doc/html/rfc8259#section-2 https://datatracker.ietf.org/doc/html/rfc8259#section-2 gives an array as (roughly) > ws %x5B ws value (ws %x2C ws value)* ws %x5D ws Where `ws` explicitly includes `%x09`. Which seems to cover this case?
- Someone 5y ago
  The grammar in https://www.json.org/json-en.html https://www.json.org/json-en.html disagrees. It has json element element ws value ws ws ‘0009’ ws
- tonyg 5y ago
  Per RFC 8259: ws = *( %x20 / ; Space %x09 / ; Horizontal tab %x0A / ; Line feed or New line %x0D ) ; Carriage return