4 ms·

This was all very interesting, but that polyglot json/yaml/xml payload was a big surprise to me! I had no idea that go's default xml parser would accept proceed

by anitil 1y ago

This was all very interesting, but that polyglot json/yaml/xml payload was a big surprise to me! I had no idea that go's default xml parser would accept proceeding and trailing garbage. I'd always thought of json as one of the simpler formats to parse, but I suppose the real world would beg to differ. It's interesting that decisions made about seemingly-innocuous conditions like 'what if there are duplicate keys' have a long tail of consequences

shakna 1y ago
Tangent for breaking Python's JSON parser: This has worked for five years. The docs do not say that parsing an invalid piece will result in a RecursionError. They specify JSONDecodeError and UnicodeDecodeError. (There is a RecursionError reference to a key that is off by default - but if its off, we can still raise this...) #!/bin/sh # Python will hit it's recursion limit # If you supply just 4 less than the recursion limit # I assume this means there's a few objects on the call stack first # Probably: __main__, print, json.loads, and input. n="$(python3 -c 'import math; import sys; sys.stdout.write(str(math.floor(sys.getrecursionlimit() - 4)))')" echo "N: $n" # Obviously invalid, but unparseable without matching pair # JSON's grammar is... Not good at being partially parsed. left="$(yes [ | head -n "$n" | tr -d '\n')" # Rather than exploding with the expected decodeError # This will explode with a RecursionError # Which naturally thrashes the memory cache. echo "$left" | python3 -c 'import json; print(json.loads(input()))'
- anitil 1y ago
  This is why we can't have nice things
- jwilk 1y ago
  Reported as <https://github.com/python/cpython/issues/135835 https://github.com/python/cpython/issues/135835>.
  - shakna 1y ago
    I did report this five years ago. And got told that CPython don't consider it a problem. Anything is allowed to raise RecursionError without documentation, in the same way anything is allowed to raise MemoryError. I don't really agree, as "surprising" is a stench in any API. But it's their project.
mdaniel 1y ago
> I'd always thought of json as one of the simpler formats to parse, but I suppose the real world would beg to differ Parsing JSON Is a Minefield (2018) - https://news.ycombinator.com/item?id=40555431 https://news.ycombinator.com/item?id=40555431 - June, 2024 (56 comments) et al https://hn.algolia.com/?query=parsing%20json%20is%20a%20minefield&type=story https://hn.algolia.com/?query=parsing%20json%20is%20a%20mine...

by anitil 1y ago

shakna 1y ago

Tangent for breaking Python's JSON parser: This has worked for five years. The docs do not say that parsing an invalid piece will result in a RecursionError. They specify JSONDecodeError and UnicodeDecodeError. (There is a RecursionError reference to a key that is off by default - but if its off, we can still raise this...) #!/bin/sh # Python will hit it's recursion limit # If you supply just 4 less than the recursion limit # I assume this means there's a few objects on the call stack first # Probably: __main__, print, json.loads, and input. n="$(python3 -c 'import math; import sys; sys.stdout.write(str(math.floor(sys.getrecursionlimit() - 4)))')" echo "N: $n" # Obviously invalid, but unparseable without matching pair # JSON's grammar is... Not good at being partially parsed. left="$(yes [ | head -n "$n" | tr -d '\n')" # Rather than exploding with the expected decodeError # This will explode with a RecursionError # Which naturally thrashes the memory cache. echo "$left" | python3 -c 'import json; print(json.loads(input()))'

anitil 1y ago
This is why we can't have nice things
jwilk 1y ago
Reported as <https://github.com/python/cpython/issues/135835 https://github.com/python/cpython/issues/135835>.
- shakna 1y ago
  I did report this five years ago. And got told that CPython don't consider it a problem. Anything is allowed to raise RecursionError without documentation, in the same way anything is allowed to raise MemoryError. I don't really agree, as "surprising" is a stench in any API. But it's their project.

mdaniel 1y ago

> I'd always thought of json as one of the simpler formats to parse, but I suppose the real world would beg to differ Parsing JSON Is a Minefield (2018) - https://news.ycombinator.com/item?id=40555431 https://news.ycombinator.com/item?id=40555431 - June, 2024 (56 comments) et al https://hn.algolia.com/?query=parsing%20json%20is%20a%20minefield&type=story https://hn.algolia.com/?query=parsing%20json%20is%20a%20mine...