4 ms·

Most of your problems aren't problems > 3. Do you pass around user-provided JSON data accross your system? How many JSON nesting levels does your implementatio

Most of your problems aren't problems > 3. Do you pass around user-provided JSON data accross your system? How many JSON nesting levels does your implementation allow? What happens if it's exceeded? What happens if different parts of your processing system have different limits? What about other unspecified limits like serialized size, string length? XML has the same issue, that's why SAX exists, it works the same way with JSON. > 2. JSON allows for duplicate/repeated keys, and allows for the parser to basically do anything when that happens. Do you know how the parser implementation you use handles this? Are you sure there's no differences between that implementation and other implementations used in your system (eg. between execution and validation)? What about other undefined behaviour, like permitted number ranges? A parser should... parse and not interpret data or it isn't a parser. it's a deserializer. Well how many languages allow duplicate keys for maps anyway? this isn't an issue in practice. Basically, the answer to all your problems is to use an evented parser instead of a deserializer.

anonymoushn 5y ago
> Basically, the answer to all your problems is to use an evented parser instead of a deserializer. This seems a bit unreasonable? People face the same issue where different pieces of software treat duplicate HTTP headers differently, and usually they haven't written each piece of software they use to parse HTTP headers
kortex 5y ago
> this isn't an issue in practice. It absolutely is an issue in practice. If system A handles dupes by accepting the first and ignoring the rest, and system B implements last-key-wins, then that's a potential source of bugs. The system might not fully parse to a map. It may, for example, do string-level modification of json strings. Is that disgusting and wrong? Yes. Have I seen it in prod? Also yes.
- throw_m239339 5y ago
  > It absolutely is an issue in practice. If system A handles dupes by accepting the first and ignoring the rest, and system B implements last-key-wins, then that's a potential source of bugs. The system might not fully parse to a map. But the system shouldn't be automatically be parsing a "json map" to a map at first place: {"foo":"bar","foo":"baz","foo":"qix","fiz":"buzz"} Shouldn't be deserialized into a map. but a Array<Map<string,string>> like structure. A SAX style parser for JSON can help do that. Thus the issue is the choice of parser indeed. Not JSON.
  - q3k 5y ago
    > Shouldn't be deserialized into a map. but a Array<Map<string,string>> like structure. But that's the thing: you might actually expect/want a Map<string,string>, but a malicious/broken system might emit something that cannot be deserialized into a Map<string,string>. It's then the JSON parser's/deserializer's job to figure out what to do, as the standards say to do whatever. That in turn causes different parsers/deserializers to behave differently (whatever the implementer thought makes sense), which is a source of interoperability bugs.
    - throw_m239339 5y ago
      > But that's the thing: you might actually expect/want a Map<string,string>, but a malicious/broken system might emit something that cannot be deserialized into a Map<string,string>. It's then the JSON parser's/deserializer's job to figure out what to do, as the standards say to do whatever. That in turn causes different parsers/deserializers to behave differently (whatever the implementer thought makes sense), which is a source of interoperability bugs. I disagree, people are mixing up parsing and deserializing. The JSON spec isn't at fault here. The JSON spec is only concerned with defining the parsing, not the deserialization, because obviously, a JSON array isn't a PHP array or a Ruby array, a JSON map isn't a PHP object or a Go map at first place. The problem isn't with JSON but how some JSON deserializers work. Again, a deserializer isn't a parser.
      - q3k 5y ago
        > The problem isn't with JSON but how some JSON deserializers work. That makes no observable difference to the end-user of JSON wishing to use it as an interchange format. The standard might as well be perfect, but if nearly all of its implementations (yes, extending that into deserialization, not just parsing - because that's how most people use JSON!) are problematic, then the standard is effectively also problematic. This is why I also always include Python's broken implementation in my JSON rant - it's not indicative of the standard(s) being bad, but the ecosystem being bad.
        5y ago
- nuerow 5y ago
  > It absolutely is an issue in practice. It really isn't. At most, it's a problem caused by picking a broken implementation that doesn't meet your needs, but that's a self-inflicted problem, not a JSON problem.
detaro 5y ago
> Basically, the answer to all your problems is to use an evented parser instead of a deserializer. Which "nobody" does, so it is a problem in practice.
- throw_m239339 5y ago
  > Which "nobody" does, so it is a problem in practice. who's nobody? if developers care about performances, they obviously do. What if the json file is 500MB of logs? Furthermore, all these JSON deserialization lib tricks might work in some languages that are dynamic or support runtime reflection, it doesn't for other languages where using a proper evented parser is mandatory.
recursive 5y ago
> use an evented parser I've never heard of this. A google search isn't particularly illuminating. What is an "evented parser"?
- dragonwriter 5y ago
  > What is an "evented parser"? Also knowns as a “streaming parser”, its a parser that takes in a data stream and produces a stream of events which client code can handle; it allows more flexible handling than deserializers, including ability to handle arbitrarily large input. SAX is a streaming/evented parser API for XML, and there are similar ones for other formats.
- throw_m239339 5y ago
  Google "event-based parser"
- rovolo 5y ago
  The sibling comments are good descriptions, but here are example interfaces: - Normal parser https://www.javadoc.io/static/com.google.code.gson/gson/2.8.8/com.google.gson/com/google/gson/JsonParser.html https://www.javadoc.io/static/com.google.code.gson/gson/2.8.... - Streaming parser https://www.javadoc.io/doc/com.google.code.gson/gson/latest/com.google.gson/com/google/gson/stream/JsonReader.html https://www.javadoc.io/doc/com.google.code.gson/gson/latest/...
- Smaug123 5y ago
  Just a parser which fires events you can listen on when its internal state machine changes state.

anonymoushn 5y ago

> Basically, the answer to all your problems is to use an evented parser instead of a deserializer. This seems a bit unreasonable? People face the same issue where different pieces of software treat duplicate HTTP headers differently, and usually they haven't written each piece of software they use to parse HTTP headers

kortex 5y ago

> this isn't an issue in practice. It absolutely is an issue in practice. If system A handles dupes by accepting the first and ignoring the rest, and system B implements last-key-wins, then that's a potential source of bugs. The system might not fully parse to a map. It may, for example, do string-level modification of json strings. Is that disgusting and wrong? Yes. Have I seen it in prod? Also yes.

throw_m239339 5y ago
> It absolutely is an issue in practice. If system A handles dupes by accepting the first and ignoring the rest, and system B implements last-key-wins, then that's a potential source of bugs. The system might not fully parse to a map. But the system shouldn't be automatically be parsing a "json map" to a map at first place: {"foo":"bar","foo":"baz","foo":"qix","fiz":"buzz"} Shouldn't be deserialized into a map. but a Array<Map<string,string>> like structure. A SAX style parser for JSON can help do that. Thus the issue is the choice of parser indeed. Not JSON.
- q3k 5y ago
  > Shouldn't be deserialized into a map. but a Array<Map<string,string>> like structure. But that's the thing: you might actually expect/want a Map<string,string>, but a malicious/broken system might emit something that cannot be deserialized into a Map<string,string>. It's then the JSON parser's/deserializer's job to figure out what to do, as the standards say to do whatever. That in turn causes different parsers/deserializers to behave differently (whatever the implementer thought makes sense), which is a source of interoperability bugs.
  - throw_m239339 5y ago
    > But that's the thing: you might actually expect/want a Map<string,string>, but a malicious/broken system might emit something that cannot be deserialized into a Map<string,string>. It's then the JSON parser's/deserializer's job to figure out what to do, as the standards say to do whatever. That in turn causes different parsers/deserializers to behave differently (whatever the implementer thought makes sense), which is a source of interoperability bugs. I disagree, people are mixing up parsing and deserializing. The JSON spec isn't at fault here. The JSON spec is only concerned with defining the parsing, not the deserialization, because obviously, a JSON array isn't a PHP array or a Ruby array, a JSON map isn't a PHP object or a Go map at first place. The problem isn't with JSON but how some JSON deserializers work. Again, a deserializer isn't a parser.
    - q3k 5y ago
      > The problem isn't with JSON but how some JSON deserializers work. That makes no observable difference to the end-user of JSON wishing to use it as an interchange format. The standard might as well be perfect, but if nearly all of its implementations (yes, extending that into deserialization, not just parsing - because that's how most people use JSON!) are problematic, then the standard is effectively also problematic. This is why I also always include Python's broken implementation in my JSON rant - it's not indicative of the standard(s) being bad, but the ecosystem being bad.
      - 5y ago
nuerow 5y ago
> It absolutely is an issue in practice. It really isn't. At most, it's a problem caused by picking a broken implementation that doesn't meet your needs, but that's a self-inflicted problem, not a JSON problem.

detaro 5y ago

> Basically, the answer to all your problems is to use an evented parser instead of a deserializer. Which "nobody" does, so it is a problem in practice.

throw_m239339 5y ago
> Which "nobody" does, so it is a problem in practice. who's nobody? if developers care about performances, they obviously do. What if the json file is 500MB of logs? Furthermore, all these JSON deserialization lib tricks might work in some languages that are dynamic or support runtime reflection, it doesn't for other languages where using a proper evented parser is mandatory.

recursive 5y ago

> use an evented parser I've never heard of this. A google search isn't particularly illuminating. What is an "evented parser"?

dragonwriter 5y ago
> What is an "evented parser"? Also knowns as a “streaming parser”, its a parser that takes in a data stream and produces a stream of events which client code can handle; it allows more flexible handling than deserializers, including ability to handle arbitrarily large input. SAX is a streaming/evented parser API for XML, and there are similar ones for other formats.
throw_m239339 5y ago
Google "event-based parser"
rovolo 5y ago
The sibling comments are good descriptions, but here are example interfaces: - Normal parser https://www.javadoc.io/static/com.google.code.gson/gson/2.8.8/com.google.gson/com/google/gson/JsonParser.html https://www.javadoc.io/static/com.google.code.gson/gson/2.8.... - Streaming parser https://www.javadoc.io/doc/com.google.code.gson/gson/latest/com.google.gson/com/google/gson/stream/JsonReader.html https://www.javadoc.io/doc/com.google.code.gson/gson/latest/...
Smaug123 5y ago
Just a parser which fires events you can listen on when its internal state machine changes state.