3 ms·

> Why unsafe to parse? Try to write a parser for an arbitrarily complex and nested JSON object without running into runtime errors even after the fourth-or-so

> Why unsafe to parse? Try to write a parser for an arbitrarily complex and nested JSON object without running into runtime errors even after the fourth-or-so revision. There are too many "yes buts" in the living standard. > that overhead is not much worse It _is_ much worse than the two-byte "overhead" of one s-exp node.

tonyg 7y ago
> It _is_ much worse than the two-byte "overhead" of one s-exp node. I don't believe you. Atoms are roughly the same, and compounds, well: [1,2,3] vs (1 2 3) {"a":1,"b":2} vs (("a"1)("b"2)) I suppose if you have a lot of booleans you might save a little by using T/NIL or #t/#f, but otherwise it's more or less a wash. (You might suggest plists, ("a"1"b"2), for dictionaries, in which case you save a whopping two bytes per entry. But I'd choose the nested-list format over plists, if I were encoding a dictionary into an S-expression.)
- viraptor 7y ago
  > you save a whopping two bytes per entry Which pretty much disappears when compressed anyway.
lloeki 7y ago
In specific contexts such as this one you don't need arbitrarily complex JSON, you just need to validate that the JSON structure respects not just JSON itself but also the protocol specification at hand while you parse, and bail out when it doesn't, so you can have a dedicated, fast parser tailored for the use case, instead of first parsing general JSON then validate its content. Anyway, for being a Fasmail user, JMAP rips IMAP to shreds in terms of performance. I've been following JMAP from afar and waiting for Fastmail to publish this for a long time.
- rhabarba 7y ago
  So you're suggesting to write a tailored JSON parser for each use case? Sounds convincing over my preferred approach...
  - lloeki 7y ago
    Not for each use case of everything, anywhere, of course. This specific use case is important enough to warrant a dedicated parser though, and has benefits both in performance and in security. People have ben actively doing this for XML with streaming parsers that emit events, where you don't just validate XML itself but the semantics of the payload you're receiving, and bail out early instead of parsing the whole XML into a DOM then validating the DOM structure and values.
some_random 7y ago
>There are too many "yes buts" in the living standard. Could you give an example? The only deviation from the standard I've run into is JSONL being called JSON, which isn't actually meaningfully harder to parse.
afiori 7y ago
Is nesting in JSON more complex than nesting in sexpr? I feel like it should be possible to translate from valid JSON to valid sexpr with a regexp parser (just match parenthesis and stuff) > There are too many "yes buts" in the living standard. I am not an expert and do not want to sound dismissive, but I feel like most of those "yes buts" are a solution to otherwise incompatible implementations. I fail to see where S-expressions have an advantage here. Is there a widely recognized standard that most implementations respect?
geofft 7y ago
> Try to write a parser for an arbitrarily complex and nested JSON object without running into runtime errors even after the fourth-or-so revision. Do you mean parsing the JSON bytes itself into maps and lists and atoms, or turning that into an application-useful object? If the former, are you claiming that Python's standard json library, libjansson, JSON::XS, etc. have bugs? (I don't know why you would roll your own JSON parser in production code.) If the latter, how are S-expressions any different? (And there are non-buggy libraries that do both, e.g., serde.)

tonyg 7y ago

> It _is_ much worse than the two-byte "overhead" of one s-exp node. I don't believe you. Atoms are roughly the same, and compounds, well: [1,2,3] vs (1 2 3) {"a":1,"b":2} vs (("a"1)("b"2)) I suppose if you have a lot of booleans you might save a little by using T/NIL or #t/#f, but otherwise it's more or less a wash. (You might suggest plists, ("a"1"b"2), for dictionaries, in which case you save a whopping two bytes per entry. But I'd choose the nested-list format over plists, if I were encoding a dictionary into an S-expression.)

viraptor 7y ago
> you save a whopping two bytes per entry Which pretty much disappears when compressed anyway.

lloeki 7y ago

In specific contexts such as this one you don't need arbitrarily complex JSON, you just need to validate that the JSON structure respects not just JSON itself but also the protocol specification at hand while you parse, and bail out when it doesn't, so you can have a dedicated, fast parser tailored for the use case, instead of first parsing general JSON then validate its content. Anyway, for being a Fasmail user, JMAP rips IMAP to shreds in terms of performance. I've been following JMAP from afar and waiting for Fastmail to publish this for a long time.

rhabarba 7y ago
So you're suggesting to write a tailored JSON parser for each use case? Sounds convincing over my preferred approach...
- lloeki 7y ago
  Not for each use case of everything, anywhere, of course. This specific use case is important enough to warrant a dedicated parser though, and has benefits both in performance and in security. People have ben actively doing this for XML with streaming parsers that emit events, where you don't just validate XML itself but the semantics of the payload you're receiving, and bail out early instead of parsing the whole XML into a DOM then validating the DOM structure and values.

some_random 7y ago

>There are too many "yes buts" in the living standard. Could you give an example? The only deviation from the standard I've run into is JSONL being called JSON, which isn't actually meaningfully harder to parse.

afiori 7y ago

Is nesting in JSON more complex than nesting in sexpr? I feel like it should be possible to translate from valid JSON to valid sexpr with a regexp parser (just match parenthesis and stuff) > There are too many "yes buts" in the living standard. I am not an expert and do not want to sound dismissive, but I feel like most of those "yes buts" are a solution to otherwise incompatible implementations. I fail to see where S-expressions have an advantage here. Is there a widely recognized standard that most implementations respect?

geofft 7y ago

> Try to write a parser for an arbitrarily complex and nested JSON object without running into runtime errors even after the fourth-or-so revision. Do you mean parsing the JSON bytes itself into maps and lists and atoms, or turning that into an application-useful object? If the former, are you claiming that Python's standard json library, libjansson, JSON::XS, etc. have bugs? (I don't know why you would roll your own JSON parser in production code.) If the latter, how are S-expressions any different? (And there are non-buggy libraries that do both, e.g., serde.)