5 ms·

Parsing JSON Is a Minefield (2018)

• chrisjj 2y ago
  Nice. Feedback: > I wrote yet another JSON parser (section 6) Link defunct.
  • douglee650 2y ago
    ``` One day a student came to Moon and said: “I understand how to make a better garbage collector. We must keep a reference count of the pointers to each cons.” Moon patiently told the student the following story: “One day a student came to Moon and said: ‘I understand how to make a better garbage collector... ```
• jsnell 2y ago
  (2016). Previous significant discussions: https://news.ycombinator.com/item?id=12796556 https://news.ycombinator.com/item?id=12796556 https://news.ycombinator.com/item?id=20724672 https://news.ycombinator.com/item?id=20724672 https://news.ycombinator.com/item?id=28826600 https://news.ycombinator.com/item?id=28826600
  • dang 2y ago
    Thanks! Macroexpanded: Parsing JSON Is a Minefield (2016) - https://news.ycombinator.com/item?id=28826600 https://news.ycombinator.com/item?id=28826600 - Oct 2021 (173 comments) Parsing JSON Is a Minefield (2018) - https://news.ycombinator.com/item?id=20724672 https://news.ycombinator.com/item?id=20724672 - Aug 2019 (178 comments) Parsing JSON is a Minefield - https://news.ycombinator.com/item?id=16897061 https://news.ycombinator.com/item?id=16897061 - April 2018 (246 comments) Parsing JSON is a Minefield - https://news.ycombinator.com/item?id=12796556 https://news.ycombinator.com/item?id=12796556 - Oct 2016 (292 comments)
• sureglymop 2y ago
  Wrote a json parser recently and did not think this hard about it (because the spec is so simple). Time to revisit
• rendaw 2y ago
  The primitive types JSON specifies are redundant and generally only lead to issues. Almost all JSON consumers are either deserializing to a spec that already contains type information, frequently richer, with even more variety of types (url, telephone number, UUID, not just "string"), and even without a spec code will be written to need a specific type (i.e. you're not going to write code to accept an integer when you want a person's name). It would be much simpler if all primitives were strings, and it'd probably save a few people from accidentally doing the wrong thing while dealing with prices.
  • aftbit 2y ago
    Perhaps. I've often wished that JSON supported some sort of custom types or type annotations, or failing that, at least datetimes. Some other nice extensions would be support for comments and optional trailing commas. There is something very nice and expressive about the existing JSON types. Just 6 types (null, boolean, string, number, array, and dictionary) are enough to cover a ton of use cases, and as you suggest, one can always fall back to "stringly typed" alternatives by implementing one's own serialization and deserialization for extra types.
    • ooterness 2y ago
      You may be interested in CBOR (IETF RFC 8949). CBOR features are almost one-to-one with JSON, except that the encoding is more size-efficient, it supports a few additional types (e.g., integers and floats are separate), and it allows semantic tags. https://en.wikipedia.org/wiki/CBOR https://en.wikipedia.org/wiki/CBOR
      • murmansk 2y ago
        While it might be great in theory, CBOR has own separate set of dragons waiting for you. Expectation: tags in CBOR allow you to pass semantics. Reality: multitude of tags, and absence of strict rules for the tags make it pain in the ass.
      • zzo38computer 2y ago
        There are some benefits of CBOR (having a separate integer type is good, and a byte string type is good, and they have typed numeric arrays which is good also, etc), but also some problems. For example, I might have preferred that Unicode is a tag rather than a type (other tags can be used for other character sets), and base64-encoded strings also seems unnecessary (since it is a binary format anyways, you should just use the binary data directly instead), and I think it would be better for a MIME message to be treated as a byte string instead of Unicode (fortuantely the specification allows that, but it seems to just be "added on" afterward due to a lack of consideration), and possibly maybe it might be better to disallow the types of keys to be arrays and maps. However, some of the things I mentioned above, do have benefits for interoperability with JSON, although they aren't good for a general-purpose use; I think that it would generally be better to make a good format rather than trying to work only with the bad ideas of other specifications. (Fortunately, I think what I described above could be implemented using a subset of CBOR.) However, using these formats (whether CBOR or JSON) is often more complicated than should be needed for a specific use anyways.
• aftbit 2y ago
  I'm a little sad to see that no implementations of JavaScript were on the tested parser list. I'd be interested to see where browsers and nodejs `JSON.parse` as well as `eval` parsers fall. As the author mentioned, some of the JSON features are not valid JavaScript but I wonder which of these test cases fail `eval`. Note, just so nobody reminds me, don't parse JSON with eval for security reasons. I'm just curious how it would work from a parser completeness point of view.
• deleted 2y ago
  [deleted]
• Thaxll 2y ago
  XML was indeed better.
  • IshKebab 2y ago
    It absolutely wasn't, primarily because the XML data model is so mismatched with the object structures you find in programming languages. It does at least support comments though. Biggest flaw in JSON by far.
    • wtetzner 2y ago
      > primarily because the XML data model is so mismatched with the object structures you find in programming languages I dunno, it matches up reasonably well with languages that have nestable custom types. XML labels nodes, and JSON labels edges. They both have pluses and minuses.
      • IshKebab 2y ago
        XML nodes have attributes and contents. No programming language I know of works like that.
        • wtetzner 2y ago
          It doesn't seem like an especially important distinction. I've never understood why people always make such a big deal about it.
          • IshKebab 2y ago
            It's a big pain when decoding XML. That's not the only impedance mismatch. The fact that XML is just a soup of objects is not how programming language objects work either. https://docs.rs/serde-xml-rs/0.6.0/serde_xml_rs/#caveats https://docs.rs/serde-xml-rs/0.6.0/serde_xml_rs/#caveats Look at how much more complex this is than the equivalent JSON code, which requires none of these annotations: https://docs.rs/strong-xml/latest/strong_xml/ https://docs.rs/strong-xml/latest/strong_xml/
• ryjo 2y ago
  Writing a JSON parser is a good way to teach yourself better programming practices. I attribute my understanding of pointer arithmetic and i/o streams to my own efforts in parsing/generating JSON.
• jwells89 2y ago
  JSON has its issues, but modern languages including facilities to work with it (fewer dependencies to wrangle is always great) and the way typesafe (de)serializers can be synthesized automatically with a little thoughtfulness in design instead of needing to be manually written (see Swift’s Codable and Kotlin/Java’s Moshi, for example) can in my opinion make it compelling enough to overlook its warts. It doesn’t fit everywhere of course but it’s more than good enough for a vast range of applications.
• theamk 2y ago
  This is interesting, but seems pretty irrelevant for the real world (kinda like "i = ++i + ++i;" C puzzle). The answer to those dangers is "don't do it then". Use your stdlib to emit json, don't use string functions to modify json, assume any number is no better tha float64, and base64 your binary data - and you will never have to worry about this "minefield" (the only possible problem is if you are designing a security system, but even then, since all the ambiguity is whether to reject the string, it will cause DOS at worst)
• mariusor 2y ago
  Funny how Baader-Meinhof works, I just finished writing a JSON toy parser earlier today. I guess I'll add the mentioned corner cases to the testsuite, and watch them fail. :D
  • hughw 2y ago
    And so I just now learned that the Baader-Meinhof Gang of the 1970s gave its name to the phenomenon of frequency illusion, where once you hear about a thing you notice many more references to it.
• RedShift1 2y ago
  Parsing any format is a minefield though...
• thecleaner 2y ago
  Btw if we use parser generators like antlr for this purpose, is it still a minefield ? Can someone point some vulnerabilities I can study ?
• deleted 2y ago
  [deleted]
• zzo38computer 2y ago
  There is problem with JSON, such as: - The numbers is floating points, but cannot be Infinity and NaN. It is not a integer type, so long integers might not work properly. (There are other problems with numbers too, as mentioned in that article.) - The strings is Unicode. Non-Unicode (including binary data) doesn't do properly, and even Unicode can have problems (some of which are mentioned in that article, but there are others too). - Keys are only strings, not numbers. - Syntax convenience isn't so well, e.g. doesn't have comments, optional trailing commas, etc. - The format is difficult for reasons explained in that article, too. One possible alternative would be a format based on a subset of PostScript (instead of JavaScript), e.g. (a part of a example from Wikipedia): << /first_name (John) /last_name (Smith) /is_alive true /age 27 /phone_numbers [ << /type (home) /number (212 555-1234) >> << /type (office) /number (646 555-4567) >> ] /spouse null >> PostScript also has binary format, comments (with a percentage sign), hex string literals, etc. (And, commas are not used, so the problem with trailing commas also does not apply.) (Nevertheless, I did write a JSON parser (and also a JSON writer) in PostScript.) It is also possible to use binary formats, CSV, etc, depending on what exactly is needed by the program; for many reasons, one format cannot solve everything.
  • BugsJustFindMe 2y ago
    > The numbers is floating points...long integers might not work properly I personally hate the usual interpretation as float and see it as a common but extremely-implementation-induced failure. It's far better interpreted as an arbitrary precision numeric type, not float or int. The spec even says as much and only says that implementations mostly suck so watch out. IMO precision myopia is why we end up with e.g. Python's refusal-by-default to (de)serialize from/to Decimal.
  • nurettin 2y ago
    Why not make non-strict parsers that will handle unicodes, longs, binary, ignore comments and allow trailing commas? If you set bend_over_backwards=true, it will do strict parsing for the poor souls who need that. edit: I didn't mention integer keys, because object members canonically start with a letter.
  • 2y ago
• stevejb 2y ago
  I definitely agree with a lot of the comments here especially the ones in the vein of “don’t do dangerous things with json. “ if you have control of the sender and the receiver, it makes sense to have fields that add a bit of extra type information e.g. this is an integer or this is a float with this much precision
  • _lvbh 2y ago
    At that point just use protobuf
• throwaway984393 2y ago
  [dead]