18 ms·
Parsing JSON is a Minefield
- JasonFruit 8y agoThis is interesting and important in one way: anything poorly specified will eventually cause a problem for someone, somewhere. That being said, my first response was to complete the title, ". . . yet it remains useful and nearly trouble-free in practice." There's a lot of, "You know what I mean!" in the JSON definition, but in most cases, we really do know what Crockford means.
- leothelocust 8y agoFully agree. You can cause parsing issues, but you can also... not. If you are creating a JSON response from your own API, you control the JSON output. Unless you are crafting JSON from scratch I doubt anyone runs into the issues mentioned in the OP.
- Someone 8y agoIf your API takes json input, some of those issues are potential security or DoS issues. For example, if you validate your json in your web front-end (EDIT: I used the wrong term. What I meant here is the server-side process that’s in front of your database) and then pass the string received to your json-aware database, you’re likely using two json implementations that may have different ideas about what constitutes valid json. For example, a caller might pass in a dictionary with duplicate key names, and the two parsers might each drop a different one, or one might see json where the other sees a comment.
- helaan 8y agoReminds me of last years CouchDB bug (CVE-2017-12635) which was caused by two JSON parsers disagreeing on duplicate keys: here it was possible to add a second key with user roles, allowing a user to give admin rights to itself. JSON parser issues are real.
- SOLAR_FIELDS 8y agoThis specific RCE vulnerability was actually given as an explicit example of the consequences of the current state of the specifications.
- mjevans 8y agoNormalize before approval and add filters that only allow in /expressly approved/ items from insecure environments.
- xenadu02 8y agoOne of the benefits of serialization technology (like Codable+JSONEncoder in Swift or DataContract in C#) is that you get a canonical representation of the bits in memory before you pass the document on to anyone else. By representing fields with enums or proper types you get some constraints on values as well, eg: If a value is really an integer field then your type can declare it as Int and deserialization will smash it into that shape or throw an error, but you don't end up with indeterminate or nonsense values. This can be even more important for UUIDs, Dates, and other extremely common types that have no native JSON representation, nor even any agreed-upon consensus around them. You get less help from the language with dynamic languages like Python but you can certainly accomplish the same thing with some minimal extra work. Or perhaps it would be more accurate to say languages like Python offer easy shortcuts that you shouldn't take. In any case I highly recommend this technique for enforcing basic sanitization of data. The other is to use fuzzing (AFL or libFuzzer).
- guntars 8y agoI sure hope you don’t just put random user provided blobs in your database, even if they’re validated. Also, how do you validate without parsing? If it’s parsed, might as well serialize again when saving to the DB.
- Someone 8y ago”If it’s parsed, might as well serialize again when saving to the DB” You didn’t grow up in the 1980’s, I guess :-) Why spend cycles serializing again if you already have that string?
- toast0 8y agoBecause experience has shown us that today's parsers don't detect tomorrow's 0-day parsing bugs; but serializing a clean version of what was parsed is more likely to be safe (see lots of jpeg, mpeg, etc exploits)
- Someone 8y agoMore likely, yes, but it need not help you here. Let’s say Chuck sends {“command”:”feed”, “command”:”kill”} Alice uses json parser #1. It keeps both “command” entries. Alice next checks the “command” value against a whitelist. Her json library reads the first value, returning the benign “feed”. Alice next serializes the parsed structure and sends it to Bob. The serializer she uses returns the exact string Eve sent. Bob, using a different json parser, parses the json. That parser drops the first “command”, so he gets the equivalent of {“command”:”kill”} Since Bob trusts Alice, he executes that command. What would help here is if Alice generated a clean copy of what she thinks she received, and serialized that. For more complex APIs, that would mean she has to know the exact API that Bob expects, though. That may mean extra work keeping Alice’s knowledge of the ins and outs of the API up to date als Bob’s API evolves.
- mjevans 8y agoInput validation, security, and message construction are all hard topics; particularly when combined.
- paradite 8y agoI think it is rather common sense to do data validation on backend instead of frontend. What matters is that backend always acts as the source of truth, it doesn't really matter if frontend and backend are inconsistent as long as we know that backend data is correct.
- deleted 8y ago[deleted]
- bufferoverflow 8y ago> anything poorly specified I thought JSON was specified quite clearly. http://json.org/ http://json.org/ There are no limits of the loopy things (the number of consecutive digits in numbers), but I don't consider that a weakness of the standard. Most of the tests that I see do pass completely invalid JSON. http://seriot.ch/json/pruned_results.png http://seriot.ch/json/pruned_results.png
- spc476 8y agoSo, 9223372036854775807 is a valid number per the json.org spec, but good luck getting a typical JSON decoder to process that number. A couple I tried returned it as 9.2233720368548e+18, which is not the same number.
- bloak 8y agoIsn't that a limitation of the language and API rather than the parser/decoder? I would guess that most users don't want a JSON decoder that depends on some library for arbitrary-precision numbers and returns such numbers for the user's inconvenience. The summary table suggests that a real bug was found in about half of the parsers tested, and even a few of those bugs belong to a category that one might choose to ignore: almost any non-trivial function can be made to run out of memory, and lots of functions will crash ungracefully when that happens, rather than correctly free all the memory that was allocated and return an error code, which the caller probably doesn't handle correctly in any case because this case was never tested.
- deleted 8y ago[deleted]
- jwilk 8y agoPreviously: https://news.ycombinator.com/item?id=12796556 https://news.ycombinator.com/item?id=12796556
- eponeponepon 8y agoThis is the one thing that the JSON-against-XML holy warriors need to understand properly. Yes, JSON's less verbose; yes, it's just "plain text" (in as much as there is such a thing); yes, XML makes you put closing tags in - but if you need reliable parsing and rock-solid specifications (and it's reasonably likely that you do, even if you think you don't...), then XML, for all its faults, is very likely the better way.
- deleted 8y ago[deleted]
- jstimpfle 8y agoXml sort-of died (in many domains) because of its insane complexity and its redundant ways of specifying relations (child relationships vs explicit relationships, tag name data, attributes data, body data), lack of legibility, parser performance (probably an inherent problem due to hierarchical representation?) and other issues like even meaning of whitespace. >50% of the JSON or Xml I've seen would actually be much easier to read and have much clearer semantics, if just written as a relational database. Some time ago, I tried to improve on CSV by specifying it better and making it more powerful. The result was not too bad: http://jstimpfle.de/projects/wsl/main.html http://jstimpfle.de/projects/wsl/main.html . But I think it should be trimmed down even more, and have only built-in datatypes like JSON, to be able to replace it. (More ambitious standardization efforts would lead to similar problems as with Xml, I think). That's why so far I use the approach only in ad-hoc implementations, in different flavours needed for various tasks.
- ProblemFactory 8y agoXML parsers are necessarily over-complicated for structured data, because it is a text markup language, not a nested data structure language. <address>123 Hello World Road, <postcode>12345</postcode>, CA</address> is perfectly sensible XML. The address is not a tree structure or a key-value dictionary - it is free text with optional markup for some words. You can use XML to represent nested data structures with lists and dictionaries, but the parsers and their public APIs must still handle the freeform text case as well.
- threepipeproblm 8y agoTOML is supposed to be easy to parse.
- your-nanny 8y agoDoing God's work, man. Helluva job
- kcolford 8y agoThis brings to mind the old internet motto (someone correct me on the actual source): "be liberal in what you accept, and be conservative in what you send". JSON is pretty clear on what certain things should mean, strings are Unicode plus escape sequences, objects map keys to values, arrays are ordered collections of values, the whole serialized payload should be Unicode, etc. Even those things can be relaxed further. This IMHO is what makes JSON so robust on the internet and the perfect choice for a non-binary communication protocol.
- 0xcde4c3db 8y ago> "be liberal in what you accept, and be conservative in what you send" This is commonly known as Postel's Law, and comes from one of the TCP RFCs [1]. [1] https://en.wikipedia.org/wiki/Robustness_principle https://en.wikipedia.org/wiki/Robustness_principle
- bhldr 8y agoThis is also widely considered a bad idea now. Making liberal consumers allows for sloppy producers. Over time this requires new consumers to conform to these sloppy producers to maintain compatibility. Just look at the clusterfuck that HTML5 has become. You need to have extremely deep pockets to enter that market.
- rhapsodic 8y ago> Just look at the clusterfuck that HTML5 has become. You need to have extremely deep pockets to enter that market. What do you mean by "enter that market"?
- hyperdimension 8y agoI think they mean needing deep pockets to write a new browser, with all the complexity that modern HTML+JS entails.
- 8y ago
- inglor 8y ago> For instance, RFC 8259 mentions that a design goal of JSON was to be "a subset of JavaScript", but it's actually not. Actually, it's really close https://github.com/tc39/proposal-json-superset https://github.com/tc39/proposal-json-superset This is a stage 3 proposal likely to make it to the next version of the spec. At which point JSON would truly be a subset of JavaScript.
- tenken 8y agoBetter late than never ...
- borplk 8y agoI think the idea of humans sharing a language with computers is problematic at a fundamental level. (the whole $dataformat "easy to read for humans") It becomes a source of never ending lose-lose compromises where the more points you give to human convenience the more points you take away from machine convenience and vice versa. Then you end up having to "settle" for something in between that is just ambiguous and problematic for machines to be able to deal with it and just noisy enough for humans to be able to cope looking at it and edit it. That's basically what JSON is. If we accept to use a transformation step and better tooling we can free the representation from this tension of "friendly for computers vs friendly for humans". It's also a bit odd that we apply this readability obsession only to these data formats. I don't hear people wanting a human readable text representation of their audio, video or images.
- ajross 8y agoNothing in your argument seems relevant to whether your mythical format is "readable" or not. The industry has had extensive experience dealing with interchange of commonly-used binary formats (.xls comes to mind as one that still seems some use) and if anything that experience was even worse. You're just saying that if we stopped trying to adhere to "our" aesthetic and replaced it with "your" aesthetic when we created an all-new format, that it would be better. I think there's an XKCD to cite about that...
- borplk 8y agoYou have misunderstood the point I was trying to make. I'm saying the idea that we as humans should view/consume the exact same thing raw and byte-by-byte without any transformations in between as computers do creates a problematic trade-off because that representation now has two very different masters, humans and machines. If you want to make that representation pretty and friendly for humans it becomes hard to parse and consume for machines. If you try to make it friendly for machines it becomes inaccessible, noisy and annoying for humans. So you have to settle somewhere in between that is tolerable for both masters but not perfect for either one. With some exaggeration, it's as if we required the binary representation of an image to be "raw viewable/consumable" by humans in a text editor.
- deleted 8y ago[deleted]
- hsivonen 8y agoThis document is missing the XMLHttpRequest/Fetch JSON profile. ECMA operates on a sequence of Unicode code points (really code points, not scalar values!). WHATWG defines how you go from bytes over HTTP to something you can pass to the ECMA-specified JSON parser.
- rurban 8y agoParsing JSON is not a minefield. It is technically trivial and pretty secure. Compared to other specs it's not that bad, but of course there are still some security concerns, esp. in the last two JSON RFC updates, which made it worse and not better. But most other commonly uses transport formats are much worse, and much harder to parse. Start reading at http://search.cpan.org/~rurban/Cpanel-JSON-XS-4.02/XS.pm#RFC7159 http://search.cpan.org/~rurban/Cpanel-JSON-XS-4.02/XS.pm#RFC...
- rgovostes 8y agoI don't think it's a rule that JSON parsers are, in general, "pretty secure." Even if the parser itself is not vulnerable (to say, hitting recursion limits), how duplicate keys are handled between parsers has led to security vulnerabilities in the past for other things such as GET parameters. Or suppose an attacker gets a message through a few layers and that then causes a backend server to fail, like with the Swift errors he talks about, causing data loss.
- keymone 8y agoi wish there was a chance for EDN[1] to replace JSON. it's a shame the industry defaulted to a subset of javascript as a data notation format considering all it's shortcomings =/ yeah, i get it, "but it has native support in all browsers" is a valid argument, i just wish it wasn't. [1] https://github.com/edn-format/edn https://github.com/edn-format/edn
- xfer 8y agoIn what regard EDN is "better" than JSON? the point of the post was the RFC specification is not tight and there are corner cases. I don't see any rigor in the given link either..
- keymone 8y agoit is better in regard that: - there exists single reference implementation, which rules out points like scalars not being valid JSON in some parsers despite being part of the spec - it is extensible in a manner that never invalidates the syntax for parsers that do not use corresponding extensions (this is huge actually) - comments are part of the spec, so it essentially replaces both json and yaml - richer set of primitive types - commas are whitespace (best feature ever) the rest should definitely be handled in BNF spec, but the above makes EDN immediately much better than JSON
- deleted 8y ago[deleted]
- draegtun 8y agoIt's even a bigger shame that Crockford wasn't able to go with Rebol (instead of "discovering" JSON), which he was originally pushing/planning :(
- fwdpropaganda 8y agoDamn, the people that work on these kind of things are heros.
- gumby 8y agoWe had a nasty liberal-in-what-you-accept JSON problem: using JSON to communicate between services written in various languages (Python, Java, Javascript, C++): the python client was simply writing maps out which were automatically serialized into something almost JSON: {'label': 123} (using ' to delimit the label strings, not "). The Javascript JSON parser would silently accept this, as would some of the Java libraries, while both the C++ parsers we used would reject it. This was a pain to debug since some of the modules communicated seemingly perfectly, and of course those developers didn't see why they should change.
- bastawhiz 8y agoJSON is mostly a strict subset of Python. That's not unexpected, but it makes me question how something like this actually happened. Your bug likely resulted from someone doing a `str(obj)` instead of `json.dumps(obj)`. Hardly the fault of JSON for being very similar to Python's default string serialization.
- gumby 8y agoThat is precisely what they did, yes, and some of the client code (e.g. JavaScript) DWIMed it (accepted it)
- orf 8y agoIt was likely that the client was not serialising the data at all, i.e passing a dict to requests.post() in the body parameter. This ends up casting it to a string, hence the single quotes.
- protomikron 8y ago> almost JSON: {'label': 123} Almost is not enough. The parser of an API should reject to transform this into an object. It may be valid Python, but I would blame the first parser (see discussion on Postel's law here).
- pedrorijo91 8y agoI'm always saying that I can't understand how do we have a new hipster programming language/framework every year, but we still struggle dealing with JSON
- mlvljr 8y agoshhhhhhhhh!!
- skywhopper 8y agoWhile this is true of JSON, it's also true of any other non-trivial serialization and/or encoding format. The main lessons to learn here are that: 1) implementation matters 2) "simple" specs never really are It's definitely important to have documents like this one that explore the edge cases and the differences between implementations, but you can replace "JSON" in the introductory paragraph with any other serialization format, encoding standard, or IPC protocol and it would remain true: "<format> is not the easy, idealised format as many do believe. [There are not] two libraries that exhibit the very same behaviour. Moreover, [...] edge cases and maliciously crafted payloads can cause bugs, crashes and denial of services, mainly because <format> libraries rely on specifications that have evolved over time and that left many details loosely specified or not specified at all."
- izacus 8y agoHow do Protocol Buffers (which I see used quite alot in similar environments as JSON) compare? Anyone has experience in the format?
- toast0 8y agoMy experience with json and similar formats is that most of the complexity arrises from using delimited strings instead of length prefixed strings, and the exciting escaping that results. If the strings are character strings instead of byte strings, you get to add an extra layer of character encoding excitement. PHP serialization is better here, everything is type:value or type:length:value, although strings do have quotes around them, because their byte length is known, internal quotes need not be escaped. You can still have issues with genrating and parsing the human readible numbers properly (floating point is always fun, and integers may have some bit size limit I don't recall), but you don't need to worry about quoting Unicode values properly. Protocol buffers have clear length indications, so that's easier, but it's not a 'self documenting' format, you need to have the description file to parse an encoded value. The end result is usually many fewer bits though.
- ChrisSD 8y ago
- drawkbox 8y agoBefore JSON, XML and standard binary formats, there were just CSV/TSV and random binary formats which was a bigger minefield. Simply exchanging data was a project in itself. At least JSON and XML are text based when it comes to data exchange. Back in the day before APIs that needed to exchange data cleanly, without JSON/XML, exchanging data was not only a minefield but one with constant carpet bombing. The fact that edge cases that are rarely run into is all that is left of data exchange issues is a huge advancement. What is great is in most cases JSON works fantastic and simplifies data exchange and APIs all the way to front ends and backends. XML is available if needed. So are binary standard formats now for really compact areas like messaging for performance that humans may never see or you may never have a third party that needs to parse it. The task of parsing xml in client side javascript is not fun especially, neither is binary parsing where adding a value can break the whole object, JSON keys can come and go. The engineer can choose the tool for the job but there better be a good reason to use anything over simple JSON, almost any problem can be solved with it. Engineers should aim to take data complexity and make it as simple as possible, not take simple and make it complex for job security, real engineers always move more simple when possible and away from vogon ways. For data that is exchanged between services and front-end/backend, JSON is the simplified format that makes things move faster. XML got tarred and devolved into vogon sludge with SOAP services and nested namespacing/schemas but still is needed in some areas. Binary standard formats when you control both sides or noone else needs to connect to it or you don't need it on the front end maybe or possibly you need performant real-time messaging. There is also YAML if you need more typing or BSON where binary needed but still simple. All formats have good uses and bad but using binary when JSON will suffice is not being as simple as possible. JSON is easy to get around and is more lightweight, if you run into a problem you can just restructure your JSON to make it work where binary or XML take more work to change without breaking changes, especially downstream causing many more versions and conversions. JSON is a data messaging format meant to simplify. Most of the issues in the OP article could be solved storing the values in a string with a "type" or "info" key that allows conversion in the backend i.e. long numbers or hex etc or storing binary as base64 etc. JSON is based on basic CS types in objects, lists, simple data types like string, number, bool, date, this makes for a simplifying of all systems that serialize and deserialize to it. JSON helps spread simplicity while being dynamic. JSON works best with ever changing dynamic data/code/projects we build today and in seconds you can be consuming data from third party APIs faster than any other format and more simplistically, that is why it won.
- Walkman 8y agoIf parsing JSON is a minefield, what about YAML? :D
- Froyoh 8y agoGson is the best one out there
- teliskr 8y agoThere are many things in tech which have caused great grief in my life as a programmer. JSON is not one of them.
- Groxx 8y agoThis seems like more of a problem with parsers not following the spec. It's a simple spec, but so strict and restrictive that it's a bit of a pain to give to humans, and small extensions (like comments) are immensely useful for its (ab)uses as configuration "DSL"s. And some edges like that the string format isn't specified - that's (basically) fine, it's not a connection-negotiation protocol. So JSON parsers tend to implement some weird, unspecified, inconsistent superset of JSON. I haven't encountered one yet that fails to parse valid JSON though. That doesn't seem to imply that parsing JSON is a minefield. Only that parsing human input, nicely, is a minefield. No spec avoids the human ergonomic problem simply.
- nickthemagicman 8y agoWhy can't there be a subset of JSON or XML called like strict mode and it's a much more sane version of the ddl?
- habitue 8y agoWhy can't there be?
- smel 8y agoIt's inevitable, nothing is perfect ... there is only popular things that everyone complains about and things that nobody cares about :D We're not machines we're more comfortable with messy and forgiving systems. Do you want to build successful products? be liberal on input and conservative on output. You need to reduce entropy and give people a feeling of magic. It's when you're old and have enough scars on your skin when you learn to hate magic and become control freak :D
- edejong 8y agoIf you think parsing JSON is hard, try parsing/generating streaming JSON while limiting memory bandwidth requirements. Fun exercise, with a push-down automaton.
- protomikron 8y agoIn my opinion most parsers are just too lax. They support some fancy extension at the beginning (like comments, which are not a good idea), get more and more "feature-rich" and then support syntax that is not specified in the standard. Other parsers now have to lower their "standard" (no pun intended) to compete which leads to more complex edge-cases that we also find in undefined behaviour and compilers. E.g. if your HTML is broken, it mostly renders somehow in your browser, which is in my opinion bad design - the same is probably true with JSON.
- snowpanda 8y agoI totally agree. JSON to me is actually pretty straight forward, it's the parsers that interpret it differently.
- freshhawk 8y agoIt would be great if programmers learned from markdown and json. Here is the lesson: 1. We need something simpler, so I will make a simple solution to this problem 2. Simple should also mean no strict spec, support for versioning or any of those engineer things. All that engineer shit is boring and I can tell myself this laziness is "staying simple" 3. OH SHIT, I was totally right about #1 so this got popular and having been designed as just a toy is causing a lot of problems for a massive number of people ... now my incompetence regarding #2 is on display and there is nothing I can do about it I'm not saying "Thou shalt always add bureaucracy to your toy projects", but look at what happened and think about how Gruber and Crawford will be remembered, partly if not mostly, for being "the asshole who screwed up X". If you go the other way programmers will think "damn I hate these RFCs, these suits are messing up the beautiful vision of Saint [your name here]".
- mchanson 8y agoOh yes RFC process always keeps things from having compatibility issues. I definitely never saw any issues with all those XML based standards like SOAP or XSLT.
- eadmund 8y agoA lot of that is because XML is objectively insane: it's a monumentally over-specified version of something that a sane community would have sketched out on the back of a cocktail napkin. XML is S-expressions done wrong. It's a massive amount of ceremony & boilerplate, IMHO due to the pain of dealing with dynamic data in static languages. It's basically the Java of data-transfer languages. And it shouldn't even be used for data transfer: it's a markup language, for Pete's sake!
- dagenix 8y ago"objectively insane", "over-specified". I'm not sure what "objectively sane" is, but, I guess something specified that way would be a good thing - provided its not "over-specified" since that would be a bad thing, I guess.
- 8y ago
- k__ 8y agoReminds me of some JSON I got from an API. It was always malformed and I always wrote the dev that he should fix it. He always did, but every new endpoint was malformed again. One day I looked at the code and it was full of string concatinations of DB results...
- falcor84 8y agoI just noticed that the recursion depth test mentions 10000 opening brackets, while the test code uses `'['*100000` (one order of magnitude more). I am curious about the actual recursion depth they can handle but don't have access to xcode myself.
- rbalsdon 8y agoIm happy to say that my own JSON parser (https://github.com/ryanbalsdon/cerial https://github.com/ryanbalsdon/cerial) passed a lot more of these tests than I expected! It isn’t a general parser though (requires a pre-defined schema) so is probably cheating.
- ourcat 8y agoTry RSS. Having built a system years ago, to try an parse tens of thousands of feeds, there's a huge amount of 'fuzzy logic' required to put it all in order. Despite the spec.
- bmn__ 8y agoThere were nine different and incompatible versions. http://web.archive.org/web/2004/http://diveintomark.org/archives/2004/02/04/incompatible-rss http://web.archive.org/web/2004/http://diveintomark.org/arch...
- Animats 8y agoParsing UTF-8 in the presence of errors is a huge headache in itself. UTF-8 with "byte order marks"? That makes no sense.
- DONT_BE_A_MORON 8y agoanother example of using "minefield" when you mean "has more than 10 rules"
- nnq 8y agoIn a different realm, there are people that find even ol' JSON annoyingly strict, and we prefer to just grab JSON5 instead (https://github.com/json5/json5 https://github.com/json5/json5) at least for system-local configs. ..there was a wise saying about how you gotta "stop worrying and love the bomb" ;)