9 ms·
The JSON spec [1] clearly states, in section 9: An implementation may set limits on the maximum depth of nesting. So, this is kind of a pointless exercise
by thegeomaster 7y ago
The JSON spec [1] clearly states, in section 9:
An implementation may set limits on the maximum depth of
nesting.
So, this is kind of a pointless exercise. The author would do well to inform themselves better before slinging epithets such as "bad" for a library.
[1]: https://tools.ietf.org/html/rfc7159 https://tools.ietf.org/html/rfc7159
- saagarjha 7y agoI think you’re missing a link.
- Svip 7y agoIt's funny, the spec they references[0] makes no mention of parsers or nesting. But RFC 8259[1] clearly does. Moreover, I notice they were both published in December 2017. Why two specs for the same standard? [0] http://www.ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf http://www.ecma-international.org/publications/files/ECMA-ST... [1] https://tools.ietf.org/html/rfc8259 https://tools.ietf.org/html/rfc8259
- gsnedders 7y agoAs is almost always the case when there are multiple specs purporting to describe the same thing: politics. The Ecma spec is meant to be a spec defining a format and its semantics; the RFC places requirements on implementations of that format as well as describing it's semantics.
- snek 7y agoThere was a rumor that was IETF was going to specify a bunch of new breaking changes in RFC 7195 (March 2014), and because of that (irrational) fear ECMA published the 1st edition of 404 very quickly in October of 2013. RFC 8259 and the 2nd edition of ECMA-404 contain the same change, a single sentence, specifying that JSON must use UTF-8.
- deleted 7y ago[deleted]
- stevula 7y agohttps://tools.ietf.org/html/rfc7159#section-9 https://tools.ietf.org/html/rfc7159#section-9
- skybrian 7y agoMaybe the headline is bad, but making a table of implemention limits of various JSON parsers seems useful? Maybe we focus on headlines too much?
- webkike 7y agoI dunno, the author claims they're sorting the table from worst to best, instead of by minimum maximum depth to maximum maximum depth
- vesinisa 7y agoNevertheless, the parser failing at only 100 levels of nesting is shockingly bad.
- ryanianian 7y agoLots of recursive data-structures (trees/graphs/*) lend themselves nicely to lots of nesting. Such documents aren't really intended to be read/written by humans.
- vesinisa 7y agoNo, and if they can not be even read by a machine, it's bad.
- cortesoft 7y agoIf they aren't meant to be read/written by humans, JSON is probably a bad choice.
- crazygringo 7y agoI couldn't disagree more. If you need to serialize and deserialize data between tools written in different languages -- e.g. write data to a file and read it in another tool -- then what would be better than JSON? Nothing about JSON is inherently good or bad for deeply hierarchical/recursive data.
- shakna 7y agoDeeply hierarchical/recursive data is equally well served by XML and Protobuf. However, JSON itself can be... Painful for some subsets of data that you may wish to move between applications. Binary data, well-typed data. JSON tends [0] uses floats, so you may need to adjust your expectations about how to store numbers if precision matters. You can workaround a lot of these, such as through various encodings, but they are workarounds. A lot of those workarounds depend on re-encoding, which unfortunately means using strings, which can run afoul of "An implementation may set limits on the length and character contents of strings." and silently drop important bytes, or simply cut off the end of the payload. [0] The grammar is well-defined, the behaviour is not. "This specification allows implementations to set limits on the range and precision of numbers accepted." https://tools.ietf.org/html/rfc8259#page-7 https://tools.ietf.org/html/rfc8259#page-7
- lovasoa 7y agoMaybe I shouldn't have called this repository "bad json parsers". I just added the following to the README: > The name of this repository is intentionally provocative. It's goal is to document limitations of existing json parsers, not to denigrate the work of the many contributors who worked on the various libraries presented here.
- elpocko 7y agoIts*. "It is goal" makes no sense.
- lovasoa 7y agofixed
- bb88 7y agoFirst appreciate the change to the readme. I think that goes a long way. Second, "bad" here is highly subjective. I've never ever hit the recursive depth limit in python with JSON, and if I do, I'll probably think about doing it a different way, in a way that won't require python to do deep recursion.
- deleted 7y ago[deleted]
- Igelau 7y agoThe word you're looking for is clickbait ;) "We Ranked The Top JSON Parsers By Call Stack Depth... Number 6 Will Shock You!"
- bryanrasmussen 7y agoBad is a value statement, they didn't say not to spec, invalid, or similar.
- gameswithgo 7y agoIt is better to not set limits.
- skybrian 7y agoThere is always a limit. The question is whether you want to use up all available memory reading bad input, or reject it sooner. If you're worried about denial-of-service attacks, it makes sense to be explicit about how big an input you'll accept.
- paulddraper 7y agoYeah, this is dumb.* I'd sooner criticize a parser for mangling numbers above 2^53 (also allowed by the spec) than not allowing thousands of nested arrays. --- * Except that this is potential DoS vector. From that angle, it has some interest.
- Svip 7y ago> I'd sooner criticize a parser for mangling numbers above 2^53 (also allowed by the spec) The spec has something specifically to say about this[0]: > Note that when such software is used, numbers that are integers and are in the range [-(253)+1, (253)-1] are interoperable in the sense that implementations will agree exactly on their numeric values. Basically, allow up to 2^53, and you ought to be fine. [0] https://tools.ietf.org/html/rfc8259#section-6 https://tools.ietf.org/html/rfc8259#section-6
- paulddraper 7y agoAgreed, such a parse would be "to spec". It says "An implementation may set limits on the maximum depth of nesting. An implementation may set limits on the range and precision of numbers." On a practical level, parsers not handling 64-bit ints has bit me more than parsers not handling 1000+ levels of nesting.
- axman6 7y agoAs a side note, the Haskell implementation also supports numbers of arbitrary size, again limited by memory, for both parsing and encoding.
- ken 7y agoI would be perfectly OK with that answer if they documented that they use the C stack and therefore have a small finite depth limit, and/or threw some sort of TooMuchNesting exception before they hit that limit. AFAICT, many/most of these libraries do neither of the above. There's nothing in the documentation to suggest they'll have trouble with deeply nested inputs, and their response to such a situation is to simply crash the process. I'd call that a bug. I don't consider it acceptable for a library to simply say "the spec says we're technically allowed to 'set limits' on what we accept, so here's some nasal demons". If you've got limits, you've got to document them, or at least make them recoverable.
- deleted 7y ago[deleted]
- asveikau 7y agoIt's hard for a library to portably know when the C stack will exhaust or give a concrete estimate that makes sense for documentation purposes. Firstly, that concept is not exposed at all to standard complaint C code. The standard is vague enough to allow for many possible variations of the program stack abstraction. So you would not be portable, standard C. Second, the library author doesn't know ahead of time how much stack space you, the caller, have consumed before calling the library. Do you call it deep inside some ui framework with a big stack? With a huge i/o buffer on the stack? They don't know. Translating all of this into "this is how deeply nested your json can be" is not a sane exercise, at best you can give a machine-specific guess that will vary a bit with your own application's stack needs. But perhaps one can side-step all of that and get most of the way with an artificial, arbitrary-constant runtime depth check that is safely below likely actual stack limits.
- ken 7y agoSure, that’s all true. So don’t use the C stack for parsing. It’s not difficult.
- taneq 7y ago> I don't consider it acceptable for a library to simply say "the spec says we're technically allowed to 'set limits' on what we accept, so here's some nasal demons". Welcome to C, enjoy your stay.
- andrepd 7y agoFrom TFA >The two most recent JSON standards RFC 8259 and RFC 7159 both say "An implementation may set limits on the maximum depth of nesting". However, the ECMA-404 specification doesn't contain any limit on how deeply nested JSON structures can be.
- mantap 7y agoNot stating a limit doesn't mean there's NO limit. It means the limit is implementation defined. If the way you interpret a spec implies a requirement of infinite memory or infinite stack, then the way you interpret it is wrong.
- EugeneOZ 7y agoReading "MUST" when specification says "may" is also wrong.
- ddlatham 7y agoTo be fair, that text was added after the above comment was posted. https://github.com/lovasoa/bad_json_parsers/commit/16f6ed75b4edcac4998878229783eadaa7212f18 https://github.com/lovasoa/bad_json_parsers/commit/16f6ed75b...
- mike_hock 7y agoThe commenter would do well to RTFA before slinging complaints that have already been addressed in the article. The article is cool-headed and factual, so don't get hung up on the repo name.
- ddlatham 7y agoTo be fair, the article was updated to address those complaints after the comment you're responding to was made. https://github.com/lovasoa/bad_json_parsers/commit/16f6ed75b4edcac4998878229783eadaa7212f18 https://github.com/lovasoa/bad_json_parsers/commit/16f6ed75b...
- nabla9 7y agoI just wrote standard conforming JSON parser: cat standard-still-bad.sh #!/bin/sh echo "ERROR: JSON parser implementation limit." 1>&2 exit 1 # terminate and indicate error --- https://tools.ietf.org/html/rfc7159#section-9 https://tools.ietf.org/html/rfc7159#section-9 > An implementation may set limits on the size of texts that it accepts. An implementation may set limits on the maximum depth of nesting. An implementation may set limits on the range and precision of numbers. An implementation may set limits on the length and character contents of strings.
- otterley 7y ago"Conforming" is not identical to "useful." A car that doesn't start is still a car.
- bspammer 7y agoThe author has now added this as an aside: https://github.com/lovasoa/bad_json_parsers/commit/16f6ed75b4edcac4998878229783eadaa7212f18 https://github.com/lovasoa/bad_json_parsers/commit/16f6ed75b...
- blablabla123 7y agoIt is pointless, also nobody ever cares much how well a text editor can handle obscure text files (file too large, broken encoding, ....) What matters is how it handles well-formatted files, the others aren't worth opening in an editor anyways :)
- jrockway 7y agoI don't like it when programs use physical limits to trigger physical errors based on the input. "JSON nesting depth of 23456 levels exceeds maximum memory of 23456MB" is a very different thing from being kill -9'd by the OOM killer (after of course it gets your ssh session). If you decide your limit in advance and bail out of parsing, that's one thing. If you just use up all the stack and cross your fingers, that's a pain.
- thechao 7y agoI know this is too late to be useful, but it only takes 1b per nesting level to track the nesting depth in O(1) time, or just a 64b integer to track the nesting depth in O(n) time. I wonder if there a sublinear algorithm? At best to me, right now, it could made to look like bitstream compression.