11 ms·
Comments in JSON
- varikin 13y agoThis sounds great until some parser uses the comment definition instead of the value. Is it defined in the spec that parsers need to use the last defined value for a key?
- dak1 13y agoSince the order of an object's keys is not guaranteed, it seems like even if a parser respected the last-defined rule, you could still potentially end up with the wrong field last.
- deleted 13y ago[deleted]
- ygra 13y agoNot really defined, but since an object is defined as an unordered collection of key/value pairs, a conforming parser could probably shuffle the pairs before parsing them.
- treerex 13y agoI suppose it could, but the point of the object being defined as an unordered collection is because the most straight-forward way of implementing this is through a hash table, where the order of the keys cannot be guaranteed without additional work. I'm sure they didn't consider a parser randomly permuting the lexical order of the pairs as something a sane person would do.
- ygra 13y agoGranted, hosay123's comment [982] is much more valid, though. [982] https://news.ycombinator.com/item?id=6147084 https://news.ycombinator.com/item?id=6147084
- IanCal 13y agoWell it could perfectly sensibly do this: if not key in hash: hash[key] = value That's a sensible approach, valid as per the spec. > I'm sure they didn't consider a parser randomly permuting the lexical order of the pairs as something a sane person would do. It could sort the keys, in which case the order is no longer guaranteed (again this doesn't seem insane). The proposal is to rely on undefined behaviour for comments. I'm amazed we're still talking about this.
- rwmj 13y agoAbout as defined as anything else in JSON, eg. the range of integers.
- masklinn 13y agoActually, duplicate keys is very specifically recommended against in the RFC, and left entirely unspecified.
- davidradcliffe 13y agoNeat trick! Not sure I'd trust it, and might be confusing for anyone reading who didn't know this.
- nonchalance 13y agoThe JSON RFC (http://www.ietf.org/rfc/rfc4627.txt?number=4627 http://www.ietf.org/rfc/rfc4627.txt?number=4627) says The names within an object SHOULD be unique. SHOULD is defined (http://www.ietf.org/rfc/rfc2119 http://www.ietf.org/rfc/rfc2119) as 3. SHOULD This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course. Salient point is that you would need to ensure that you are only using JSON parsers that tolerate duplicate names (and use the last value)
- IanCal 13y ago> Salient point is that you would need to ensure that you are only using JSON parsers that tolerate duplicate names (and use the last value) To drive this home a bit more forcefully, it requires knowing the behaviour of your parser where it is marked as "undefined" in the spec. If that isn't enough to stop you, DON'T USE JSON. A patch level change in a library could break your code in a non-obvious way and it would be your fault. If you want comments, DON'T USE JSON, JSON DOESN'T HAVE THEM.
- bzbarsky 13y agoNote that if your parser is the ES-standard JSON.parse, then the behavior here is in fact defined by ES5 section 15.12.2, even with duplicate names.
- juandopazo 13y agoAnd the big point here is that the members of the RFC group were considering breaking the EcmaScript standard and change it to MUST which would break existing programs and the "workaround" in the article.
- tonyg 13y agoI wish they had! I wonder why they didn't? JSON is already a subset; limiting it to non-duplicated keys would just tighten it a little.
- hosay123 13y agoThis would completely break any event driven (streaming) parser.
- the_gipsy 13y agoOr a parser that simply discards existing keys.
- IanCal 13y agoWhich, importantly, would be perfectly fine according to the spec (as I understand it).
- masklinn 13y agoIndeed, the spec states that keys SHOULD be unique (with RFC 2119 meaning) and leaves behavior unspecified in case of duplicate key.
- IanCal 13y agoMy favourite example of dealing with undefined behaviour is this: In practice, many C implementations recognize, for example, #pragma once as a rough equivalent of #include guards — but GCC 1.17, upon finding a #pragma directive, would instead attempt to launch commonly distributed Unix games such as NetHack and Rogue, or start Emacs running a simulation of the Towers of Hanoi.[7] Source: http://en.wikipedia.org/wiki/Undefined_behavior http://en.wikipedia.org/wiki/Undefined_behavior
- jgeerts 13y agoIt's overwriting existing keys, which is fine imo. When I use a map in any language and put a new value with a new key, expected behavior is that the previous key is overwritten.
- basicallydan 13y agoThis is a nice trick, but probably only should be used in systems where the set people touching the code is a limited, rarely-changing set of people and anything using the JSON is strictly going to treat the last defined value as the value to use. Dragons lurk elsewhere!
- LinaLauneBaer 13y agoThere is a interview with the inventor of JSON somewhere. In that interview he explained why he did not allow comments in JSON like in XML. He said - if I remember correctly - that it was intentional to not have comments in JSON. The reason way that comments could be misused to add additional information for a parser. For example in XML you could use comments and a special parser could use these comments to create code while parsing. He did not want that. He wanted every JSON parser to be a JSON parser and nothing more. If you wanted to have comments in JSON he said that you could simply make the comments inline and have a convention for the keys which are comments for example every key ending with _comment could have a value which is then seen as a comment by the application but not by the parser.
- jaredmcateer 13y agoYes the JSON spec was designed with interoperability in mind, I don't believe Crockford claims to have invented JSON, merely discovered it. That said if you want your Static JSON objects to have comments, just pipe the JSON object through a minifier to strip comments before parsing.
- jerf 13y agoHe both invented and discovered it. Yes, the object literal syntax existed, but he also carefully (and IMHO correctly) specified a strict subset as well, for these interoperability reasons. For instance, Javascript is happy with {a: 1}, but that is not legal JSON. It's a very well done standard.
- ryanpetrich 13y agoJSON is not actually a strict subset. Certain characters when left unescaped in a JSON string make for invalid JavaScript: http://timelessrepo.com/json-isnt-a-javascript-subset http://timelessrepo.com/json-isnt-a-javascript-subset
- jerf 13y agoIndeed, and I apologize for my ambiguity, as you are correct. By "strict subset" what I meant was a subset that attempts to reduce options, so that legality and illegality is easier to discern. That is, where Javascript accepts apostrophe and double-quote to delimit strings, JSON only accepts double-quotes, thus, "stricter" than real Javascript. You are of course correct that JSON turns out not to quite be a strict subset in the set theory sense of "strict subset", though obviously that's a bug in the spec rather than a deliberate design decision.
- avolcano 13y agoCan we all just agree, as a community, to add comment support to our JSON parsers? Hell, I'd do a PR on V8 if I knew C++. It's ridiculous that I can't document notes on dependencies in my NPM package.json, or add a little reminder to my Sublime Text configuration as to why I set some value, because we're using JSON parsers that can't handle the concept of ignoring a line with a couple slashes prefixing it. IMO - either we add comments to JSON, or we stop using it for hand-edited configuration.
- deleted 13y ago[deleted]
- buro9 13y agoTOML is nice for config files: https://github.com/mojombo/toml https://github.com/mojombo/toml
- phpnode 13y agoCrockford's rationale for not supporting comments is that people use them to add meta data to the object (e.g. type annotations) which makes it hard to consume with different parsers.
- avolcano 13y agoTrusting the community to do the right thing is better than handicapping your users. Regardless, of course, people add metadata to JSON already - there's zero reason you can't "_type": "int". It's a completely arbitrary reason.
- phpnode 13y agoright - but that is valid syntax! any json parser can understand that, and that's what he recommends doing instead. But if you're doing this in comments, you end up writing your own mini language to describe your annotations, and nothing else knows how to parse it. that should clearly be avoided.
- jasonlotito 13y agoMy first thought in seeing this was that objects aren't guaranteed to maintain order: "An object is an unordered set of name/value pairs" - http://www.json.org http://www.json.org
- _ZeD_ 13y agowhile this is true, I think it's irrelevant: the "trick" is about "abusing" * the fact parser work from top to bottom of the text AND * the fact that assigning the same key many times with different values update the key with the last value your quote regards the order in witch the different keys are saved.
- masklinn 13y agoBoth are only "correct" for specific implementation, this is not specified behavior (and duplicate keys is strongly recommended against by the key)
- _ZeD_ 13y agoabsolutely. This is nothing more than a clever trick, but I would never rely on it. Honestly, tough, I think all major JSON parser behave following the two assumption.
- masklinn 13y agostreaming parsers can't follow the assumption short of becoming useless. They're either going to send only the first instance or going to send two different events.
- jfoutz 13y agoThere is an intrinsic order in the text though. it's up to the parser to keep clobbering a value every time a new value comes in for a given key. This seems like a bad idea. It seems heavily reliant on edge case behavior. But hey, might work well for the original author.
- kalleboo 13y agoNote that these comments would disappear the second you use a JSON-aware tool to manipulate one of these files.
- mtkd 13y agoYou hope it is the comment dupe that disappears and not the field you want.
- JulianMorrison 13y agoThis definitely qualifies for a Zen style thwack over the head with a stick and a reprimand of "stop being clever!"
- NathanKP 13y agoThis hack, while nice, is still just a work around. I highly recommend that if you can, in as many places as possible use YAML instead of JSON. JSON works great for on the fly communication with frontends that are running JavaScript, or for communication between JavaScript processes like Node.js servers. But for configuration files and other things that need comments YAML is many times better, both for it's clean, Markdown reminiscent structure, and its native comment support. Node.js has a great module called js-yaml (https://github.com/nodeca/js-yaml https://github.com/nodeca/js-yaml) which automatically registers handlers for .yml and .yaml files, allowing you to require them in your Node.js code just like you can with JSON files. It also comes with a YAML parser for the browser side of things, so if you want you could even communicate YAML directly from the server to the client side, although frankly I don't see much advantage to sending YAML over the wire instead of JSON. (And as others have mentioned below untrusted YAML sources could insert malicious objects in YAML, so I wouldn't recommend this technique.) You can even use YAML for your package.json in a Node program: (https://npmjs.org/package/npm-yaml https://npmjs.org/package/npm-yaml)
- homakov 13y agoThis hack, while nice, is still just a work around. I highly recommend that if you can, in as many places as possible use YAML instead of JSON. Rails RCE, sup
- NathanKP 13y agoI've actually never developed anything serious in Rails. I just don't like the framework, and the performance of Rails leaves a lot to be desired in my opinion. I'm a 100% Node.js convert these days. But I do like the Rails convention of using YAML format and have adopted that in my own code as much as possible.
- IanCal 13y agoI think he's referring to the rails YAML exploit [0] because you can use yaml to create objects, like this: --- !ruby/hash:ActionDispatch::Routing::RouteSet::NamedRouteCollection 'foo; eval(eval(puts '=== hello there'.inspect);': !ruby/object:OpenStruct table: :defaults: {} Allowing people to run arbitrary code on rails servers. [0] http://rubysource.com/anatomy-of-an-exploit-an-in-depth-look-at-the-rails-yaml-vulnerability/ http://rubysource.com/anatomy-of-an-exploit-an-in-depth-look...
- asnyder 13y agoYou should use standard JS comments and process them out. Douglas Crockford's offical answer on comments, https://plus.google.com/118095276221607585885/posts/RK8qyGVaGSr https://plus.google.com/118095276221607585885/posts/RK8qyGVa.... Essentially just process them out beforehand with something like jsmin, pretty straightforward.
- jmcdonald-ut 13y agoI'm sure there are counter points to what I'm about to bring up, but three observations: 1. In my experience JSON is frequently output programmatically, and taken in programmatically. Comments are not useful in these cases. 2. The only time comments could be perceived as useful then would be when parsing JSON by eye or hand. However, it is not difficult to parse JSON and understand it unless the keys have used obfuscated names. If key naming is obfuscated, comments aren't really the correct solution. 3. "An object is an unordered set of name/value pairs", as mentioned by jasonlotito and others earlier. There is no guarantee that a JSON parser will give you the right value if there are two of the same keys in the same scope.
- masklinn 13y ago> There is no guarantee that a JSON parser will give you the right value if there are two of the same keys in the same scope. In fact, reading the RFC: > The names within an object SHOULD be unique. I'm pretty sure an implementation could refuse to parse the form altogether.
- rpledge 13y agoSHOULD is a horrible word to put in any spec if it doesn't specify what the result will be if that recommendation is violated
- IanCal 13y agoSHOULD is defined in RFC 2119 as 3. SHOULD This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course. The consequences are undefined, I feel, for a reason. You can't put them all down on paper, it depends on what all the parsers do. The parsers can accept or reject things with duplicate keys, or they can play a nice little ditty through the speakers. All it means is a parser isn't required to reject JSON with multiple keys. It can, however, do whatever the fuck it wants with them. If the wording was precise, then it should be a MUST. SHOULD indicates a terrible world of unknown consequences.
- jgeerts 13y agoIt is a 'hack' as discussed in the article and I will probably never use it. JSON should be either self explanatory or documented, I don't see any reason why you would add this unnecessary clutter to these messages. It is already hard to read as is and it's making it worse to read and confusing, if some big service would start using this, you would have to know about this 'hack' otherwise he would have to look up what the hell is going on. Also, this is the same information for each call and thus redundant, makes your messages larger when an advantage of JSON is that it's generally a small message.
- kgabis 13y agoWell, here we go: https://github.com/kgabis/parson/issues/7 https://github.com/kgabis/parson/issues/7
- sktrdie 13y agoThis is a horrible hack. You should use JSON-LD [1] to describe the fields of your JSON. It's a W3C standard! Also, it's not defined in the JSON standard in which order an implementation needs to parse the JSON fields/keys. So you could end up with potentially wrong results! 1. http://json-ld.org/ http://json-ld.org/
- zemo 13y agoif I ever saw this in a project, I would remove those comments in a heartbeat. The behavior here is specific to the json parser. JavaScript is not the entirety of programming. It does break the json parser in the Go standard library, in a totally nonobvious way: http://play.golang.org/p/BsDd47vWna http://play.golang.org/p/BsDd47vWna I would be surprised if it doesn't break many parsers, especially json parsers in static languages. If you want that sort of behavior, don't use json.
- knodi 13y agoThis is a recipe for disaster.
- rcarmo 13y agoI do something else that is a lot more readable: { "#": "this is a comment for the next line", "url": "http://foo.bar" } Simple.
- IanCal 13y agoHopefully you don't use the same key multiple times, as that's not guaranteed to work in different parsers.
- peterkelly 13y ago> Believe it or not, it turns out JSON parsers work the same way Please don't do this. There's almost certainly some parsers out there currently that don't work like this, and if not, there likely will be one day.
- nrivadeneira 13y agoTerrible spec-violating hack aside, the idea of the author soliciting upvotes on StackOverflow doesn't sit well with me. I'd hate for SO solutions to become diluted by answers from users who are 'marketing' for upvotes.
- kstenerud 13y agoInstead of using tricks that rely on parser implementation behaviors, why not just put an actual comment field in the object? { "myvalue_comment": "This is a comment", "myvalue": 42 }
- MatthewPhillips 13y agoThat example is fine, but you wouldn't want a long comment getting loaded into memory because the parser doesn't know any better.
- kstenerud 13y agoWhy not? It's just a configuration file.
- dnautics 13y agofor that matter, just do: { "comment":"this is a comment"; "value": 45; "comment":"this is also a comment"; "value2": 64; "comment":"we like overloading the comment field"; "stringval":"but these stay the same"; }
- IanCal 13y agoThen the parser might fail, and rightly so. A comment lower down shows it failing in a simple parser in go: https://news.ycombinator.com/item?id=6147478 https://news.ycombinator.com/item?id=6147478 Keys SHOULD be unique.
- JOnAgain 13y agoThis, to me, looks like an example of relying on a nondeterministic implementation. To my knowledge, the standard doesn't prescribe that parsers take the second/last of a duplicate key. As a result, this is relying on implementation-specific choices which can lead to a terrible upgrade process. Switch to a different JSON parser, does it still work? probably. but I wouldn't bet that much. If I were implementing a JSON parser, might I throw an error on a duplicate key? maybe. Maybe I would just print a warning? If I were every going to give someone advice it would be to never do this.
- 8ig8 13y agoThat seems pretty fragile.
- M4rkH 13y agoA common practice in config files is to comment out whole sections e.g. optional proxy server settings. This sort of multi-line comment is not addressed by this hack
- adamtj 13y agoThis is misguided. You don't need comments in a JSON config file. Why? Because you don't use JSON for config files that need comments. JSON is like duc(k|t) tape. It's really easy to stick two things together with it. That doesn't mean you always should. It's the simple thing that gets the job done so you can focus on what matters. One shouldn't pick JSON for your config files and then hold it up as good design. "Look at me, I'm daring and _not using XML_!" Using JSON is crap design, but good engineering means sometimes picking something crappy and not wasting effort on things that don't matter in the end. If your configuration files become both complicated and important enough that you need comments, then you should stop using JSON. If your duck tape job starts needing additional reinforcement, then you should probably just get rid of the duct tape and do it right. If one of your requirements is a sufficiently trendy yet commentable config language, look into YAML. Also, gaffer tape. The white kind is easier to write on.
- tieTYT 13y agoYeah maybe you don't use JSON for config files that need comments, but that's because there's no documented way of how to put comments in JSON. The article solved the problem. Actually, I'm 100% playing the devils advocate here. I'll even flip-flop to prove it. Regarding the article, I doubt that every JSON parser will let this slide. To me that's an even better reason to avoid this practice.
- IanCal 13y ago> Regarding the article, I doubt that every JSON parser will let this slide. To me that's an even better reason to avoid this practice. If someone uses undefined behaviour in config files for the sake of storing a comment, I reserve the right to hunt them down if I have to maintain their code.
- glhaynes 13y agoIf crap design like JSON is the right engineering choice sometimes (and I agree that it is), that seems like an argument that adding comments in this crappy way may sometimes be the right engineering choice.
- lttlrck 13y agoNice hack but fails JSHint. [1] http://jshint.com/ http://jshint.com/
- julius 13y agoFunny story. JSLint[1] does not approve of this technique. I asked Crockford to implement the duplicate check in April 2009 via email. 20 minutes later, out of nowhere, he was done implementing that check and wrote back "Please try it now." This guy is fast. Especially nice considering we do not know each other at all. [1] http://www.jslint.com/ http://www.jslint.com/ - JS checking tool from the inventor of JSON
- WayneDB 13y agoI sent him an email once asking for the same JSLint license that he gave to IBM (you know, the one without the "do not use this for evil" clause.) He responded that he was getting annoyed by everybody asking for this, so it was going to cost me $100K to obtain such a license. I responded that I only asked for that license in order to annoy him (and thanks for the confirmation that it worked), because his immature license clause is annoying everybody else.
- opminion 13y agoJSON has comments already. It just requires you to decide what the comment marker is.
- CanSpice 13y agoGiven the RFC says "The names within an object SHOULD be unique", there's nothing stopping me from writing a parser that takes the first name/value pair and throwing all the others on the floor. Or even better, picks a random name/value pair when the same name appears. Both of these behaviours are allowed by the RFC, and would break this hack. Putting comments into JSON in this way is a hack and shouldn't be used by anybody who has any interest in writing maintainable software. Relying on ambiguities in an RFC and someone saying "JSON parsers work the same way" is a good way to end up with a really obscure bug in the future.
- bzbarsky 13y agoAssuming you mean RFC 4627, you're quoting the restrictions on what character streams can be called "JSON". The "should" means that if your names are not unique you can still call it "JSON", but you should think twice about it. The parsing behavior for JSON is not defined at all in RFC 4627, actually. Browsers (and Node, since it's using a browser js engine) use the parsing specification in ECMA-262 edition 5 section 15.12.2. Note that ES5 section 15.12 in general is much stricter than RFC 4627, as it explicitly points out if you read it.
- serichsen 13y agoAt least in ECMA-262 5, Ch. 15.12.2, there is a NOTE: "In the case where there are duplicate name Strings within an object, lexically preceding values for the same key shall be overwritten." It still does not feel right.
- znmeb 13y agoThis is a celebration of programmers' ability to generate unmaintainable code by exploiting implementation dependencies. People get fired for pulling this horseshit every day!
- quantumpotato_ 13y agoI thought JSON is mainly for machine to machine consumption.. who reads comments?
- wickedlogic 13y agoDon't use them, there is no such thing. Make your comments first class citizens in the data.