6 ms·
The Fixing-JSON Conversation
- niftich 10y agoHe summarized the most upvoted posts from the last thread [1] really well. [1] https://news.ycombinator.com/item?id=12328088 https://news.ycombinator.com/item?id=12328088 Regarding datetimes, it's worth pointing out the conversation that TOML had about it. It's a pretty long read [2][3][4][5] with lots of points raised for and against, but it also shows some of the process of how consensus was eventually forged: through trial-and-error, some enlightening realizations, expert opinions, and a willingness to leave some aspects of the behavior it up to parser, to avoid requiring all other languages to reimplement half of Java 8 Time. [2] https://github.com/toml-lang/toml/pull/414 https://github.com/toml-lang/toml/pull/414 [3] https://github.com/toml-lang/toml/pull/362 https://github.com/toml-lang/toml/pull/362 [4] https://github.com/toml-lang/toml/issues/412 https://github.com/toml-lang/toml/issues/412 [5] https://github.com/toml-lang/toml/issues/263 https://github.com/toml-lang/toml/issues/263 The salient point being that RFC 3339 does not in truth describe exactly one datatype, so you can't just reference the spec and hope everyone reads it the same way. EDIT: Specifically, RFC 3339 says: "Date and time expressions indicate an instant in time. Description of time periods, or intervals, is not covered here.", but then goes on to define [6] a number of different syntaxes in ABNF, to indicate the subsets of ISO 8601 that "SHOULD be used in new protocols on the Internet." It essentially never defines what a 'valid' RFC 3339 object looks like, it doesn't explicitly say which ones are considered complete representations, so it's not clear if, say, '2016' is a valid RFC 3339 object... but the ones towards the bottom contain more than one discrete term, and can be presumed to be 'complete' representations. These are: [A] partial-time: HH:MM:SS(.SSS) [B] full-date: YYYY-MM-DD [C] full-time: 'partial-time' +/- offsetFromUTC(HH:MM) [D] date-time: 'full-date' "T" 'full-time' Out of these, [D] is clearly a timestamp of an absolute instant in time, but the rest are debatable. [6] https://tools.ietf.org/html/rfc3339#section-5.6 https://tools.ietf.org/html/rfc3339#section-5.6
- outsidetheparty 10y ago> He summarized the most upvoted posts from the last thread [1] really well. I feel like he glossed right past the objections to the biggest and (to my mind) most destructive proposed change, the commas-to-whitespace thing; in fact doubles down on it (let's just declare that commas are whitespace! That surely won't confuse anyone!)
- thymelord 10y agoI've written a few JSON parsers over the years that treat commas as whitespace. The grammar is simpler and the parser is faster as result. As long as one always emits standards-compliant JSON there's no problem. Had the JSON standard supported ECMAScript array holes [1,,,2,,3] this grammar shortcut would not have been possible. But luckily that's not the case.
- outsidetheparty 10y agoWell, that's the thing, though: sure, for machine parsing it really doesn't matter what you use as a delimiter. But the only reason to get rid of commas is to eliminate the trailing comma problem, which only occurs when hand-editing JSON. Replacing that with whitespace, or worse both whitespace and commas, would be a lot more prone to hand-editing errors, I think, than would the much less drastic change of allowing trailing commas. Or better still of just leaving JSON as is and letting people use a more robust protocol, if that fits their needs, or pre-parsing whatever special snowflake variations they want into standards-compliant JSON. I'm more or less in agreement with the commenter on his site who said > this entire proposal pretty much comes down to "I like JSON, but need more and am too lazy to write the extra 3 line wrapper to process type 'x'." I'd say no thanks. https://www.tbray.org/ongoing/When/201x/2016/08/20/Fixing-JSON#c1471881712.859312 https://www.tbray.org/ongoing/When/201x/2016/08/20/Fixing-JS...
- thymelord 10y agoTimestamps are so complicated once you factor in timezones and daylight savings that it doesn't belong in JSON. Time zones are not static. They can change from country to country, or even states within countries. Ditto for when daylight savings is enacted during the year - even changing over the years. There is no rhyme or reason to any of this. The data for this has to be stored in tables and time zone meanings can change retroactively. The only reliable time stamp is UTC without leap seconds. (Speaking of leap seconds, who thought seconds going from 0 - 60 rather than 0 - 59 was a good idea?) Accurate time is one of the most difficult things to model in computer science.
- realharo 10y agoTime is actually quite simple if you have a good mental model of what you're trying to represent and don't try to mix different concepts into a single value. This talk explains it VERY nicely: https://www.youtube.com/watch?v=2rnIHsqABfM https://www.youtube.com/watch?v=2rnIHsqABfM Basically just decide whether you're trying to store an absolute time (a timestamp will do) or a civil time (year, month, day, etc.) and treat them as two separate data types. (If you just use "civil time + offset from UTC" like RFC 3339 does, then you can convert it to an absolute time, but you can convert only that one specific value using that offset, and not any other - i.e. that offset is not a substitute for an actual timezone identifier.)
- coldtea 10y ago>Time is actually quite simple if you have a good mental model of what you're trying to represent and don't try to mix different concepts into a single value. Not even close. http://www.creativedeletion.com/2015/01/28/falsehoods-programmers-date-time-zones.html http://www.creativedeletion.com/2015/01/28/falsehoods-progra...
- realharo 10y agoIf you watch the video I linked above, he explicitly mentions that 99% of the time you should not be dealing with any of those things manually - that if you find yourself working with offsets and DST values, it's a sign that you're most likely doing something wrong.
- sanqui 10y agoPersonally, I would really like to see integer object keys (as opposed to only string keys). For simple numeric transformations, strings feel really heavy and require annoying conversion in languages. E.g. {"10": 60, "42": 2}.
- niftich 10y agoThe flipside is that an integer-keyed map is similar in meaning to an array, which associates by virtue of placement, an integer with the value sitting at that index. While it's possible to spec it to forbid this interpretation, Lua has made this interpretation a language feature, and it'd become impossible to construct an unambigous parser/printer in Lua for this new format.
- thaumasiotes 10y ago> it'd become impossible to construct an unambigous parser/printer in Lua for this new format How so? JSON is just text; you can parse it however you like. For example, I can write a parser that produces the byte sequence "It checks out." for any valid JSON input. A Lua JSON parser that represents objects with integer keys as maps with string keys, and a Lua JSON parser that represents objects with integer keys as sparse arrays, are both unambiguous and both correct (as to a hypothetical JSON which allowed integer keys). JSON is a format, not a Lua structure. Even if your JSON data is { 10: 10, "10": "ten" } there's no problem being able to write an unambiguous parser. Define what your parser does in that situation, and it's unambiguous.
- velox_io 10y agoJSON (for the most part) is a nice format to work with, aside from loosely defined datetimes as mention. The two areas where I believe the format can greatly be improved; 1# having a standard to define the structure (sometimes schemas can be handy!); 2# a stranded binary format, yes right now with have UBJSON (which doesn't have a date format, this is worse in binary) and BSON (which contains some MongoDB specific stuff). I'm not saying they don't have their place, but.. Protocol Buffers are more akeen to .net or Java serialization, in the they're quite fragile if used with different versions and/ or with different vendors.
- thymelord 10y agoMessagePack is better supported than UBJSON and has far more implementations in almost every language. http://msgpack.org/index.html http://msgpack.org/index.html JSON Schema draft 4 is the defacto schema standard for JSON. http://json-schema.org/latest/json-schema-core.html http://json-schema.org/latest/json-schema-core.html Having used both XML schema and this one, I much prefer using JSON schema.
- Zardoz84 10y agoSDLang !!!! : https://sdlang.org/ https://sdlang.org/ Full example : https://github.com/Abscissa/SDLang-D/wiki/Language-Guide#example-sdl-file https://github.com/Abscissa/SDLang-D/wiki/Language-Guide#exa... Examples: Creating a Tree plants { trees { deciduous { elm oak } } } Creating a Matrix myMatrix { 4 2 5 2 8 2 4 2 1 } A Tree of Nodes with Values and Attributes folder "myFiles" color="yellow" protection=on { folder "my images" { file "myHouse.jpg" color=true date=2005/11/05 file "myCar.jpg" color=false date=2002/01/05 } folder "my documents" { document "resume.pdf" } } Date and Date/Time Literals (and comments!) # create a tag called "date" with a date value of Dec 5, 2005 date 2005/12/05 # a date time literal without a timezone here 2005/12/05 14:12:23.345 # a date time literal with a timezone in_japan 2005/12/05 14:12:23.345-JST
- _pmf_ 10y agoThat's beautiful!
- masklinn 10y agoFor simple data exchange? Way over-engineered is what it is, TFA specifically noted that they want to stay in the overall conceptual complexity of JSON: > “Just use X” […] Nah, most of them are way, way richer than JSON, often with fully-worked-out type systems and Conceptual Tutorials and so on.
- RoryH 10y agoI think now is a good time to re-quote the man himself... Douglas Crockford.. https://plus.google.com/+DouglasCrockfordEsq/posts/RK8qyGVaGSr https://plus.google.com/+DouglasCrockfordEsq/posts/RK8qyGVaG... I removed comments from JSON because I saw people were using them to hold parsing directives, a practice which would have destroyed interoperability. I know that the lack of comments makes some people sad, but it shouldn't. Suppose you are using JSON to keep configuration files, which you would like to annotate. Go ahead and insert all the comments you like. Then pipe it through JSMin before handing it to your JSON parser.
- realharo 10y agoThe second part basically says "don't use JSON for config files, use some unspecified JSON superset".
- coldtea 10y ago>I removed comments from JSON because I saw people were using them to hold parsing directives, a practice which would have destroyed interoperability. I call BS. If people want to have custom parsing directives they can send them out of band, encode them in the filename, or whatever. But they don't. And I've not seen this happening with most other serialisation formats either, so why would JSON be a particular target? After all it's value comes from being trivially parsable across languages, and that would be killed by custom parsing directives. Those wanting those would also implement their own parsers etc. Addition: Besides, reading comments to decide how to parse, implies either "comments on top of the file" or a "2 stage parsing". With 2 stage parsing, you could implement comments and whetever else yourself, even in pure JSON anyway. As for "comments on top of the file", well, just disallow them (only allow comments after the first JSON object starts), and no issue with "parsing directives" anymore...
- willvarfar 10y agoIf // and /* are used as comments, then most of this new extended-JSON will still be valid Javascript. If # is used as comments, then this breaks documents being Javascript. The post says that "don't eval() JSON ever", but that's like Crockford leaving out comments originally in order to stop them being abused as processor directives...
- daenney 10y agoLike the post says, JSON is already not guaranteed to be valid JS so this isn't really a problem. The fact that 99% of the time it works to just eval it is great and granted, the "feature" that triggers this is incompatibility is a bit obscure. But if you just do the right thing from the start you'll never have a thing to worry about in the first place, # comments or not.
- abofh 10y agoThis is a bit of a strawman argument though. JSON is rarely used in a generic form - but instead as a format between endpoints (servers, clients, whathaveyou). When you control the server, you control the API - so random UTF-8 magic crap? Wasn't valid in the first place. Just because you can express things that aren't javascript with it doesn't mean that ever actually turns up in practice -- more importantly 95% of the time, you control both the producer and consumer of JSON because it's consumed internally anyhow - at which point - you're not _going_ to write illegal json to yourself. This is a true statement, but virtually nothing in practice cares.
- josephg 10y agoI'm jealous. I've had it show up in production. I think we were using script tags with sanitized JSON to re-hydrate an isomorphically rendered page. Of course, a user entered one of the valid json, invalid JS characters in a field, which made its way into our database. Once there we started having weird errors show up between the frontend and client. That little gem took us days to track down. JSON should be either a subset of javascript, or obviously not a subset of javascript. 99% compatible systems are dangerous landmines. That you haven't been blown up yet doesn't make it ok.
- fiatjaf 10y ago“Just use X” · For values of X including Hjson, Amazon Ion, edn, Transit, YAML, and TOML. ¶ Nah, most of them are way, way richer than JSON, often with fully-worked-out type systems and Conceptual Tutorials and so on. What? MOST OF THEM? YAML is not, Hjson is not, TOML is not.
- gengkev 10y agoYAML... really? Looking at the examples in the Wikipedia article (https://en.wikipedia.org/wiki/YAML https://en.wikipedia.org/wiki/YAML) gives me a headache. Fortunately, most actual YAML files I've seen are not that complicated.
- fiatjaf 10y agohttps://en.wikipedia.org/wiki/JSON#YAML_sample https://en.wikipedia.org/wiki/JSON#YAML_sample
- fiatjaf 10y agoThis guy is an idiot anyway. There's no way to "fix JSON". All you can do is create a new language, it doesn't matter if you call it JSON 2.0, it will still be incompatible with all the JSON parsers of today. I don't get why he is so mad at people suggesting him to use one of the JSON supersets that exist today.
- wtbob 10y agoI think the idea about removing whitespace is kinda hilarious, because that would mean one could write: ["a" "b" "c"] or: {"a": "foo" "b": "bar" "c": "baz"} The advantage over: (a b c) or: (a foo b bar c baz) or: ((a foo) (b bar) (c baz)) or: ((a . foo) (b . bar) (c . baz)) seems … non-existent.
- tarnacious_ 10y agoI don't think # or // for comments is a very good idea as it would also make newline characters significant. I find it useful to be able store a JSON object per-line.