11 ms·

Some other fun facts about JSON, its mainstream implementations and using it reliably: 1. json.dump(s) in Python by default emits non-standards-compliant JSON,

by q3k 5y ago

Some other fun facts about JSON, its mainstream implementations and using it reliably: 1. json.dump(s) in Python by default emits non-standards-compliant JSON, ie. will happily serialize NaN/Inf/-Inf. You want to set allow_nan=False to be compliant. Otherwise this _will_ annoy someone who has to consume your shoddy pseudo-JSON from a standards-compliant library. 2. JSON allows for duplicate/repeated keys, and allows for the parser to basically do anything when that happens. Do you know how the parser implementation you use handles this? Are you sure there's no differences between that implementation and other implementations used in your system (eg. between execution and validation)? What about other undefined behaviour, like permitted number ranges? 3. Do you pass around user-provided JSON data accross your system? How many JSON nesting levels does your implementation allow? What happens if it's exceeded? What happens if different parts of your processing system have different limits? What about other unspecified limits like serialized size, string length? My general opinion is that it's extremely hard to reliably use JSON as an interchange format reliably when multiple systems and/or parser implementations are involved. It's based on a set of underdefined specifications that leaves critical behaviour undefined, effectively making it impossible to have 100% interoperable implementations. It doesn't help that one of the mainstream implementations (in Python) is just non-compliant by default. I highly encourage any greenfield project to look into well designed and better specified alternatives.

appleflaxen 5y ago
> look into well designed and better specified alternatives From an end user who knows little about these parsing issues, what alternatives are you happy with?
EdwardDiego 5y ago
> How many JSON nesting levels does your implementation allow? What happens if it's exceeded Haha, I've met a few stack overflows in this area.
- tehbeard 5y ago
  While there's a lot of issues with JSON, this one also applies to any other interchange format that supports nesting, including the much beloved XML. Protobuf might also have this, idk if it does any static analysis for infinite depth.
  - q3k 5y ago
    The problem doesn't really exist in Protobuf, as protobuf (de)serialization is performed based on an IDL definition of the message type. Whatever that IDL specifies, a corresponding typed definition and (de)serialization function will be generated for your programming language, and that implementation will ignore any fields that weren't part of the IDL. The (de)serializing code is statically generated ahead of time, and is treated like any other code that operates on potentially nested data structures. What this means is that if your IDL specifies deep nesting (or recursive nesting), then it means your application is expected to handle this “by contract”, and attempts to deserialize will rightfully fail in case of out-of-memory / stack overflow errors. There's no danger of an implementation 'accidentally' deserializing something nested that was passsed from the outside, as anything unknown to the IDL is simply ignored. Finally, there's no XML-like self-references in Protobuf, so it's not possible to have an infinitely deep structure, or a combinatorial explosion like with billion laughs - just a very deeply nested one, and only if allowed in the IDL, and only up to whatever message size limit you're allowing.
    - tehbeard 5y ago
      Thank you for the 2nd + 3rd paragraphs, those were parts of protobufs design I wasn't really aware of from a cursory glance. I'm a little suprised to learn there's no self-reference support in protobuf, as I wouldn't have assumed parsing that would be an issue (as all it really is is a pointer to an existing object in the message to say, put a ref. to it here), though I guess it might be a problem in supporting certain languages.
      - q3k 5y ago
        > I'm a little suprised to learn there's no self-reference support in protobuf, as I wouldn't have assumed parsing that would be an issue (as all it really is is a pointer to an existing object in the message to say, put a ref. to it here), though I guess it might be a problem in supporting certain languages. That's a tradeoff more designs should have, IMO: reduce the feature set as much as possible, but in return make the implementation vastly simpler. :) I assume it's not only about support in programming languages, but also exactly to eliminate the entire class of bugs that stems from back/forward-references in serialized data, and to generally keep the wire format as simple (to parse and to implement a parser for) as possible. The few usecases that could make use of references are not worth the pain inflicted on everyone if they were implemented.
- thechao 5y ago
  I had a little non-recursive JSON parser hanging around. When you have "nested" levels you've really only got two choices: object, or array. That implies that to track nesting, you just need an array of 1b values. In order to shave the yak properly, I built "nesting compressor" that detected runs of array/object and represented them using a 64b RLE; or, it bailed out, and then just used on-the-fly compression with zstd. Obviously, any sort of JSON file that fit on a disk I can afford can be parsed into memory in a tiny fraction of its on-disk representation. I modified `yes` to just stream `[` out; the JSON parser handled it just fine — it takes a while to roll a 64b counter.
kortex 5y ago
> I highly encourage any greenfield project to look into well designed and better specified alternatives. Like what?
- jerf 5y ago
  Part of the problem is that there's at least half-a-dozen high quality answers out of the gate (gRPC, FlatBuffers, Protocol Buffers, XML in some cases, Thrift), and an even-longer long tail after that. It's made harder when four different teams who deeply loathe JSON and independently decide to use something "better" can legitimately use four completely different technologies if they don't communicate with each other.
  - 35fbe7d3d5b9 5y ago
    To your comment above – you can bodge around interop problems with JSON in ways that you cannot with some of these other technologies. I like to joke that I invented ndjson over a decade ago when I accidentally forgot to put things in an array before `json.dumps`, I just wasn't smart enough to call it a standard. But when you do end up with ndjson when you wanted an array of results, or vice versa, JSON makes it easy to munge things to where you need. Compare that to something like protobuf: it's not a self-synchronizing stream, so if you send someone multiple messages without framing them (prefix by length or delimited are popular approaches), they're going to decode a single message that doesn't make much sense on the other end. And they won't be able to fix it at all. So I guess JSON is New Jersey style design[1]. [1]: https://dreamsongs.com/RiseOfWorseIsBetter.html https://dreamsongs.com/RiseOfWorseIsBetter.html
    - kortex 5y ago
      Well, you invented one of the best things since sliced bread! I love NDjson, being able to parse a sequence of {} objects as an array is just frankly more natural. A coworker got some absurd speedup going from some massive json array to ndjson. Honestly if json had as part of its spec line-delimited arrays, and accepting NaN, it'd be close to perfect. Oh and native ints, but that is JS's problem. Well, and a single, canonical spec. And a hard limit (however high) on nesting depth. And some other things. Ok, maybe it's far from perfect.
jerf 5y ago
"My general opinion is that it's extremely hard to reliably use JSON as an interchange format reliably when multiple systems and/or parser implementations are involved." I suspect one of the reasons that JSON has been so successful is precisely this fuzziness, though. Every language can do something a little slightly different and it'll work at first when you send it to somebody else. You get up and off the ground really quickly, and can fix up issues as you go. If you try to specify something with a stronger schema right off the bat, I find a number of problems immediately emerge that tend to slow the process down. It may be foreign to programmers on HN who have embraced a strong static type mindset, or dynamic programmers who have learned the hard way that sometimes you need to be more precise about your types, but there's still a lot of programmers out there who will wonder why you're asking them whether this is an int or a float is relevant. I came in to work this morning to an alert system telling me that a field that a particular system has been sending as an integer for a couple of months now over many thousands of pushes, "number of bytes transferred", is apparently capable of being a float once every several thousand times for some reason. There's a lot of programmers who will send a string, or a null, or maybe a float, or maybe it's always an integer, and deeply don't understand why you care what it's getting serialized as. And that's just an example of some of the issues, not a complete list. Trying to specify with some stronger system moves a lot of these issues up front. (If your organization has internalized that's just how it has to be done, great! I bet you encountered a lot of these bumps on the way, though.) This isn't a celebration of JSON per se... this is really a rather cynical take. I don't know that we need to type everything to the n'th degree in the first meeting, but "why can't we just let our dynamically-typed language send this number as a string sometimes?" is definitely something I've had to discuss. (Now, I don't get a lot of resistance per se, but it's something I have to bring up.) I'm not presenting this as a good thing, but as a theory that JSON's success is actually in large part because of its loosey-gooseyness, and not despite it, regardless of how we may feel about it.
- lifthrasiir 5y ago
  I'm not convinced. There are a lot less JSON implementatations than JSON users, so we should have been possible to guide implementations with a means of proper specification and test suites. Note that the OP is possibly the first ever complete test suite for JSON after 15 full years. It is not like seeding initial implementations (that can serve as models for future implementors) is particularly hard either; Douglas Crockford himself wrote two implementations in C and JavaScript.
35fbe7d3d5b9 5y ago
> I highly encourage any greenfield project to look into well designed and better specified alternatives. By way of recommendation: I reach for protobufs to do data interchange between polyglot systems and have yet to be disappointed. Even if you aren't getting into gRPC, having data interchange backed by codegen and an IDL removes a lot of the risk you get with data interchange.
- avmich 5y ago
  In my experience JSON allowed absence of codegen and superior schema definition capabilities to protobuf, and also nice transformations with parts of jq built into JSON libraries. Try to limit structure complexity to something which can be verified before usage, yes. YMMV.
- theamk 5y ago
  In my experience, protobuf has a minimum project complexity threshold before it starts to make sense. Yes, if both sides of your interchange are systems which have build infra setup, it provides a better experience. But if you need to access data from outside of your usual projects, or from shell, or from random data analysis notebooks, It becomes a major pain. Recent example: we’ve had an orchestrator script which was written in “python with stdlib only” - no build step, download an archive, extract and run. This script had to talk to third-party program which would export protobuf only. This was a major pain as yon can imagine.
makeitdouble 5y ago
It’s a pyramid. At the bottom you have CSV which is popular beyond belief, and has no real specification, with common cases wildly handled differently across libraries. In the middle you have JSON which isn’t 100% interoperable, but goes 98% of the way. And you have XML and protobufs at the top tip, who have strong mechanisms available for interoperability but at an operational cost that rarely justifies the upgrade from JSON. I suspect it will take a lot more that “well designed” and “better specified” to justify moving away from JSON as the default stepup from chaotic CSV like formats.
- heresie-dabord 5y ago
  I would describe it as a _pyramid of bounded viability_ , from the minimally viable to the feature-burdened maximum. CSV excels (ha) as a minimally viable exchange format for data. Combined with awk, grep, sed, bash, and Perl, and some simple SVG or D3 with SVG, the analytical solution is fast, scalable, and automatable. But CSV has limits. The column headers are the schema. Beyond these bounds, we have the other formats. JSON is messier. Its strength is in network/browser encapsulations and operations. As I have seen it used, people insert an array as a kind of schema, and they stay away from complex nesting where scalability starts failing and other difficult tooling must be summoned to compensate (parsing tools such as jq). Beyond JSON's bounds, we have XML and associated tooling. XML is versatile and expressive. XML and JSON can be written simply but both can be abused by programmers who aren't thinking beyond their own cursor. This is a rich set of tooling for data representation. In the end, one of the main advantages of CSV is that it remains a format that brings little tooling baggage ("ecosystem") to the task.
- maple3142 5y ago
  I am not sure if parsing XML is better than parsing JSON. Many languages or libraries' XML parser are dangerous by default. You usually need to manually configure your XML parser to be secure from XML-related attacks. Fortunately, some languages and libraries are going to make XML have a securer defaults, this is a good change. IMO, I think XML shouldn't have include many questionable features from security perspective.
  - 5y ago
lifthrasiir 5y ago
And all these problems trace back to Douglas Crockford. He didn't know how to make a proper serialization format [1] and also an interoperable standard (for the latter, Tim Bray tried very hard to make it slightly better [2]). He just noticed that a (supposed) subset of JavaScript can be easily turned into a serialization format with `eval` and went to publicize it, only noticing the issues later and still pursuring its standardization as is. I hate him. [1] My additional complaints: https://news.ycombinator.com/item?id=24953981 https://news.ycombinator.com/item?id=24953981 [2] https://www.tbray.org/ongoing/When/201x/2014/03/05/RFC7159-JSON https://www.tbray.org/ongoing/When/201x/2014/03/05/RFC7159-J...
- gmac 5y ago
  I was using JSON before it was 'invented', as was basically anyone sending data to the browser in JS format. Holding specific people responsible is pretty absurd.
  - lifthrasiir 5y ago
    JSON before the standardization had an obvious data model and specification: ECMAScript. (I don't think JSON was widely used outside of JavaScript back then.) ECMAScript is particularly strictly defined even compared to other language standards, so it should have been possible to extract the relevant portions of ECMAScript into a proper standard. Crockford didn't. JSON as specified by Crockford was not even a proper subset of ECMAScript until ECMAScript itself retrofitted its syntax.
magicalhippo 5y ago
> My general opinion is that it's extremely hard to reliably use JSON as an interchange format reliably when multiple systems and/or parser implementations are involved. XML is very precisely defined in comparison to JSON. Yet we've had one customer who had a system that couldn't handle XML files with newlines in them at all, and several which sometimes sends ISO 8859-1 (Latin 1) encoded data in some fields of a XML file with encoding="UTF-8" in the header... We of course also have some nice fixed-field integrations, based on customer's specs, where the system suddenly sends multiple mangled characters if any non-ASCII character is present, causing the fields to suddenly not be so fixed anymore... It behaves very much like UTF-8 interpreted as Latin-1, except with something else than Latin-1. Anyway, I've given up trying to be strict at this point. We will have to wash incoming data, it's apparently inevitable.
- spookthesunset 5y ago
  I mean even if it is well defined that doesn’t mean the devs are using the languages native parser library. I’ve encounter at least two projects where the devs rolled their own XML “parser” using regex and “substring” functions. Why? “The xml library was too bloated… much easier to write it ourself”. Suffice to say, they had tons to problems.
throw_m239339 5y ago
Most of your problems aren't problems > 3. Do you pass around user-provided JSON data accross your system? How many JSON nesting levels does your implementation allow? What happens if it's exceeded? What happens if different parts of your processing system have different limits? What about other unspecified limits like serialized size, string length? XML has the same issue, that's why SAX exists, it works the same way with JSON. > 2. JSON allows for duplicate/repeated keys, and allows for the parser to basically do anything when that happens. Do you know how the parser implementation you use handles this? Are you sure there's no differences between that implementation and other implementations used in your system (eg. between execution and validation)? What about other undefined behaviour, like permitted number ranges? A parser should... parse and not interpret data or it isn't a parser. it's a deserializer. Well how many languages allow duplicate keys for maps anyway? this isn't an issue in practice. Basically, the answer to all your problems is to use an evented parser instead of a deserializer.
- anonymoushn 5y ago
  > Basically, the answer to all your problems is to use an evented parser instead of a deserializer. This seems a bit unreasonable? People face the same issue where different pieces of software treat duplicate HTTP headers differently, and usually they haven't written each piece of software they use to parse HTTP headers
- kortex 5y ago
  > this isn't an issue in practice. It absolutely is an issue in practice. If system A handles dupes by accepting the first and ignoring the rest, and system B implements last-key-wins, then that's a potential source of bugs. The system might not fully parse to a map. It may, for example, do string-level modification of json strings. Is that disgusting and wrong? Yes. Have I seen it in prod? Also yes.
  - throw_m239339 5y ago
    > It absolutely is an issue in practice. If system A handles dupes by accepting the first and ignoring the rest, and system B implements last-key-wins, then that's a potential source of bugs. The system might not fully parse to a map. But the system shouldn't be automatically be parsing a "json map" to a map at first place: {"foo":"bar","foo":"baz","foo":"qix","fiz":"buzz"} Shouldn't be deserialized into a map. but a Array<Map<string,string>> like structure. A SAX style parser for JSON can help do that. Thus the issue is the choice of parser indeed. Not JSON.
zelphirkalt 5y ago
Some good points there. And now imagine people wanting to needlessly use YAML for configuration, which adds loads of edge cases on top of that.
stinos 5y ago
You want to set allow_nan=False to be compliant. Otherwise this _will_ annoy someone who has to consume your shoddy pseudo-JSON from a standards-compliant library Funny (well, not really) thing is NaN and Inf are perfectly valid floating point numbers acoording to most (?) standards used on computers. To the point that I don't understand why it was left out of JSON. So unless you're 100% sure you won't encounter these numbers the choice is between not being able to use JSON, or finding hacks around (and using null isn't one of them since you have 3 numbers to represent), or just using non-compliant-yet-often-accepted JSON and possibly annoying someone whos parser doesn't handle it. And for me there have been quite a lot of cases were I just quickly needed something simple to interface between components so when finding out they all support JSON+Nan/Inf then the choice is usually made quickly.
- nomel 5y ago
  > To the point that I don't understand why it was left out of JSON I think you're forgetting the birthplace of JSON. Who deals with the concept of infinity and NaN in the context of web front ends?
  - lifthrasiir 5y ago
    Ranges are pretty common in APIs and both -Infinity and Infinity can naturally arise from one-sided ranges. Since they are absent in JSON, they are frequently replaced with null, ad-hoc sentinel values with uncoded assumptions (e.g. timestamps should be always positive) and missing fields.
  - stinos 5y ago
    I get that, but to go from "oh this won't be very common" to willingly "let's just leave this out" is something else. At least in my mind :) Or was it an oversight?
    - mst 5y ago
      I suspect it was a bet on worse is better. Whether it was a good bet is debatable, but given Crockford's focus on "try and leave out as much as possible" I can certainly see it making sense at the time.
Cloudef 5y ago
Any JSON parser that tries to handle numbers without big number support is broken. This is why I always raise eyebrows if I see json library that doesn't allow me to deal with the number myself by retrieving it as a string.
- JoBrad 5y ago
  Agreed. Even worse, one that corrupts data it doesn’t know how to handle.
nuerow 5y ago
> My general opinion is that it's extremely hard to reliably use JSON as an interchange format reliably when multiple systems and/or parser implementations are involved. None of the points you made were regarding JSON, and we're focused on either hypothetical or very specific implementation problems. Even though it's true that you can experience problems if you pick a broken noncompliant implementation and fail to test, that's hardly a JSON issue. I mean, are we supposed to think that XML has a problem if you pick an implementation that fails to handle multiple entities or outputs broken XML?

by q3k 5y ago

appleflaxen 5y ago

> look into well designed and better specified alternatives From an end user who knows little about these parsing issues, what alternatives are you happy with?

EdwardDiego 5y ago

> How many JSON nesting levels does your implementation allow? What happens if it's exceeded Haha, I've met a few stack overflows in this area.

tehbeard 5y ago
While there's a lot of issues with JSON, this one also applies to any other interchange format that supports nesting, including the much beloved XML. Protobuf might also have this, idk if it does any static analysis for infinite depth.
- q3k 5y ago
  The problem doesn't really exist in Protobuf, as protobuf (de)serialization is performed based on an IDL definition of the message type. Whatever that IDL specifies, a corresponding typed definition and (de)serialization function will be generated for your programming language, and that implementation will ignore any fields that weren't part of the IDL. The (de)serializing code is statically generated ahead of time, and is treated like any other code that operates on potentially nested data structures. What this means is that if your IDL specifies deep nesting (or recursive nesting), then it means your application is expected to handle this “by contract”, and attempts to deserialize will rightfully fail in case of out-of-memory / stack overflow errors. There's no danger of an implementation 'accidentally' deserializing something nested that was passsed from the outside, as anything unknown to the IDL is simply ignored. Finally, there's no XML-like self-references in Protobuf, so it's not possible to have an infinitely deep structure, or a combinatorial explosion like with billion laughs - just a very deeply nested one, and only if allowed in the IDL, and only up to whatever message size limit you're allowing.
  - tehbeard 5y ago
    Thank you for the 2nd + 3rd paragraphs, those were parts of protobufs design I wasn't really aware of from a cursory glance. I'm a little suprised to learn there's no self-reference support in protobuf, as I wouldn't have assumed parsing that would be an issue (as all it really is is a pointer to an existing object in the message to say, put a ref. to it here), though I guess it might be a problem in supporting certain languages.
    - q3k 5y ago
      > I'm a little suprised to learn there's no self-reference support in protobuf, as I wouldn't have assumed parsing that would be an issue (as all it really is is a pointer to an existing object in the message to say, put a ref. to it here), though I guess it might be a problem in supporting certain languages. That's a tradeoff more designs should have, IMO: reduce the feature set as much as possible, but in return make the implementation vastly simpler. :) I assume it's not only about support in programming languages, but also exactly to eliminate the entire class of bugs that stems from back/forward-references in serialized data, and to generally keep the wire format as simple (to parse and to implement a parser for) as possible. The few usecases that could make use of references are not worth the pain inflicted on everyone if they were implemented.
thechao 5y ago
I had a little non-recursive JSON parser hanging around. When you have "nested" levels you've really only got two choices: object, or array. That implies that to track nesting, you just need an array of 1b values. In order to shave the yak properly, I built "nesting compressor" that detected runs of array/object and represented them using a 64b RLE; or, it bailed out, and then just used on-the-fly compression with zstd. Obviously, any sort of JSON file that fit on a disk I can afford can be parsed into memory in a tiny fraction of its on-disk representation. I modified `yes` to just stream `[` out; the JSON parser handled it just fine — it takes a while to roll a 64b counter.

kortex 5y ago

> I highly encourage any greenfield project to look into well designed and better specified alternatives. Like what?

jerf 5y ago
Part of the problem is that there's at least half-a-dozen high quality answers out of the gate (gRPC, FlatBuffers, Protocol Buffers, XML in some cases, Thrift), and an even-longer long tail after that. It's made harder when four different teams who deeply loathe JSON and independently decide to use something "better" can legitimately use four completely different technologies if they don't communicate with each other.
- 35fbe7d3d5b9 5y ago
  To your comment above – you can bodge around interop problems with JSON in ways that you cannot with some of these other technologies. I like to joke that I invented ndjson over a decade ago when I accidentally forgot to put things in an array before `json.dumps`, I just wasn't smart enough to call it a standard. But when you do end up with ndjson when you wanted an array of results, or vice versa, JSON makes it easy to munge things to where you need. Compare that to something like protobuf: it's not a self-synchronizing stream, so if you send someone multiple messages without framing them (prefix by length or delimited are popular approaches), they're going to decode a single message that doesn't make much sense on the other end. And they won't be able to fix it at all. So I guess JSON is New Jersey style design[1]. [1]: https://dreamsongs.com/RiseOfWorseIsBetter.html https://dreamsongs.com/RiseOfWorseIsBetter.html
  - kortex 5y ago
    Well, you invented one of the best things since sliced bread! I love NDjson, being able to parse a sequence of {} objects as an array is just frankly more natural. A coworker got some absurd speedup going from some massive json array to ndjson. Honestly if json had as part of its spec line-delimited arrays, and accepting NaN, it'd be close to perfect. Oh and native ints, but that is JS's problem. Well, and a single, canonical spec. And a hard limit (however high) on nesting depth. And some other things. Ok, maybe it's far from perfect.

jerf 5y ago

"My general opinion is that it's extremely hard to reliably use JSON as an interchange format reliably when multiple systems and/or parser implementations are involved." I suspect one of the reasons that JSON has been so successful is precisely this fuzziness, though. Every language can do something a little slightly different and it'll work at first when you send it to somebody else. You get up and off the ground really quickly, and can fix up issues as you go. If you try to specify something with a stronger schema right off the bat, I find a number of problems immediately emerge that tend to slow the process down. It may be foreign to programmers on HN who have embraced a strong static type mindset, or dynamic programmers who have learned the hard way that sometimes you need to be more precise about your types, but there's still a lot of programmers out there who will wonder why you're asking them whether this is an int or a float is relevant. I came in to work this morning to an alert system telling me that a field that a particular system has been sending as an integer for a couple of months now over many thousands of pushes, "number of bytes transferred", is apparently capable of being a float once every several thousand times for some reason. There's a lot of programmers who will send a string, or a null, or maybe a float, or maybe it's always an integer, and deeply don't understand why you care what it's getting serialized as. And that's just an example of some of the issues, not a complete list. Trying to specify with some stronger system moves a lot of these issues up front. (If your organization has internalized that's just how it has to be done, great! I bet you encountered a lot of these bumps on the way, though.) This isn't a celebration of JSON per se... this is really a rather cynical take. I don't know that we need to type everything to the n'th degree in the first meeting, but "why can't we just let our dynamically-typed language send this number as a string sometimes?" is definitely something I've had to discuss. (Now, I don't get a lot of resistance per se, but it's something I have to bring up.) I'm not presenting this as a good thing, but as a theory that JSON's success is actually in large part because of its loosey-gooseyness, and not despite it, regardless of how we may feel about it.

lifthrasiir 5y ago
I'm not convinced. There are a lot less JSON implementatations than JSON users, so we should have been possible to guide implementations with a means of proper specification and test suites. Note that the OP is possibly the first ever complete test suite for JSON after 15 full years. It is not like seeding initial implementations (that can serve as models for future implementors) is particularly hard either; Douglas Crockford himself wrote two implementations in C and JavaScript.

35fbe7d3d5b9 5y ago

> I highly encourage any greenfield project to look into well designed and better specified alternatives. By way of recommendation: I reach for protobufs to do data interchange between polyglot systems and have yet to be disappointed. Even if you aren't getting into gRPC, having data interchange backed by codegen and an IDL removes a lot of the risk you get with data interchange.

avmich 5y ago
In my experience JSON allowed absence of codegen and superior schema definition capabilities to protobuf, and also nice transformations with parts of jq built into JSON libraries. Try to limit structure complexity to something which can be verified before usage, yes. YMMV.
theamk 5y ago
In my experience, protobuf has a minimum project complexity threshold before it starts to make sense. Yes, if both sides of your interchange are systems which have build infra setup, it provides a better experience. But if you need to access data from outside of your usual projects, or from shell, or from random data analysis notebooks, It becomes a major pain. Recent example: we’ve had an orchestrator script which was written in “python with stdlib only” - no build step, download an archive, extract and run. This script had to talk to third-party program which would export protobuf only. This was a major pain as yon can imagine.

makeitdouble 5y ago

It’s a pyramid. At the bottom you have CSV which is popular beyond belief, and has no real specification, with common cases wildly handled differently across libraries. In the middle you have JSON which isn’t 100% interoperable, but goes 98% of the way. And you have XML and protobufs at the top tip, who have strong mechanisms available for interoperability but at an operational cost that rarely justifies the upgrade from JSON. I suspect it will take a lot more that “well designed” and “better specified” to justify moving away from JSON as the default stepup from chaotic CSV like formats.

heresie-dabord 5y ago
I would describe it as a _pyramid of bounded viability_ , from the minimally viable to the feature-burdened maximum. CSV excels (ha) as a minimally viable exchange format for data. Combined with awk, grep, sed, bash, and Perl, and some simple SVG or D3 with SVG, the analytical solution is fast, scalable, and automatable. But CSV has limits. The column headers are the schema. Beyond these bounds, we have the other formats. JSON is messier. Its strength is in network/browser encapsulations and operations. As I have seen it used, people insert an array as a kind of schema, and they stay away from complex nesting where scalability starts failing and other difficult tooling must be summoned to compensate (parsing tools such as jq). Beyond JSON's bounds, we have XML and associated tooling. XML is versatile and expressive. XML and JSON can be written simply but both can be abused by programmers who aren't thinking beyond their own cursor. This is a rich set of tooling for data representation. In the end, one of the main advantages of CSV is that it remains a format that brings little tooling baggage ("ecosystem") to the task.
maple3142 5y ago
I am not sure if parsing XML is better than parsing JSON. Many languages or libraries' XML parser are dangerous by default. You usually need to manually configure your XML parser to be secure from XML-related attacks. Fortunately, some languages and libraries are going to make XML have a securer defaults, this is a good change. IMO, I think XML shouldn't have include many questionable features from security perspective.
- 5y ago

lifthrasiir 5y ago

And all these problems trace back to Douglas Crockford. He didn't know how to make a proper serialization format [1] and also an interoperable standard (for the latter, Tim Bray tried very hard to make it slightly better [2]). He just noticed that a (supposed) subset of JavaScript can be easily turned into a serialization format with `eval` and went to publicize it, only noticing the issues later and still pursuring its standardization as is. I hate him. [1] My additional complaints: https://news.ycombinator.com/item?id=24953981 https://news.ycombinator.com/item?id=24953981 [2] https://www.tbray.org/ongoing/When/201x/2014/03/05/RFC7159-JSON https://www.tbray.org/ongoing/When/201x/2014/03/05/RFC7159-J...

gmac 5y ago
I was using JSON before it was 'invented', as was basically anyone sending data to the browser in JS format. Holding specific people responsible is pretty absurd.
- lifthrasiir 5y ago
  JSON before the standardization had an obvious data model and specification: ECMAScript. (I don't think JSON was widely used outside of JavaScript back then.) ECMAScript is particularly strictly defined even compared to other language standards, so it should have been possible to extract the relevant portions of ECMAScript into a proper standard. Crockford didn't. JSON as specified by Crockford was not even a proper subset of ECMAScript until ECMAScript itself retrofitted its syntax.

magicalhippo 5y ago

> My general opinion is that it's extremely hard to reliably use JSON as an interchange format reliably when multiple systems and/or parser implementations are involved. XML is very precisely defined in comparison to JSON. Yet we've had one customer who had a system that couldn't handle XML files with newlines in them at all, and several which sometimes sends ISO 8859-1 (Latin 1) encoded data in some fields of a XML file with encoding="UTF-8" in the header... We of course also have some nice fixed-field integrations, based on customer's specs, where the system suddenly sends multiple mangled characters if any non-ASCII character is present, causing the fields to suddenly not be so fixed anymore... It behaves very much like UTF-8 interpreted as Latin-1, except with something else than Latin-1. Anyway, I've given up trying to be strict at this point. We will have to wash incoming data, it's apparently inevitable.

spookthesunset 5y ago
I mean even if it is well defined that doesn’t mean the devs are using the languages native parser library. I’ve encounter at least two projects where the devs rolled their own XML “parser” using regex and “substring” functions. Why? “The xml library was too bloated… much easier to write it ourself”. Suffice to say, they had tons to problems.

throw_m239339 5y ago

Most of your problems aren't problems > 3. Do you pass around user-provided JSON data accross your system? How many JSON nesting levels does your implementation allow? What happens if it's exceeded? What happens if different parts of your processing system have different limits? What about other unspecified limits like serialized size, string length? XML has the same issue, that's why SAX exists, it works the same way with JSON. > 2. JSON allows for duplicate/repeated keys, and allows for the parser to basically do anything when that happens. Do you know how the parser implementation you use handles this? Are you sure there's no differences between that implementation and other implementations used in your system (eg. between execution and validation)? What about other undefined behaviour, like permitted number ranges? A parser should... parse and not interpret data or it isn't a parser. it's a deserializer. Well how many languages allow duplicate keys for maps anyway? this isn't an issue in practice. Basically, the answer to all your problems is to use an evented parser instead of a deserializer.

anonymoushn 5y ago
> Basically, the answer to all your problems is to use an evented parser instead of a deserializer. This seems a bit unreasonable? People face the same issue where different pieces of software treat duplicate HTTP headers differently, and usually they haven't written each piece of software they use to parse HTTP headers
kortex 5y ago
> this isn't an issue in practice. It absolutely is an issue in practice. If system A handles dupes by accepting the first and ignoring the rest, and system B implements last-key-wins, then that's a potential source of bugs. The system might not fully parse to a map. It may, for example, do string-level modification of json strings. Is that disgusting and wrong? Yes. Have I seen it in prod? Also yes.
- throw_m239339 5y ago
  > It absolutely is an issue in practice. If system A handles dupes by accepting the first and ignoring the rest, and system B implements last-key-wins, then that's a potential source of bugs. The system might not fully parse to a map. But the system shouldn't be automatically be parsing a "json map" to a map at first place: {"foo":"bar","foo":"baz","foo":"qix","fiz":"buzz"} Shouldn't be deserialized into a map. but a Array<Map<string,string>> like structure. A SAX style parser for JSON can help do that. Thus the issue is the choice of parser indeed. Not JSON.

zelphirkalt 5y ago

Some good points there. And now imagine people wanting to needlessly use YAML for configuration, which adds loads of edge cases on top of that.

stinos 5y ago

You want to set allow_nan=False to be compliant. Otherwise this _will_ annoy someone who has to consume your shoddy pseudo-JSON from a standards-compliant library Funny (well, not really) thing is NaN and Inf are perfectly valid floating point numbers acoording to most (?) standards used on computers. To the point that I don't understand why it was left out of JSON. So unless you're 100% sure you won't encounter these numbers the choice is between not being able to use JSON, or finding hacks around (and using null isn't one of them since you have 3 numbers to represent), or just using non-compliant-yet-often-accepted JSON and possibly annoying someone whos parser doesn't handle it. And for me there have been quite a lot of cases were I just quickly needed something simple to interface between components so when finding out they all support JSON+Nan/Inf then the choice is usually made quickly.

nomel 5y ago
> To the point that I don't understand why it was left out of JSON I think you're forgetting the birthplace of JSON. Who deals with the concept of infinity and NaN in the context of web front ends?
- lifthrasiir 5y ago
  Ranges are pretty common in APIs and both -Infinity and Infinity can naturally arise from one-sided ranges. Since they are absent in JSON, they are frequently replaced with null, ad-hoc sentinel values with uncoded assumptions (e.g. timestamps should be always positive) and missing fields.
- stinos 5y ago
  I get that, but to go from "oh this won't be very common" to willingly "let's just leave this out" is something else. At least in my mind :) Or was it an oversight?
  - mst 5y ago
    I suspect it was a bet on worse is better. Whether it was a good bet is debatable, but given Crockford's focus on "try and leave out as much as possible" I can certainly see it making sense at the time.

Cloudef 5y ago

Any JSON parser that tries to handle numbers without big number support is broken. This is why I always raise eyebrows if I see json library that doesn't allow me to deal with the number myself by retrieving it as a string.

JoBrad 5y ago
Agreed. Even worse, one that corrupts data it doesn’t know how to handle.

nuerow 5y ago

> My general opinion is that it's extremely hard to reliably use JSON as an interchange format reliably when multiple systems and/or parser implementations are involved. None of the points you made were regarding JSON, and we're focused on either hypothetical or very specific implementation problems. Even though it's true that you can experience problems if you pick a broken noncompliant implementation and fail to test, that's hardly a JSON issue. I mean, are we supposed to think that XML has a problem if you pick an implementation that fails to handle multiple entities or outputs broken XML?