8 ms·

JSON is an incredibly inefficient format for shareable data: it is annoying to write, unsafe to parse and it even comes with a lot of overhead (colons, quotes,

JSON is an incredibly inefficient format for shareable data: it is annoying to write, unsafe to parse and it even comes with a lot of overhead (colons, quotes, brackets and the like). I'd prefer s-expressions.

the8472 7y ago
CBOR is a better wire format in all aspects except readability when all you have is a text viewer. It's a superset replacement when it comes to supported datatypes
- rhabarba 7y ago
  Looks like a worse (= less widely supported) msgpack to me...?
  - piotrkubisa 7y ago
    It looks it performs way better than msgpack in following benchmarks [0]. However, it's true that it's not very popular - personally, I rarely see any software that utilize CBOR. [0]: http://tutorials.jenkov.com/ion/ion-performance-benchmarks.html http://tutorials.jenkov.com/ion/ion-performance-benchmarks.h...
heinrichhartman 7y ago
Have you ever seen s-expressions being used as data exchange format in the wild? (outside of the LISP/emacs world) Not sure there is a decent standard spec and a good Java (say) parser around. Is there?
- rhabarba 7y ago
  JSON is mostly irrelevant outside the JavaScript world as well. (Don't quote me please.) There are no RFCs for s-expressions, but that's fixable, I guess. And yes - there are a few Java parsers for them.
  - diroussel 7y ago
    What about all those java, ruby, python and C# services that expose JSON based REST services. JSON is widely used for network API, config files and data stores.
    - rhabarba 7y ago
      "Too many people already do the wrong thing" is not a good excuse to stick with the wrong thing.
      - afiori 7y ago
        But you have yet to say why JSON is worse than your preferred alternative. I agree that it is not perfect (for many use cases I prefer TOML personally), but I cannot see how it is worse that S-expressions (except for the trailing comma issue, that is a sin in itself)
      - dragonwriter 7y ago
        No, but that's not the claim: the claim was JSON is irrelevant outside of JavaScript, which is clearly false given all the use in non-JS environments. Whether it's the “wrong thing” is another issue, though I've yet to see a standard of “wrong” proposed in the thread for which that is accurate.
      - geofft 7y ago
        It is not the wrong thing. You were arguing that it was wrong because it was rarely used; you have yet to prove your point that it is in any way wrong. And, perhaps, all those people know better than you.
      - heinrichhartman 7y ago
        LOL! DNS, HTTP, XML, Linux, C, are all terrible if you look close enough. So ... should we ditch all of this? Wide adoption and popularity trumps "being wrong" everyday.
        7y ago
- bb010g 7y ago
  edn's pretty alright. https://github.com/edn-format/edn https://github.com/edn-format/edn Here's a Java parser: https://github.com/bpsm/edn-java https://github.com/bpsm/edn-java
  - heinrichhartman 7y ago
    very cool!
rlonstein 7y ago
The obvious defense is that it's already in common use and widely (if unevenly) supported.
afiori 7y ago
Why unsafe to parse? > it even comes with a lot of overhead (colons, quotes, brackets and the like). according to Sqlite devs the that overhead is not much worse that for a binary encoding. quoting: https://www.sqlite.org/json1.html https://www.sqlite.org/json1.html > The json1 extension does not (currently) support a binary encoding of JSON. Experiments have been unable to find a binary encoding that is significantly smaller or faster than a plain text encoding. (The present implementation parses JSON text at over 1 GB/s.)
- rhabarba 7y ago
  > Why unsafe to parse? Try to write a parser for an arbitrarily complex and nested JSON object without running into runtime errors even after the fourth-or-so revision. There are too many "yes buts" in the living standard. > that overhead is not much worse It _is_ much worse than the two-byte "overhead" of one s-exp node.
  - tonyg 7y ago
    > It _is_ much worse than the two-byte "overhead" of one s-exp node. I don't believe you. Atoms are roughly the same, and compounds, well: [1,2,3] vs (1 2 3) {"a":1,"b":2} vs (("a"1)("b"2)) I suppose if you have a lot of booleans you might save a little by using T/NIL or #t/#f, but otherwise it's more or less a wash. (You might suggest plists, ("a"1"b"2), for dictionaries, in which case you save a whopping two bytes per entry. But I'd choose the nested-list format over plists, if I were encoding a dictionary into an S-expression.)
    - viraptor 7y ago
      > you save a whopping two bytes per entry Which pretty much disappears when compressed anyway.
  - lloeki 7y ago
    In specific contexts such as this one you don't need arbitrarily complex JSON, you just need to validate that the JSON structure respects not just JSON itself but also the protocol specification at hand while you parse, and bail out when it doesn't, so you can have a dedicated, fast parser tailored for the use case, instead of first parsing general JSON then validate its content. Anyway, for being a Fasmail user, JMAP rips IMAP to shreds in terms of performance. I've been following JMAP from afar and waiting for Fastmail to publish this for a long time.
chrismorgan 7y ago
I’m a developer at Fastmail, and I’m going to bite. Personally, I prefer binary formats where feasible, because I like pursuing efficiency to extremes, and write all my own code in Rust. Professionally, I happen to write mostly JavaScript, and no Rust at all, but we do still focus on performance at Fastmail. (It’s in the name!) If you’re comparing JSON to s-expressions, it’s a wash in efficiency. They’re both “inefficient”, both representing generic data structures in regular text. They use slightly different punctuation, but that’s about it. The differences are negligible. But when it comes to support, JSON is well-supported in all mainstream languages, while s-expressions have no central specification on precise syntax, and you’ll need to do your own thing. One of the deliberate design advantages of JMAP over IMAP is doing away with custom parsers, and using the standard tools people are familiar with, HTTPS and JSON. When talking efficiency, you might then ask why not to use Cap’n Proto, protobufs or one of those sorts of things. Due to the need for extensibility (e.g. in Fastmail we add a few custom fields to JMAP core and mail data structures), rigid structures are unsuitable; you do need arbitrary key-value arrangement, despite it being larger over the wire. All in all, JSON is a robust choice for just about any API. Fun fact: at Fastmail we compress both request and response, even using a custom dictionary primed on real JMAP stuff, to improve the compression ratio. I understand standard DEFLATE was about equivalent in size to CBOR (a binary representation of JSON), and the 2KB dictionary brought it down even further, can’t remember how much it was off the top of my head. Neil Jenkins gave a talk about this and related matters last month, https://www.webdirections.org/code/speakers/neil-jenkins.php https://www.webdirections.org/code/speakers/neil-jenkins.php, no public video sadly. I’ve thought about writing a blog post about it all. An unfortunate side-effect of this compression is that you can’t inspect traffic in the network panel of the dev tools; you can add ?compress=0 to disable it, if you’re an interested Fastmail user.
- nerdponx 7y ago
  Is Messagepack not a useful alternative to JSON in your opinion?
  - chrismorgan 7y ago
    MessagePack and CBOR are equivalent. I haven’t ever had much to do with them (never used them, only known about them), but I’m inclined to think that simple compression achieves between “roughly as much” and “more” in general, and leaves you with the industry standard of JSON underneath. Do not consider me an expert by any means, and I’m fairly ambivalent on your question; but I don’t see a particularly useful niche for them between plain JSON and a binary representation matching a predefined static schema, all potentially with compression.
verisimilitudes 7y ago
I agree. I'm in the process of writing an efficient JSON parser, since using the simple approach is too slow, I find, and because a database I want to introspect on uses it. To process JSON, you must inspect every character. Douglas Crockford brags about how smart he is for disallowing comments, but there's whitespace allowed everywhere. I actually have a (I think) rather novel approach and there's some potential for little optimizations here and there, but it disgusts me that I'm doing this at all. The optimizations popular, shitty formats like JSON receives are purely wasteful. It's suitable, though. The WWW wasn't made right from the start and there's always an idiot somewhere looking to add more shit to the pile. Not only does JavaScript exist, but it morphed into this ugly, inefficient, very poor ''format''. We don't have Lisp machines, but new processors are getting instructions to cope with JavaScript idiocy. A numerical (binary) format is clearly best. It's small, random-access when properly designed, and efficient, but hey, everything's being pushed into port 80 so why not push all data into JSON? Now, the rest of this message is in response to @chrismorgan from an organization pushing this filth. >Personally, I prefer binary formats where feasible, because I like pursuing efficiency to extremes, and write all my own code in Rust. We agree on that, then, but I prefer to write that Rust is just the effeminate man's Ada. >If you’re comparing JSON to s-expressions, it’s a wash in efficiency. They’re both “inefficient”, both representing generic data structures in regular text. They use slightly different punctuation, but that’s about it. The differences are negligible. This is wrong. Firstly, we should clarify what is meant by S-expressions. Let's assume the S-expressions have lists, strings, symbols, and numbers. That's already simpler than JSON. You're not hampered by JavaScript's floating point nonsense, either. There's fewer edge cases and less escaping. JSON is, at best, a worse-is-better form of S-expressions. >But when it comes to support, JSON is well-supported in all mainstream languages, while s-expressions have no central specification on precise syntax, and you’ll need to do your own thing. One of the deliberate design advantages of JMAP over IMAP is doing away with custom parsers, and using the standard tools people are familiar with, HTTPS and JSON. So, this ties into what I wrote before I even read your post, exactly. Cretins continue to push everything into the WWW protocols and act like their garbage is standard, simple, acceptable, or wanted; it's only technically standard, but it's a bad standard. I'd bet my foot you also like ''DNS over HTTPS''. A numerical (binary) format should be used, because it's the best option and people can be expected to bother writing an acceptor for such important data. There are plenty of languages that can easily accept a numerical format, but not JSON; think of, say, Forth. Oh, but everyone writes in Rust and JavaScript nowadays, right. >When talking efficiency, you might then ask why not to use Cap’n Proto, protobufs or one of those sorts of things. Due to the need for extensibility (e.g. in Fastmail we add a few custom fields to JMAP core and mail data structures), rigid structures are unsuitable; you do need arbitrary key-value arrangement, despite it being larger over the wire. All of this is unnecessary. Email doesn't need this drivel. >All in all, JSON is a robust choice for just about any API. Of course; forty years ago, UNIX was the solution for every problem. The WWW and JavaScript are nothing more than a continuation of this worse-is-better filth. >Fun fact: at Fastmail we compress both request and response, even using a custom dictionary primed on real JMAP stuff, to improve the compression ratio. I understand standard DEFLATE was about equivalent in size to CBOR (a binary representation of JSON), and the 2KB dictionary brought it down even further, can’t remember how much it was off the top of my head. I hate how everyone's response to inefficiency is ''Compress it!'' and ''Cache it!''. There's no reason it should be so inefficient to start with.
jerf 7y ago
I don't know if this addresses everything, but from the spec: 1.5. JSON as the Data Encoding Format JSON is a text-based data interchange format as specified in [RFC8259]. The Internet JSON (I-JSON) format defined in [RFC7493] is a strict subset of this, adding restrictions to avoid potentially confusing scenarios (for example, it mandates that an object MUST NOT have two members with the same name). All data sent from the client to the server or from the server to the client (except binary file upload/download) MUST be valid I-JSON according to the RFC and is therefore case sensitive and encoded in UTF-8 [RFC3629]. RFC-7493: https://tools.ietf.org/html/rfc7493 https://tools.ietf.org/html/rfc7493 A quick scan of it says to me it doesn't get all the issues but it does get the biggest one, which IMHO are the ambiguities in what constitutes a "number" in JSON. I-JSON explicitly limits it to 64-bit floats.
inimino 7y ago
> annoying to write That's subjective and it seems plenty of people have the opposite opinion. > unsafe to parse Is this based on theoretical concerns or practical experience? > comes with a lot of overhead Keep in mind the alternatives it is largely replacing. XML, text-based email protocols each with their own unique grammars, etc. > I'd prefer s-expressions. Ah yes, the plaintive cry of the lisper in the desert. ;-) I'm afraid in the 60+ years that s-exps have had to catch on in the mainstream, they haven't. I can't explain this result but nonetheless it seems irrational to expect it to suddenly change.
m463 7y ago
From a developer perspective, I like json a LOT more than XML-based data. s-expressions seem like an out-of-left-field choice.
deleted 7y ago
[deleted]

the8472 7y ago

CBOR is a better wire format in all aspects except readability when all you have is a text viewer. It's a superset replacement when it comes to supported datatypes

rhabarba 7y ago
Looks like a worse (= less widely supported) msgpack to me...?
- piotrkubisa 7y ago
  It looks it performs way better than msgpack in following benchmarks [0]. However, it's true that it's not very popular - personally, I rarely see any software that utilize CBOR. [0]: http://tutorials.jenkov.com/ion/ion-performance-benchmarks.html http://tutorials.jenkov.com/ion/ion-performance-benchmarks.h...

heinrichhartman 7y ago

Have you ever seen s-expressions being used as data exchange format in the wild? (outside of the LISP/emacs world) Not sure there is a decent standard spec and a good Java (say) parser around. Is there?

rhabarba 7y ago
JSON is mostly irrelevant outside the JavaScript world as well. (Don't quote me please.) There are no RFCs for s-expressions, but that's fixable, I guess. And yes - there are a few Java parsers for them.
- diroussel 7y ago
  What about all those java, ruby, python and C# services that expose JSON based REST services. JSON is widely used for network API, config files and data stores.
  - rhabarba 7y ago
    "Too many people already do the wrong thing" is not a good excuse to stick with the wrong thing.
    - afiori 7y ago
      But you have yet to say why JSON is worse than your preferred alternative. I agree that it is not perfect (for many use cases I prefer TOML personally), but I cannot see how it is worse that S-expressions (except for the trailing comma issue, that is a sin in itself)
    - dragonwriter 7y ago
      No, but that's not the claim: the claim was JSON is irrelevant outside of JavaScript, which is clearly false given all the use in non-JS environments. Whether it's the “wrong thing” is another issue, though I've yet to see a standard of “wrong” proposed in the thread for which that is accurate.
    - geofft 7y ago
      It is not the wrong thing. You were arguing that it was wrong because it was rarely used; you have yet to prove your point that it is in any way wrong. And, perhaps, all those people know better than you.
    - heinrichhartman 7y ago
      LOL! DNS, HTTP, XML, Linux, C, are all terrible if you look close enough. So ... should we ditch all of this? Wide adoption and popularity trumps "being wrong" everyday.
      - 7y ago
bb010g 7y ago
edn's pretty alright. https://github.com/edn-format/edn https://github.com/edn-format/edn Here's a Java parser: https://github.com/bpsm/edn-java https://github.com/bpsm/edn-java
- heinrichhartman 7y ago
  very cool!

rlonstein 7y ago

The obvious defense is that it's already in common use and widely (if unevenly) supported.

afiori 7y ago

Why unsafe to parse? > it even comes with a lot of overhead (colons, quotes, brackets and the like). according to Sqlite devs the that overhead is not much worse that for a binary encoding. quoting: https://www.sqlite.org/json1.html https://www.sqlite.org/json1.html > The json1 extension does not (currently) support a binary encoding of JSON. Experiments have been unable to find a binary encoding that is significantly smaller or faster than a plain text encoding. (The present implementation parses JSON text at over 1 GB/s.)

rhabarba 7y ago
> Why unsafe to parse? Try to write a parser for an arbitrarily complex and nested JSON object without running into runtime errors even after the fourth-or-so revision. There are too many "yes buts" in the living standard. > that overhead is not much worse It _is_ much worse than the two-byte "overhead" of one s-exp node.
- tonyg 7y ago
  > It _is_ much worse than the two-byte "overhead" of one s-exp node. I don't believe you. Atoms are roughly the same, and compounds, well: [1,2,3] vs (1 2 3) {"a":1,"b":2} vs (("a"1)("b"2)) I suppose if you have a lot of booleans you might save a little by using T/NIL or #t/#f, but otherwise it's more or less a wash. (You might suggest plists, ("a"1"b"2), for dictionaries, in which case you save a whopping two bytes per entry. But I'd choose the nested-list format over plists, if I were encoding a dictionary into an S-expression.)
  - viraptor 7y ago
    > you save a whopping two bytes per entry Which pretty much disappears when compressed anyway.
- lloeki 7y ago
  In specific contexts such as this one you don't need arbitrarily complex JSON, you just need to validate that the JSON structure respects not just JSON itself but also the protocol specification at hand while you parse, and bail out when it doesn't, so you can have a dedicated, fast parser tailored for the use case, instead of first parsing general JSON then validate its content. Anyway, for being a Fasmail user, JMAP rips IMAP to shreds in terms of performance. I've been following JMAP from afar and waiting for Fastmail to publish this for a long time.

chrismorgan 7y ago

I’m a developer at Fastmail, and I’m going to bite. Personally, I prefer binary formats where feasible, because I like pursuing efficiency to extremes, and write all my own code in Rust. Professionally, I happen to write mostly JavaScript, and no Rust at all, but we do still focus on performance at Fastmail. (It’s in the name!) If you’re comparing JSON to s-expressions, it’s a wash in efficiency. They’re both “inefficient”, both representing generic data structures in regular text. They use slightly different punctuation, but that’s about it. The differences are negligible. But when it comes to support, JSON is well-supported in all mainstream languages, while s-expressions have no central specification on precise syntax, and you’ll need to do your own thing. One of the deliberate design advantages of JMAP over IMAP is doing away with custom parsers, and using the standard tools people are familiar with, HTTPS and JSON. When talking efficiency, you might then ask why not to use Cap’n Proto, protobufs or one of those sorts of things. Due to the need for extensibility (e.g. in Fastmail we add a few custom fields to JMAP core and mail data structures), rigid structures are unsuitable; you do need arbitrary key-value arrangement, despite it being larger over the wire. All in all, JSON is a robust choice for just about any API. Fun fact: at Fastmail we compress both request and response, even using a custom dictionary primed on real JMAP stuff, to improve the compression ratio. I understand standard DEFLATE was about equivalent in size to CBOR (a binary representation of JSON), and the 2KB dictionary brought it down even further, can’t remember how much it was off the top of my head. Neil Jenkins gave a talk about this and related matters last month, https://www.webdirections.org/code/speakers/neil-jenkins.php https://www.webdirections.org/code/speakers/neil-jenkins.php, no public video sadly. I’ve thought about writing a blog post about it all. An unfortunate side-effect of this compression is that you can’t inspect traffic in the network panel of the dev tools; you can add ?compress=0 to disable it, if you’re an interested Fastmail user.

nerdponx 7y ago
Is Messagepack not a useful alternative to JSON in your opinion?
- chrismorgan 7y ago
  MessagePack and CBOR are equivalent. I haven’t ever had much to do with them (never used them, only known about them), but I’m inclined to think that simple compression achieves between “roughly as much” and “more” in general, and leaves you with the industry standard of JSON underneath. Do not consider me an expert by any means, and I’m fairly ambivalent on your question; but I don’t see a particularly useful niche for them between plain JSON and a binary representation matching a predefined static schema, all potentially with compression.

verisimilitudes 7y ago

I agree. I'm in the process of writing an efficient JSON parser, since using the simple approach is too slow, I find, and because a database I want to introspect on uses it. To process JSON, you must inspect every character. Douglas Crockford brags about how smart he is for disallowing comments, but there's whitespace allowed everywhere. I actually have a (I think) rather novel approach and there's some potential for little optimizations here and there, but it disgusts me that I'm doing this at all. The optimizations popular, shitty formats like JSON receives are purely wasteful. It's suitable, though. The WWW wasn't made right from the start and there's always an idiot somewhere looking to add more shit to the pile. Not only does JavaScript exist, but it morphed into this ugly, inefficient, very poor ''format''. We don't have Lisp machines, but new processors are getting instructions to cope with JavaScript idiocy. A numerical (binary) format is clearly best. It's small, random-access when properly designed, and efficient, but hey, everything's being pushed into port 80 so why not push all data into JSON? Now, the rest of this message is in response to @chrismorgan from an organization pushing this filth. >Personally, I prefer binary formats where feasible, because I like pursuing efficiency to extremes, and write all my own code in Rust. We agree on that, then, but I prefer to write that Rust is just the effeminate man's Ada. >If you’re comparing JSON to s-expressions, it’s a wash in efficiency. They’re both “inefficient”, both representing generic data structures in regular text. They use slightly different punctuation, but that’s about it. The differences are negligible. This is wrong. Firstly, we should clarify what is meant by S-expressions. Let's assume the S-expressions have lists, strings, symbols, and numbers. That's already simpler than JSON. You're not hampered by JavaScript's floating point nonsense, either. There's fewer edge cases and less escaping. JSON is, at best, a worse-is-better form of S-expressions. >But when it comes to support, JSON is well-supported in all mainstream languages, while s-expressions have no central specification on precise syntax, and you’ll need to do your own thing. One of the deliberate design advantages of JMAP over IMAP is doing away with custom parsers, and using the standard tools people are familiar with, HTTPS and JSON. So, this ties into what I wrote before I even read your post, exactly. Cretins continue to push everything into the WWW protocols and act like their garbage is standard, simple, acceptable, or wanted; it's only technically standard, but it's a bad standard. I'd bet my foot you also like ''DNS over HTTPS''. A numerical (binary) format should be used, because it's the best option and people can be expected to bother writing an acceptor for such important data. There are plenty of languages that can easily accept a numerical format, but not JSON; think of, say, Forth. Oh, but everyone writes in Rust and JavaScript nowadays, right. >When talking efficiency, you might then ask why not to use Cap’n Proto, protobufs or one of those sorts of things. Due to the need for extensibility (e.g. in Fastmail we add a few custom fields to JMAP core and mail data structures), rigid structures are unsuitable; you do need arbitrary key-value arrangement, despite it being larger over the wire. All of this is unnecessary. Email doesn't need this drivel. >All in all, JSON is a robust choice for just about any API. Of course; forty years ago, UNIX was the solution for every problem. The WWW and JavaScript are nothing more than a continuation of this worse-is-better filth. >Fun fact: at Fastmail we compress both request and response, even using a custom dictionary primed on real JMAP stuff, to improve the compression ratio. I understand standard DEFLATE was about equivalent in size to CBOR (a binary representation of JSON), and the 2KB dictionary brought it down even further, can’t remember how much it was off the top of my head. I hate how everyone's response to inefficiency is ''Compress it!'' and ''Cache it!''. There's no reason it should be so inefficient to start with.

jerf 7y ago

I don't know if this addresses everything, but from the spec: 1.5. JSON as the Data Encoding Format JSON is a text-based data interchange format as specified in [RFC8259]. The Internet JSON (I-JSON) format defined in [RFC7493] is a strict subset of this, adding restrictions to avoid potentially confusing scenarios (for example, it mandates that an object MUST NOT have two members with the same name). All data sent from the client to the server or from the server to the client (except binary file upload/download) MUST be valid I-JSON according to the RFC and is therefore case sensitive and encoded in UTF-8 [RFC3629]. RFC-7493: https://tools.ietf.org/html/rfc7493 https://tools.ietf.org/html/rfc7493 A quick scan of it says to me it doesn't get all the issues but it does get the biggest one, which IMHO are the ambiguities in what constitutes a "number" in JSON. I-JSON explicitly limits it to 64-bit floats.

inimino 7y ago

> annoying to write That's subjective and it seems plenty of people have the opposite opinion. > unsafe to parse Is this based on theoretical concerns or practical experience? > comes with a lot of overhead Keep in mind the alternatives it is largely replacing. XML, text-based email protocols each with their own unique grammars, etc. > I'd prefer s-expressions. Ah yes, the plaintive cry of the lisper in the desert. ;-) I'm afraid in the 60+ years that s-exps have had to catch on in the mainstream, they haven't. I can't explain this result but nonetheless it seems irrational to expect it to suddenly change.

m463 7y ago

From a developer perspective, I like json a LOT more than XML-based data. s-expressions seem like an out-of-left-field choice.

deleted 7y ago

[deleted]