11 ms·

The JSON Meta Application Protocol (JMAP)

160 points by kuba-orlik 7y ago

rhabarba 7y ago
JSON is an incredibly inefficient format for shareable data: it is annoying to write, unsafe to parse and it even comes with a lot of overhead (colons, quotes, brackets and the like). I'd prefer s-expressions.
- the8472 7y ago
  CBOR is a better wire format in all aspects except readability when all you have is a text viewer. It's a superset replacement when it comes to supported datatypes
  - rhabarba 7y ago
    Looks like a worse (= less widely supported) msgpack to me...?
    - piotrkubisa 7y ago
      It looks it performs way better than msgpack in following benchmarks [0]. However, it's true that it's not very popular - personally, I rarely see any software that utilize CBOR. [0]: http://tutorials.jenkov.com/ion/ion-performance-benchmarks.html http://tutorials.jenkov.com/ion/ion-performance-benchmarks.h...
- heinrichhartman 7y ago
  Have you ever seen s-expressions being used as data exchange format in the wild? (outside of the LISP/emacs world) Not sure there is a decent standard spec and a good Java (say) parser around. Is there?
  - rhabarba 7y ago
    JSON is mostly irrelevant outside the JavaScript world as well. (Don't quote me please.) There are no RFCs for s-expressions, but that's fixable, I guess. And yes - there are a few Java parsers for them.
    - diroussel 7y ago
      What about all those java, ruby, python and C# services that expose JSON based REST services. JSON is widely used for network API, config files and data stores.
      - rhabarba 7y ago
        "Too many people already do the wrong thing" is not a good excuse to stick with the wrong thing.
        afiori 7y ago
        But you have yet to say why JSON is worse than your preferred alternative. I agree that it is not perfect (for many use cases I prefer TOML personally), but I cannot see how it is worse that S-expressions (except for the trailing comma issue, that is a sin in itself)
        dragonwriter 7y ago
        No, but that's not the claim: the claim was JSON is irrelevant outside of JavaScript, which is clearly false given all the use in non-JS environments. Whether it's the “wrong thing” is another issue, though I've yet to see a standard of “wrong” proposed in the thread for which that is accurate.
        geofft 7y ago
        It is not the wrong thing. You were arguing that it was wrong because it was rarely used; you have yet to prove your point that it is in any way wrong. And, perhaps, all those people know better than you.
        heinrichhartman 7y ago
  - bb010g 7y ago
    edn's pretty alright. https://github.com/edn-format/edn https://github.com/edn-format/edn Here's a Java parser: https://github.com/bpsm/edn-java https://github.com/bpsm/edn-java
    - heinrichhartman 7y ago
      very cool!
- rlonstein 7y ago
  The obvious defense is that it's already in common use and widely (if unevenly) supported.
- afiori 7y ago
  Why unsafe to parse? > it even comes with a lot of overhead (colons, quotes, brackets and the like). according to Sqlite devs the that overhead is not much worse that for a binary encoding. quoting: https://www.sqlite.org/json1.html https://www.sqlite.org/json1.html > The json1 extension does not (currently) support a binary encoding of JSON. Experiments have been unable to find a binary encoding that is significantly smaller or faster than a plain text encoding. (The present implementation parses JSON text at over 1 GB/s.)
  - rhabarba 7y ago
    > Why unsafe to parse? Try to write a parser for an arbitrarily complex and nested JSON object without running into runtime errors even after the fourth-or-so revision. There are too many "yes buts" in the living standard. > that overhead is not much worse It _is_ much worse than the two-byte "overhead" of one s-exp node.
    - tonyg 7y ago
      > It _is_ much worse than the two-byte "overhead" of one s-exp node. I don't believe you. Atoms are roughly the same, and compounds, well: [1,2,3] vs (1 2 3) {"a":1,"b":2} vs (("a"1)("b"2)) I suppose if you have a lot of booleans you might save a little by using T/NIL or #t/#f, but otherwise it's more or less a wash. (You might suggest plists, ("a"1"b"2), for dictionaries, in which case you save a whopping two bytes per entry. But I'd choose the nested-list format over plists, if I were encoding a dictionary into an S-expression.)
      - viraptor 7y ago
        > you save a whopping two bytes per entry Which pretty much disappears when compressed anyway.
    - lloeki 7y ago
      In specific contexts such as this one you don't need arbitrarily complex JSON, you just need to validate that the JSON structure respects not just JSON itself but also the protocol specification at hand while you parse, and bail out when it doesn't, so you can have a dedicated, fast parser tailored for the use case, instead of first parsing general JSON then validate its content. Anyway, for being a Fasmail user, JMAP rips IMAP to shreds in terms of performance. I've been following JMAP from afar and waiting for Fastmail to publish this for a long time.
- chrismorgan 7y ago
  I’m a developer at Fastmail, and I’m going to bite. Personally, I prefer binary formats where feasible, because I like pursuing efficiency to extremes, and write all my own code in Rust. Professionally, I happen to write mostly JavaScript, and no Rust at all, but we do still focus on performance at Fastmail. (It’s in the name!) If you’re comparing JSON to s-expressions, it’s a wash in efficiency. They’re both “inefficient”, both representing generic data structures in regular text. They use slightly different punctuation, but that’s about it. The differences are negligible. But when it comes to support, JSON is well-supported in all mainstream languages, while s-expressions have no central specification on precise syntax, and you’ll need to do your own thing. One of the deliberate design advantages of JMAP over IMAP is doing away with custom parsers, and using the standard tools people are familiar with, HTTPS and JSON. When talking efficiency, you might then ask why not to use Cap’n Proto, protobufs or one of those sorts of things. Due to the need for extensibility (e.g. in Fastmail we add a few custom fields to JMAP core and mail data structures), rigid structures are unsuitable; you do need arbitrary key-value arrangement, despite it being larger over the wire. All in all, JSON is a robust choice for just about any API. Fun fact: at Fastmail we compress both request and response, even using a custom dictionary primed on real JMAP stuff, to improve the compression ratio. I understand standard DEFLATE was about equivalent in size to CBOR (a binary representation of JSON), and the 2KB dictionary brought it down even further, can’t remember how much it was off the top of my head. Neil Jenkins gave a talk about this and related matters last month, https://www.webdirections.org/code/speakers/neil-jenkins.php https://www.webdirections.org/code/speakers/neil-jenkins.php, no public video sadly. I’ve thought about writing a blog post about it all. An unfortunate side-effect of this compression is that you can’t inspect traffic in the network panel of the dev tools; you can add ?compress=0 to disable it, if you’re an interested Fastmail user.
  - nerdponx 7y ago
    Is Messagepack not a useful alternative to JSON in your opinion?
    - chrismorgan 7y ago
      MessagePack and CBOR are equivalent. I haven’t ever had much to do with them (never used them, only known about them), but I’m inclined to think that simple compression achieves between “roughly as much” and “more” in general, and leaves you with the industry standard of JSON underneath. Do not consider me an expert by any means, and I’m fairly ambivalent on your question; but I don’t see a particularly useful niche for them between plain JSON and a binary representation matching a predefined static schema, all potentially with compression.
- verisimilitudes 7y ago
  I agree. I'm in the process of writing an efficient JSON parser, since using the simple approach is too slow, I find, and because a database I want to introspect on uses it. To process JSON, you must inspect every character. Douglas Crockford brags about how smart he is for disallowing comments, but there's whitespace allowed everywhere. I actually have a (I think) rather novel approach and there's some potential for little optimizations here and there, but it disgusts me that I'm doing this at all. The optimizations popular, shitty formats like JSON receives are purely wasteful. It's suitable, though. The WWW wasn't made right from the start and there's always an idiot somewhere looking to add more shit to the pile. Not only does JavaScript exist, but it morphed into this ugly, inefficient, very poor ''format''. We don't have Lisp machines, but new processors are getting instructions to cope with JavaScript idiocy. A numerical (binary) format is clearly best. It's small, random-access when properly designed, and efficient, but hey, everything's being pushed into port 80 so why not push all data into JSON? Now, the rest of this message is in response to @chrismorgan from an organization pushing this filth. >Personally, I prefer binary formats where feasible, because I like pursuing efficiency to extremes, and write all my own code in Rust. We agree on that, then, but I prefer to write that Rust is just the effeminate man's Ada. >If you’re comparing JSON to s-expressions, it’s a wash in efficiency. They’re both “inefficient”, both representing generic data structures in regular text. They use slightly different punctuation, but that’s about it. The differences are negligible. This is wrong. Firstly, we should clarify what is meant by S-expressions. Let's assume the S-expressions have lists, strings, symbols, and numbers. That's already simpler than JSON. You're not hampered by JavaScript's floating point nonsense, either. There's fewer edge cases and less escaping. JSON is, at best, a worse-is-better form of S-expressions. >But when it comes to support, JSON is well-supported in all mainstream languages, while s-expressions have no central specification on precise syntax, and you’ll need to do your own thing. One of the deliberate design advantages of JMAP over IMAP is doing away with custom parsers, and using the standard tools people are familiar with, HTTPS and JSON. So, this ties into what I wrote before I even read your post, exactly. Cretins continue to push everything into the WWW protocols and act like their garbage is standard, simple, acceptable, or wanted; it's only technically standard, but it's a bad standard. I'd bet my foot you also like ''DNS over HTTPS''. A numerical (binary) format should be used, because it's the best option and people can be expected to bother writing an acceptor for such important data. There are plenty of languages that can easily accept a numerical format, but not JSON; think of, say, Forth. Oh, but everyone writes in Rust and JavaScript nowadays, right. >When talking efficiency, you might then ask why not to use Cap’n Proto, protobufs or one of those sorts of things. Due to the need for extensibility (e.g. in Fastmail we add a few custom fields to JMAP core and mail data structures), rigid structures are unsuitable; you do need arbitrary key-value arrangement, despite it being larger over the wire. All of this is unnecessary. Email doesn't need this drivel. >All in all, JSON is a robust choice for just about any API. Of course; forty years ago, UNIX was the solution for every problem. The WWW and JavaScript are nothing more than a continuation of this worse-is-better filth. >Fun fact: at Fastmail we compress both request and response, even using a custom dictionary primed on real JMAP stuff, to improve the compression ratio. I understand standard DEFLATE was about equivalent in size to CBOR (a binary representation of JSON), and the 2KB dictionary brought it down even further, can’t remember how much it was off the top of my head. I hate how everyone's response to inefficiency is ''Compress it!'' and ''Cache it!''. There's no reason it should be so inefficient to start with.
- jerf 7y ago
  I don't know if this addresses everything, but from the spec: 1.5. JSON as the Data Encoding Format JSON is a text-based data interchange format as specified in [RFC8259]. The Internet JSON (I-JSON) format defined in [RFC7493] is a strict subset of this, adding restrictions to avoid potentially confusing scenarios (for example, it mandates that an object MUST NOT have two members with the same name). All data sent from the client to the server or from the server to the client (except binary file upload/download) MUST be valid I-JSON according to the RFC and is therefore case sensitive and encoded in UTF-8 [RFC3629]. RFC-7493: https://tools.ietf.org/html/rfc7493 https://tools.ietf.org/html/rfc7493 A quick scan of it says to me it doesn't get all the issues but it does get the biggest one, which IMHO are the ambiguities in what constitutes a "number" in JSON. I-JSON explicitly limits it to 64-bit floats.
- inimino 7y ago
  > annoying to write That's subjective and it seems plenty of people have the opposite opinion. > unsafe to parse Is this based on theoretical concerns or practical experience? > comes with a lot of overhead Keep in mind the alternatives it is largely replacing. XML, text-based email protocols each with their own unique grammars, etc. > I'd prefer s-expressions. Ah yes, the plaintive cry of the lisper in the desert. ;-) I'm afraid in the 60+ years that s-exps have had to catch on in the mainstream, they haven't. I can't explain this result but nonetheless it seems irrational to expect it to suddenly change.
- m463 7y ago
  From a developer perspective, I like json a LOT more than XML-based data. s-expressions seem like an out-of-left-field choice.
- deleted 7y ago
  [deleted]
beefhash 7y ago
I'm really not sure what I feel about the naming. It's one letter off (both on the keyboard and alphabetically) from IMAP. Is this an attempt to coattail the widespread recognition of IMAP or just an extremely unfortunate name collision?
- rhabarba 7y ago
  There is a use case example for replacing IMAP: https://jmap.io/#representation-of-email https://jmap.io/#representation-of-email
- diroussel 7y ago
  The original authors of JMAP were IMAP experts who were increasingly frustrated with IMAP. So not it’s not a coincidence. It just started as IMAP over JSON and then ask they fixed issues and added features it became its own thing.
- lloeki 7y ago
  {a,b{,a},k,z,{,t}c,fi}sh come to mind and it's fine. The name makes perfect sense, given it's an alternative to IMAP built on JSON messages.
- chrismorgan 7y ago
  It has been jested in the Fastmail office that one of the requirements for a spec to be adopted by the IETF is that it have a clever name. Have a look around the mail and calendar specs, for example; there are many amusing names. JMAP is a generic protocol, but its primary motivating example and the thing that has influenced the design most is email, as a replacement for SMTP (between the user and server, not between servers) and IMAP. The name JMAP was chosen quite early on. I + 1 = J. The words the acronym stands for have changed a couple of times along the way!
- jcranmer 7y ago
  JMAP is pretty explicitly meant to replace IMAP. IMAP was originally conceived essentially to allow a very thin client that proxies every user command back to the user store, and it also embeds some reliance on old implementation artifacts (UIDVALIDITY, anyone?). There are a lot of "fun" corner cases--I once found a message that was interpreted 5 different ways by 4 different implementations. (It doesn't help that MIME structure is actually pretty loopy and maps very poorly to how all modern clients view mail messages, resulting in many interesting interpretations). Modern email clients are all structured very differently from the "thin client" perspective--everyone keeps their own copy of the email (metadata) database locally, and use IMAP essentially as a database synchronization protocol, but IMAP is pretty bad at synchronization, at least until the LEMONADE additions came in ~a decade ago. IMAP is especially bad at handling multiple folders correctly, since it assumes that every folder is independent of the other ones, and you can only sit and wait for messages on one folder per connection at a time.
andrewvc 7y ago
The introduction, for those not wanting to click too deeply into the link: The JSON Meta Application Protocol (JMAP) is used for synchronising data, such as mail, calendars, or contacts, between a client and a server. It is optimised for mobile and web environments and aims to provide a consistent interface to different data types. This specification is for the generic mechanism of data synchronisation. Further specifications define the data models for different data types that may be synchronised via JMAP. JMAP is designed to make efficient use of limited network resources. Multiple API calls may be batched in a single request to the server, reducing round trips and improving battery life on mobile devices. Push connections remove the need for polling, and an efficient delta update mechanism ensures a minimum amount of data is transferred. JMAP is designed to be horizontally scalable to a very large number of users. This is facilitated by separate endpoints for users after login, the separation of binary and structured data, and a data model for sharing that does not allow data dependencies between accounts.
chrismorgan 7y ago
For context: RFC 8620 is the JMAP core, just published. It defines an RPC framework, an object synchronisation protocol, and miscellaneous other things like push channels. It has no data model associated with it, beyond handling blobs. The JMAP mail spec, which is what most people will be more interested in, is awaiting final sign-off before publication as RFC 8621. We’re using JMAP in production at Fastmail on our Fastmail and Topicbox products, right through the stack. Both of their web interfaces speak JMAP to the server. Unfortunately at this time it’s not ready for public use: definition of authentication technique is one thing that is deliberately left out of JMAP, and we’re currently using our own system which has no public documentation (and we don’t expose /.well-known/jmap yet either). The intrepid user may be able to figure it out, but we won’t be providing any support for it yet. We use the JMAP mail spec, the draft calendar and contacts specs, and a handful of our own specs for our own data types, also not publicly documented at this time. JMAP is much more complex than the standard sort of simple REST API, but there is cause for all aspects of its complexity; the most notable complexity in it is its record state management, part of it being an object synchronisation protocol; an advantage of that is that robust offline support is feasible, even fairly straightforward. I’m looking forward to working on offline support in the Fastmail web app. I personally think the increased complexity is well worth-while, and I have several products I’ve been sketching out and working on personally (completely unrelated to the world of email or to my employer Fastmail), and all the ones with any kind of client-server API I intend to build atop JMAP, because they’ll benefit heavily from it with respect to features like real-time interface updates and offline support (for web, desktop and command line programs).
- lloeki 7y ago
  I've hacked IMAP long enough to want it thoroughly dead, so I've been following JMAP from afar and waiting for Fastmail to publish this for a long time. Props to the team, and I hope the (publicly) missing parts will come soon enough.
  - chrismorgan 7y ago
    Note that IETF working groups publish drafts all along the way, so there’s https://datatracker.ietf.org/doc/draft-ietf-jmap-mail/ https://datatracker.ietf.org/doc/draft-ietf-jmap-mail/ for the current text of the mail spec, which is probably what will be published.
jorangreef 7y ago
IMAP's killer design flaw was that it imposed a strict UID consistency requirement between replica servers if client sync was to work correctly, i.e. you needed a master-slave setup or Paxos, you couldn't use a true multi-master setup or CRDTs with eventual consistency because of the possibility that clients might miss UIDs. There have been attempts to solve this with even-odd UIDs etc. but those broke down on further analysis. There have also been papers claiming to implement multi-master IMAP servers over CRDTs but these never addressed the strict UID consistency requirement and never tested with actual client implementations for sync correctness. How does JMAP address the distributed multi-master server sync problem, assuming eventual consistency?
- throw0101a 7y ago
  > IMAP's killer design flaw was that it imposed a strict UID consistency requirement between replica servers if client sync was to work correctly That may be a problem for folks running things at the proverbial web scale, but if it's just a Dovecot server handling a couple of dozen or hundred accounts, is it that big of an issue?
  - jorangreef 7y ago
    > is it that big of an issue? No, unless you're designing a protocol to replace IMAP and you want all the proverbial web scale systems to adopt your new protocol. And then again, interoperability and universal adoption is probably something the smaller Dovecot installations would want. Master-master conflict-free eventual consistency systems are easier and safer to operate, there's no nightmare damage from split-brain and things just work, even in the worst scenario. Master-slave failover is notoriously prone to error, even for small deployments.
    - dvdhnt 7y ago
      This isn't on topic of GP, but you seem to have a handle on what this is, so I thought I'd ask... The conversation seems to focus on JMAP as a replacement for IMAP and primarily as an email protocol. The RFC identifies itself as - > a protocol for clients to efficiently query, > fetch, and modify JSON-based data objects, with support for push > notification of changes and fast resynchronisation and for out-of- > band binary data upload/download. It reads like it applies to APIs in general, or is this only for mail/messaging services? edit: I see now, this is the core spec, not specifically the mail spec as referenced in another comment. Thanks. source: https://news.ycombinator.com/item?id=20478227 https://news.ycombinator.com/item?id=20478227
sctb 7y ago
Recent discussions about JMAP in general: https://news.ycombinator.com/item?id=19839104 https://news.ycombinator.com/item?id=19839104 https://news.ycombinator.com/item?id=18996200 https://news.ycombinator.com/item?id=18996200 https://news.ycombinator.com/item?id=18766709 https://news.ycombinator.com/item?id=18766709
felixfbecker 7y ago
Would love to read blog posts about how to use the API to e.g. manage calendar events, and then build some CLI tools around that!
- nmjenkins 7y ago
  The JMAP Calendar spec is the next one being standardised. There's a first draft at https://tools.ietf.org/html/draft-ietf-jmap-calendars-00 https://tools.ietf.org/html/draft-ietf-jmap-calendars-00 but quite a lot more to be added still. Expect a final standard probably first half of next year.
dragonwriter 7y ago
Is there an extensible JMAP core (what this RFC presents) server framework available? I know there are JMAP mail/contacts/calendar server implementations, but it'd be interesting to have a fairly clean starting point for other potential applications...
deleted 7y ago
[deleted]

160 points by kuba-orlik 7y ago

JSON is an incredibly inefficient format for shareable data: it is annoying to write, unsafe to parse and it even comes with a lot of overhead (colons, quotes, brackets and the like). I'd prefer s-expressions.

the8472 7y ago
CBOR is a better wire format in all aspects except readability when all you have is a text viewer. It's a superset replacement when it comes to supported datatypes
- rhabarba 7y ago
  Looks like a worse (= less widely supported) msgpack to me...?
  - piotrkubisa 7y ago
    It looks it performs way better than msgpack in following benchmarks [0]. However, it's true that it's not very popular - personally, I rarely see any software that utilize CBOR. [0]: http://tutorials.jenkov.com/ion/ion-performance-benchmarks.html http://tutorials.jenkov.com/ion/ion-performance-benchmarks.h...
heinrichhartman 7y ago
Have you ever seen s-expressions being used as data exchange format in the wild? (outside of the LISP/emacs world) Not sure there is a decent standard spec and a good Java (say) parser around. Is there?
- rhabarba 7y ago
  JSON is mostly irrelevant outside the JavaScript world as well. (Don't quote me please.) There are no RFCs for s-expressions, but that's fixable, I guess. And yes - there are a few Java parsers for them.
  - diroussel 7y ago
    What about all those java, ruby, python and C# services that expose JSON based REST services. JSON is widely used for network API, config files and data stores.
    - rhabarba 7y ago
      "Too many people already do the wrong thing" is not a good excuse to stick with the wrong thing.
      - afiori 7y ago
        But you have yet to say why JSON is worse than your preferred alternative. I agree that it is not perfect (for many use cases I prefer TOML personally), but I cannot see how it is worse that S-expressions (except for the trailing comma issue, that is a sin in itself)
      - dragonwriter 7y ago
        No, but that's not the claim: the claim was JSON is irrelevant outside of JavaScript, which is clearly false given all the use in non-JS environments. Whether it's the “wrong thing” is another issue, though I've yet to see a standard of “wrong” proposed in the thread for which that is accurate.
      - geofft 7y ago
        It is not the wrong thing. You were arguing that it was wrong because it was rarely used; you have yet to prove your point that it is in any way wrong. And, perhaps, all those people know better than you.
      - heinrichhartman 7y ago
- bb010g 7y ago
  edn's pretty alright. https://github.com/edn-format/edn https://github.com/edn-format/edn Here's a Java parser: https://github.com/bpsm/edn-java https://github.com/bpsm/edn-java
  - heinrichhartman 7y ago
    very cool!
rlonstein 7y ago
The obvious defense is that it's already in common use and widely (if unevenly) supported.
afiori 7y ago
Why unsafe to parse? > it even comes with a lot of overhead (colons, quotes, brackets and the like). according to Sqlite devs the that overhead is not much worse that for a binary encoding. quoting: https://www.sqlite.org/json1.html https://www.sqlite.org/json1.html > The json1 extension does not (currently) support a binary encoding of JSON. Experiments have been unable to find a binary encoding that is significantly smaller or faster than a plain text encoding. (The present implementation parses JSON text at over 1 GB/s.)
- rhabarba 7y ago
  > Why unsafe to parse? Try to write a parser for an arbitrarily complex and nested JSON object without running into runtime errors even after the fourth-or-so revision. There are too many "yes buts" in the living standard. > that overhead is not much worse It _is_ much worse than the two-byte "overhead" of one s-exp node.
  - tonyg 7y ago
    > It _is_ much worse than the two-byte "overhead" of one s-exp node. I don't believe you. Atoms are roughly the same, and compounds, well: [1,2,3] vs (1 2 3) {"a":1,"b":2} vs (("a"1)("b"2)) I suppose if you have a lot of booleans you might save a little by using T/NIL or #t/#f, but otherwise it's more or less a wash. (You might suggest plists, ("a"1"b"2), for dictionaries, in which case you save a whopping two bytes per entry. But I'd choose the nested-list format over plists, if I were encoding a dictionary into an S-expression.)
    - viraptor 7y ago
      > you save a whopping two bytes per entry Which pretty much disappears when compressed anyway.
  - lloeki 7y ago
    In specific contexts such as this one you don't need arbitrarily complex JSON, you just need to validate that the JSON structure respects not just JSON itself but also the protocol specification at hand while you parse, and bail out when it doesn't, so you can have a dedicated, fast parser tailored for the use case, instead of first parsing general JSON then validate its content. Anyway, for being a Fasmail user, JMAP rips IMAP to shreds in terms of performance. I've been following JMAP from afar and waiting for Fastmail to publish this for a long time.
chrismorgan 7y ago
I’m a developer at Fastmail, and I’m going to bite. Personally, I prefer binary formats where feasible, because I like pursuing efficiency to extremes, and write all my own code in Rust. Professionally, I happen to write mostly JavaScript, and no Rust at all, but we do still focus on performance at Fastmail. (It’s in the name!) If you’re comparing JSON to s-expressions, it’s a wash in efficiency. They’re both “inefficient”, both representing generic data structures in regular text. They use slightly different punctuation, but that’s about it. The differences are negligible. But when it comes to support, JSON is well-supported in all mainstream languages, while s-expressions have no central specification on precise syntax, and you’ll need to do your own thing. One of the deliberate design advantages of JMAP over IMAP is doing away with custom parsers, and using the standard tools people are familiar with, HTTPS and JSON. When talking efficiency, you might then ask why not to use Cap’n Proto, protobufs or one of those sorts of things. Due to the need for extensibility (e.g. in Fastmail we add a few custom fields to JMAP core and mail data structures), rigid structures are unsuitable; you do need arbitrary key-value arrangement, despite it being larger over the wire. All in all, JSON is a robust choice for just about any API. Fun fact: at Fastmail we compress both request and response, even using a custom dictionary primed on real JMAP stuff, to improve the compression ratio. I understand standard DEFLATE was about equivalent in size to CBOR (a binary representation of JSON), and the 2KB dictionary brought it down even further, can’t remember how much it was off the top of my head. Neil Jenkins gave a talk about this and related matters last month, https://www.webdirections.org/code/speakers/neil-jenkins.php https://www.webdirections.org/code/speakers/neil-jenkins.php, no public video sadly. I’ve thought about writing a blog post about it all. An unfortunate side-effect of this compression is that you can’t inspect traffic in the network panel of the dev tools; you can add ?compress=0 to disable it, if you’re an interested Fastmail user.
- nerdponx 7y ago
  Is Messagepack not a useful alternative to JSON in your opinion?
  - chrismorgan 7y ago
    MessagePack and CBOR are equivalent. I haven’t ever had much to do with them (never used them, only known about them), but I’m inclined to think that simple compression achieves between “roughly as much” and “more” in general, and leaves you with the industry standard of JSON underneath. Do not consider me an expert by any means, and I’m fairly ambivalent on your question; but I don’t see a particularly useful niche for them between plain JSON and a binary representation matching a predefined static schema, all potentially with compression.
verisimilitudes 7y ago
I agree. I'm in the process of writing an efficient JSON parser, since using the simple approach is too slow, I find, and because a database I want to introspect on uses it. To process JSON, you must inspect every character. Douglas Crockford brags about how smart he is for disallowing comments, but there's whitespace allowed everywhere. I actually have a (I think) rather novel approach and there's some potential for little optimizations here and there, but it disgusts me that I'm doing this at all. The optimizations popular, shitty formats like JSON receives are purely wasteful. It's suitable, though. The WWW wasn't made right from the start and there's always an idiot somewhere looking to add more shit to the pile. Not only does JavaScript exist, but it morphed into this ugly, inefficient, very poor ''format''. We don't have Lisp machines, but new processors are getting instructions to cope with JavaScript idiocy. A numerical (binary) format is clearly best. It's small, random-access when properly designed, and efficient, but hey, everything's being pushed into port 80 so why not push all data into JSON? Now, the rest of this message is in response to @chrismorgan from an organization pushing this filth. >Personally, I prefer binary formats where feasible, because I like pursuing efficiency to extremes, and write all my own code in Rust. We agree on that, then, but I prefer to write that Rust is just the effeminate man's Ada. >If you’re comparing JSON to s-expressions, it’s a wash in efficiency. They’re both “inefficient”, both representing generic data structures in regular text. They use slightly different punctuation, but that’s about it. The differences are negligible. This is wrong. Firstly, we should clarify what is meant by S-expressions. Let's assume the S-expressions have lists, strings, symbols, and numbers. That's already simpler than JSON. You're not hampered by JavaScript's floating point nonsense, either. There's fewer edge cases and less escaping. JSON is, at best, a worse-is-better form of S-expressions. >But when it comes to support, JSON is well-supported in all mainstream languages, while s-expressions have no central specification on precise syntax, and you’ll need to do your own thing. One of the deliberate design advantages of JMAP over IMAP is doing away with custom parsers, and using the standard tools people are familiar with, HTTPS and JSON. So, this ties into what I wrote before I even read your post, exactly. Cretins continue to push everything into the WWW protocols and act like their garbage is standard, simple, acceptable, or wanted; it's only technically standard, but it's a bad standard. I'd bet my foot you also like ''DNS over HTTPS''. A numerical (binary) format should be used, because it's the best option and people can be expected to bother writing an acceptor for such important data. There are plenty of languages that can easily accept a numerical format, but not JSON; think of, say, Forth. Oh, but everyone writes in Rust and JavaScript nowadays, right. >When talking efficiency, you might then ask why not to use Cap’n Proto, protobufs or one of those sorts of things. Due to the need for extensibility (e.g. in Fastmail we add a few custom fields to JMAP core and mail data structures), rigid structures are unsuitable; you do need arbitrary key-value arrangement, despite it being larger over the wire. All of this is unnecessary. Email doesn't need this drivel. >All in all, JSON is a robust choice for just about any API. Of course; forty years ago, UNIX was the solution for every problem. The WWW and JavaScript are nothing more than a continuation of this worse-is-better filth. >Fun fact: at Fastmail we compress both request and response, even using a custom dictionary primed on real JMAP stuff, to improve the compression ratio. I understand standard DEFLATE was about equivalent in size to CBOR (a binary representation of JSON), and the 2KB dictionary brought it down even further, can’t remember how much it was off the top of my head. I hate how everyone's response to inefficiency is ''Compress it!'' and ''Cache it!''. There's no reason it should be so inefficient to start with.
jerf 7y ago
I don't know if this addresses everything, but from the spec: 1.5. JSON as the Data Encoding Format JSON is a text-based data interchange format as specified in [RFC8259]. The Internet JSON (I-JSON) format defined in [RFC7493] is a strict subset of this, adding restrictions to avoid potentially confusing scenarios (for example, it mandates that an object MUST NOT have two members with the same name). All data sent from the client to the server or from the server to the client (except binary file upload/download) MUST be valid I-JSON according to the RFC and is therefore case sensitive and encoded in UTF-8 [RFC3629]. RFC-7493: https://tools.ietf.org/html/rfc7493 https://tools.ietf.org/html/rfc7493 A quick scan of it says to me it doesn't get all the issues but it does get the biggest one, which IMHO are the ambiguities in what constitutes a "number" in JSON. I-JSON explicitly limits it to 64-bit floats.
inimino 7y ago
> annoying to write That's subjective and it seems plenty of people have the opposite opinion. > unsafe to parse Is this based on theoretical concerns or practical experience? > comes with a lot of overhead Keep in mind the alternatives it is largely replacing. XML, text-based email protocols each with their own unique grammars, etc. > I'd prefer s-expressions. Ah yes, the plaintive cry of the lisper in the desert. ;-) I'm afraid in the 60+ years that s-exps have had to catch on in the mainstream, they haven't. I can't explain this result but nonetheless it seems irrational to expect it to suddenly change.
m463 7y ago
From a developer perspective, I like json a LOT more than XML-based data. s-expressions seem like an out-of-left-field choice.
deleted 7y ago
[deleted]

beefhash 7y ago

I'm really not sure what I feel about the naming. It's one letter off (both on the keyboard and alphabetically) from IMAP. Is this an attempt to coattail the widespread recognition of IMAP or just an extremely unfortunate name collision?

rhabarba 7y ago
There is a use case example for replacing IMAP: https://jmap.io/#representation-of-email https://jmap.io/#representation-of-email
diroussel 7y ago
The original authors of JMAP were IMAP experts who were increasingly frustrated with IMAP. So not it’s not a coincidence. It just started as IMAP over JSON and then ask they fixed issues and added features it became its own thing.
lloeki 7y ago
{a,b{,a},k,z,{,t}c,fi}sh come to mind and it's fine. The name makes perfect sense, given it's an alternative to IMAP built on JSON messages.
chrismorgan 7y ago
It has been jested in the Fastmail office that one of the requirements for a spec to be adopted by the IETF is that it have a clever name. Have a look around the mail and calendar specs, for example; there are many amusing names. JMAP is a generic protocol, but its primary motivating example and the thing that has influenced the design most is email, as a replacement for SMTP (between the user and server, not between servers) and IMAP. The name JMAP was chosen quite early on. I + 1 = J. The words the acronym stands for have changed a couple of times along the way!
jcranmer 7y ago
JMAP is pretty explicitly meant to replace IMAP. IMAP was originally conceived essentially to allow a very thin client that proxies every user command back to the user store, and it also embeds some reliance on old implementation artifacts (UIDVALIDITY, anyone?). There are a lot of "fun" corner cases--I once found a message that was interpreted 5 different ways by 4 different implementations. (It doesn't help that MIME structure is actually pretty loopy and maps very poorly to how all modern clients view mail messages, resulting in many interesting interpretations). Modern email clients are all structured very differently from the "thin client" perspective--everyone keeps their own copy of the email (metadata) database locally, and use IMAP essentially as a database synchronization protocol, but IMAP is pretty bad at synchronization, at least until the LEMONADE additions came in ~a decade ago. IMAP is especially bad at handling multiple folders correctly, since it assumes that every folder is independent of the other ones, and you can only sit and wait for messages on one folder per connection at a time.

andrewvc 7y ago

The introduction, for those not wanting to click too deeply into the link: The JSON Meta Application Protocol (JMAP) is used for synchronising data, such as mail, calendars, or contacts, between a client and a server. It is optimised for mobile and web environments and aims to provide a consistent interface to different data types. This specification is for the generic mechanism of data synchronisation. Further specifications define the data models for different data types that may be synchronised via JMAP. JMAP is designed to make efficient use of limited network resources. Multiple API calls may be batched in a single request to the server, reducing round trips and improving battery life on mobile devices. Push connections remove the need for polling, and an efficient delta update mechanism ensures a minimum amount of data is transferred. JMAP is designed to be horizontally scalable to a very large number of users. This is facilitated by separate endpoints for users after login, the separation of binary and structured data, and a data model for sharing that does not allow data dependencies between accounts.

chrismorgan 7y ago

For context: RFC 8620 is the JMAP core, just published. It defines an RPC framework, an object synchronisation protocol, and miscellaneous other things like push channels. It has no data model associated with it, beyond handling blobs. The JMAP mail spec, which is what most people will be more interested in, is awaiting final sign-off before publication as RFC 8621. We’re using JMAP in production at Fastmail on our Fastmail and Topicbox products, right through the stack. Both of their web interfaces speak JMAP to the server. Unfortunately at this time it’s not ready for public use: definition of authentication technique is one thing that is deliberately left out of JMAP, and we’re currently using our own system which has no public documentation (and we don’t expose /.well-known/jmap yet either). The intrepid user may be able to figure it out, but we won’t be providing any support for it yet. We use the JMAP mail spec, the draft calendar and contacts specs, and a handful of our own specs for our own data types, also not publicly documented at this time. JMAP is much more complex than the standard sort of simple REST API, but there is cause for all aspects of its complexity; the most notable complexity in it is its record state management, part of it being an object synchronisation protocol; an advantage of that is that robust offline support is feasible, even fairly straightforward. I’m looking forward to working on offline support in the Fastmail web app. I personally think the increased complexity is well worth-while, and I have several products I’ve been sketching out and working on personally (completely unrelated to the world of email or to my employer Fastmail), and all the ones with any kind of client-server API I intend to build atop JMAP, because they’ll benefit heavily from it with respect to features like real-time interface updates and offline support (for web, desktop and command line programs).

lloeki 7y ago
I've hacked IMAP long enough to want it thoroughly dead, so I've been following JMAP from afar and waiting for Fastmail to publish this for a long time. Props to the team, and I hope the (publicly) missing parts will come soon enough.
- chrismorgan 7y ago
  Note that IETF working groups publish drafts all along the way, so there’s https://datatracker.ietf.org/doc/draft-ietf-jmap-mail/ https://datatracker.ietf.org/doc/draft-ietf-jmap-mail/ for the current text of the mail spec, which is probably what will be published.

jorangreef 7y ago

IMAP's killer design flaw was that it imposed a strict UID consistency requirement between replica servers if client sync was to work correctly, i.e. you needed a master-slave setup or Paxos, you couldn't use a true multi-master setup or CRDTs with eventual consistency because of the possibility that clients might miss UIDs. There have been attempts to solve this with even-odd UIDs etc. but those broke down on further analysis. There have also been papers claiming to implement multi-master IMAP servers over CRDTs but these never addressed the strict UID consistency requirement and never tested with actual client implementations for sync correctness. How does JMAP address the distributed multi-master server sync problem, assuming eventual consistency?

throw0101a 7y ago
> IMAP's killer design flaw was that it imposed a strict UID consistency requirement between replica servers if client sync was to work correctly That may be a problem for folks running things at the proverbial web scale, but if it's just a Dovecot server handling a couple of dozen or hundred accounts, is it that big of an issue?
- jorangreef 7y ago
  > is it that big of an issue? No, unless you're designing a protocol to replace IMAP and you want all the proverbial web scale systems to adopt your new protocol. And then again, interoperability and universal adoption is probably something the smaller Dovecot installations would want. Master-master conflict-free eventual consistency systems are easier and safer to operate, there's no nightmare damage from split-brain and things just work, even in the worst scenario. Master-slave failover is notoriously prone to error, even for small deployments.
  - dvdhnt 7y ago
    This isn't on topic of GP, but you seem to have a handle on what this is, so I thought I'd ask... The conversation seems to focus on JMAP as a replacement for IMAP and primarily as an email protocol. The RFC identifies itself as - > a protocol for clients to efficiently query, > fetch, and modify JSON-based data objects, with support for push > notification of changes and fast resynchronisation and for out-of- > band binary data upload/download. It reads like it applies to APIs in general, or is this only for mail/messaging services? edit: I see now, this is the core spec, not specifically the mail spec as referenced in another comment. Thanks. source: https://news.ycombinator.com/item?id=20478227 https://news.ycombinator.com/item?id=20478227

sctb 7y ago

Recent discussions about JMAP in general: https://news.ycombinator.com/item?id=19839104 https://news.ycombinator.com/item?id=19839104 https://news.ycombinator.com/item?id=18996200 https://news.ycombinator.com/item?id=18996200 https://news.ycombinator.com/item?id=18766709 https://news.ycombinator.com/item?id=18766709

felixfbecker 7y ago

Would love to read blog posts about how to use the API to e.g. manage calendar events, and then build some CLI tools around that!

nmjenkins 7y ago
The JMAP Calendar spec is the next one being standardised. There's a first draft at https://tools.ietf.org/html/draft-ietf-jmap-calendars-00 https://tools.ietf.org/html/draft-ietf-jmap-calendars-00 but quite a lot more to be added still. Expect a final standard probably first half of next year.

dragonwriter 7y ago

Is there an extensible JMAP core (what this RFC presents) server framework available? I know there are JMAP mail/contacts/calendar server implementations, but it'd be interesting to have a fairly clean starting point for other potential applications...

deleted 7y ago

[deleted]