10 ms·

Progress on JMAP, better standard for synchronising mail, calendars and contacts

181 points by carey 11y ago

brudgers 11y ago
JMAP homepage: http://jmap.io/ http://jmap.io/
2bluesc 11y ago
Reading about JMAP last year during their holiday series of blog posts persuaded me to jump on-board for their mail service and to support the evolution of JMAP. No regrets!
zx2c4 11y ago
This is great to hear. I'm glad the project is progressing. For those not in the know, JMAP is a protocol for mail that, unlike IMAP, has threads and labels built-in at its core. In other words, Gmail without Gmail. And it's about time somebody came along and built something sane for this. There have been a lot of valiant attempts at this. Thunderbird has a few buggy extensions that attempt it. Notmuch is a very well written library and command line MUA with a very fast on-disk storage format (with ncurses, emacs, and vim interfaces, as well as neat FUSE file system toys), but it is not a protocol, and therefore does not scale well beyond the limited set of MUAs for it that mostly need a shell or a flimsy web app. Mailpile is an attempt at making a Web 2.0 style Gmail clone, but it's only an MUA and the on-disk format is flimsy at best. JMAP defines the protocol first. Then, multiple people can take a stab at the most sane implementation. I like this approach quite a bit.
- teacup50 11y ago
  Why not extend IMAP? It is by definition highly extensible, and unlike JMAP, it's not wedged into complex poorly suited protocols like HTTP. Clients leave IMAP ports open indefinately; IMAP IDLE provides push e-mail over this mechanism. Reading about JMAP, this just seems like adding unnecessary web overhead and complexity to an already complex problem space.
  - jlgaddis 11y ago
    FastMail's "why" is better explained in a post from last year, "JMAP -- A better way to email" [0]. [0]: https://blog.fastmail.com/2014/12/23/jmap-a-better-way-to-email/ https://blog.fastmail.com/2014/12/23/jmap-a-better-way-to-em...
  - wtetzner 11y ago
    Well, HTTP/2 with server push should handle the "keep a connection open" and "let the server push things to the client" issues.
  - rewqfdsa 11y ago
    Because programmers graduating today literally can't imagine a world full of non-HTTP protocols.
  - _pmf_ 11y ago
    > It is by definition highly extensible, and unlike JMAP, it's not wedged into complex poorly suited protocols like HTTP. Why building on mature concepts that have proven themselves in practice over several decades? It's much more in line with the current web culture if smart rockstar developers half-assedly cobble together a RESTful shit fest.
  - buster 11y ago
    It's just a smart move to have the API your webmail uses be a "standard"... I don't understand why i should implement Fastmails web API, when there are several major protocols that are supported by a huge ecosystem already: IMAP, CalDAV, CardDAV.
  - wnoise 11y ago
    While I think extending IMAP is the better solution, it is already too complex, as many clients and servers treat things just slightly different enough to be really annoying.
  - brongondwana 11y ago
    So we do that too - indeed I've done most of the work getting the Cyrus IMAPd server support of IMAP to the level of compliance that it has today, and am still involved in the standardisation battles. But I see from the responses to this comment that it's easier to make snarky comments on the internet than to understand where IMAP just doesn't do what we needed from a protocol, and why we designed JMAP.
LeoPanthera 11y ago
This might be a naive question, but I don't know any better: Why can't your mailbox be represented on a server as a directory, perhaps like Maildir format, and then synchronised to your device using a directory sync tool, such as rsync? Changes, or online access, could be done using any standard remote file access method, such as sftp. Are there downsides to this approach?
- nmjenkins 11y ago
  This will be way too slow, and is not at all feasible on a device like a mobile phone where you want only the recent email cached locally, but want to be able to stream on demand the rest with minimal latency (so it appears to the user as though it's all loaded locally under ideal network conditions). The server needs to be able to sort the messages, calculate threading, unread counts etc.
- jesstaa 11y ago
  You still need an index of your mail for searches and extra metadata for tags, threading, unread status etc. So you need a protocol for requesting this search and a format what how the client can use the result of that search to access the matching emails. You need to be able to push email status changes out to connected devices, I don't want an alert for an email on phone if I've already read it on my laptop. You need decisions around how attachments are stored. Most of the time you'll want to store the attachments separately from the text content of the email so clients don't have to download the attachments to access the text, so you'll need to decide on how this location would be communicated to a client. Plain files are rather bad for multi-user access, if you have multiple users/devices all accessing the same email they're going to need some protocol for synchronization so you don't get data corruption or lost changes.
  - viperscape 11y ago
    It would be interesting to think about email and sync using git on the backend.
    - 11y ago
protomyth 11y ago
Sadly, I don't see a lot of uptake on a new protocol unless Microsoft supports it in Outlook or someone develops a plug-in. Also, the mobile situation is going to be a tough haul.
SwellJoe 11y ago
I'm pretty excited about JMAP. We've been planning an overhaul of Usermin, our web-based mail client, and now that we have a UI developer, we'll be embarking on more ambitious features, like threading, off-line abilities, etc. Having threading handled automatically is already reason enough to use this, as it's a much harder problem than it seems at first glance to solve in a consistent way across many clients and servers. The fact that it also handles calendars and contacts is icing on the cake.
djsumdog 11y ago
Interesting. I use Radicalie for Cal/CardDav. Maybe if I get some bandwidth, I could look at potentially contribution a JMAP extension.
nemoniac 11y ago
What clients/apps currently speak JMAP?
imron 11y ago
It's great to see a company building technology and open protocols without walled gardens and lockin.
StavrosK 11y ago
My favorite part about this is that one can implement and start using it at their discretion (i.e. it doesn't require the other server to also speak JMAP). That way I can have a mobile app that speaks JMAP and an IMAP-to-JMAP bridge that syncs my Gmail to it. Does anyone know of such a bridge? I would love to run one for myself.
- brongondwana 11y ago
  https://proxy.jmap.io/ https://proxy.jmap.io/ - it contains links to the source code if you'd like to run it yourself. It's just a few thousand lines of perl, and I wrote it in two groups of about 2 weeks each time (about 6 months apart) - and each time very much part time amongst other things. It's actually not that hard if you already have IMAP, CalDAV and CardDAV client libraries.
Touche 11y ago
Any news on whether the big mail providers have an interest (Microsoft, Gmail, Yahoo, etc.)?
Qantourisc 11y ago
If you want to make this a standard I recommend making a RFC for JAMP. (Then again they have docs, this would just make it seem more official.)
marknadal 11y ago
There seems to be a lot of interest in sync related protocols in this thread and dismissal of JMAP as a result. This is bad, because the current state if synchronization protocols still kinda sucks - full disclosure, I work on http://GitHub.com/amark/gun http://GitHub.com/amark/gun which is a JavaScript based syncing database. Getting synch right in p2p or federated systems (email) entails more detail than centralized ones. So any work on synchronization should be seen as favorable compared to how bad other protocols are for sync.
nine_k 11y ago
What somehow bothers me is that the new protocol is all-in-one. Instead of building on / integrating with CalDAV / CardDAV and being modular, they are rolling everything in the same protocol. This is possibly convenient for a protocol controlled by one commercial entity, but why not just use Outlook's then? Piggybacking it over HTTP also looks a bit strange, but probably helps corporate firewall penetration.

181 points by carey 11y ago

JMAP homepage: http://jmap.io/ http://jmap.io/

Reading about JMAP last year during their holiday series of blog posts persuaded me to jump on-board for their mail service and to support the evolution of JMAP. No regrets!

zx2c4 11y ago

This is great to hear. I'm glad the project is progressing. For those not in the know, JMAP is a protocol for mail that, unlike IMAP, has threads and labels built-in at its core. In other words, Gmail without Gmail. And it's about time somebody came along and built something sane for this. There have been a lot of valiant attempts at this. Thunderbird has a few buggy extensions that attempt it. Notmuch is a very well written library and command line MUA with a very fast on-disk storage format (with ncurses, emacs, and vim interfaces, as well as neat FUSE file system toys), but it is not a protocol, and therefore does not scale well beyond the limited set of MUAs for it that mostly need a shell or a flimsy web app. Mailpile is an attempt at making a Web 2.0 style Gmail clone, but it's only an MUA and the on-disk format is flimsy at best. JMAP defines the protocol first. Then, multiple people can take a stab at the most sane implementation. I like this approach quite a bit.

teacup50 11y ago
Why not extend IMAP? It is by definition highly extensible, and unlike JMAP, it's not wedged into complex poorly suited protocols like HTTP. Clients leave IMAP ports open indefinately; IMAP IDLE provides push e-mail over this mechanism. Reading about JMAP, this just seems like adding unnecessary web overhead and complexity to an already complex problem space.
- jlgaddis 11y ago
  FastMail's "why" is better explained in a post from last year, "JMAP -- A better way to email" [0]. [0]: https://blog.fastmail.com/2014/12/23/jmap-a-better-way-to-email/ https://blog.fastmail.com/2014/12/23/jmap-a-better-way-to-em...
- wtetzner 11y ago
  Well, HTTP/2 with server push should handle the "keep a connection open" and "let the server push things to the client" issues.
- rewqfdsa 11y ago
  Because programmers graduating today literally can't imagine a world full of non-HTTP protocols.
- _pmf_ 11y ago
  > It is by definition highly extensible, and unlike JMAP, it's not wedged into complex poorly suited protocols like HTTP. Why building on mature concepts that have proven themselves in practice over several decades? It's much more in line with the current web culture if smart rockstar developers half-assedly cobble together a RESTful shit fest.
- buster 11y ago
  It's just a smart move to have the API your webmail uses be a "standard"... I don't understand why i should implement Fastmails web API, when there are several major protocols that are supported by a huge ecosystem already: IMAP, CalDAV, CardDAV.
- wnoise 11y ago
  While I think extending IMAP is the better solution, it is already too complex, as many clients and servers treat things just slightly different enough to be really annoying.
- brongondwana 11y ago
  So we do that too - indeed I've done most of the work getting the Cyrus IMAPd server support of IMAP to the level of compliance that it has today, and am still involved in the standardisation battles. But I see from the responses to this comment that it's easier to make snarky comments on the internet than to understand where IMAP just doesn't do what we needed from a protocol, and why we designed JMAP.

LeoPanthera 11y ago

This might be a naive question, but I don't know any better: Why can't your mailbox be represented on a server as a directory, perhaps like Maildir format, and then synchronised to your device using a directory sync tool, such as rsync? Changes, or online access, could be done using any standard remote file access method, such as sftp. Are there downsides to this approach?

nmjenkins 11y ago
This will be way too slow, and is not at all feasible on a device like a mobile phone where you want only the recent email cached locally, but want to be able to stream on demand the rest with minimal latency (so it appears to the user as though it's all loaded locally under ideal network conditions). The server needs to be able to sort the messages, calculate threading, unread counts etc.
jesstaa 11y ago
You still need an index of your mail for searches and extra metadata for tags, threading, unread status etc. So you need a protocol for requesting this search and a format what how the client can use the result of that search to access the matching emails. You need to be able to push email status changes out to connected devices, I don't want an alert for an email on phone if I've already read it on my laptop. You need decisions around how attachments are stored. Most of the time you'll want to store the attachments separately from the text content of the email so clients don't have to download the attachments to access the text, so you'll need to decide on how this location would be communicated to a client. Plain files are rather bad for multi-user access, if you have multiple users/devices all accessing the same email they're going to need some protocol for synchronization so you don't get data corruption or lost changes.
- viperscape 11y ago
  It would be interesting to think about email and sync using git on the backend.
  - 11y ago

protomyth 11y ago

Sadly, I don't see a lot of uptake on a new protocol unless Microsoft supports it in Outlook or someone develops a plug-in. Also, the mobile situation is going to be a tough haul.

SwellJoe 11y ago

I'm pretty excited about JMAP. We've been planning an overhaul of Usermin, our web-based mail client, and now that we have a UI developer, we'll be embarking on more ambitious features, like threading, off-line abilities, etc. Having threading handled automatically is already reason enough to use this, as it's a much harder problem than it seems at first glance to solve in a consistent way across many clients and servers. The fact that it also handles calendars and contacts is icing on the cake.

djsumdog 11y ago

Interesting. I use Radicalie for Cal/CardDav. Maybe if I get some bandwidth, I could look at potentially contribution a JMAP extension.

nemoniac 11y ago

What clients/apps currently speak JMAP?

imron 11y ago

It's great to see a company building technology and open protocols without walled gardens and lockin.

StavrosK 11y ago

My favorite part about this is that one can implement and start using it at their discretion (i.e. it doesn't require the other server to also speak JMAP). That way I can have a mobile app that speaks JMAP and an IMAP-to-JMAP bridge that syncs my Gmail to it. Does anyone know of such a bridge? I would love to run one for myself.

brongondwana 11y ago
https://proxy.jmap.io/ https://proxy.jmap.io/ - it contains links to the source code if you'd like to run it yourself. It's just a few thousand lines of perl, and I wrote it in two groups of about 2 weeks each time (about 6 months apart) - and each time very much part time amongst other things. It's actually not that hard if you already have IMAP, CalDAV and CardDAV client libraries.

Touche 11y ago

Any news on whether the big mail providers have an interest (Microsoft, Gmail, Yahoo, etc.)?

Qantourisc 11y ago

If you want to make this a standard I recommend making a RFC for JAMP. (Then again they have docs, this would just make it seem more official.)

marknadal 11y ago

There seems to be a lot of interest in sync related protocols in this thread and dismissal of JMAP as a result. This is bad, because the current state if synchronization protocols still kinda sucks - full disclosure, I work on http://GitHub.com/amark/gun http://GitHub.com/amark/gun which is a JavaScript based syncing database. Getting synch right in p2p or federated systems (email) entails more detail than centralized ones. So any work on synchronization should be seen as favorable compared to how bad other protocols are for sync.

nine_k 11y ago

What somehow bothers me is that the new protocol is all-in-one. Instead of building on / integrating with CalDAV / CardDAV and being modular, they are rolling everything in the same protocol. This is possibly convenient for a protocol controlled by one commercial entity, but why not just use Outlook's then? Piggybacking it over HTTP also looks a bit strange, but probably helps corporate firewall penetration.