7 ms·

JMAP: A better way to email (2014)

253 points by robert_foss 9y ago

LeoPanthera 9y ago
More recent info at http://jmap.io http://jmap.io
irl_ 9y ago
If you want to try out JMAP on a provider that is not FastMail, you use a proxy: https://proxy.jmap.io/ https://proxy.jmap.io/ I've not seen it deployed anywhere else yet as a production thing, and I don't know how much progress has been made on standardisation of it, but it does seem to offer vast improvements over IMAP.
- lemming 9y ago
  Do Fastmail themselves let you use jmap to connect now? They didn't a while back but I haven't checked recently.
  - Nemo157 9y ago
    I was looking for a way a week ago and couldn't find it, but maybe it's just hidden really well.
  - brongondwana 9y ago
    Not yet, though it's being used internally for a few things already. The ietf working group and the Cyrus IMAP source code are the most up to date right now. I'll be updating the proxy (perl, open source) soon.
- brongondwana 9y ago
  The proxy is a pretty crappy Perl JMAP server layered on top of a middling quality IMAP sync layer. The CalDAV/CardDAV bits are a bit better because it has nicer synchronisation primatives, but it's still code I slapped together over a couple of weeks of evenings. It's fine for playing with, but I wouldn't try to use it for large mailboxes. There's not much optimisation in there, it's doing pretty basic sqlite queries against a fairly normalised database, so there's no denomalisation for efficiency.
mostafah 9y ago
Email protocols are very old and network APIs have improved a lot during the last two decades. So, when FastMail (I’m a happy customer) introduced JMAP I was excited about it. But even then I didn’t anticipate much success for it. Big email providers (Gmail, iCloud, Outlook, …) don’t care about these things. And it’s a shame that email providers and clients are still wrestling with ancient protocols to provide a decent experience for saving drafts, search, folders/labels, and other basic things.
- StavrosK 9y ago
  I hope it gains more traction. In the mean time, I've switched, to Fastmail, and I couldn't be happier. It really is much faster than Gmail, I was very surprised because I didn't think email could be so fast. I guess Gmail had become the new normal.
  - jpgvm 9y ago
    Except for search really. I love FastMail but they need to work on the search speed. Atleast it's faster than Outlook.com though, that search speed is beyond unacceptable.
    - brongondwana 9y ago
      If you open a support ticket I can look at your search issues, ask for Bron. It shouldn't be slow unless you're deliberately bypassing the Xapian indexes by using substr searches.
      - Semaphor 9y ago
        How long is search supposed to take? Entering something common (like "gmail") takes around 5 seconds for me with 32k results. My old google mail account takes 1.5s to display the first page.
        mike-cardwell 9y ago
nmjenkins 9y ago
Just as an update since then: a JMAP working group was chartered at the IETF a few months ago, and is working towards bringing the spec to RFC status: https://datatracker.ietf.org/wg/jmap/about/ https://datatracker.ietf.org/wg/jmap/about/
- brongondwana 9y ago
  Thanks for posting that, was about to say the same thing. Am one of the co-chairs. AMA.
deleted 9y ago
[deleted]
Mojah 9y ago
Happy to see this get more attention again. I highlighted it in last week's cron.weekly [1] too. I'm still unsure of the viability at this point, but time will tell. [1] https://www.cronweekly.com/issue-78/ https://www.cronweekly.com/issue-78/
shmerl 9y ago
Does this trend of replacing TCP/IP with HTTP start being more common, or it just looks to me that way? I get a feeling that HTTP is stretched too much outside of what it's designed for, and it results in messy design.
- brongondwana 9y ago
  There are benefits to using http as a substrate, like request bundling on mobile to reduce radio and battery usage. JMAP is transport agnostic in its core form, but json for structure over https for transport is the current best choice for simplicity and developer friendliness
- vbezhenar 9y ago
  Most important thing is JavaScript from browser can use only HTTP, so if you want to build web-based mail client, you won't have to build proxy HTTP service, you can just implement protocol on client. Another important thing is proxy, many organizations don't allow anything but HTTP, so it's easier to just use it. HTTP is not that bad, anyway. You don't have to use every quirk.
  - djsumdog 9y ago
    I agree with your comments about web-based e-mail clients, but: > many organizations don't allow anything but HTTP, so it's easier to just use it. Remember when we allowed for newer protocols to have their own ports and didn't blacklist everything but HTTP? Why are we doing this again, now? Please don't say security/firewalls or else I'm going to ram an ice pick in my eye. If your organisation is blocking ports, they're probably filtering HTTP. Companies that care about filtering deny access to most personal e-mail on work machines. So basically you're saying "let's abuse HTTP so people can violate company policy." You may or may not also being saying, "Let's build a brand new awesome shiny latest and greatest protocol...and permanently tie it to the technical debt wreckage of HTTP/2. oh and Google could just fix their terrible broken IMAP implementation.
- Arnt 9y ago
  (HTTPS, actually.) HTTPS (when combined with JSON) provides requests and responses, without harsh size limits, with nice addressing. TCP provides a byte stream. A lot of people are finding it easier to express their upper-level protocols in terms of requests and responses than in terms of a byte stream. Having to cope with NAT middleboxes breaking the byte stream doesn't help, of course.
im0nde 9y ago
What about encrypting/signing? Does it address that too?
- brongondwana 9y ago
  Explicitly not. Key management is a far from solved problem. We're not trying to fix everything.
mike-cardwell 9y ago
After reading the "Why is JMAP better than IMAP?" section on http://jmap.io/ http://jmap.io/, what it seems to boil down to is small efficiency gains and an easier time for people writing clients. It doesn't look to me like users of clients will really gain much from their client using JMAP instead of IMAP. There's a comment about not requiring a persistent connection being better for mobile clients, but I've been using IMAP push with K-9 on my phone for a long time now, and I've never noticed a degraded battery life from using it. A quick look at the protocol looks like in order to get a list of mailboxes, you have to ask for them all. In IMAP, you can say, show me top level mailboxes, show me direct child mailboxes of mailbox x, etc. In a previous role, I was responsible for working on a mail archiving solution which had a large hierarchy of folders (into the thousands IIRC) in a single account. Access to the archive was via IMAP. Looks to me that JMAP here would require the client to potentially fetch megabytes of data just to start showing a list of folders, where the solution we had with IMAP probably took a couple of kilobytes at most to get started. It does make me think that the protocol may have been designed with less thought for uncommon use cases than IMAP was, but I've only spent a few minutes looking at the protocol, so I could be wrong.
- brongondwana 9y ago
  You're very welcome to comment on the IETF mailing list if you think you can make a compelling argument for having getMailboxes also take a filter allowing you to build a tree with "parentId": [null] and then use the IDs from each of those responses to create a "parentIds": [x, y, z] query. Of course nothing stops you storing thousands of mailboxes in a non-heirarchical layout and making IMAP have large LIST fetches too. (I do wonder - did your solution via IMAP still fetch everything each time, because in that case all you're doing is spreading that megabyte out over multiple round trips)
  - mike-cardwell 9y ago
    > You're very welcome to comment on the IETF mailing list if you think you can make a compelling argument for having getMailboxes also take a filter allowing you to build a tree with "parentId": [null] and then use the IDs from each of those responses to create a "parentIds": [x, y, z] query. I'm not invested enough to sign up to a mailing list and try to convince people to change a protocol. I'm only invested enough to fire off a comment on a forum I am already signed up to. I explained my use case above. Whether or not it is compelling is up to the reader to decide. Maybe I'm the only person in the World who has dealt with large hierarchies of mailboxes. > Of course nothing stops you storing thousands of mailboxes in a non-heirarchical layout and making IMAP have large LIST fetches too. Yes. I would issue the LIST command, and as the IMAP server started streaming the results to the client, I would immediately process each mailbox as soon as it was down, rather than waiting for them all to be retrieved. I fear that with a JSON based protocol, most server and client implementations wouldn't deal with streaming (although they technically could), and would just deal with completed blobs. Does the Cyrus implementation construct the entire blob of JSON before beginning sending it to the client, or does it start constructing and streaming the JSON as soon as possible, before it has even fetched the entire list of mailboxes from whatever the backend data store is? The JMAP streaming solution seems like a more complicated solution for developers to implement than the IMAP streaming solution. > (I do wonder - did your solution via IMAP still fetch everything each time, because in that case all you're doing is spreading that megabyte out over multiple round trips) People were accessing this mailbox with many different clients. How each of those clients worked under the hood, I do not know. I did build a web frontend, and that web frontend worked efficiently, fetching only the folders that were being displayed to the user, and their immediate children. A user clicked a folder to expand it, and they would immediately see the pre-fetched child folders, and a new request to fetch those folders children was then triggered.
brongondwana 9y ago
Thanks HN - while I'm away at Mt Gambier supporting my kid in http://generationsinjazz.com.au/ http://generationsinjazz.com.au/ you post not one, but TWO things about my hobby horses. Turns out 6000 people in a field tends to overload the local cell, so I've had very spotty coverage. Sitting in a bus with unreliable phone connection on my way back home trying to reply to everything now. Cheers.
- jorangreef 9y ago
  "Turns out 6000 people in a field tends to overload the local cell" Next time, try dropping down to 2G and you should make it through fine.
  - brongondwana 9y ago
    Good idea :) Though putting the phone away and cheering on our kids was pretty worthwhile too.
tyingq 9y ago
I found this observation interesting: "Our servers are in New York, and our developers are in Australia. Even on a good day, the ping times are over 230 milliseconds. Anything more than the absolute minimum number of round trips is felt very keenly." I wonder if some web apps might be built better if developers were bandwidth throttled and latency hindered artificially on their own machines, for say, one day a week.
- arkitaip 9y ago
  This is a useful built-in feature of the dev tools on Chrome so for devs who care about their users it's trivial to use. I think moving towards progressive web apps (PWA) should make the web much faster because good performance over poor bandwidths is one of the cornerstones of PWA.
  - tyingq 9y ago
    Yes, this thing in chrome: https://developers.google.com/web/tools/chrome-devtools/network-performance/imgs/throttle-selection.png https://developers.google.com/web/tools/chrome-devtools/netw... PWA should help somewhat, but there is still quite a bit of javascript bloat and "too many assets" to combat. When my phone runs out of the included 4G and drops to 3G, many sites are unbearably slow.
- asadlionpk 9y ago
  I think Facebook does/used to do something like this[1] [1] https://blogs.wsj.com/digits/2015/10/27/facebook-slows-the-internet-for-staffers-on-2g-tuesdays/ https://blogs.wsj.com/digits/2015/10/27/facebook-slows-the-i...
- mdaniel 9y ago
  Don't forget that developers make decisions within the boundaries prescribed by the release bookends, meaning it can certainly valuable to the developer but not valuable to the team or larger organization. This very situation reminds me of the quote "forgive the length of this letter, for I didn't have time to make it shorter." Taking steps to optimize load times or page size requires effort, and then maintaining constant diligence against future laziness
Bino 9y ago
Why dont improve on IMAP with extensions, I can't see how the "extending IMAP only makes it worse" argument hold. How would adding an addition protocol (JMAP) to a buggy client make the situation any better? I can see how it's easier for fastmail as a company to try to get its changes out by making another protocol, but would we not benefit better by trying to improve on IMAP? it's very unlikely this will ever replace IMAP and SMTPS from history internet always try to iterate and improve not replace standards completely? There is great effort to improve on SMTP and encryption lately, what can we learn from that?

253 points by robert_foss 9y ago

More recent info at http://jmap.io http://jmap.io

irl_ 9y ago

If you want to try out JMAP on a provider that is not FastMail, you use a proxy: https://proxy.jmap.io/ https://proxy.jmap.io/ I've not seen it deployed anywhere else yet as a production thing, and I don't know how much progress has been made on standardisation of it, but it does seem to offer vast improvements over IMAP.

lemming 9y ago
Do Fastmail themselves let you use jmap to connect now? They didn't a while back but I haven't checked recently.
- Nemo157 9y ago
  I was looking for a way a week ago and couldn't find it, but maybe it's just hidden really well.
- brongondwana 9y ago
  Not yet, though it's being used internally for a few things already. The ietf working group and the Cyrus IMAP source code are the most up to date right now. I'll be updating the proxy (perl, open source) soon.
brongondwana 9y ago
The proxy is a pretty crappy Perl JMAP server layered on top of a middling quality IMAP sync layer. The CalDAV/CardDAV bits are a bit better because it has nicer synchronisation primatives, but it's still code I slapped together over a couple of weeks of evenings. It's fine for playing with, but I wouldn't try to use it for large mailboxes. There's not much optimisation in there, it's doing pretty basic sqlite queries against a fairly normalised database, so there's no denomalisation for efficiency.

mostafah 9y ago

Email protocols are very old and network APIs have improved a lot during the last two decades. So, when FastMail (I’m a happy customer) introduced JMAP I was excited about it. But even then I didn’t anticipate much success for it. Big email providers (Gmail, iCloud, Outlook, …) don’t care about these things. And it’s a shame that email providers and clients are still wrestling with ancient protocols to provide a decent experience for saving drafts, search, folders/labels, and other basic things.

StavrosK 9y ago
I hope it gains more traction. In the mean time, I've switched, to Fastmail, and I couldn't be happier. It really is much faster than Gmail, I was very surprised because I didn't think email could be so fast. I guess Gmail had become the new normal.
- jpgvm 9y ago
  Except for search really. I love FastMail but they need to work on the search speed. Atleast it's faster than Outlook.com though, that search speed is beyond unacceptable.
  - brongondwana 9y ago
    If you open a support ticket I can look at your search issues, ask for Bron. It shouldn't be slow unless you're deliberately bypassing the Xapian indexes by using substr searches.
    - Semaphor 9y ago
      How long is search supposed to take? Entering something common (like "gmail") takes around 5 seconds for me with 32k results. My old google mail account takes 1.5s to display the first page.
      - mike-cardwell 9y ago

nmjenkins 9y ago

Just as an update since then: a JMAP working group was chartered at the IETF a few months ago, and is working towards bringing the spec to RFC status: https://datatracker.ietf.org/wg/jmap/about/ https://datatracker.ietf.org/wg/jmap/about/

brongondwana 9y ago
Thanks for posting that, was about to say the same thing. Am one of the co-chairs. AMA.

deleted 9y ago

[deleted]

Mojah 9y ago

Happy to see this get more attention again. I highlighted it in last week's cron.weekly [1] too. I'm still unsure of the viability at this point, but time will tell. [1] https://www.cronweekly.com/issue-78/ https://www.cronweekly.com/issue-78/

shmerl 9y ago

Does this trend of replacing TCP/IP with HTTP start being more common, or it just looks to me that way? I get a feeling that HTTP is stretched too much outside of what it's designed for, and it results in messy design.

brongondwana 9y ago
There are benefits to using http as a substrate, like request bundling on mobile to reduce radio and battery usage. JMAP is transport agnostic in its core form, but json for structure over https for transport is the current best choice for simplicity and developer friendliness
vbezhenar 9y ago
Most important thing is JavaScript from browser can use only HTTP, so if you want to build web-based mail client, you won't have to build proxy HTTP service, you can just implement protocol on client. Another important thing is proxy, many organizations don't allow anything but HTTP, so it's easier to just use it. HTTP is not that bad, anyway. You don't have to use every quirk.
- djsumdog 9y ago
  I agree with your comments about web-based e-mail clients, but: > many organizations don't allow anything but HTTP, so it's easier to just use it. Remember when we allowed for newer protocols to have their own ports and didn't blacklist everything but HTTP? Why are we doing this again, now? Please don't say security/firewalls or else I'm going to ram an ice pick in my eye. If your organisation is blocking ports, they're probably filtering HTTP. Companies that care about filtering deny access to most personal e-mail on work machines. So basically you're saying "let's abuse HTTP so people can violate company policy." You may or may not also being saying, "Let's build a brand new awesome shiny latest and greatest protocol...and permanently tie it to the technical debt wreckage of HTTP/2. oh and Google could just fix their terrible broken IMAP implementation.
Arnt 9y ago
(HTTPS, actually.) HTTPS (when combined with JSON) provides requests and responses, without harsh size limits, with nice addressing. TCP provides a byte stream. A lot of people are finding it easier to express their upper-level protocols in terms of requests and responses than in terms of a byte stream. Having to cope with NAT middleboxes breaking the byte stream doesn't help, of course.

im0nde 9y ago

What about encrypting/signing? Does it address that too?

brongondwana 9y ago
Explicitly not. Key management is a far from solved problem. We're not trying to fix everything.

mike-cardwell 9y ago

After reading the "Why is JMAP better than IMAP?" section on http://jmap.io/ http://jmap.io/, what it seems to boil down to is small efficiency gains and an easier time for people writing clients. It doesn't look to me like users of clients will really gain much from their client using JMAP instead of IMAP. There's a comment about not requiring a persistent connection being better for mobile clients, but I've been using IMAP push with K-9 on my phone for a long time now, and I've never noticed a degraded battery life from using it. A quick look at the protocol looks like in order to get a list of mailboxes, you have to ask for them all. In IMAP, you can say, show me top level mailboxes, show me direct child mailboxes of mailbox x, etc. In a previous role, I was responsible for working on a mail archiving solution which had a large hierarchy of folders (into the thousands IIRC) in a single account. Access to the archive was via IMAP. Looks to me that JMAP here would require the client to potentially fetch megabytes of data just to start showing a list of folders, where the solution we had with IMAP probably took a couple of kilobytes at most to get started. It does make me think that the protocol may have been designed with less thought for uncommon use cases than IMAP was, but I've only spent a few minutes looking at the protocol, so I could be wrong.

brongondwana 9y ago
You're very welcome to comment on the IETF mailing list if you think you can make a compelling argument for having getMailboxes also take a filter allowing you to build a tree with "parentId": [null] and then use the IDs from each of those responses to create a "parentIds": [x, y, z] query. Of course nothing stops you storing thousands of mailboxes in a non-heirarchical layout and making IMAP have large LIST fetches too. (I do wonder - did your solution via IMAP still fetch everything each time, because in that case all you're doing is spreading that megabyte out over multiple round trips)
- mike-cardwell 9y ago
  > You're very welcome to comment on the IETF mailing list if you think you can make a compelling argument for having getMailboxes also take a filter allowing you to build a tree with "parentId": [null] and then use the IDs from each of those responses to create a "parentIds": [x, y, z] query. I'm not invested enough to sign up to a mailing list and try to convince people to change a protocol. I'm only invested enough to fire off a comment on a forum I am already signed up to. I explained my use case above. Whether or not it is compelling is up to the reader to decide. Maybe I'm the only person in the World who has dealt with large hierarchies of mailboxes. > Of course nothing stops you storing thousands of mailboxes in a non-heirarchical layout and making IMAP have large LIST fetches too. Yes. I would issue the LIST command, and as the IMAP server started streaming the results to the client, I would immediately process each mailbox as soon as it was down, rather than waiting for them all to be retrieved. I fear that with a JSON based protocol, most server and client implementations wouldn't deal with streaming (although they technically could), and would just deal with completed blobs. Does the Cyrus implementation construct the entire blob of JSON before beginning sending it to the client, or does it start constructing and streaming the JSON as soon as possible, before it has even fetched the entire list of mailboxes from whatever the backend data store is? The JMAP streaming solution seems like a more complicated solution for developers to implement than the IMAP streaming solution. > (I do wonder - did your solution via IMAP still fetch everything each time, because in that case all you're doing is spreading that megabyte out over multiple round trips) People were accessing this mailbox with many different clients. How each of those clients worked under the hood, I do not know. I did build a web frontend, and that web frontend worked efficiently, fetching only the folders that were being displayed to the user, and their immediate children. A user clicked a folder to expand it, and they would immediately see the pre-fetched child folders, and a new request to fetch those folders children was then triggered.

brongondwana 9y ago

Thanks HN - while I'm away at Mt Gambier supporting my kid in http://generationsinjazz.com.au/ http://generationsinjazz.com.au/ you post not one, but TWO things about my hobby horses. Turns out 6000 people in a field tends to overload the local cell, so I've had very spotty coverage. Sitting in a bus with unreliable phone connection on my way back home trying to reply to everything now. Cheers.

jorangreef 9y ago
"Turns out 6000 people in a field tends to overload the local cell" Next time, try dropping down to 2G and you should make it through fine.
- brongondwana 9y ago
  Good idea :) Though putting the phone away and cheering on our kids was pretty worthwhile too.

tyingq 9y ago

I found this observation interesting: "Our servers are in New York, and our developers are in Australia. Even on a good day, the ping times are over 230 milliseconds. Anything more than the absolute minimum number of round trips is felt very keenly." I wonder if some web apps might be built better if developers were bandwidth throttled and latency hindered artificially on their own machines, for say, one day a week.

arkitaip 9y ago
This is a useful built-in feature of the dev tools on Chrome so for devs who care about their users it's trivial to use. I think moving towards progressive web apps (PWA) should make the web much faster because good performance over poor bandwidths is one of the cornerstones of PWA.
- tyingq 9y ago
  Yes, this thing in chrome: https://developers.google.com/web/tools/chrome-devtools/network-performance/imgs/throttle-selection.png https://developers.google.com/web/tools/chrome-devtools/netw... PWA should help somewhat, but there is still quite a bit of javascript bloat and "too many assets" to combat. When my phone runs out of the included 4G and drops to 3G, many sites are unbearably slow.
asadlionpk 9y ago
I think Facebook does/used to do something like this[1] [1] https://blogs.wsj.com/digits/2015/10/27/facebook-slows-the-internet-for-staffers-on-2g-tuesdays/ https://blogs.wsj.com/digits/2015/10/27/facebook-slows-the-i...
mdaniel 9y ago
Don't forget that developers make decisions within the boundaries prescribed by the release bookends, meaning it can certainly valuable to the developer but not valuable to the team or larger organization. This very situation reminds me of the quote "forgive the length of this letter, for I didn't have time to make it shorter." Taking steps to optimize load times or page size requires effort, and then maintaining constant diligence against future laziness

Bino 9y ago

Why dont improve on IMAP with extensions, I can't see how the "extending IMAP only makes it worse" argument hold. How would adding an addition protocol (JMAP) to a buggy client make the situation any better? I can see how it's easier for fastmail as a company to try to get its changes out by making another protocol, but would we not benefit better by trying to improve on IMAP? it's very unlikely this will ever replace IMAP and SMTPS from history internet always try to iterate and improve not replace standards completely? There is great effort to improve on SMTP and encryption lately, what can we learn from that?