16 ms·

JMAP – a better way to email

410 points by brongondwana 12y ago

ratsmack 12y ago
This looks interesting, as long as it is more efficient as a protocol and API. On another note, the entire email infrastructure need to be redesigned in my opinion.
- alfiedotwtf 12y ago
  > as long as it is more efficient as a protocol and API It is :)
tinco 12y ago
Wasn't there a startup recently that pivoted from being a mail client into building a generalized API for interfacing with mail services like gmail/yahoo/outlook? I forgot the name but I'm super interested in efforts like this. I have the feeling the next big social network disruption is going to be leveraging e-mail in a big way. E-mail is the established quasi-p2p (it's a network of centralized services) platform that everyone has an account and a very complete social network on. I'm not saying it would be easy to launch a Facebook competitor from e-mail, but I believe it's still got the potential to launch one last social network and someone just has to do it. I'm hoping the key lies somewhere in building the right sort of mail client, that's got the reliability and extent of e-mail, adding on top some hook feature and layering in perhaps a dash of privacy, cryptography and independence. There's got to be something viral in there.
- aroch 12y ago
  Inbox (MIT and Dropbox): http://techcrunch.com/2014/07/07/mit-and-dropbox-alums-launch-inbox-a-next-generation-email-platform/ http://techcrunch.com/2014/07/07/mit-and-dropbox-alums-launc... https://www.inboxapp.com/ https://www.inboxapp.com/
  - grinich 12y ago
    We didn't pivot. ;)
- username223 12y ago
  IMHO the big outstanding problems with email are spam and the difficulty of using public-key cryptography (both key distribution and mail client support). Write a dead-sexy mail client that makes the crypto completely invisible and uses either a public key directory or some kind of OAuth-ish thing, and you'd solve the latter. Better, if you made finding someone's public key take time and/or money, and simply refused to deliver unsigned mail, you'd take a big step toward killing spam.
  - jmnicolas 12y ago
    If only they could make encrypted emails as transparent as encrypted SMS like Textsecure ...
0942v8653 12y ago
Did something happen to the comments? I'm seeing 78 upvotes and no comments.
- toyg 12y ago
  I felt a great disturbance in the Force, as if millions of people raised their hands in joy and suddenly clicked to upvote. I fear something incredible has happened. Better and more robust email protocols, based on unencumbered open tech? Yes please. What else is there to say? :)
- clooney 12y ago
  Fastmail uses Hacker News to spam, I mean publish all their new developments.
  - dvfjsdhgfv 12y ago
    In general I dislike FastMail promotion on HN. But this article is different - it's some new open development, potentially very useful to many users, and its success is directly related to the number of developers who will pick it up. So at least this time it deserves some attention.
thecoffman 12y ago
I haven't had a chance to review the specifics of the protocol yet, but as a Fastmail user, I can vouch for the speed of event propagation across multiple clients. Its lightning fast; noticeably faster than say, Gmail.
driverdan 12y ago
Why would you create a new email protocol without an encryption requirement? I understand trying to fix the existing protocol problems but one of the biggest is plaintext.
- brongondwana 12y ago
  Because SSL does the job of securing data between the server and the client just fine. There are a ton of interesting problems in the email world, including all the server to server communications. JMAP is lightning focused on solving ONE of those problems - the communication between end-user devices and their mail server - and doing it well.
  - mikhailt 12y ago
    You're assuming SSL is implemented without any security bugs. We saw what happened with Heartbleed and when SSL implementations are not properly reviewed and updated by a full funded team.
    - nmjenkins 12y ago
      So you're assuming that an independent encryption layer that has to be reimplemented by every email client dev is going to be more secure than a widely studied protocol, implemented at the OS level. Right…
      - mikhailt 12y ago
        No, I'm not assuming anything and didn't even say anything like that. My point is that using SSL on its own as the only line of defense should not be an excuse not to have anything else. It's like saying my apartment doesn't have a spec for a security system nor a safe because the lock on the door does enough of a job to secure it. Also, a widely studied protocol spec means nothing. The bugs are from the humans coding the implementations, it doesn't matter what level it is, they will have some bugs. Nobody can code a perfect secure implementation but we can have some kind of redundancies in the system, where if one security level fails, the rest can still have some reasonable security left. Relying on SSL alone is not enough. But I don't think JMAP is the right place to do this, we may need something else in addition to JMAP. At least JMAP is extendable, so that's one good thing it has.
kijin 12y ago
JMAP is a rather misleading name. "J" is misleading because it's not JSON that distinguishes JMAP from IMAP, but rather the use of HTTP, along with recently added HTTP features such as push. JSON is just the message packing scheme. HTTP is what does all the heavy lifting. "M" is misleading because this protocol is designed for a lot more than mail. It also handles contacts and calendars, and I wouldn't be surprised if Fastmail made their file storage service accessible through JMAP as well. What's even worse is that once you come up with a protocol that covers everything that Fastmail does, everyone and their dog will try to add more services to the protocol. Instant messaging? Got it. Collaborative editing a la Google Drive? No problem. And before you know it, the protocol is bloated as hell and you've basically reinvented HTTP. Please don't try to do everything. Do one thing, do it well, and put a strict limit to the scope of the project for the time being. A few more minutes of battery life on a phone is not worth polluting the world with yet another example of massive scope creep. IMAP+SMTP using a stateless protocol would be cool, but anything more and I'm not sure.
- quink 12y ago
  First thing I thought: It's one better than IMAP and meant to replace it. I++ == J. JMAP.
  - gboss 12y ago
    You mean ++I == J
- toyg 12y ago
  It must be a generational thing: for me, anything starting with "J" must have something to do with Java. ... But to be honest, I don't really care. A lot of people don't know what IMAP and POP and SMTP actually stand for -- they don't care, and neither do I.
  - mdadm 12y ago
    I thought that it'd be lowercase "j" that signifies Java, not a capital "J"...
- 12y ago
HarrietJones 12y ago
This is good, and I love FastMail, but the one thing that really, really gladdens my heart with this is that they're pushing it as an open standard. Thanks FastMail. Sincerely, thanks.
elwell 12y ago
> By using the platform push channels, JMAP avoids having to hold its own connection open. Can anyone explain what they mean by "platform push channels"? I don't understand how you can have realtime updates without leaving a connection open or periodically polling. Do they mean that they use the same mode of transport as Push Notifications?
- robn_fastmail 12y ago
  Android, iOS, (Windows Phone, Blackberry, etc) have services where a server application can push a small data packet to a specific device. That device wakes an application in response which can then do things. We use this right now in our mobile apps. The apps respond by calling back to FastMail to check for updates. More info: http://blog.fastmail.com/2014/12/02/dec-3-push-it-real-good/ http://blog.fastmail.com/2014/12/02/dec-3-push-it-real-good/ https://developer.android.com/google/gcm/index.html https://developer.android.com/google/gcm/index.html https://developer.apple.com/library/ios/documentation/NetworkingInternet/Conceptual/RemoteNotificationsPG/Chapters/ApplePushService.html https://developer.apple.com/library/ios/documentation/Networ...
maxk42 12y ago
It's missing the one killer feature that would fix the spam problem: approved senders. If you had a "friends" list like Facebook where only people you've approved may send messages to you, we'd cut down on 90% of spam immediately. Between that and the lack of mandatory encryption this is just another tedious protocol to implement.
- LeoPanthera 12y ago
  That's not a practical feature in the real world. Sometimes you need email from unapproved senders. Spam filtering is better accomplished before or after delivery, not during.
  - maxk42 12y ago
    There's nothing preventing an unapproved sender from requesting contact privileges, but once denied, their messages and further requests would be ignored by default. Each server and user would have their own identity management concerns. Basically someone can send a "friend request" and if approved by a human (the recipient) they may communicate freely from that point.
    - ricardobeat 12y ago
      How is this different from adding an address to your blacklist after the first message?
      - maxk42 12y ago
        They can't deliver their spam message to begin with. You'd need to approve the contact then you could still block them if they spam you. There's still friend-request spam on facebook, but not nearly as much as email because you can just ignore the people you don't know and block people you do know if they begin to spam you.
        feld 12y ago
        The request for contact would be just as bad as the spam itself...
mortenjorck 12y ago
This looks fantastic and is a huge, long-overdue breath of fresh air in the protocol space. But realistically, what will it take for major vendors like Apple and Google to start adding support for it?
dannysu 12y ago
Great job guys! Things like this together with CalDAV and CardDAV are reasons why I pay you guys for service. I just tried the following experiments with my Gmail and FastMail account. There's a reason why FastMail just feels faster! Try this: 1) While having the Gmail iOS app open, mark an email as read on the web 2) See how long it takes for Gmail to reflect the change 3) Now try with FastMail iOS app, see that it's faster Try this too: 1) While having the Gmail iOS app open, delete an email on the web 2) See that Gmail app never updates until you manually refresh 3) Now try with FastMail iOS app, see that it happens instantly How about the other direction? 1) Read an email on your phone 2) See that Gmail on the web never updates until manual refresh 3) Now try with FastMail... Instant update!
- veb 12y ago
  Thank you for this comment. I do wonder why they simply didn't say this at the beginning of the article. I understood what they were up to purely because I'm a technical guy, but still I didn't want to actually have to watch the video -- namely because I'm deaf, and far too often there's no captions to explain what's going on so I prefer to read (and so do people at work, school, no sound etc), and it takes a lot less time! I'm really considering moving from Google Apps to Fastmail... who else has made this jump? Totally worth it?
  - hsshah 12y ago
    Every time I read about Fastmail, I want to make the jump from Google Apps. However, I don't do it because of Google Drive/Docs/Spreadsheet. Any recommendation for their replacement?!
    - veb 12y ago
      Ha. I'm the same, every time I read about them I want to make the jump - but I'm not reliant on any of those, except mail fortunately. I just haven't quite done it yet. From people I've spoken to, everyone praises Fastmail... seems like we all need that extra push.
ebabchick 12y ago
For the well acquainted -- how does this relate to Inbox (https://www.inboxapp.com/ https://www.inboxapp.com/)? I have not spent enough time with either to make a comparison
- geofft 12y ago
  I haven't actually used either of these products, so I might be totally wrong, but.... Inbox is a separate layer on top of existing email servers. My impression is that the pain point they're addressing is, you want to build some awesome email integration app and sell it to companies, but you're looking at how complicated it would be to speak IMAP and IMAP-with-quirks and ActiveSync and the Gmail API and everything else (in order to build a high-quality integration) and that's scaring you. You can outsource that problem to Inbox, they'll make sure they get good-quality data from any email server reliably, and you just speak an HTTP API. This involves deploying a separate server which caches email. You talk to the "sync engine" over a reasonable protocol, and the sync engine takes care of talking an unreasonable protocol to wherever the data actually lives. JMAP is an API to directly access an email store. It's something that an email provider (Fastmail, most obviously) will directly support, but it also only helps you for those providers that actually speak JMAP. If you're an email server developer, an ISP, etc., then JMAP is cool and Inbox is not really useful -- there's not much of a point in having an email server that speaks IMAP and then also sticking a frontend to convert IMAP to something else. You might as well just make your email server speak a reasonable protocol directly. If you're an app author, Inbox is useful today, and JMAP is only useful, maybe, if all your users are Fastmail users. If you're an in-house developer at a large company that's running Exchange, and realistically you have no hope of getting the email admins to switch email servers (and you wouldn't want to run the email service yourself, even if they offered), JMAP is useless to you, and Inbox is super compelling. (There is probably an argument that the Inbox "sync engine" should speak JMAP on the front end / those APIs should converge. Certainly the Inbox sync engine should speak JMAP on the back end, so Inbox can advertise support for Fastmail.)
- grinich 12y ago
  Michael from Inbox here— a few key differences, from the horse’s mouth. :) First of all— I want to say that we’re huge fans of Fastmail and really admire their team and focus. Rob and Neil actually came by for lunch this summer and hacked in our office for the day. We’ve discussed JMAP back and forth with them for a while, and specifically had a long thread when we added calendar endpoints to the Inbox platform. [1] The fundamental differences between the Inbox APIs and JMAP are the kinds of applications built with them, the open source code, and the email providers supported. The Inbox APIs are really designed to let developers access email data in non-traditional ways. Our site reads “Say goodbye to IMAP” but we’re not so much replacing IMAP, as creating a new protocol which liberates email data to a higher form. We think about email as “the database of your life” which means unconventional “clients” and access/manipulation patterns. That’s one of the reasons we have a more REST-inspired design, whereas JMAP is closer to RPC. The Inbox API endpoints strive to represent canonical objects that developers simply treat as resources. JMAP is tailored specifically for traditional email clients, so it excels at fast operations (like archive -> reload), but compromises in other areas of simplicity and clarity IMHO. The other main difference is code. The Inbox sync engine is open source with about 15k lines of code on GitHub.[2] You can run it yourself right now, and it works on all providers, including Fastmail! That means your users don’t need to be on any specific provider or backend. If you don’t want to run it yourself, you can use our hosted version which also includes support for enterprise environments that use Microsoft Exchange. [3] I’ve been lurking on the IETF mailing lists for several years now, and one thing I learned is that spec documents hardly move the world forward. Developers need code that is battle tested and they can depend on. What use is a API spec if you can’t build an app with it? The JMAP system is a great spec for FastMail’s internal mail service, and it works to power their own clients, but unless they make some drastic changes to the code and business, it’s not a platform for building 3rd party mail apps. As an aside, I totally wish I had more time to write. I envy how prolific the Fastmail blog is these days. :) Keep up the great work Bron, Rob, Neil, and the rest! [0] https://www.inboxapp.com https://www.inboxapp.com [1] https://www.inboxapp.com/docs/api#calendar https://www.inboxapp.com/docs/api#calendar [2] https://github.com/inboxapp/inbox https://github.com/inboxapp/inbox [3] https://www.inboxapp.com/features https://www.inboxapp.com/features
hedgehog 12y ago
Neat. Two questions about the digest: 1. You recommend using a digest of the message to generate the message ID. Did you consider mandating that scheme or going farther and using a Merkle tree for the mailbox representation? It seems like this would allow for generating single requests that can fetch all new items. 2. Why SHA-1? Edit: Also, thanks for keeping Fastmail running. As a customer of about ten years it's much appreciated.
- brongondwana 12y ago
  No mandated format for message-ID. If you are proxying an existing store with a stable message-ID, you can just use that. My partially written gmail proxy uses the MSGID from Gmail, for example. SHA1 was a good balance between speed and unforgability at the time. There's still no known collision, and I'm working on putting a piece of randomness into the LMTP headers that are added to every message on delivery into Cyrus to make it even safer.
iwantagrinder 12y ago
I' m interested in this from a security perspective. What does this new protocol offer in terms of better control around what makes it to the inbox? Would IMAP>JMAP translation before hitting the user give us better ability to filter out malicious items/spam?
- SwellJoe 12y ago
  I think the only thing this particular protocol should be concerned with is reliably providing a method for communicating between client and server. Most spam and malicious mail filtering should happen as early in the process as possible, hopefully during the initial connection. There is a a protocol named Sieve ( http://www.ietf.org/rfc/rfc5228.txt http://www.ietf.org/rfc/rfc5228.txt ) which provides for delivery stage filtering rules. It is similar in capability and usage to procmail, but formalized and somewhat more modern (and not capable of running arbitrary system commands), and supported by some modern IMAP/POP servers, like Dovecot. Presumably if JMAP gets integrated into those servers, Sieve would also be available. And, I would hope there wouldn't be an IMAP/JMAP translation layer, but instead servers would implement JMAP directly. Though I guess in the short term there might be some sort of proxy.
  - gecko 12y ago
    Sieve is the native filtering engine used by FastMail, so I'd say it integrates quite well with JMAP, from my experience. :)
- groby_b 12y ago
  So am I. From looking at http://jmap.io/spec.html#authentication http://jmap.io/spec.html#authentication, it looks like the password will be transmitted in plain text. (See the text below the 200 response) That makes me extremely queasy. Yes, HTTPS theoretically provides transport layer security, but a single breach of transport layer security results in the attackers permanent access to your mail. I.e. run a MITM attack in a coffee shop, snoop up JMAP passwords, make use of them at a latter point in time. I really hope I'm missing something important, or misread the spec.
sjtrny 12y ago
http://xkcd.com/927/ http://xkcd.com/927/
- LeoPanthera 12y ago
  Funny, but how many open mail retrieval protocols are in common use? I can think of maybe two - IMAP and POP3, and a small selection of proprietary ones, dominated by Exchange/ActiveSync. The time is right for a better standard.
- brongondwana 12y ago
  My presentation to the Thunderbird folks led with that, and we would have used it at Inbox Love too if we'd had time, but a 5 minute presentation had to get straight to the point. It's a good comic. Good point. We didn't take this decision lightly. There are significant reasons why all the existing standards don't do what's needed, and we made sure we built a technology preview and used it in the real world first - check the video or try out FastMail yourself.
plg 12y ago
Amazing. Modern, innovative, service-oriented products and a business model based on customers giving the company money in return for services. I hope that this is a reminder to startups out there that you don't have to trick customers into handing over their personal information all the while pretending to offer your services "for free", in order to gain market share and be profitable. The "traditional" business model can be quite successful.
- Touche 12y ago
  I hope you're right but I fear that Fastmail isn't successful in a way that startup founders are going for. Advancing technology unfortunately isn't a priority for most founders.
  - robn_fastmail 12y ago
    We're fully owned by the staff, with no external funding. We've been nicely profitable since the beginning. Not huge profits but enough to pay for a nice office, decent salaries and few gadgets. So yes, if your goal is to attract massive amounts of venture capital and then get bought out, don't copy us. If you want to have some fun doing something you love without having to worry about anything else then this is definitely the way to do it :)
ummjackson 12y ago
FastMail continue to impress me. Good work folks.
nona 12y ago
I hate to be critical considering the considerable work they've put in, but I don't buy their "JMAP is actually more REST-like than most “RESTful” APIs" statement. But maybe I'm missing a whole lot. REST doesn't preclude doing multiple things in one round trip. Well it does seem JMAP is a bit more flexible when it comes to multiplexing very different actions. But the error handling doesn't seem to tell you which command has failed. You still can't get away from certain ordering issues. And to be honest, I don't see how it's going to be that easy to cache. This really is RPC – using it as a straight replacement for IMAP might work well, but I wouldn't want to use it as a platform to build something very different than an email client on it.
- nmjenkins 12y ago
  Each call is tagged by the client and the same tag is added to the response by the server, so you can tell exactly what error corresponds to which call. Caching is pretty easy: you just keep a cache of each object type. The delta update mechanism means you can very accurately invalidate what you need to keep it up to date. The system is actually really flexible and a great fit for any CRUD based app; JMAP is really just a combination of a very powerful and efficient database access protocol, and the definition of some objects to represent email, contacts and calendars.
  - icebraining 12y ago
    Not just some objects, but different methods to access different objects, and it's those methods that implement the semantics - the base of JMAP is just an encoding for bulk RPC. What I mean is, if tomorrow I add a file with the mediatype application/vnd.icebraining to my server, an intermediate HTTP proxy can still cache it even though it knows nothing about my format. But if I add a new object (say, Note) to my JMAP server, along with the appropriate methods (getNotes, getNoteUpdates, etc), an intermediate JMAP server will be unable to cache them, since it needs to know the semantics of the new methods. So frankly, I don't see how their protocol follows an uniform interface at all. I still think it's pretty neat, though, don't take me wrong. I just want people to stop abusing REST. There's nothing wrong with RPC!
Animats 12y ago
What does this do that IMAP and SMTP don't do already? All you need is an IMAP server, accessed from all your devices. The one that comes with Android isn't bad. Thunderbird works fine on the desktop. You don't have to use GMail with Android; when you first power up your Android device, click "Later" when it asks for a Google login. Then delete the Google One-Time Startup app. Google won't bother you again.
- corford 12y ago
  tldr; version: It reduces the number of network roundtrips (and server overhead) needed to keep clients in sync (great for mobile) and addresses the fragmentation issue between the various server & client implementations of the sprawling IMAP protocol. By moving to JSON and HTTP it also makes the barrier to entry a lot lower for client developers and opens up the possibility to improve speed yet further with GZIP and websockets. All that while being able to stay compatible with legacy IMAP servers via a reference proxy provided for free by Fastmail. Edit: Oops I've just noticed your handle so apologies for assuming you didn't read the article. I guess you've read it and still don't see the benefit. In which case would love to know why you think it's a wasted effort (personally, I think it looks like a welcome refresh).
  - Animats 12y ago
    The Ruby version of the IMAP client isn't that bad. Yes, IMAP doesn't use the current fad for marshaling, JSON. But a few years ago the web crowd would have wanted XML, and a few years from now they'll probably want some binary protocol Google comes up with. "Gzip" compression isn't a big win for objects the size of email messages, anyway. IMAP servers that don't use a real database are sure to be a mess, but that's not the fault of the protocol. I'd rather not have some startup in the middle of the mail chain. They'll start thinking they own mail.
- feld 12y ago
  IMAP is a terrible protocol. It's a poorly designed filesystem with all the baggage that comes with it. Read the IMAP overview in the IMAP ruby module -- it's disgusting and was never meant for the modern world. http://rxr.whitequark.org/mri/source/lib/net/imap.rb http://rxr.whitequark.org/mri/source/lib/net/imap.rb
msh 12y ago
I guess this tries to solve the same issue as microsoft active sync, just in a open specification way.
mike-cardwell 12y ago
I understand you guys use and contribute to Cyrus. Is there a JMAP interface planned for Cyrus? Or for any other mail servers, like Dovecot?
- aroman 12y ago
  from the article itself: "Finally, we know IMAP, SMTP and the DAVs aren’t going away any time soon. No protocol will succeed unless it provides an upgrade path from where we are now, and a compelling reason to switch. We will provide a proxy which can talk to existing servers and present them over JMAP."
  - mike-cardwell 12y ago
    Your comment implies that the quote you provided answered my question. It didn't.
- robn_fastmail 12y ago
  We're working on a proxy that will work with any IMAP server that of course will work with Cyrus. I think its likely that we'll look into building support directly into Cyrus at some point but we don't have any concrete plans yet.
- brongondwana 12y ago
  http://git.cyrusimap.org/cyrus-imapd/tree/imap/http_jmap.c?h=caldav-2.4 http://git.cyrusimap.org/cyrus-imapd/tree/imap/http_jmap.c?h... Totally a stub right now, but yes - we would like to have it directly in Cyrus...

410 points by brongondwana 12y ago

This looks interesting, as long as it is more efficient as a protocol and API. On another note, the entire email infrastructure need to be redesigned in my opinion.

alfiedotwtf 12y ago
> as long as it is more efficient as a protocol and API It is :)

tinco 12y ago

Wasn't there a startup recently that pivoted from being a mail client into building a generalized API for interfacing with mail services like gmail/yahoo/outlook? I forgot the name but I'm super interested in efforts like this. I have the feeling the next big social network disruption is going to be leveraging e-mail in a big way. E-mail is the established quasi-p2p (it's a network of centralized services) platform that everyone has an account and a very complete social network on. I'm not saying it would be easy to launch a Facebook competitor from e-mail, but I believe it's still got the potential to launch one last social network and someone just has to do it. I'm hoping the key lies somewhere in building the right sort of mail client, that's got the reliability and extent of e-mail, adding on top some hook feature and layering in perhaps a dash of privacy, cryptography and independence. There's got to be something viral in there.

aroch 12y ago
Inbox (MIT and Dropbox): http://techcrunch.com/2014/07/07/mit-and-dropbox-alums-launch-inbox-a-next-generation-email-platform/ http://techcrunch.com/2014/07/07/mit-and-dropbox-alums-launc... https://www.inboxapp.com/ https://www.inboxapp.com/
- grinich 12y ago
  We didn't pivot. ;)
username223 12y ago
IMHO the big outstanding problems with email are spam and the difficulty of using public-key cryptography (both key distribution and mail client support). Write a dead-sexy mail client that makes the crypto completely invisible and uses either a public key directory or some kind of OAuth-ish thing, and you'd solve the latter. Better, if you made finding someone's public key take time and/or money, and simply refused to deliver unsigned mail, you'd take a big step toward killing spam.
- jmnicolas 12y ago
  If only they could make encrypted emails as transparent as encrypted SMS like Textsecure ...

0942v8653 12y ago

Did something happen to the comments? I'm seeing 78 upvotes and no comments.

toyg 12y ago
I felt a great disturbance in the Force, as if millions of people raised their hands in joy and suddenly clicked to upvote. I fear something incredible has happened. Better and more robust email protocols, based on unencumbered open tech? Yes please. What else is there to say? :)
clooney 12y ago
Fastmail uses Hacker News to spam, I mean publish all their new developments.
- dvfjsdhgfv 12y ago
  In general I dislike FastMail promotion on HN. But this article is different - it's some new open development, potentially very useful to many users, and its success is directly related to the number of developers who will pick it up. So at least this time it deserves some attention.

thecoffman 12y ago

I haven't had a chance to review the specifics of the protocol yet, but as a Fastmail user, I can vouch for the speed of event propagation across multiple clients. Its lightning fast; noticeably faster than say, Gmail.

driverdan 12y ago

Why would you create a new email protocol without an encryption requirement? I understand trying to fix the existing protocol problems but one of the biggest is plaintext.

brongondwana 12y ago
Because SSL does the job of securing data between the server and the client just fine. There are a ton of interesting problems in the email world, including all the server to server communications. JMAP is lightning focused on solving ONE of those problems - the communication between end-user devices and their mail server - and doing it well.
- mikhailt 12y ago
  You're assuming SSL is implemented without any security bugs. We saw what happened with Heartbleed and when SSL implementations are not properly reviewed and updated by a full funded team.
  - nmjenkins 12y ago
    So you're assuming that an independent encryption layer that has to be reimplemented by every email client dev is going to be more secure than a widely studied protocol, implemented at the OS level. Right…
    - mikhailt 12y ago
      No, I'm not assuming anything and didn't even say anything like that. My point is that using SSL on its own as the only line of defense should not be an excuse not to have anything else. It's like saying my apartment doesn't have a spec for a security system nor a safe because the lock on the door does enough of a job to secure it. Also, a widely studied protocol spec means nothing. The bugs are from the humans coding the implementations, it doesn't matter what level it is, they will have some bugs. Nobody can code a perfect secure implementation but we can have some kind of redundancies in the system, where if one security level fails, the rest can still have some reasonable security left. Relying on SSL alone is not enough. But I don't think JMAP is the right place to do this, we may need something else in addition to JMAP. At least JMAP is extendable, so that's one good thing it has.

kijin 12y ago

JMAP is a rather misleading name. "J" is misleading because it's not JSON that distinguishes JMAP from IMAP, but rather the use of HTTP, along with recently added HTTP features such as push. JSON is just the message packing scheme. HTTP is what does all the heavy lifting. "M" is misleading because this protocol is designed for a lot more than mail. It also handles contacts and calendars, and I wouldn't be surprised if Fastmail made their file storage service accessible through JMAP as well. What's even worse is that once you come up with a protocol that covers everything that Fastmail does, everyone and their dog will try to add more services to the protocol. Instant messaging? Got it. Collaborative editing a la Google Drive? No problem. And before you know it, the protocol is bloated as hell and you've basically reinvented HTTP. Please don't try to do everything. Do one thing, do it well, and put a strict limit to the scope of the project for the time being. A few more minutes of battery life on a phone is not worth polluting the world with yet another example of massive scope creep. IMAP+SMTP using a stateless protocol would be cool, but anything more and I'm not sure.

quink 12y ago
First thing I thought: It's one better than IMAP and meant to replace it. I++ == J. JMAP.
- gboss 12y ago
  You mean ++I == J
toyg 12y ago
It must be a generational thing: for me, anything starting with "J" must have something to do with Java. ... But to be honest, I don't really care. A lot of people don't know what IMAP and POP and SMTP actually stand for -- they don't care, and neither do I.
- mdadm 12y ago
  I thought that it'd be lowercase "j" that signifies Java, not a capital "J"...
12y ago

HarrietJones 12y ago

This is good, and I love FastMail, but the one thing that really, really gladdens my heart with this is that they're pushing it as an open standard. Thanks FastMail. Sincerely, thanks.

elwell 12y ago

> By using the platform push channels, JMAP avoids having to hold its own connection open. Can anyone explain what they mean by "platform push channels"? I don't understand how you can have realtime updates without leaving a connection open or periodically polling. Do they mean that they use the same mode of transport as Push Notifications?

robn_fastmail 12y ago
Android, iOS, (Windows Phone, Blackberry, etc) have services where a server application can push a small data packet to a specific device. That device wakes an application in response which can then do things. We use this right now in our mobile apps. The apps respond by calling back to FastMail to check for updates. More info: http://blog.fastmail.com/2014/12/02/dec-3-push-it-real-good/ http://blog.fastmail.com/2014/12/02/dec-3-push-it-real-good/ https://developer.android.com/google/gcm/index.html https://developer.android.com/google/gcm/index.html https://developer.apple.com/library/ios/documentation/NetworkingInternet/Conceptual/RemoteNotificationsPG/Chapters/ApplePushService.html https://developer.apple.com/library/ios/documentation/Networ...

maxk42 12y ago

It's missing the one killer feature that would fix the spam problem: approved senders. If you had a "friends" list like Facebook where only people you've approved may send messages to you, we'd cut down on 90% of spam immediately. Between that and the lack of mandatory encryption this is just another tedious protocol to implement.

LeoPanthera 12y ago
That's not a practical feature in the real world. Sometimes you need email from unapproved senders. Spam filtering is better accomplished before or after delivery, not during.
- maxk42 12y ago
  There's nothing preventing an unapproved sender from requesting contact privileges, but once denied, their messages and further requests would be ignored by default. Each server and user would have their own identity management concerns. Basically someone can send a "friend request" and if approved by a human (the recipient) they may communicate freely from that point.
  - ricardobeat 12y ago
    How is this different from adding an address to your blacklist after the first message?
    - maxk42 12y ago
      They can't deliver their spam message to begin with. You'd need to approve the contact then you could still block them if they spam you. There's still friend-request spam on facebook, but not nearly as much as email because you can just ignore the people you don't know and block people you do know if they begin to spam you.
      - feld 12y ago
        The request for contact would be just as bad as the spam itself...

mortenjorck 12y ago

This looks fantastic and is a huge, long-overdue breath of fresh air in the protocol space. But realistically, what will it take for major vendors like Apple and Google to start adding support for it?

dannysu 12y ago

Great job guys! Things like this together with CalDAV and CardDAV are reasons why I pay you guys for service. I just tried the following experiments with my Gmail and FastMail account. There's a reason why FastMail just feels faster! Try this: 1) While having the Gmail iOS app open, mark an email as read on the web 2) See how long it takes for Gmail to reflect the change 3) Now try with FastMail iOS app, see that it's faster Try this too: 1) While having the Gmail iOS app open, delete an email on the web 2) See that Gmail app never updates until you manually refresh 3) Now try with FastMail iOS app, see that it happens instantly How about the other direction? 1) Read an email on your phone 2) See that Gmail on the web never updates until manual refresh 3) Now try with FastMail... Instant update!

veb 12y ago
Thank you for this comment. I do wonder why they simply didn't say this at the beginning of the article. I understood what they were up to purely because I'm a technical guy, but still I didn't want to actually have to watch the video -- namely because I'm deaf, and far too often there's no captions to explain what's going on so I prefer to read (and so do people at work, school, no sound etc), and it takes a lot less time! I'm really considering moving from Google Apps to Fastmail... who else has made this jump? Totally worth it?
- hsshah 12y ago
  Every time I read about Fastmail, I want to make the jump from Google Apps. However, I don't do it because of Google Drive/Docs/Spreadsheet. Any recommendation for their replacement?!
  - veb 12y ago
    Ha. I'm the same, every time I read about them I want to make the jump - but I'm not reliant on any of those, except mail fortunately. I just haven't quite done it yet. From people I've spoken to, everyone praises Fastmail... seems like we all need that extra push.

ebabchick 12y ago

For the well acquainted -- how does this relate to Inbox (https://www.inboxapp.com/ https://www.inboxapp.com/)? I have not spent enough time with either to make a comparison

geofft 12y ago
I haven't actually used either of these products, so I might be totally wrong, but.... Inbox is a separate layer on top of existing email servers. My impression is that the pain point they're addressing is, you want to build some awesome email integration app and sell it to companies, but you're looking at how complicated it would be to speak IMAP and IMAP-with-quirks and ActiveSync and the Gmail API and everything else (in order to build a high-quality integration) and that's scaring you. You can outsource that problem to Inbox, they'll make sure they get good-quality data from any email server reliably, and you just speak an HTTP API. This involves deploying a separate server which caches email. You talk to the "sync engine" over a reasonable protocol, and the sync engine takes care of talking an unreasonable protocol to wherever the data actually lives. JMAP is an API to directly access an email store. It's something that an email provider (Fastmail, most obviously) will directly support, but it also only helps you for those providers that actually speak JMAP. If you're an email server developer, an ISP, etc., then JMAP is cool and Inbox is not really useful -- there's not much of a point in having an email server that speaks IMAP and then also sticking a frontend to convert IMAP to something else. You might as well just make your email server speak a reasonable protocol directly. If you're an app author, Inbox is useful today, and JMAP is only useful, maybe, if all your users are Fastmail users. If you're an in-house developer at a large company that's running Exchange, and realistically you have no hope of getting the email admins to switch email servers (and you wouldn't want to run the email service yourself, even if they offered), JMAP is useless to you, and Inbox is super compelling. (There is probably an argument that the Inbox "sync engine" should speak JMAP on the front end / those APIs should converge. Certainly the Inbox sync engine should speak JMAP on the back end, so Inbox can advertise support for Fastmail.)
grinich 12y ago
Michael from Inbox here— a few key differences, from the horse’s mouth. :) First of all— I want to say that we’re huge fans of Fastmail and really admire their team and focus. Rob and Neil actually came by for lunch this summer and hacked in our office for the day. We’ve discussed JMAP back and forth with them for a while, and specifically had a long thread when we added calendar endpoints to the Inbox platform. [1] The fundamental differences between the Inbox APIs and JMAP are the kinds of applications built with them, the open source code, and the email providers supported. The Inbox APIs are really designed to let developers access email data in non-traditional ways. Our site reads “Say goodbye to IMAP” but we’re not so much replacing IMAP, as creating a new protocol which liberates email data to a higher form. We think about email as “the database of your life” which means unconventional “clients” and access/manipulation patterns. That’s one of the reasons we have a more REST-inspired design, whereas JMAP is closer to RPC. The Inbox API endpoints strive to represent canonical objects that developers simply treat as resources. JMAP is tailored specifically for traditional email clients, so it excels at fast operations (like archive -> reload), but compromises in other areas of simplicity and clarity IMHO. The other main difference is code. The Inbox sync engine is open source with about 15k lines of code on GitHub.[2] You can run it yourself right now, and it works on all providers, including Fastmail! That means your users don’t need to be on any specific provider or backend. If you don’t want to run it yourself, you can use our hosted version which also includes support for enterprise environments that use Microsoft Exchange. [3] I’ve been lurking on the IETF mailing lists for several years now, and one thing I learned is that spec documents hardly move the world forward. Developers need code that is battle tested and they can depend on. What use is a API spec if you can’t build an app with it? The JMAP system is a great spec for FastMail’s internal mail service, and it works to power their own clients, but unless they make some drastic changes to the code and business, it’s not a platform for building 3rd party mail apps. As an aside, I totally wish I had more time to write. I envy how prolific the Fastmail blog is these days. :) Keep up the great work Bron, Rob, Neil, and the rest! [0] https://www.inboxapp.com https://www.inboxapp.com [1] https://www.inboxapp.com/docs/api#calendar https://www.inboxapp.com/docs/api#calendar [2] https://github.com/inboxapp/inbox https://github.com/inboxapp/inbox [3] https://www.inboxapp.com/features https://www.inboxapp.com/features

hedgehog 12y ago

Neat. Two questions about the digest: 1. You recommend using a digest of the message to generate the message ID. Did you consider mandating that scheme or going farther and using a Merkle tree for the mailbox representation? It seems like this would allow for generating single requests that can fetch all new items. 2. Why SHA-1? Edit: Also, thanks for keeping Fastmail running. As a customer of about ten years it's much appreciated.

brongondwana 12y ago
No mandated format for message-ID. If you are proxying an existing store with a stable message-ID, you can just use that. My partially written gmail proxy uses the MSGID from Gmail, for example. SHA1 was a good balance between speed and unforgability at the time. There's still no known collision, and I'm working on putting a piece of randomness into the LMTP headers that are added to every message on delivery into Cyrus to make it even safer.

iwantagrinder 12y ago

I' m interested in this from a security perspective. What does this new protocol offer in terms of better control around what makes it to the inbox? Would IMAP>JMAP translation before hitting the user give us better ability to filter out malicious items/spam?

SwellJoe 12y ago
I think the only thing this particular protocol should be concerned with is reliably providing a method for communicating between client and server. Most spam and malicious mail filtering should happen as early in the process as possible, hopefully during the initial connection. There is a a protocol named Sieve ( http://www.ietf.org/rfc/rfc5228.txt http://www.ietf.org/rfc/rfc5228.txt ) which provides for delivery stage filtering rules. It is similar in capability and usage to procmail, but formalized and somewhat more modern (and not capable of running arbitrary system commands), and supported by some modern IMAP/POP servers, like Dovecot. Presumably if JMAP gets integrated into those servers, Sieve would also be available. And, I would hope there wouldn't be an IMAP/JMAP translation layer, but instead servers would implement JMAP directly. Though I guess in the short term there might be some sort of proxy.
- gecko 12y ago
  Sieve is the native filtering engine used by FastMail, so I'd say it integrates quite well with JMAP, from my experience. :)
groby_b 12y ago
So am I. From looking at http://jmap.io/spec.html#authentication http://jmap.io/spec.html#authentication, it looks like the password will be transmitted in plain text. (See the text below the 200 response) That makes me extremely queasy. Yes, HTTPS theoretically provides transport layer security, but a single breach of transport layer security results in the attackers permanent access to your mail. I.e. run a MITM attack in a coffee shop, snoop up JMAP passwords, make use of them at a latter point in time. I really hope I'm missing something important, or misread the spec.

sjtrny 12y ago

http://xkcd.com/927/ http://xkcd.com/927/

LeoPanthera 12y ago
Funny, but how many open mail retrieval protocols are in common use? I can think of maybe two - IMAP and POP3, and a small selection of proprietary ones, dominated by Exchange/ActiveSync. The time is right for a better standard.
brongondwana 12y ago
My presentation to the Thunderbird folks led with that, and we would have used it at Inbox Love too if we'd had time, but a 5 minute presentation had to get straight to the point. It's a good comic. Good point. We didn't take this decision lightly. There are significant reasons why all the existing standards don't do what's needed, and we made sure we built a technology preview and used it in the real world first - check the video or try out FastMail yourself.

plg 12y ago

Amazing. Modern, innovative, service-oriented products and a business model based on customers giving the company money in return for services. I hope that this is a reminder to startups out there that you don't have to trick customers into handing over their personal information all the while pretending to offer your services "for free", in order to gain market share and be profitable. The "traditional" business model can be quite successful.

Touche 12y ago
I hope you're right but I fear that Fastmail isn't successful in a way that startup founders are going for. Advancing technology unfortunately isn't a priority for most founders.
- robn_fastmail 12y ago
  We're fully owned by the staff, with no external funding. We've been nicely profitable since the beginning. Not huge profits but enough to pay for a nice office, decent salaries and few gadgets. So yes, if your goal is to attract massive amounts of venture capital and then get bought out, don't copy us. If you want to have some fun doing something you love without having to worry about anything else then this is definitely the way to do it :)

ummjackson 12y ago

FastMail continue to impress me. Good work folks.

nona 12y ago

I hate to be critical considering the considerable work they've put in, but I don't buy their "JMAP is actually more REST-like than most “RESTful” APIs" statement. But maybe I'm missing a whole lot. REST doesn't preclude doing multiple things in one round trip. Well it does seem JMAP is a bit more flexible when it comes to multiplexing very different actions. But the error handling doesn't seem to tell you which command has failed. You still can't get away from certain ordering issues. And to be honest, I don't see how it's going to be that easy to cache. This really is RPC – using it as a straight replacement for IMAP might work well, but I wouldn't want to use it as a platform to build something very different than an email client on it.

nmjenkins 12y ago
Each call is tagged by the client and the same tag is added to the response by the server, so you can tell exactly what error corresponds to which call. Caching is pretty easy: you just keep a cache of each object type. The delta update mechanism means you can very accurately invalidate what you need to keep it up to date. The system is actually really flexible and a great fit for any CRUD based app; JMAP is really just a combination of a very powerful and efficient database access protocol, and the definition of some objects to represent email, contacts and calendars.
- icebraining 12y ago
  Not just some objects, but different methods to access different objects, and it's those methods that implement the semantics - the base of JMAP is just an encoding for bulk RPC. What I mean is, if tomorrow I add a file with the mediatype application/vnd.icebraining to my server, an intermediate HTTP proxy can still cache it even though it knows nothing about my format. But if I add a new object (say, Note) to my JMAP server, along with the appropriate methods (getNotes, getNoteUpdates, etc), an intermediate JMAP server will be unable to cache them, since it needs to know the semantics of the new methods. So frankly, I don't see how their protocol follows an uniform interface at all. I still think it's pretty neat, though, don't take me wrong. I just want people to stop abusing REST. There's nothing wrong with RPC!

Animats 12y ago

What does this do that IMAP and SMTP don't do already? All you need is an IMAP server, accessed from all your devices. The one that comes with Android isn't bad. Thunderbird works fine on the desktop. You don't have to use GMail with Android; when you first power up your Android device, click "Later" when it asks for a Google login. Then delete the Google One-Time Startup app. Google won't bother you again.

corford 12y ago
tldr; version: It reduces the number of network roundtrips (and server overhead) needed to keep clients in sync (great for mobile) and addresses the fragmentation issue between the various server & client implementations of the sprawling IMAP protocol. By moving to JSON and HTTP it also makes the barrier to entry a lot lower for client developers and opens up the possibility to improve speed yet further with GZIP and websockets. All that while being able to stay compatible with legacy IMAP servers via a reference proxy provided for free by Fastmail. Edit: Oops I've just noticed your handle so apologies for assuming you didn't read the article. I guess you've read it and still don't see the benefit. In which case would love to know why you think it's a wasted effort (personally, I think it looks like a welcome refresh).
- Animats 12y ago
  The Ruby version of the IMAP client isn't that bad. Yes, IMAP doesn't use the current fad for marshaling, JSON. But a few years ago the web crowd would have wanted XML, and a few years from now they'll probably want some binary protocol Google comes up with. "Gzip" compression isn't a big win for objects the size of email messages, anyway. IMAP servers that don't use a real database are sure to be a mess, but that's not the fault of the protocol. I'd rather not have some startup in the middle of the mail chain. They'll start thinking they own mail.
feld 12y ago
IMAP is a terrible protocol. It's a poorly designed filesystem with all the baggage that comes with it. Read the IMAP overview in the IMAP ruby module -- it's disgusting and was never meant for the modern world. http://rxr.whitequark.org/mri/source/lib/net/imap.rb http://rxr.whitequark.org/mri/source/lib/net/imap.rb

msh 12y ago

I guess this tries to solve the same issue as microsoft active sync, just in a open specification way.

mike-cardwell 12y ago

I understand you guys use and contribute to Cyrus. Is there a JMAP interface planned for Cyrus? Or for any other mail servers, like Dovecot?

aroman 12y ago
from the article itself: "Finally, we know IMAP, SMTP and the DAVs aren’t going away any time soon. No protocol will succeed unless it provides an upgrade path from where we are now, and a compelling reason to switch. We will provide a proxy which can talk to existing servers and present them over JMAP."
- mike-cardwell 12y ago
  Your comment implies that the quote you provided answered my question. It didn't.
robn_fastmail 12y ago
We're working on a proxy that will work with any IMAP server that of course will work with Cyrus. I think its likely that we'll look into building support directly into Cyrus at some point but we don't have any concrete plans yet.
brongondwana 12y ago
http://git.cyrusimap.org/cyrus-imapd/tree/imap/http_jmap.c?h=caldav-2.4 http://git.cyrusimap.org/cyrus-imapd/tree/imap/http_jmap.c?h... Totally a stub right now, but yes - we would like to have it directly in Cyrus...