12 ms·

JMAP: It’s Like IMAP but Not Really (2019)

184 points by jbellis 4y ago

bee_rider 4y ago
So, it isn't just IMAP but for electrical engineers? (/s)
- deleted 4y ago
  [deleted]
deleted 4y ago
[deleted]
YPPH 4y ago
Is it too late for any significant improvements to email? It seems like, to our lament, we're headed in the direction of proprietary instant chat services and protocols. Also, you can't overlook that a significant number of people use Gmail/Google Apps and Outlook.com/Office 365 which each use a proprietary communication protocol that is unlikely to change in the future. So, it does make me wonder, who does JMAP benefit?
- mcint 4y ago
  JMAP benefits people who want to use broadly accepted, open standards. Especially standards better adapted to present and future needs: mobile use - lower latency & fewer round trips, lower end-user device power usage. JMAP benefits FastMail (and others). The CEO of FastMail is also a heavy-lifter in standardizing JMAP, Bron Gondwana. It's probably appropriate to mention he's also a lead developer of the Cyrus IMAP free-software project, so it's hard to accuse him of making a private moat for his company. Self-hosters, and small businesses, who can run software, and possibly contribute, but not write their own email software from scratch—-benefit. JMAP is targeting email, and soon calendar, contacts, and maybe more general messaging, to run better suited to current network and user conditions. https://jmap.io/ https://jmap.io/ https://www.ietf.org/blog/jmap/ https://www.ietf.org/blog/jmap/ Who does TCP benefit, JSON, .well-known? who does vcf/vCard benefit, or .ics (even if it started as a proprietary format)?
  - mcint 4y ago
    The multiplexing of requests has me pretty interested. If I made myself the spare time, or had a business model, I would love to implement chat bridges (for personal, self-hosted) small-scale use, using JMAP. I like, I'm hopeful for the potential of, the promise of, REST-ful state transfer but with multiplexing of requests internal to the protocol. Think of the protocol as sending a git-patch to modify state, instead of sending a diff per file, "resource". Analogous to how QUIC (which is still adding standardization features) works around the problem of head of line blocking on TCP connections (and on http2's HTTP sequentially multiplex-able response streams), a core theme supporting in existing JMAP for email & attachments is multiple application requests per web-request, on-the-wire payload. Said another way, I appreciate the multiplexing in: - per web request, - per application request (JMAP adds multiplexing here) - per resource, accessible to the user, on the server, - you can create, read, update, delete, move, etc.
  - brongondwana 4y ago
    Thanks for the callout :) I'm not doing so much development on Cyrus any more - I still attend the weekly team meetings and do code and design review, but trying to keep my focus more on company operations and standardisation work. JMAP core and email standards haven't changed in the past few years because they're done! In the JMAP working group at the IETF we're currently heavily focused on getting the calendaring and contacts bits finished. That unlocks significantly more benefit for a JMAP client because you don't need multiple different authentications. As for email clients - yeah, it's been slow going, partly because when COVID hit we all shrank back into our shells a bit, and certainly my travel and networking was cut back! It's good to see how it's perceived from outside our little bubble where JMAP is doing great and we're moving more and more of Fastmail's internal APIs to be JMAP-style because it's so easy to multiplex requests together and route them separately through our middleware into different systems, then combine the results back to together in a single response to the calling system or browser.
    - soco 4y ago
      Bringing the documentation up to date would be then a logical step.
- georgyo 4y ago
  > Also, you can't overlook that a significant number of people use Gmail/Google Apps and Outlook.com/Office 365 which each use a proprietary communication protocol that is unlikely to change in the future. All of those support IMAP currently, an open standard. Exchange has MAPI as well. Email is the best federated system we might ever have, and it has thousands of clients that can work with nearly any provider. JMAP has not taken off, but there is a chicken and egg problem. Most clients don't support it so there is no pressure on services to support it. Though I imagine if Gmail supported it then suddenly all the clients would support it
  - josephg 4y ago
    > All of those support IMAP currently, an open standard. Exchange has MAPI as well. Gmail supports IMAP, with custom extensions to support threading. As far as I understand it they use a custom protocol for their own gmail apps because of IMAP's shortcomings. Its basically impossible to build a modern email app directly on top of gmail's public APIs. You can do it - but only by mirroring and re-indexing all the email.
- Gigachad 4y ago
  Email has already largely become a newsletter and notification distribution system. Average people don't write or read emails anymore outside of some work settings but even then its usually Slack or MS Teams. Technology and people move quickly so fossilized systems such as email get left behind. There is too much friction to improve a system which has a large number of different implementations. Where any change is hotly debated and people still resist features added to email over a decade ago. Naturally the proprietary systems win when they can roll out a new feature and have it universally work on day one.
  - alberth 4y ago
    > Average people don't write or read emails anymore outside of some work settings but even then its usually Slack or MS Teams. Do others believe/experience this? I ask because this sounds like a very Bay Area comment.
    - Gigachad 4y ago
      I'm Australian and have worked for a variety of companies and its been over a year since someone internal at the company sent me a non broadcast newsletter email. Email is much like SMS these days, you use it to arrange job interviews or talk occasionally between businesses but you would never email someone you know directly.
      - alberth 4y ago
        Do you mind if I ask how many employees (order of magnitude) these companies have. Eg 100, 1000, 10000, etc.
        Gigachad 4y ago
        Previous was 50, current is 500. All communication is over MS Teams now.
      - wildfire 4y ago
        I'm Australian and have the direct counter experience. Most of the business I've worked for consider Slack as ephermal. Email is for significant communications. Outside of work, I'm using daily for things like booking bands, talking to strata, etc. While email in a lot of large companies might tend to be becoming a one-way distribution system; it is still vibrant and heavily used in smaller enterprises.
verisimilitudes 4y ago
Yeah, what can't be fixed by shoveling JSON shit into HTTP instead of a dedicated protocol? JMAP was stupid when the Fastmail CEO or whoever was here pushing it, and it's still stupid.
- ikt 4y ago
  Why do you call JSON shit? It's incredibly useful and I use it all the time, am I missing something?
  - shakna 4y ago
    JSON has a few rough points that make it frustrating to parse. Like for example, most values requiring open and ending tags, no trailing commas allowed (so you must either be lax and non-compliant or backtrack the parser on last element), and other similar edge cases. This results in making it extraordinarily easy to break most parsers in use, today. For example, to kill Python: n="$(python3 -c 'import math; import sys; sys.stdout.write(str(math.floor(sys.getrecursionlimit() - 4)))')" left="$(yes [ | head -n "$n" | tr -d '\n')" echo "$left" | python3 -c 'import json; print(json.loads(input()))' That's a less than 1000-character long object, that can explode. Whilst there are more robust parsers you can get for Python, it doesn't change the fact that generating bad JSON is _easy_. So when you're handling something like email, on a mobile device which may drop a partial connection, you need a hell of a lot of extra work to show the partial download, which might even be everything except the closing braces, or you have to discard the whole thing - which isn't kind to your user.
    - duskwuff 4y ago
      > JSON has a few rough points that make it frustrating to parse. Even so, parsing JSON is a well-understood problem, and there are tons of mature libraries to parse and generate it. I'd much rather work with a JSON-based protocol than one which made up its own wacky data format. The same goes for the use of HTTP. It may not be the ideal protocol for accessing mail, but everyone knows how to work with HTTP, and tools to interact with it are plentiful.
kyrra 4y ago
2019.
layer8 4y ago
See here for a list of software supporting JMAP: https://jmap.io/software.html https://jmap.io/software.html
derekzhouzhen 4y ago
There are 3 different kinds of email services that only have limited interoperability: * Corporate email services, aka Outlook/Exchange * Surveillance email services, aka Gmail, etc * Real email services that expose the raw email database, such as Maildir, to the users. For the first 2 kinds, I'd rather POP and keep my interaction with them to the minimum. For the last kind, IMAP/JMAP is too limiting in capability.
syntaxfree 4y ago
Now, to build a better IMAP on top of UMAP.
superkuh 4y ago
JMAP sounds like a crutch for mobile devices that can't keep a TCP connection open. The internet is slowly being gimped to support mobile devices.
- aetherspawn 4y ago
  Pretty much every device is mobile now. For example, people will probably start reading emails on the center console of their autonomous cars on the way to work over 5G. Nevertheless, a valid current-day example for non-mobile devices requiring a tolerance to dropped connections: I regularly dock and undock my laptop when heading to meeting rooms, and simply switching from docked Ethernet to WiFi causes a network switch that breaks a lot of apps. Engineering tolerance for this kind of thing is only good.
  - superkuh 4y ago
    Yes. That's reality. Most devices people use are terrible, terrible computers and they can't do what computers used to be able to do. This is a bad thing.
    - Gigachad 4y ago
      Users don't care what you consider to be a good computer. They use the proprietary email provider and it just works and shows emails on their car. While FOSS fanboys are sitting around complaining about the decay of tech because of JSON protocols.
    - gregmac 4y ago
      Honestly this sounds like a "if your computer weighs less than 5 tons and can fit on a desk, it's not a real computer" argument someone would have made 30 years ago. Reality is people use multiple different and mobile devices, roam across connections, and intermittently go offline. Building software and protocols that are designed around this is beneficial to every use case - even those using whatever you consider a non-"terrible computer" to be.
      - 4y ago
nick__m 4y ago
Almost all requests in JMAP are to the same URL using an HTTP POST to submit a JSON body of “methods”. So it's an adhoc reimplementation of SOAP using json instead of xml?
- masklinn 4y ago
  Tunnelling everything through post is the normal way to do e.g. rpc over http. It’s not a soap thing, it’s also how xml-rpc, json-rpc, and graphql work. More generally it’s how you layer over http when you can’t necessarily guarantee respecting its requirements like purity, idempotence, or cacheability (or don’t want to bother), or when you fear hitting it’s practical implementation limits.
  - p_l 4y ago
    And SOAP inherited it from xml-rpc.
- Octoth0rpe 4y ago
  And more importantly, that's _extremely_ un-REST-like, at least in the modern use of the term.
encryptluks2 4y ago
I want hashes for email and not just UIDs. Does any protocol do this currently where it hashes the entire message?
- andris9 4y ago
  Email hashes are not very useful as additional headers are added to the email every step on the way. Even if you send an email to yourself, the copy in the Sent Mail folder has a different hash than the one in Inbox.
  - encryptluks2 4y ago
    They are useful for verification though that an email is the original, and for detecting changed messages when syncing.
    - sbuk 4y ago
      DKIM offers something similar.
    - 7steps2much 4y ago
      At least for the verifying that an email is original I agree with the other poster, DKIM exists for that purpose, to sign emails while in transit. As for changed messages while synching, do your messages change after delivery? If not, a simple delivered after X would be sufficient.
      - encryptluks2 4y ago
        Yes messages can change after syncing. Using mbsync for example, if you change the contents of an email locally, there are no ways to detect the change. I guess the preferred solution may be to do a more generic sync directly from the mail server and bypass IMAP entirely.. not possible with most email providers like Gmail, Microsoft, etc. Hopeful to setup my own email server someday.
1995mike 4y ago
newscracker 4y ago
JMAP sounded promising several years ago, and it was great to hear it published as a standard. But it seems like it’ll just remain as what Fastmail (the email provider) uses. Maybe the JMAP team is very small and can’t get more people. Maybe Fastmail doesn’t have enough money and/or enough interest to make it easier to use and to evangelize it with major email providers and email client developers. I don’t see Outlook365 or Gmail (or even Apple for its iCloud email that has a relatively smaller user base) announcing any implementations of JMAP on their services. I don’t see smaller paid email service providers announce support for it. As it stands now, I don’t see a future for JMAP outside of Fastmail. A popular client like Mozilla Thunderbird still hasn’t started implementing JMAP [1] and one of the reasons is because…the client and server documentation for JMAP haven’t been updated for a few years. [2] So yeah, JMAP is like IMAP, but not really. We probably need a successor to JMAP that is liked enough to be implemented by email service providers and by email clients. [1]: https://bugzilla.mozilla.org/show_bug.cgi?id=1322991 https://bugzilla.mozilla.org/show_bug.cgi?id=1322991 [2]: https://github.com/jmapio/jmap/issues/323 https://github.com/jmapio/jmap/issues/323
- deleted 4y ago
  [deleted]
- brongondwana 4y ago
  I have more faith in JMAP's eventual success than you do, but I do agree that it's frustratingly slow. We knew that adoption would be frustrating slow of course, but I do hope that in 3 years' time we'll have a much better story than we have had in the three years since initial standardisation of the core and mail parts. Certainly once calendars and contacts are done, and hopefully tasks as well, that will make a more complete ecosystem that is even more attractive than just the email component!
  - Loic 4y ago
    My personal take is that JMAP would start being widely used if we had some simple Python/Go bindings to easily manipulate the emails. My first reaction as I have seen this standard popping up was not "my email will be faster/better", but "I will be able to setup a nice support@mycompany.com" email address and have both email/web integration with it. If developers start to embrace JMAP for a derivative usage, I think it could drive the adoption.
deleted 4y ago
[deleted]
aorth 4y ago
On a related note: the author links to his open source web mail platform Cypht. It looks cool and sounds promising. Anyone tried it? I can't remember if it has been on HN. The git history of the project is very active.
- degrees57 4y ago
  I have the Helm email appliance, and what it does _not_ come with is a webmail interface. (It does have Nextcloud, though). I put Cypht on a different box to access my Helm mail, and it was very useful. At the moment it is down because I did a migration from one host to another, there is some sort of login problem on the new box with the old database. RSN I'll have time to diagnose and fix.
- jasonmunro 4y ago
  Hello, author here! Not sure who posted this old blog article but great to see an interesting discussion and new interest in the project. Development is active and we always appreciate new feedback!
qwerty456127 4y ago
I would go further and also replace SMTP with the same for clients. As I understand we still need SMTP to pass mail between servers but a client already having access to IMAP/JMAP/whatever could just put the mail into the outbox folder for the server to deliver it.
- jeltz 4y ago
  Agreed, I think the separation between SMTP (as used for submission) and IMAP makes sending emails unnecessarily complicated. Edit: From a quick glance it seems like JMAP actually supports submission.
asim 4y ago
Whoa I love SMTP and email, as a distributed systems nerd I'm a huge fan of the federated protocol and it's simplicity but JMAP I'm sorry is garbage. The http/json part is great, the potential to simplify email access and use is great, even using DNS SRV records is a potential good use of DNS and a compliment to MX records. But the actual JMAP protocol is horrible. This is something built incrementally for fastmail, it's not how you design a protocol. We need to go back to square one and redesign and access and communication protocol as one thing that's ultra simplistic as a building block and starts with very basic primitives. JMAP combines too many things while being very ad-hoc and custom not taking into account a lot of progress made in things like gRPC protos, etc. Tempted to build it from scratch just to prove a point.
lewisl9029 4y ago
Does JMAP do anything about the fact that it takes on average multiple whole number seconds, with frequent spikes into 10s of seconds, for email delivered to show up in people's inboxes? Source: https://status.postmarkapp.com/ https://status.postmarkapp.com/ If not, is anyone doing anything to improve this situation? Sending packets over the internet across the farthest ends of the earth takes about 300ms roundtrip. What's preventing email from approaching this? Latency is the biggest UX issue facing email IMHO, and a huge part of why it is continuing to lose ground vs alternative messaging systems, most of which are proprietary, but offer sub-second latency delivery. The world will become a worse place if the trend continues to its logical conclusion and proprietary silos eventually fully supplants email's remaining use cases. The high average latency and even higher variance leads to people second guessing if an email they sent will ever arrive, and if an email they're expecting has ever been sent, resulting in crude workarounds like resend buttons becoming commonplace and seen as necessary, perpetuating the uncertainty around deliverability, and even further eroding trust in the system. But most importantly, the latency problem makes email unsuitable for the most compelling use case for messaging platforms: real-time conversation. Holding a spontaneous real-time conversation over email is painful, as anyone who has tried can probably attest. Both sides are constantly left wondering if the other side is simply taking their sweet time to reply, or if the email is just taking even longer than email usually takes to get delivered. Conversations that could have taken mere seconds end up taking minutes as a result. This is why email has been consistently losing to messaging apps for fun conversations between friends where email's latency would suck much of the fun out of them, to Slack for mission critical work conversations where wasting time means literally wasting money, to live chat widgets for businesses looking to provide a customer support experience that delights, whereas conversations over email frequently tends to frustrate, the list goes on... Solving the latency problem for email I believe is the key to slowing and possibly reversing the trend of email's slow but steady decline, and society will benefit massively from the continued prosperity of this rare miracle of an open system that managed to survive for so long despite all its shortcomings.
- denton-scratch 4y ago
  > unsuitable for the most compelling use case for messaging platforms: real-time conversation. If that's what you need, you use a real-time messaging application. That's not what email is for. I disapprove of the incorporation of email functionality behind a messaging UI; I get fed up of sending a properly-composed email, and getting a one-liner reply, because it's hard to compose a thought-out, formatted email on a phone.
andyjohnson0 4y ago
The article explains the technical advantages of JMAP, but I'm not clear how (if at all) it benefits email users. Without clear end-user advantages its hard to see why email providers would want to implement it when they already support just-good-enough IMAP.
- danielheath 4y ago
  Key things that are hard in IMAP and easy in JMAP are well summarized in this comment: https://lobste.rs/s/er97xr/so_when_did_pop_imap_become_legacy#c_bcye2y https://lobste.rs/s/er97xr/so_when_did_pop_imap_become_legac...
  - andyjohnson0 4y ago
    Global search is clearly important. The other "hard thing" are a little... niche? AFAIK search in IMAP is folder-based, so you have to iterate over the folders, do multiple searches, and merge the results. Not elegant, and probably not fast, but it doesn't seem like a protocol killer to me.
    - danielheath 4y ago
      One persons 'niche' is anothers 'essential feature'. Fastmail are surely in a better position than you or I to figure out which kinds of searches are commonly performed!
jgalt212 4y ago
JMAP is nice if you sell self-hosting data mining software. One less connector to write.
0x445442 4y ago
> Almost all requests in JMAP are to the same URL using an HTTP POST to submit a JSON body of “methods”. A method is an action or query you want to perform like “give me the 20 newest messages in this mailbox” Can the HTTP spec just add a CALL verb already?
jasonmunro 4y ago
hrm, handful of new github stars in one day? That's odd. Must have gotten picked up by something. Oh, HN repost of a blog post I wrote 3 years ago, cool! To sum up my thoughts about this thread: Appreciate the great discussion! Wherever you land on the validity of JMAP, as a long (long) time IMAP client developer I stand by my position that JMAP is a huge improvement over legacy IMAP. Not perfect, but walks a reasonable line between what IMAP client devs understand and the modern world of APIs 30 years later. Is it REST compliant or a SOAP derivative? Most likely no and yes. Does it only benefit client developers? Probably, but that is a big benefit to the E-mail ecosystem. IMO the biggest barrier to new and innovative E-mail clients is the obtuseness of the protocol itself, and JMAP goes a long way to bridge that gap.

184 points by jbellis 4y ago

bee_rider 4y ago

So, it isn't just IMAP but for electrical engineers? (/s)

deleted 4y ago
[deleted]

deleted 4y ago

[deleted]

YPPH 4y ago

Is it too late for any significant improvements to email? It seems like, to our lament, we're headed in the direction of proprietary instant chat services and protocols. Also, you can't overlook that a significant number of people use Gmail/Google Apps and Outlook.com/Office 365 which each use a proprietary communication protocol that is unlikely to change in the future. So, it does make me wonder, who does JMAP benefit?

mcint 4y ago
JMAP benefits people who want to use broadly accepted, open standards. Especially standards better adapted to present and future needs: mobile use - lower latency & fewer round trips, lower end-user device power usage. JMAP benefits FastMail (and others). The CEO of FastMail is also a heavy-lifter in standardizing JMAP, Bron Gondwana. It's probably appropriate to mention he's also a lead developer of the Cyrus IMAP free-software project, so it's hard to accuse him of making a private moat for his company. Self-hosters, and small businesses, who can run software, and possibly contribute, but not write their own email software from scratch—-benefit. JMAP is targeting email, and soon calendar, contacts, and maybe more general messaging, to run better suited to current network and user conditions. https://jmap.io/ https://jmap.io/ https://www.ietf.org/blog/jmap/ https://www.ietf.org/blog/jmap/ Who does TCP benefit, JSON, .well-known? who does vcf/vCard benefit, or .ics (even if it started as a proprietary format)?
- mcint 4y ago
  The multiplexing of requests has me pretty interested. If I made myself the spare time, or had a business model, I would love to implement chat bridges (for personal, self-hosted) small-scale use, using JMAP. I like, I'm hopeful for the potential of, the promise of, REST-ful state transfer but with multiplexing of requests internal to the protocol. Think of the protocol as sending a git-patch to modify state, instead of sending a diff per file, "resource". Analogous to how QUIC (which is still adding standardization features) works around the problem of head of line blocking on TCP connections (and on http2's HTTP sequentially multiplex-able response streams), a core theme supporting in existing JMAP for email & attachments is multiple application requests per web-request, on-the-wire payload. Said another way, I appreciate the multiplexing in: - per web request, - per application request (JMAP adds multiplexing here) - per resource, accessible to the user, on the server, - you can create, read, update, delete, move, etc.
- brongondwana 4y ago
  Thanks for the callout :) I'm not doing so much development on Cyrus any more - I still attend the weekly team meetings and do code and design review, but trying to keep my focus more on company operations and standardisation work. JMAP core and email standards haven't changed in the past few years because they're done! In the JMAP working group at the IETF we're currently heavily focused on getting the calendaring and contacts bits finished. That unlocks significantly more benefit for a JMAP client because you don't need multiple different authentications. As for email clients - yeah, it's been slow going, partly because when COVID hit we all shrank back into our shells a bit, and certainly my travel and networking was cut back! It's good to see how it's perceived from outside our little bubble where JMAP is doing great and we're moving more and more of Fastmail's internal APIs to be JMAP-style because it's so easy to multiplex requests together and route them separately through our middleware into different systems, then combine the results back to together in a single response to the calling system or browser.
  - soco 4y ago
    Bringing the documentation up to date would be then a logical step.
georgyo 4y ago
> Also, you can't overlook that a significant number of people use Gmail/Google Apps and Outlook.com/Office 365 which each use a proprietary communication protocol that is unlikely to change in the future. All of those support IMAP currently, an open standard. Exchange has MAPI as well. Email is the best federated system we might ever have, and it has thousands of clients that can work with nearly any provider. JMAP has not taken off, but there is a chicken and egg problem. Most clients don't support it so there is no pressure on services to support it. Though I imagine if Gmail supported it then suddenly all the clients would support it
- josephg 4y ago
  > All of those support IMAP currently, an open standard. Exchange has MAPI as well. Gmail supports IMAP, with custom extensions to support threading. As far as I understand it they use a custom protocol for their own gmail apps because of IMAP's shortcomings. Its basically impossible to build a modern email app directly on top of gmail's public APIs. You can do it - but only by mirroring and re-indexing all the email.
Gigachad 4y ago
Email has already largely become a newsletter and notification distribution system. Average people don't write or read emails anymore outside of some work settings but even then its usually Slack or MS Teams. Technology and people move quickly so fossilized systems such as email get left behind. There is too much friction to improve a system which has a large number of different implementations. Where any change is hotly debated and people still resist features added to email over a decade ago. Naturally the proprietary systems win when they can roll out a new feature and have it universally work on day one.
- alberth 4y ago
  > Average people don't write or read emails anymore outside of some work settings but even then its usually Slack or MS Teams. Do others believe/experience this? I ask because this sounds like a very Bay Area comment.
  - Gigachad 4y ago
    I'm Australian and have worked for a variety of companies and its been over a year since someone internal at the company sent me a non broadcast newsletter email. Email is much like SMS these days, you use it to arrange job interviews or talk occasionally between businesses but you would never email someone you know directly.
    - alberth 4y ago
      Do you mind if I ask how many employees (order of magnitude) these companies have. Eg 100, 1000, 10000, etc.
      - Gigachad 4y ago
        Previous was 50, current is 500. All communication is over MS Teams now.
    - wildfire 4y ago
      I'm Australian and have the direct counter experience. Most of the business I've worked for consider Slack as ephermal. Email is for significant communications. Outside of work, I'm using daily for things like booking bands, talking to strata, etc. While email in a lot of large companies might tend to be becoming a one-way distribution system; it is still vibrant and heavily used in smaller enterprises.

verisimilitudes 4y ago

Yeah, what can't be fixed by shoveling JSON shit into HTTP instead of a dedicated protocol? JMAP was stupid when the Fastmail CEO or whoever was here pushing it, and it's still stupid.

ikt 4y ago
Why do you call JSON shit? It's incredibly useful and I use it all the time, am I missing something?
- shakna 4y ago
  JSON has a few rough points that make it frustrating to parse. Like for example, most values requiring open and ending tags, no trailing commas allowed (so you must either be lax and non-compliant or backtrack the parser on last element), and other similar edge cases. This results in making it extraordinarily easy to break most parsers in use, today. For example, to kill Python: n="$(python3 -c 'import math; import sys; sys.stdout.write(str(math.floor(sys.getrecursionlimit() - 4)))')" left="$(yes [ | head -n "$n" | tr -d '\n')" echo "$left" | python3 -c 'import json; print(json.loads(input()))' That's a less than 1000-character long object, that can explode. Whilst there are more robust parsers you can get for Python, it doesn't change the fact that generating bad JSON is _easy_. So when you're handling something like email, on a mobile device which may drop a partial connection, you need a hell of a lot of extra work to show the partial download, which might even be everything except the closing braces, or you have to discard the whole thing - which isn't kind to your user.
  - duskwuff 4y ago
    > JSON has a few rough points that make it frustrating to parse. Even so, parsing JSON is a well-understood problem, and there are tons of mature libraries to parse and generate it. I'd much rather work with a JSON-based protocol than one which made up its own wacky data format. The same goes for the use of HTTP. It may not be the ideal protocol for accessing mail, but everyone knows how to work with HTTP, and tools to interact with it are plentiful.

kyrra 4y ago

2019.

layer8 4y ago

See here for a list of software supporting JMAP: https://jmap.io/software.html https://jmap.io/software.html

derekzhouzhen 4y ago

There are 3 different kinds of email services that only have limited interoperability: * Corporate email services, aka Outlook/Exchange * Surveillance email services, aka Gmail, etc * Real email services that expose the raw email database, such as Maildir, to the users. For the first 2 kinds, I'd rather POP and keep my interaction with them to the minimum. For the last kind, IMAP/JMAP is too limiting in capability.

syntaxfree 4y ago

Now, to build a better IMAP on top of UMAP.

superkuh 4y ago

JMAP sounds like a crutch for mobile devices that can't keep a TCP connection open. The internet is slowly being gimped to support mobile devices.

aetherspawn 4y ago
Pretty much every device is mobile now. For example, people will probably start reading emails on the center console of their autonomous cars on the way to work over 5G. Nevertheless, a valid current-day example for non-mobile devices requiring a tolerance to dropped connections: I regularly dock and undock my laptop when heading to meeting rooms, and simply switching from docked Ethernet to WiFi causes a network switch that breaks a lot of apps. Engineering tolerance for this kind of thing is only good.
- superkuh 4y ago
  Yes. That's reality. Most devices people use are terrible, terrible computers and they can't do what computers used to be able to do. This is a bad thing.
  - Gigachad 4y ago
    Users don't care what you consider to be a good computer. They use the proprietary email provider and it just works and shows emails on their car. While FOSS fanboys are sitting around complaining about the decay of tech because of JSON protocols.
  - gregmac 4y ago
    Honestly this sounds like a "if your computer weighs less than 5 tons and can fit on a desk, it's not a real computer" argument someone would have made 30 years ago. Reality is people use multiple different and mobile devices, roam across connections, and intermittently go offline. Building software and protocols that are designed around this is beneficial to every use case - even those using whatever you consider a non-"terrible computer" to be.
    - 4y ago

nick__m 4y ago

Almost all requests in JMAP are to the same URL using an HTTP POST to submit a JSON body of “methods”. So it's an adhoc reimplementation of SOAP using json instead of xml?

masklinn 4y ago
Tunnelling everything through post is the normal way to do e.g. rpc over http. It’s not a soap thing, it’s also how xml-rpc, json-rpc, and graphql work. More generally it’s how you layer over http when you can’t necessarily guarantee respecting its requirements like purity, idempotence, or cacheability (or don’t want to bother), or when you fear hitting it’s practical implementation limits.
- p_l 4y ago
  And SOAP inherited it from xml-rpc.
Octoth0rpe 4y ago
And more importantly, that's _extremely_ un-REST-like, at least in the modern use of the term.

encryptluks2 4y ago

I want hashes for email and not just UIDs. Does any protocol do this currently where it hashes the entire message?

andris9 4y ago
Email hashes are not very useful as additional headers are added to the email every step on the way. Even if you send an email to yourself, the copy in the Sent Mail folder has a different hash than the one in Inbox.
- encryptluks2 4y ago
  They are useful for verification though that an email is the original, and for detecting changed messages when syncing.
  - sbuk 4y ago
    DKIM offers something similar.
  - 7steps2much 4y ago
    At least for the verifying that an email is original I agree with the other poster, DKIM exists for that purpose, to sign emails while in transit. As for changed messages while synching, do your messages change after delivery? If not, a simple delivered after X would be sufficient.
    - encryptluks2 4y ago
      Yes messages can change after syncing. Using mbsync for example, if you change the contents of an email locally, there are no ways to detect the change. I guess the preferred solution may be to do a more generic sync directly from the mail server and bypass IMAP entirely.. not possible with most email providers like Gmail, Microsoft, etc. Hopeful to setup my own email server someday.

1995mike 4y ago

newscracker 4y ago

JMAP sounded promising several years ago, and it was great to hear it published as a standard. But it seems like it’ll just remain as what Fastmail (the email provider) uses. Maybe the JMAP team is very small and can’t get more people. Maybe Fastmail doesn’t have enough money and/or enough interest to make it easier to use and to evangelize it with major email providers and email client developers. I don’t see Outlook365 or Gmail (or even Apple for its iCloud email that has a relatively smaller user base) announcing any implementations of JMAP on their services. I don’t see smaller paid email service providers announce support for it. As it stands now, I don’t see a future for JMAP outside of Fastmail. A popular client like Mozilla Thunderbird still hasn’t started implementing JMAP [1] and one of the reasons is because…the client and server documentation for JMAP haven’t been updated for a few years. [2] So yeah, JMAP is like IMAP, but not really. We probably need a successor to JMAP that is liked enough to be implemented by email service providers and by email clients. [1]: https://bugzilla.mozilla.org/show_bug.cgi?id=1322991 https://bugzilla.mozilla.org/show_bug.cgi?id=1322991 [2]: https://github.com/jmapio/jmap/issues/323 https://github.com/jmapio/jmap/issues/323

deleted 4y ago
[deleted]
brongondwana 4y ago
I have more faith in JMAP's eventual success than you do, but I do agree that it's frustratingly slow. We knew that adoption would be frustrating slow of course, but I do hope that in 3 years' time we'll have a much better story than we have had in the three years since initial standardisation of the core and mail parts. Certainly once calendars and contacts are done, and hopefully tasks as well, that will make a more complete ecosystem that is even more attractive than just the email component!
- Loic 4y ago
  My personal take is that JMAP would start being widely used if we had some simple Python/Go bindings to easily manipulate the emails. My first reaction as I have seen this standard popping up was not "my email will be faster/better", but "I will be able to setup a nice support@mycompany.com" email address and have both email/web integration with it. If developers start to embrace JMAP for a derivative usage, I think it could drive the adoption.

deleted 4y ago

[deleted]

aorth 4y ago

On a related note: the author links to his open source web mail platform Cypht. It looks cool and sounds promising. Anyone tried it? I can't remember if it has been on HN. The git history of the project is very active.

degrees57 4y ago
I have the Helm email appliance, and what it does _not_ come with is a webmail interface. (It does have Nextcloud, though). I put Cypht on a different box to access my Helm mail, and it was very useful. At the moment it is down because I did a migration from one host to another, there is some sort of login problem on the new box with the old database. RSN I'll have time to diagnose and fix.
jasonmunro 4y ago
Hello, author here! Not sure who posted this old blog article but great to see an interesting discussion and new interest in the project. Development is active and we always appreciate new feedback!

qwerty456127 4y ago

I would go further and also replace SMTP with the same for clients. As I understand we still need SMTP to pass mail between servers but a client already having access to IMAP/JMAP/whatever could just put the mail into the outbox folder for the server to deliver it.

jeltz 4y ago
Agreed, I think the separation between SMTP (as used for submission) and IMAP makes sending emails unnecessarily complicated. Edit: From a quick glance it seems like JMAP actually supports submission.

asim 4y ago

Whoa I love SMTP and email, as a distributed systems nerd I'm a huge fan of the federated protocol and it's simplicity but JMAP I'm sorry is garbage. The http/json part is great, the potential to simplify email access and use is great, even using DNS SRV records is a potential good use of DNS and a compliment to MX records. But the actual JMAP protocol is horrible. This is something built incrementally for fastmail, it's not how you design a protocol. We need to go back to square one and redesign and access and communication protocol as one thing that's ultra simplistic as a building block and starts with very basic primitives. JMAP combines too many things while being very ad-hoc and custom not taking into account a lot of progress made in things like gRPC protos, etc. Tempted to build it from scratch just to prove a point.

lewisl9029 4y ago

Does JMAP do anything about the fact that it takes on average multiple whole number seconds, with frequent spikes into 10s of seconds, for email delivered to show up in people's inboxes? Source: https://status.postmarkapp.com/ https://status.postmarkapp.com/ If not, is anyone doing anything to improve this situation? Sending packets over the internet across the farthest ends of the earth takes about 300ms roundtrip. What's preventing email from approaching this? Latency is the biggest UX issue facing email IMHO, and a huge part of why it is continuing to lose ground vs alternative messaging systems, most of which are proprietary, but offer sub-second latency delivery. The world will become a worse place if the trend continues to its logical conclusion and proprietary silos eventually fully supplants email's remaining use cases. The high average latency and even higher variance leads to people second guessing if an email they sent will ever arrive, and if an email they're expecting has ever been sent, resulting in crude workarounds like resend buttons becoming commonplace and seen as necessary, perpetuating the uncertainty around deliverability, and even further eroding trust in the system. But most importantly, the latency problem makes email unsuitable for the most compelling use case for messaging platforms: real-time conversation. Holding a spontaneous real-time conversation over email is painful, as anyone who has tried can probably attest. Both sides are constantly left wondering if the other side is simply taking their sweet time to reply, or if the email is just taking even longer than email usually takes to get delivered. Conversations that could have taken mere seconds end up taking minutes as a result. This is why email has been consistently losing to messaging apps for fun conversations between friends where email's latency would suck much of the fun out of them, to Slack for mission critical work conversations where wasting time means literally wasting money, to live chat widgets for businesses looking to provide a customer support experience that delights, whereas conversations over email frequently tends to frustrate, the list goes on... Solving the latency problem for email I believe is the key to slowing and possibly reversing the trend of email's slow but steady decline, and society will benefit massively from the continued prosperity of this rare miracle of an open system that managed to survive for so long despite all its shortcomings.

denton-scratch 4y ago
> unsuitable for the most compelling use case for messaging platforms: real-time conversation. If that's what you need, you use a real-time messaging application. That's not what email is for. I disapprove of the incorporation of email functionality behind a messaging UI; I get fed up of sending a properly-composed email, and getting a one-liner reply, because it's hard to compose a thought-out, formatted email on a phone.

andyjohnson0 4y ago

The article explains the technical advantages of JMAP, but I'm not clear how (if at all) it benefits email users. Without clear end-user advantages its hard to see why email providers would want to implement it when they already support just-good-enough IMAP.

danielheath 4y ago
Key things that are hard in IMAP and easy in JMAP are well summarized in this comment: https://lobste.rs/s/er97xr/so_when_did_pop_imap_become_legacy#c_bcye2y https://lobste.rs/s/er97xr/so_when_did_pop_imap_become_legac...
- andyjohnson0 4y ago
  Global search is clearly important. The other "hard thing" are a little... niche? AFAIK search in IMAP is folder-based, so you have to iterate over the folders, do multiple searches, and merge the results. Not elegant, and probably not fast, but it doesn't seem like a protocol killer to me.
  - danielheath 4y ago
    One persons 'niche' is anothers 'essential feature'. Fastmail are surely in a better position than you or I to figure out which kinds of searches are commonly performed!

jgalt212 4y ago

JMAP is nice if you sell self-hosting data mining software. One less connector to write.

0x445442 4y ago

> Almost all requests in JMAP are to the same URL using an HTTP POST to submit a JSON body of “methods”. A method is an action or query you want to perform like “give me the 20 newest messages in this mailbox” Can the HTTP spec just add a CALL verb already?

jasonmunro 4y ago

hrm, handful of new github stars in one day? That's odd. Must have gotten picked up by something. Oh, HN repost of a blog post I wrote 3 years ago, cool! To sum up my thoughts about this thread: Appreciate the great discussion! Wherever you land on the validity of JMAP, as a long (long) time IMAP client developer I stand by my position that JMAP is a huge improvement over legacy IMAP. Not perfect, but walks a reasonable line between what IMAP client devs understand and the modern world of APIs 30 years later. Is it REST compliant or a SOAP derivative? Most likely no and yes. Does it only benefit client developers? Probably, but that is a big benefit to the E-mail ecosystem. IMO the biggest barrier to new and innovative E-mail clients is the obtuseness of the protocol itself, and JMAP goes a long way to bridge that gap.