69 ms·

Though I have no complaints about the way they went about implementing it (open standards hooray!), as a mail client author and someone with lots of experience

Though I have no complaints about the way they went about implementing it (open standards hooray!), as a mail client author and someone with lots of experience with mail in general - I'm NACK to JMAP. I think that it comes with some questionable design decisions (HTTP? JSON?) which on their face I imagine look fine to the typical HNer, but in many ways is a regression from IMAP. I would prefer to see the limitations of IMAP addressed with a few carefully written extensions to the spec - which is already extensible and has a thriving set of useful extensions in the wild. By no means is IMAP a particularly good protocol, but it is extremely broadly supported and designed more with email in mind than JMAP appears to be, which seems to focus more on bringing shiny web tech to email regardless of its utility in such a context. It's also one of many closely interlinked mail-related specifications, and JMAP fails to integrate well with many related specs imo.

pjc50 7y ago
> questionable design decisions (HTTP? JSON?) Those are the premises for JMAP, from which the rest follows.
- ori_b 7y ago
  Yes, which is why the above post finds it not particularly desirable.
tjoff 7y ago
JMAP is literally the only positive thing that has happened to email in decades, please don't dismiss it (or do you know of anything else in the works that has any chance of success?). HTTP and JSON are such an insignificant price to pay for the sliver of hope that email can be saved and not being in complete control of google et al.
- asjo 7y ago
  > HTTP and JSON are such an insignificant price to pay for the sliver of hope that email can be saved and not being in complete control of google et al. How would JMAP accomplish that, in a way that other protocols don't?
  - tjoff 7y ago
    The shortcomings of IMAP and POP are reflected in all mail clients and is a large part in what has made webmail so popular. Another reason is that webmail vendors of today typically deliver a subpar IMAP/POP experience. So combining a native client and webmail when on the go is rarely ideal. JMAP is the first chance in a long time to get a decent experience that doesn't involve an expert and (even if you are willing to put in the time) tons of compromises. And aside from all that we have an opportunity and reason to revive native clients that has gone stagnant (which is pretty much all of them).
    - u801e 7y ago
      > The shortcomings of IMAP and POP are reflected in all mail clients and is a large part in what has made webmail so popular. Could you elaborate on this? I personally find using a mail client like Thunderbird to access my mail account far more responsive and seamless compared to using the webmail interface. If I don't have much bandwidth to work with, the local mail client using IMAP and SMTP will work while the webmail client will experience timeouts.
      - tjoff 7y ago
        So do I. But syncing mail with IMAP isn't good enough and many features of webmail clients do not map to IMAP clients. There are also tons of caveats during configuration so tons of people, rightfully so, don't bother.
      - amaccuish 7y ago
        IMAP is incredibly annoying. To send a message, you send via SMTP and then upload that message to your Sent folder, so it's transferred twice. For some reason the IMAP implementation I use means that when I move a message to a different folder, it uploads the message to the new folder, deletes the original message, and then downloads the message from the new folder. I have no idea if it's my server, client or the spec, but very wasteful and often fucks up.
- teknopaul 7y ago
  Some interesting work was done on adding prescence and chat to email that im disappointed not to see in jmap. Be great if we could chat to anyone with a name and dns domain instead of being stuck in a silo like facebook.
  - diggernet 7y ago
    That would indeed be awesome. Every email server a chat server. But they are not done with JMAP yet, still working on contacts and calendar. Maybe chat could be next.
- Tharkun 7y ago
  The real positive thing that needs to happen to email, is putting an end to near-monopoly of gmail. Email is great. It's decentralized. You can simply roll your own. You can use whatever client you like. Apparently even using HTTP/JSON. But you can't send mail to anyone using gmail, which is nearly everyone, unless you play by Google's rules. And even then they'll still put you in their spam folder, or just randomly blackhole you without any explanation or recourse. JMAP vs IMAP vs POP is moot as far as I'm concerned. As long as my mail client continues to work, I'm good. If this gains any traction, I'm sure dovecot and the likes will add support for it (if they haven't already). But no amount of protocol fiddling can fix the fundamental issues of email centralization.
  - tjoff 7y ago
    That's what I meant. And we need something better than IMAP/POP to have a fighting chance in challenging google. Because without it IMAP will, without doubt, loose market share even more - further cementing the dominance of larger players. That is why protocol fiddling is an essential part of it. Not enough, but without something better than IMAP might as well give up yesterday (as most already have done).
    - Tharkun 7y ago
      > we need something better than IMAP/POP to have a fighting chance in challenging google How's that? I don't see how messing with IMAP/POP/JMAP will make a difference there. Mail delivery, over SMTP, is what the Gmails and Outlooks of this world are so nasty about. They claim to protect their users from spam, but small mail servers get killed in the crossfire. Mail clients downloading mail over whatever protocol is a trivial problem by comparison, and they have zilch to do with SMTP abuse.
      - tjoff 7y ago
        Yes, but it is all about market share. If 99.9% of all mail originate from gmail+outlook then they can (and will) pretty much ignore the rest. The less dominant they are the more they are forced to be reasonable. JMAP will help diversify the market as well as simplify the process of creating competing products. And has the potential to vastly increase demand for native clients and proper support from webmail vendors. All of the above will work towards lessen the dominance of gmail etc. It isn't anything by itself but I believe it is a requirement going forward. Because IMAP doesn't cut it, so everyone has to create their own crap, which won't be as "good/easy/cheap" as gmail.
bad_user 7y ago
I think HTTP and JSON are great. You might not like it, but every programming language in existence has easy to use HTTP and JSON parsing libraries. It's basically a solved problem. As a software developer I dread working with protocols that are not via HTTP because most often then not the clients are badly designed, leaky and I can't fix it by changing the client. And I love JSON in spite of its inefficiencies, because it is easy to debug and there are really good JSON parsing libraries for the statically typed languages I'm using (Scala, Haskell). You basically declare your high level types and the library will derive (de)serialization and validation logic for you. Also consider that this isn't necessarily about building full fledged email clients. Maybe you want an automated way to read an Inbox and import certain messages in your database. Maybe you build a browser extension that counts the number of unread messages. There are many, many micro tasks that developers would be interested in doing, if they had an easy way to do it. And do you know what protocol does NOT have good libraries for every language in existence? IMAP. It's basically not a competition from my perspective, HTTP/JSON wins by default.
- TheOtherHobbes 7y ago
  "We have libraries for these technologies" is a poor substitute for "These technologies don't really solve the problems that need to be solved." The nice thing about the old internet protocols is that they tried to distinguish between abstraction and implementation layers. (Not always successfully or effectively, but they did at least try.) Modern protocol designs seem to do the opposite. They start from a default implementation layer - usually REST, HTTP, and JSON - and then try to build the abstraction layer out of that. This isn't necessarily a good approach. But it is a reflection of the difference in culture - the change from hackers who were comfortable with open-ended bottom-up development, and modern developers who seem to prefer snap-together development using standard libraries.
  - bad_user 7y ago
    > They start from a default implementation layer - usually REST, HTTP, and JSON - and then try to build the abstraction layer out of that. Also known as the Unix philosophy ;-) This isn't about comfort, but about doing what needs to be done. Every webmail interface in existence has implemented their own half baked, proprietary protocol that does what JMAP is doing. Or do you think Google's Gmail web interface is using IMAP? --- > "hackers who were comfortable with open-ended bottom-up development" That's because they lacked basic tools. Hackers in those days weren't getting much work done either.
qzygr 7y ago
What's unappealing about HTTP and JSON? What's better about having a completely custom protocol and format?
- Sir_Cmpwn 7y ago
  HTTP is not designed for persistent connections (WebSockets were hacked on later, but JMAP does not use them anyway). This turns a push protocol into a poll protocol, which is a stark regression. It's also got all sorts of other crap built-in which is really not necessary for this use-case but which an intrepid implementer will have to deal with regardless. JSON does not cope well with binary data, which is common in emails. It fails to elegantly deal with the various email encodings which exist in the wild. Consider as well that all JSON numbers are floating point numbers, despite the fact that floating point numbers provide absolutely nothing of value to this spec and in fact are more likely to introduce bugs than not. And embedded devices can't deal with floats quickly or elegantly, but still need to implement them if they want to use JSON-based protocols. And for what gain! In short, JMAP reinvents the wheel but worse for the sake of making it easier for web developers to build web shit around email. This kind of stupid change for change's sake is an epidemic in the webshit scene. IMAP is warty but it's fine. Let it be. When your only hammer is JavaScript, everyone else's thumb looks like a nail.
  - cyphar 7y ago
    I will preface this by admitting my only direct experience with IMAP as a protocol was with writing a few scripts to synchronise mail and having to fix bugs in the IMAP libraries I was using. I must be missing something obvious -- how is IMAP a push protocol? All experiences with IMAP I've had are in the form of basic request-response flows. Also, regarding JSON floats, most languages have the ability to give you errors if you encounter a float in a JSON field that is meant to be an integer. So it really shouldn't matter at all that JSON's floats are awful. I haven't looked, but does JMAP use floats? I don't particularly love HTTP nor JSON, but if that's the only problem with JMAP then it's such a massive improvement over IMAP that we should have started switching to it yesterday.
    - 7y ago
brongondwana 7y ago
You're very welcome to join the EXTRA working group at IETF and propose the extensions that you think would make IMAP better. Apart from working on JMAP, I have also written an extension to IMAP and may write another one next year bringing some of the push stuff from JMAP back to IMAP as well. But JMAP is based on many years of working on servers, middleware and (at least on the web) clients. We did actually extent IMAP with some experimental stuff first (cross folder threading and thread collapsing for example) but it never solved the batching and proxying difficulties and the cost of instantiating message numbers per-session that are un-avoidable parts of the IMAP model. By the way, engineers who work on other servers have said similar things to us - IMAP is costly to implement in terms of server resources when you're doing true concurrency, high availability, etc.
- jorangreef 7y ago
  > IMAP is costly to implement in terms of server resources when you're doing true concurrency, high availability Yes, IMAP requires expensive coordination among multi-master servers to generate UIDs in such a way that won't mess with client synchronization. In other words, IMAP is not a fully distributed design, it assumes a centralized control plane (think Raft or Paxos), i.e. a single central server with many distributed clients, which is a shame for new systems building on CRDTs. How does JMAP address this achilles heel of IMAP? Merkle tree sync? More detail on the problem: https://news.ycombinator.com/item?id=20479011 https://news.ycombinator.com/item?id=20479011
  - brongondwana 7y ago
    Individual unique IDs and /changes lists all the IDs that might have changed. One good thing about IMAP is that there's a fair bit of data which is immutable, and JMAP retains that - emails with the same ID must have the same content, so there's a limited amount of metadata that you need to resync. I wrote something about how you fix UID clashes in a split brain while staying within the IMAP data model many years ago... https://lists.andrew.cmu.edu/pipermail/cyrus-devel/2010-July/001500.html https://lists.andrew.cmu.edu/pipermail/cyrus-devel/2010-July...

pjc50 7y ago

> questionable design decisions (HTTP? JSON?) Those are the premises for JMAP, from which the rest follows.

ori_b 7y ago
Yes, which is why the above post finds it not particularly desirable.

tjoff 7y ago

JMAP is literally the only positive thing that has happened to email in decades, please don't dismiss it (or do you know of anything else in the works that has any chance of success?). HTTP and JSON are such an insignificant price to pay for the sliver of hope that email can be saved and not being in complete control of google et al.

asjo 7y ago
> HTTP and JSON are such an insignificant price to pay for the sliver of hope that email can be saved and not being in complete control of google et al. How would JMAP accomplish that, in a way that other protocols don't?
- tjoff 7y ago
  The shortcomings of IMAP and POP are reflected in all mail clients and is a large part in what has made webmail so popular. Another reason is that webmail vendors of today typically deliver a subpar IMAP/POP experience. So combining a native client and webmail when on the go is rarely ideal. JMAP is the first chance in a long time to get a decent experience that doesn't involve an expert and (even if you are willing to put in the time) tons of compromises. And aside from all that we have an opportunity and reason to revive native clients that has gone stagnant (which is pretty much all of them).
  - u801e 7y ago
    > The shortcomings of IMAP and POP are reflected in all mail clients and is a large part in what has made webmail so popular. Could you elaborate on this? I personally find using a mail client like Thunderbird to access my mail account far more responsive and seamless compared to using the webmail interface. If I don't have much bandwidth to work with, the local mail client using IMAP and SMTP will work while the webmail client will experience timeouts.
    - tjoff 7y ago
      So do I. But syncing mail with IMAP isn't good enough and many features of webmail clients do not map to IMAP clients. There are also tons of caveats during configuration so tons of people, rightfully so, don't bother.
    - amaccuish 7y ago
      IMAP is incredibly annoying. To send a message, you send via SMTP and then upload that message to your Sent folder, so it's transferred twice. For some reason the IMAP implementation I use means that when I move a message to a different folder, it uploads the message to the new folder, deletes the original message, and then downloads the message from the new folder. I have no idea if it's my server, client or the spec, but very wasteful and often fucks up.
teknopaul 7y ago
Some interesting work was done on adding prescence and chat to email that im disappointed not to see in jmap. Be great if we could chat to anyone with a name and dns domain instead of being stuck in a silo like facebook.
- diggernet 7y ago
  That would indeed be awesome. Every email server a chat server. But they are not done with JMAP yet, still working on contacts and calendar. Maybe chat could be next.
Tharkun 7y ago
The real positive thing that needs to happen to email, is putting an end to near-monopoly of gmail. Email is great. It's decentralized. You can simply roll your own. You can use whatever client you like. Apparently even using HTTP/JSON. But you can't send mail to anyone using gmail, which is nearly everyone, unless you play by Google's rules. And even then they'll still put you in their spam folder, or just randomly blackhole you without any explanation or recourse. JMAP vs IMAP vs POP is moot as far as I'm concerned. As long as my mail client continues to work, I'm good. If this gains any traction, I'm sure dovecot and the likes will add support for it (if they haven't already). But no amount of protocol fiddling can fix the fundamental issues of email centralization.
- tjoff 7y ago
  That's what I meant. And we need something better than IMAP/POP to have a fighting chance in challenging google. Because without it IMAP will, without doubt, loose market share even more - further cementing the dominance of larger players. That is why protocol fiddling is an essential part of it. Not enough, but without something better than IMAP might as well give up yesterday (as most already have done).
  - Tharkun 7y ago
    > we need something better than IMAP/POP to have a fighting chance in challenging google How's that? I don't see how messing with IMAP/POP/JMAP will make a difference there. Mail delivery, over SMTP, is what the Gmails and Outlooks of this world are so nasty about. They claim to protect their users from spam, but small mail servers get killed in the crossfire. Mail clients downloading mail over whatever protocol is a trivial problem by comparison, and they have zilch to do with SMTP abuse.
    - tjoff 7y ago
      Yes, but it is all about market share. If 99.9% of all mail originate from gmail+outlook then they can (and will) pretty much ignore the rest. The less dominant they are the more they are forced to be reasonable. JMAP will help diversify the market as well as simplify the process of creating competing products. And has the potential to vastly increase demand for native clients and proper support from webmail vendors. All of the above will work towards lessen the dominance of gmail etc. It isn't anything by itself but I believe it is a requirement going forward. Because IMAP doesn't cut it, so everyone has to create their own crap, which won't be as "good/easy/cheap" as gmail.

bad_user 7y ago

I think HTTP and JSON are great. You might not like it, but every programming language in existence has easy to use HTTP and JSON parsing libraries. It's basically a solved problem. As a software developer I dread working with protocols that are not via HTTP because most often then not the clients are badly designed, leaky and I can't fix it by changing the client. And I love JSON in spite of its inefficiencies, because it is easy to debug and there are really good JSON parsing libraries for the statically typed languages I'm using (Scala, Haskell). You basically declare your high level types and the library will derive (de)serialization and validation logic for you. Also consider that this isn't necessarily about building full fledged email clients. Maybe you want an automated way to read an Inbox and import certain messages in your database. Maybe you build a browser extension that counts the number of unread messages. There are many, many micro tasks that developers would be interested in doing, if they had an easy way to do it. And do you know what protocol does NOT have good libraries for every language in existence? IMAP. It's basically not a competition from my perspective, HTTP/JSON wins by default.

TheOtherHobbes 7y ago
"We have libraries for these technologies" is a poor substitute for "These technologies don't really solve the problems that need to be solved." The nice thing about the old internet protocols is that they tried to distinguish between abstraction and implementation layers. (Not always successfully or effectively, but they did at least try.) Modern protocol designs seem to do the opposite. They start from a default implementation layer - usually REST, HTTP, and JSON - and then try to build the abstraction layer out of that. This isn't necessarily a good approach. But it is a reflection of the difference in culture - the change from hackers who were comfortable with open-ended bottom-up development, and modern developers who seem to prefer snap-together development using standard libraries.
- bad_user 7y ago
  > They start from a default implementation layer - usually REST, HTTP, and JSON - and then try to build the abstraction layer out of that. Also known as the Unix philosophy ;-) This isn't about comfort, but about doing what needs to be done. Every webmail interface in existence has implemented their own half baked, proprietary protocol that does what JMAP is doing. Or do you think Google's Gmail web interface is using IMAP? --- > "hackers who were comfortable with open-ended bottom-up development" That's because they lacked basic tools. Hackers in those days weren't getting much work done either.

qzygr 7y ago

What's unappealing about HTTP and JSON? What's better about having a completely custom protocol and format?

Sir_Cmpwn 7y ago
HTTP is not designed for persistent connections (WebSockets were hacked on later, but JMAP does not use them anyway). This turns a push protocol into a poll protocol, which is a stark regression. It's also got all sorts of other crap built-in which is really not necessary for this use-case but which an intrepid implementer will have to deal with regardless. JSON does not cope well with binary data, which is common in emails. It fails to elegantly deal with the various email encodings which exist in the wild. Consider as well that all JSON numbers are floating point numbers, despite the fact that floating point numbers provide absolutely nothing of value to this spec and in fact are more likely to introduce bugs than not. And embedded devices can't deal with floats quickly or elegantly, but still need to implement them if they want to use JSON-based protocols. And for what gain! In short, JMAP reinvents the wheel but worse for the sake of making it easier for web developers to build web shit around email. This kind of stupid change for change's sake is an epidemic in the webshit scene. IMAP is warty but it's fine. Let it be. When your only hammer is JavaScript, everyone else's thumb looks like a nail.
- cyphar 7y ago
  I will preface this by admitting my only direct experience with IMAP as a protocol was with writing a few scripts to synchronise mail and having to fix bugs in the IMAP libraries I was using. I must be missing something obvious -- how is IMAP a push protocol? All experiences with IMAP I've had are in the form of basic request-response flows. Also, regarding JSON floats, most languages have the ability to give you errors if you encounter a float in a JSON field that is meant to be an integer. So it really shouldn't matter at all that JSON's floats are awful. I haven't looked, but does JMAP use floats? I don't particularly love HTTP nor JSON, but if that's the only problem with JMAP then it's such a massive improvement over IMAP that we should have started switching to it yesterday.
  - 7y ago

brongondwana 7y ago

You're very welcome to join the EXTRA working group at IETF and propose the extensions that you think would make IMAP better. Apart from working on JMAP, I have also written an extension to IMAP and may write another one next year bringing some of the push stuff from JMAP back to IMAP as well. But JMAP is based on many years of working on servers, middleware and (at least on the web) clients. We did actually extent IMAP with some experimental stuff first (cross folder threading and thread collapsing for example) but it never solved the batching and proxying difficulties and the cost of instantiating message numbers per-session that are un-avoidable parts of the IMAP model. By the way, engineers who work on other servers have said similar things to us - IMAP is costly to implement in terms of server resources when you're doing true concurrency, high availability, etc.

jorangreef 7y ago
> IMAP is costly to implement in terms of server resources when you're doing true concurrency, high availability Yes, IMAP requires expensive coordination among multi-master servers to generate UIDs in such a way that won't mess with client synchronization. In other words, IMAP is not a fully distributed design, it assumes a centralized control plane (think Raft or Paxos), i.e. a single central server with many distributed clients, which is a shame for new systems building on CRDTs. How does JMAP address this achilles heel of IMAP? Merkle tree sync? More detail on the problem: https://news.ycombinator.com/item?id=20479011 https://news.ycombinator.com/item?id=20479011
- brongondwana 7y ago
  Individual unique IDs and /changes lists all the IDs that might have changed. One good thing about IMAP is that there's a fair bit of data which is immutable, and JMAP retains that - emails with the same ID must have the same content, so there's a limited amount of metadata that you need to resync. I wrote something about how you fix UID clashes in a split brain while staying within the IMAP data model many years ago... https://lists.andrew.cmu.edu/pipermail/cyrus-devel/2010-July/001500.html https://lists.andrew.cmu.edu/pipermail/cyrus-devel/2010-July...