6 ms·

How to Replace IMAP

118 points by mk 17y ago

ajross 17y ago
Not to spend too much effort defending IMAP (which does, to be fair, kinda suck) but... come on. It's not that hard. It works just fine, enjoying pervasive support with very high quality clients and servers available freely on essentially all platforms. A new protocol would be ... prettier, I guess. But to pretend that the reason facebook and IM are replacing email is IMAP (and not, y'know, spam) is borderline delusional.
- sandGorgon 17y ago
  http://jerakeen.org/notes/2010/01/yay-more-email-clients/ http://jerakeen.org/notes/2010/01/yay-more-email-clients/ The Android GMail client is a perfect example of what a client looks like in this world. It talks to the (secret / private) GMail API, it does offline mail reading, and queues actions so you can archive / filter / whatever mails while offline and it’ll push changes later. You can read and write mail. It doesn’t try to do anything clever, because anything clever done on one client isn’t reproduced on any other client. And if I don’t have a client on my current computer for GMail, I can use a web browser, and still get all the features of the server. I use the web gmail interface for everything anyway, because it’s better than any GUI client I’ve got. I dont know - this guy makes for pretty convincing arguments on why email should be done right on the server rather than the client. Which effectively means a new protocol.
  - tedunangst 17y ago
    I started using this new "mail on the server" protocol way back in 2000, but back then we called it SSH. :)
  - _delirium 17y ago
    If not designed very carefully, though, it can reduce the user's ability to view things as they like them. With the current status quo of mailservers not handling message threading and such, I can choose a client that handles it how I want. If the server is doing it, it had better either do it how I want it, or provide some way to customize what it's doing. (For example, I don't like gmail's flat "conversation" view; I much prefer a "threaded conversation" view, like classic Usenet readers, or HN discussions.)
    - sandGorgon 17y ago
      reMAP claims to treat conversations as first class objects. Given suitable APIs on top, I think you could build different views on different email clients. I think what is meant by a new protocol and server innovation is a change in the data model. I do not believe they are going to dictate views.
  - coryrc 17y ago
    Mozilla Seamonkey and Thunderbird do that all with IMAP (except the web client part).
- gduffy 17y ago
  This is not about making it easier to write email clients or servers, it's about making it easier to write email apps. Currently, you either have to write a plugin for an email client or interact with IMAP directly. Opening up email to mainstream developers means ditching IMAP/MAPI/etc as the API.
  - ajross 17y ago
    And again, I don't see why that's such a huge improvement. IMAP is clunky and weird, but it's not rocket surgery. If there was a huge market opportunity for someone to write custom mailoid gadgets using IMAP, clearly it would have been done already. A better protocol will only be "better", analogous to, say, the Mac (c. 1990) being better than Windows 3.1. While true, compatibility pressures aren't going to allow someone else into the market simply for being "better". You have to be transformative, and this isn't. And in any case what's killing email isn't the lack of apps, it's the lack of authentication and moderation.
  - neilc 17y ago
    You rarely need to use IMAP directly, though: most email apps are fine using a library that manages the details of the IMAP protocol under the covers, and presents a more sane interface.
- gaborcselle 17y ago
  The combination of IMAP and MIME are hard. Yes, we could be happy with the status quo. But I do believe that one of the things that's holding up innovation around email is the lack of easy access to data. Proposed exercise: Write a script that downloads and displays your Twitter feed. Then write a script that downloads and displays your email. The email exercise will take you an order of magnitude longer. Spam filtering isn't the problem you make it out to be. I get more spam in my Facebook inbox (mostly viruses) than in Gmail. If you're hosting you're own email or are using a sucky provider, that's a different story. Email will be hard to replace long-term anyway. I doubt your bigcorp employer really going to let you send work-related Facebook messages. IM is a fundamentally different protocol with different use cases (think async vs. sync, fax vs. phone).
  - IgorPartola 17y ago
    $mbox = imap_open( $host, $login, $password ); foreach( imap_search( $mbox, 'UNSEEN', SE_UID ) as $uid ) { $header = imap_fetch_overview( $mbox, $uid, FT_UID ); echo $header[0]->subject . "\n"; } imap_close( $mbox );
    - edd 17y ago
      $status_url = 'http://twitter.com/statuses/user_timeline.json?screen_name='.$user; $timeline = json_decode(file_get_contents($status_url)); for($i = 0; $i < count($timeline); $i++){ echo $timeline[$i]->text ."\n"; } One line in it then.
      - PanMan 17y ago
        The count in the loop slows it down quite a bit:http://tips4php.com/2010/01/best-way-to-traverse-trough-an-array/ http://tips4php.com/2010/01/best-way-to-traverse-trough-an-a... Foreach is easier to write, and faster to run :)
        deleted 17y ago
        [deleted]
- ryanwaggoner 17y ago
  But to pretend that the reason facebook and IM are replacing email is IMAP (and not, y'know, spam) is borderline delusional. As is believing that it's about spam (or probably believing it all, for that matter, but let's stay on topic). I don't understand people bitching about spam in 2010. I have probably a dozen email addresses that I publish all over the web and I get maybe one spam message a month. I get way more "spam" through Twitter than I do through my email.
  - sorbits 17y ago
    I don’t understand people declare the spam problem solved. I have had false positives with every single spam filter I have tried. This is unacceptable when running a business, and if we manually have to double-check hundreds of spams each day, what is the point of the filter? Additionally we spend hours each week working with users who do not receive our emails because of spam filters unknown to them (many believe the fault is ours). It is 2010 and spam is still a major problem.
    - tbrownaw 17y ago
      > if we manually have to double-check hundreds of spams each day, what is the point of the filter? It's faster to scan a list of subject lines and senders than to look at the contents of an email, which is faster than actually deleting it. So verifying that the spam filter sorted your emails correctly is faster than sorting every message yourself.
- gibsonf1 17y ago
  It seems like the key is to work on top of an interface that does all the dirty work of negotiating with IMAP for you, such as the lisp mel-base library. Then you really don't care how messy imap might be, as you can build on top of it at increasing levels of abstraction.
nfnaaron 17y ago
I can't say whether his implementation makes sense. Regardless, I would like to see public key encryption of all email. There's no reason why messages should be plain text anywhere except at the two ends. I realize key management would be a problem, but it's probably a solvable problem. The solution would have to allow most people to be practically unaware of encryption issues.
- webignition 17y ago
  Encryption of all email would make intermediary virus scanning impossible and spam scanning next to impossible.
  - conover 17y ago
    Wouldn't it make spam impractical because you would all of a sudden need much more processing power to encrypt the messages (especially millions of them)?
mrduncan 17y ago
The description for #8 sounds more like long polling to me. For push email, I'd envision something like registering a location to send notifications to (web hooks).
- gaborcselle 17y ago
  Yeah, it is long polling. I could be misinformed, but isn't the problem with web hooks that you might not have ports open and could be behind a firewall?
  - mrduncan 17y ago
    Yea, I'd say long polling is probably a better solution in this case than web hooks since clients (likely) could be behind firewalls, NAT, and be going on and off line relatively often. Sorry, the title and description on that one just threw me off a bit - I think you chose the best option though.
tdmackey 17y ago
First the solution doesn't really fulfill all of imaps use cases (especially push mail) but that is unimportant. What is important is that the IMAP protocol isn't difficult, and I don't think that is what limits "email innovation." It is more so that it has been around since the 70s and email clients haven't been able to remove themselves from their past. It isn't the protocol limiting them from moving forward, it is the shear number of other clients that wouldn't support any of the "innovations" one would try to bring to email.
- gaborcselle 17y ago
  Would love to hear your thoughts about how to improve my proposed design. My email's on my HP in case you want to do it offline.
- beh 17y ago
  I'm very hopeful for Letters, a project just taking off under the leadership of some pretty big names: Gruber, Rands, Brent Simmons. They're hoping to collectively create a "lean and programmable IMAP email client, with plugin and automation APIs, designed for developers and power users." While still sparse and very early-stage, their vision document is ambitious: http://pastie.org/785269 http://pastie.org/785269 If something comes from this, it'll definitely be a step forward to modernize email clients. Project wiki: http://letters-wiki.heroku.com/ http://letters-wiki.heroku.com/
Joe_Bananas 17y ago
Ah, yes. A new email scheme that would require us to replace all clients and servers in one fell swoop. I can see that taking off.
- gridspy 17y ago
  All it requires is a major server (i.e gmail) providing it as an option and a major client (say a iPhone email app) supporting that option.
  - there 17y ago
    oh, that's all? it'll require an RFC or some other kind of standard, which will take a long time to finalize. then someone will have to write a reference spec server and client, then "real" versions in "real" languages, then get them both to the stability, security, and scalability of current imap clients and servers. then once big email providers (isps, google, etc.) start supporting it on the server end (gotta factor in all those different operating systems, authentication systems, storage systems, etc.), popular clients can support it. then work out all of the minor implementation details that some servers or clients get wrong (i think almost every smtp and pop3 server has hacks in it to support the stupid things outlook express does wrong), and then once everyone complains long enough, the iphone will finally get support for it (while a small but vocal minority shouts that android already supports it). there is a reason why smtp, imap, pop3, dns, http, and all of the other core internet protocols are still around after 20 years and it's not because there aren't any faults in any of them.
    - lief79 17y ago
      If you are going for a clean design, you can go in the opposite direction too. I'm fairly sure Ethernet and html evolved from a working solution. Now, granted there should be a really clean and open interface if we are going to try and make it a standard, but it's often the easier approach.
    - regularfry 17y ago
      You've got it backwards. Standardisation happens after first implementation, not before. The article author can deliver this functionality into a popular-enough client without waiting for anyone else. If the benefits turn out to be worth it, maybe it'll go somewhere.
jwhitlark 17y ago
My responses to your Design Outline. 1. I'm not convinced that http/https is the best protocol. I agree that it's the defacto standard, but if you're going to go through this, there might be a better way. (Push, in particular, is a problem). 2. Stateless is wonderful, agreed. 3. I agree with JSON over XML, but what about binary data? That is the weakest part of JSON, in my opinion. 4. I like where you're going with this, but I'm not sure this is the exact right way to handle it. 5. Agreed. 6. how about SHA-1 hashes of content? Done properly, that would eliminate any duplicate messages, and could make it very easy to include other messages. This might also allow content to be transfered between accounts only once, dropping bandwidth and speeding delivery. 7. Don't know enough about MIME to comment. 8. According to your comments on this page, you're referring to long polling. I agree that this use case needs to be addressed, but I'm not sure that this is the way to do it. Perhaps multiple options here? 9. Good idea, but do you think that stemming/language differences is going to be a problem? What about other types of content? attachments? The biggest thing missing from this list is some sort of Public Key Encryption. Anyway, it looks like a good start.
- pyre 17y ago
  > 3. I agree with JSON over XML, but what about binary data? That is the weakest part of JSON, in my opinion. What binary data? All attachments in emails are Base64 encoded. That's what MIME is all about. > 6. how about SHA-1 hashes of content? Done properly, that would eliminate any duplicate messages, and could make it very easy to include other messages. This might also allow content to be transfered between accounts only once, dropping bandwidth and speeding delivery. Developers should start to shy away from SHA-1 since it's already started to get broken.
  - jwhitlark 17y ago
    >> 3. ugh. Hadn't realized that. I like IMAP even less, now. (Told you I didn't know much about MIME ;-) >> 6. I wasn't intending it as a security mechanism, but as a way to know which bits you have and which you don't. Do you think SHA-1 is still a bad idea in that case?
JoachimSchipper 17y ago
This is a collection of the author's favourite technologies/gripes, not a useful proposal. A large number of errors can be found by considering that it'll have to talk SMTP on the back end. I'll only point to some "highlights". OAuth isn't nearly secure enough (http://hueniverse.com/2009/04/explaining-the-oauth-session-fixation-attack/ http://hueniverse.com/2009/04/explaining-the-oauth-session-f...) for your e-mail (which, don't forget, gives access to everything via password resets). Let's just ignore the part about third-party access. Encoding everything as UTF-8 is nice, but breaks the first time some idiot mail client puts Shift-JIS encoded Japanese in the Subject: field. Without any charset marker, obviously. (This goes right back to the "where does the mail come from?" issue.) IMAP has a THREAD extension (solving #4), a SEARCH extension (solving #9 and parts of others), and all sensible servers support IDLE (http://en.wikipedia.org/wiki/IMAP_IDLE http://en.wikipedia.org/wiki/IMAP_IDLE). It also supports downloading MIME parts separately, solving the part of #7 that can actually be solved (again, you'll have to accept mail.)
- __david__ 17y ago
  With regards to search, I recently installed Cyrus Squatter for full text IMAP searching and it is blazing fast. To me it seems just as fast as any of the special case local searches I've seen people build (Spotlight/recent Thunderbirds).
  - qjz 17y ago
    Any new mail storage protocol should anticipate the need to store encrypted messages at rest. Sure, it can play a part by storing an encrypted index on the server to deliver to authorized clients, but all decryption and index updates should take place client-side with the user's private key. Stable UID's can help make this a reality. There is no reason to store messages in plaintext on the server, and no new protocol should depend on it.
    - JoachimSchipper 17y ago
      Message-IDs are already stable, though.
Skeuomorph 17y ago
First, I love reMail. Gabor breathes email and search. If you own an iPhone and use Gmail or Google Apps, get reMail--you'll thank him. That said, it's easy to skim this article and nod, "Yeah! Yeah!", but when we're discussing these, laymen may need to know a few of these seem over-simplified. > "TCP connections are great, but for transferring large amounts of email securely, HTTP is the way to go." 1. http://wiki.answers.com/Q/What_is_the_difference_between_tcp_and_http http://wiki.answers.com/Q/What_is_the_difference_between_tcp... - HTTP uses a TCP connection. For that matter so does SMTP and IMAP and the proposed REMAP. 2. I generally have at least 5 devices using IMAP (on Google Apps) at once, and interleave my interactions with these devices arbitrarily, yet expect each to show me the exact same state when I look at it. Keeping them all in the apparently same state is the sort of intractable software problem bedeviled with implementation details, and I agree throwing out IMAP and starting over with lessons learned could be easier than, say, the RFC linked below. 3. http://ajaxian.com/archives/json-vs-xml-the-debate http://ajaxian.com/archives/json-vs-xml-the-debate - Seems this point is something like GIF vs JPG vs PNG, with a bit of "Markdown vs HTML" thrown in. Meanwhile, both XML and JSON are missing an important concept given labels/folders and conversations: multiple references to single objects. 4. Conversations or threads exist today as In-Reply-To: GUID and References: GUID headers, among others, but see the JSON problem above. We don't realize these headers are there because clients generally ignore these and try to group on Subject instead. Gmail grouping respecting existing headers is much better, but from usability point of view, Gmail's approach makes it challenging to bulk delete individual matching messages across a broad set of conversations. For example, to bulk move or mark all messages on a particular date results in the full conversations being moved or marked. So I like this idea as long as I can still optionally operate on sets of messages without affecting the rest of the conversation. 5. http://googlesystem.blogspot.com/2007/08/organizing-chaos-folders-labels-search.html http://googlesystem.blogspot.com/2007/08/organizing-chaos-fo... - It's not yet clear that people prefer tags to folders. After all, monkeys don't expect a banana to be in two boxes at once. I don't want to label anything with keywords. I want semantic search. 6. Excellent point. Changing GUIDs out from under us is just lame. Having had to resort to http://github.com/rgrove/larch http://github.com/rgrove/larch recently to move a decade of email onto Google Apps because Thunderbird was putting new IDs on each moved message, I fully agree this problem is evil. 7. Seems we're stuck with MIME until we get rid of every legacy email server in the world, or control what servers our recipients use. Between the new server and new clients, sure. But the server will have to know it to send it. Granted, IMAP is for receiving, and SMTP is for sending, but something in the chain has to know MIME. 8. "Call" an HTTP endpoint that "returns" when messages arrive? Just nomenclature, since IDLE is a "call" that "returns" when state changes. Either way, the socket is held open, so neither of these is "push". Maybe Gabor meant "posts back" instead of "returns"? In any case, under the hood, iPhone push and Microsoft ActiveSync are exploiting out-of-band channel such as SMS to notify the phone it should poll (or in case of ActiveSync, maybe doing a "long pull"). See http://msexchangeteam.com/archive/2004/04/26/120520.aspx http://msexchangeteam.com/archive/2004/04/26/120520.aspx versus http://tools.ietf.org/internet-drafts/draft-maes-lemonade-p-imap-12.txt http://tools.ietf.org/internet-drafts/draft-maes-lemonade-p-... 9. Will the RFC specify a search results algorithm, or will the same search return different results depending on the implementation of mail server the client is connected to? We manipulate a "client" tool to perform the search and access the email, so our natural mental model is of the client at hand, not of the remote server. This lets the brain wrap itself around Gmail's web search versus Apple Mail's search box vs OS X Spotlight vs Gabor's reMail all returning different results. Practially speaking, search at the server does make far more sense from both a data retention and security standpoint. I have 509 MB of index in Gabor's "reMail" app on my iPhone, and would rather not. Thanks for brainstorming this, and thanks for a kickass iPhone app.
dlsspy 17y ago
> All data that's ever sent to or received from the server would be in JSON format. JSON is much more human-readable than XML. I don't need my protocols to be human readable. I need my computers to read them and know that I've got good, bug-free, reusable, efficient, and easy-to understand implementations. This isn't an argument for XML over JSON -- I've made good use of both, but I don't want to consider how easy it is to type up communication protocols when I'm defining stuff the replacement for something I expect to take over the world. It's by far easier to implement the memcached binary protocol as a client or a server than it is to implement the text protocol. Easier to get right, more regular, and just generally more pleasant.
- regularfry 17y ago
  As far as I'm concerned, the argument for JSON over XML is that the parser should be much lighter. No bloody entities to worry about, for a start.
viraptor 17y ago
There are some good points, but: 1. HTTP / HTTPS - please don't... There are stupid/broken transparent proxies in many places that will effectively block this protocol. Also custom TLS connection gives some hope for domain-specific certificates instead of ip-specific - there is enough problem with them in HTTPS as it is. 3. People still don't know how to set the encoding (or what the encoding is) and will happily send you Greek in latin1 or something equally silly. 4. Would it really work for mailing lists where conversations naturally fork into many branches? 5. IMAP supports labels... even if it's crazy number-based labelling where every client needs to set its own names for the labels. We'd just need to make it not suck this time ;) So it's not all that bad in IMAP - there are many areas where we can do worse by accident. But yes - it would be nice to have server-side text searching...
- pyre 17y ago
  > 4. Would it really work for mailing lists where conversations naturally fork into many branches? I agree with this. The idea that messages must be accessed by conversation is a bit much. Support for conversation group could be nice, but ignoring that feature would have to be allowed so that clients could implement their own views. For example, the following structure would be crappy to force on clients: Label `-> conversation 1 `-> msg0 `-> msg1 `-> conversation 2 `-> msg0 `-> msg1 Needing to iterate through every conversation to pull tease out the full list of messages (in order to use a different conversation-grouping method) would be ridiculous. Not to mention the question of whether or not conversations should span labels and how that should work.
cturner 17y ago
You could implement this as a service on top of couchdb. The oreilly couchdb book that has just been released shows examples of developing apps using the javascript engine that's inside couchdb. You'd get easy access to some functionality by doing this - tagging, email versioning.

118 points by mk 17y ago

Not to spend too much effort defending IMAP (which does, to be fair, kinda suck) but... come on. It's not that hard. It works just fine, enjoying pervasive support with very high quality clients and servers available freely on essentially all platforms. A new protocol would be ... prettier, I guess. But to pretend that the reason facebook and IM are replacing email is IMAP (and not, y'know, spam) is borderline delusional.

sandGorgon 17y ago
http://jerakeen.org/notes/2010/01/yay-more-email-clients/ http://jerakeen.org/notes/2010/01/yay-more-email-clients/ The Android GMail client is a perfect example of what a client looks like in this world. It talks to the (secret / private) GMail API, it does offline mail reading, and queues actions so you can archive / filter / whatever mails while offline and it’ll push changes later. You can read and write mail. It doesn’t try to do anything clever, because anything clever done on one client isn’t reproduced on any other client. And if I don’t have a client on my current computer for GMail, I can use a web browser, and still get all the features of the server. I use the web gmail interface for everything anyway, because it’s better than any GUI client I’ve got. I dont know - this guy makes for pretty convincing arguments on why email should be done right on the server rather than the client. Which effectively means a new protocol.
- tedunangst 17y ago
  I started using this new "mail on the server" protocol way back in 2000, but back then we called it SSH. :)
- _delirium 17y ago
  If not designed very carefully, though, it can reduce the user's ability to view things as they like them. With the current status quo of mailservers not handling message threading and such, I can choose a client that handles it how I want. If the server is doing it, it had better either do it how I want it, or provide some way to customize what it's doing. (For example, I don't like gmail's flat "conversation" view; I much prefer a "threaded conversation" view, like classic Usenet readers, or HN discussions.)
  - sandGorgon 17y ago
    reMAP claims to treat conversations as first class objects. Given suitable APIs on top, I think you could build different views on different email clients. I think what is meant by a new protocol and server innovation is a change in the data model. I do not believe they are going to dictate views.
- coryrc 17y ago
  Mozilla Seamonkey and Thunderbird do that all with IMAP (except the web client part).
gduffy 17y ago
This is not about making it easier to write email clients or servers, it's about making it easier to write email apps. Currently, you either have to write a plugin for an email client or interact with IMAP directly. Opening up email to mainstream developers means ditching IMAP/MAPI/etc as the API.
- ajross 17y ago
  And again, I don't see why that's such a huge improvement. IMAP is clunky and weird, but it's not rocket surgery. If there was a huge market opportunity for someone to write custom mailoid gadgets using IMAP, clearly it would have been done already. A better protocol will only be "better", analogous to, say, the Mac (c. 1990) being better than Windows 3.1. While true, compatibility pressures aren't going to allow someone else into the market simply for being "better". You have to be transformative, and this isn't. And in any case what's killing email isn't the lack of apps, it's the lack of authentication and moderation.
- neilc 17y ago
  You rarely need to use IMAP directly, though: most email apps are fine using a library that manages the details of the IMAP protocol under the covers, and presents a more sane interface.
gaborcselle 17y ago
The combination of IMAP and MIME are hard. Yes, we could be happy with the status quo. But I do believe that one of the things that's holding up innovation around email is the lack of easy access to data. Proposed exercise: Write a script that downloads and displays your Twitter feed. Then write a script that downloads and displays your email. The email exercise will take you an order of magnitude longer. Spam filtering isn't the problem you make it out to be. I get more spam in my Facebook inbox (mostly viruses) than in Gmail. If you're hosting you're own email or are using a sucky provider, that's a different story. Email will be hard to replace long-term anyway. I doubt your bigcorp employer really going to let you send work-related Facebook messages. IM is a fundamentally different protocol with different use cases (think async vs. sync, fax vs. phone).
- IgorPartola 17y ago
  $mbox = imap_open( $host, $login, $password ); foreach( imap_search( $mbox, 'UNSEEN', SE_UID ) as $uid ) { $header = imap_fetch_overview( $mbox, $uid, FT_UID ); echo $header[0]->subject . "\n"; } imap_close( $mbox );
  - edd 17y ago
    $status_url = 'http://twitter.com/statuses/user_timeline.json?screen_name='.$user; $timeline = json_decode(file_get_contents($status_url)); for($i = 0; $i < count($timeline); $i++){ echo $timeline[$i]->text ."\n"; } One line in it then.
    - PanMan 17y ago
      The count in the loop slows it down quite a bit:http://tips4php.com/2010/01/best-way-to-traverse-trough-an-array/ http://tips4php.com/2010/01/best-way-to-traverse-trough-an-a... Foreach is easier to write, and faster to run :)
      - deleted 17y ago
        [deleted]
ryanwaggoner 17y ago
But to pretend that the reason facebook and IM are replacing email is IMAP (and not, y'know, spam) is borderline delusional. As is believing that it's about spam (or probably believing it all, for that matter, but let's stay on topic). I don't understand people bitching about spam in 2010. I have probably a dozen email addresses that I publish all over the web and I get maybe one spam message a month. I get way more "spam" through Twitter than I do through my email.
- sorbits 17y ago
  I don’t understand people declare the spam problem solved. I have had false positives with every single spam filter I have tried. This is unacceptable when running a business, and if we manually have to double-check hundreds of spams each day, what is the point of the filter? Additionally we spend hours each week working with users who do not receive our emails because of spam filters unknown to them (many believe the fault is ours). It is 2010 and spam is still a major problem.
  - tbrownaw 17y ago
    > if we manually have to double-check hundreds of spams each day, what is the point of the filter? It's faster to scan a list of subject lines and senders than to look at the contents of an email, which is faster than actually deleting it. So verifying that the spam filter sorted your emails correctly is faster than sorting every message yourself.
gibsonf1 17y ago
It seems like the key is to work on top of an interface that does all the dirty work of negotiating with IMAP for you, such as the lisp mel-base library. Then you really don't care how messy imap might be, as you can build on top of it at increasing levels of abstraction.

nfnaaron 17y ago

I can't say whether his implementation makes sense. Regardless, I would like to see public key encryption of all email. There's no reason why messages should be plain text anywhere except at the two ends. I realize key management would be a problem, but it's probably a solvable problem. The solution would have to allow most people to be practically unaware of encryption issues.

webignition 17y ago
Encryption of all email would make intermediary virus scanning impossible and spam scanning next to impossible.
- conover 17y ago
  Wouldn't it make spam impractical because you would all of a sudden need much more processing power to encrypt the messages (especially millions of them)?

mrduncan 17y ago

The description for #8 sounds more like long polling to me. For push email, I'd envision something like registering a location to send notifications to (web hooks).

gaborcselle 17y ago
Yeah, it is long polling. I could be misinformed, but isn't the problem with web hooks that you might not have ports open and could be behind a firewall?
- mrduncan 17y ago
  Yea, I'd say long polling is probably a better solution in this case than web hooks since clients (likely) could be behind firewalls, NAT, and be going on and off line relatively often. Sorry, the title and description on that one just threw me off a bit - I think you chose the best option though.

tdmackey 17y ago

First the solution doesn't really fulfill all of imaps use cases (especially push mail) but that is unimportant. What is important is that the IMAP protocol isn't difficult, and I don't think that is what limits "email innovation." It is more so that it has been around since the 70s and email clients haven't been able to remove themselves from their past. It isn't the protocol limiting them from moving forward, it is the shear number of other clients that wouldn't support any of the "innovations" one would try to bring to email.

gaborcselle 17y ago
Would love to hear your thoughts about how to improve my proposed design. My email's on my HP in case you want to do it offline.
beh 17y ago
I'm very hopeful for Letters, a project just taking off under the leadership of some pretty big names: Gruber, Rands, Brent Simmons. They're hoping to collectively create a "lean and programmable IMAP email client, with plugin and automation APIs, designed for developers and power users." While still sparse and very early-stage, their vision document is ambitious: http://pastie.org/785269 http://pastie.org/785269 If something comes from this, it'll definitely be a step forward to modernize email clients. Project wiki: http://letters-wiki.heroku.com/ http://letters-wiki.heroku.com/

Joe_Bananas 17y ago

Ah, yes. A new email scheme that would require us to replace all clients and servers in one fell swoop. I can see that taking off.

gridspy 17y ago
All it requires is a major server (i.e gmail) providing it as an option and a major client (say a iPhone email app) supporting that option.
- there 17y ago
  oh, that's all? it'll require an RFC or some other kind of standard, which will take a long time to finalize. then someone will have to write a reference spec server and client, then "real" versions in "real" languages, then get them both to the stability, security, and scalability of current imap clients and servers. then once big email providers (isps, google, etc.) start supporting it on the server end (gotta factor in all those different operating systems, authentication systems, storage systems, etc.), popular clients can support it. then work out all of the minor implementation details that some servers or clients get wrong (i think almost every smtp and pop3 server has hacks in it to support the stupid things outlook express does wrong), and then once everyone complains long enough, the iphone will finally get support for it (while a small but vocal minority shouts that android already supports it). there is a reason why smtp, imap, pop3, dns, http, and all of the other core internet protocols are still around after 20 years and it's not because there aren't any faults in any of them.
  - lief79 17y ago
    If you are going for a clean design, you can go in the opposite direction too. I'm fairly sure Ethernet and html evolved from a working solution. Now, granted there should be a really clean and open interface if we are going to try and make it a standard, but it's often the easier approach.
  - regularfry 17y ago
    You've got it backwards. Standardisation happens after first implementation, not before. The article author can deliver this functionality into a popular-enough client without waiting for anyone else. If the benefits turn out to be worth it, maybe it'll go somewhere.

jwhitlark 17y ago

My responses to your Design Outline. 1. I'm not convinced that http/https is the best protocol. I agree that it's the defacto standard, but if you're going to go through this, there might be a better way. (Push, in particular, is a problem). 2. Stateless is wonderful, agreed. 3. I agree with JSON over XML, but what about binary data? That is the weakest part of JSON, in my opinion. 4. I like where you're going with this, but I'm not sure this is the exact right way to handle it. 5. Agreed. 6. how about SHA-1 hashes of content? Done properly, that would eliminate any duplicate messages, and could make it very easy to include other messages. This might also allow content to be transfered between accounts only once, dropping bandwidth and speeding delivery. 7. Don't know enough about MIME to comment. 8. According to your comments on this page, you're referring to long polling. I agree that this use case needs to be addressed, but I'm not sure that this is the way to do it. Perhaps multiple options here? 9. Good idea, but do you think that stemming/language differences is going to be a problem? What about other types of content? attachments? The biggest thing missing from this list is some sort of Public Key Encryption. Anyway, it looks like a good start.

pyre 17y ago
> 3. I agree with JSON over XML, but what about binary data? That is the weakest part of JSON, in my opinion. What binary data? All attachments in emails are Base64 encoded. That's what MIME is all about. > 6. how about SHA-1 hashes of content? Done properly, that would eliminate any duplicate messages, and could make it very easy to include other messages. This might also allow content to be transfered between accounts only once, dropping bandwidth and speeding delivery. Developers should start to shy away from SHA-1 since it's already started to get broken.
- jwhitlark 17y ago
  >> 3. ugh. Hadn't realized that. I like IMAP even less, now. (Told you I didn't know much about MIME ;-) >> 6. I wasn't intending it as a security mechanism, but as a way to know which bits you have and which you don't. Do you think SHA-1 is still a bad idea in that case?

JoachimSchipper 17y ago

This is a collection of the author's favourite technologies/gripes, not a useful proposal. A large number of errors can be found by considering that it'll have to talk SMTP on the back end. I'll only point to some "highlights". OAuth isn't nearly secure enough (http://hueniverse.com/2009/04/explaining-the-oauth-session-fixation-attack/ http://hueniverse.com/2009/04/explaining-the-oauth-session-f...) for your e-mail (which, don't forget, gives access to everything via password resets). Let's just ignore the part about third-party access. Encoding everything as UTF-8 is nice, but breaks the first time some idiot mail client puts Shift-JIS encoded Japanese in the Subject: field. Without any charset marker, obviously. (This goes right back to the "where does the mail come from?" issue.) IMAP has a THREAD extension (solving #4), a SEARCH extension (solving #9 and parts of others), and all sensible servers support IDLE (http://en.wikipedia.org/wiki/IMAP_IDLE http://en.wikipedia.org/wiki/IMAP_IDLE). It also supports downloading MIME parts separately, solving the part of #7 that can actually be solved (again, you'll have to accept mail.)

__david__ 17y ago
With regards to search, I recently installed Cyrus Squatter for full text IMAP searching and it is blazing fast. To me it seems just as fast as any of the special case local searches I've seen people build (Spotlight/recent Thunderbirds).
- qjz 17y ago
  Any new mail storage protocol should anticipate the need to store encrypted messages at rest. Sure, it can play a part by storing an encrypted index on the server to deliver to authorized clients, but all decryption and index updates should take place client-side with the user's private key. Stable UID's can help make this a reality. There is no reason to store messages in plaintext on the server, and no new protocol should depend on it.
  - JoachimSchipper 17y ago
    Message-IDs are already stable, though.

Skeuomorph 17y ago

First, I love reMail. Gabor breathes email and search. If you own an iPhone and use Gmail or Google Apps, get reMail--you'll thank him. That said, it's easy to skim this article and nod, "Yeah! Yeah!", but when we're discussing these, laymen may need to know a few of these seem over-simplified. > "TCP connections are great, but for transferring large amounts of email securely, HTTP is the way to go." 1. http://wiki.answers.com/Q/What_is_the_difference_between_tcp_and_http http://wiki.answers.com/Q/What_is_the_difference_between_tcp... - HTTP uses a TCP connection. For that matter so does SMTP and IMAP and the proposed REMAP. 2. I generally have at least 5 devices using IMAP (on Google Apps) at once, and interleave my interactions with these devices arbitrarily, yet expect each to show me the exact same state when I look at it. Keeping them all in the apparently same state is the sort of intractable software problem bedeviled with implementation details, and I agree throwing out IMAP and starting over with lessons learned could be easier than, say, the RFC linked below. 3. http://ajaxian.com/archives/json-vs-xml-the-debate http://ajaxian.com/archives/json-vs-xml-the-debate - Seems this point is something like GIF vs JPG vs PNG, with a bit of "Markdown vs HTML" thrown in. Meanwhile, both XML and JSON are missing an important concept given labels/folders and conversations: multiple references to single objects. 4. Conversations or threads exist today as In-Reply-To: GUID and References: GUID headers, among others, but see the JSON problem above. We don't realize these headers are there because clients generally ignore these and try to group on Subject instead. Gmail grouping respecting existing headers is much better, but from usability point of view, Gmail's approach makes it challenging to bulk delete individual matching messages across a broad set of conversations. For example, to bulk move or mark all messages on a particular date results in the full conversations being moved or marked. So I like this idea as long as I can still optionally operate on sets of messages without affecting the rest of the conversation. 5. http://googlesystem.blogspot.com/2007/08/organizing-chaos-folders-labels-search.html http://googlesystem.blogspot.com/2007/08/organizing-chaos-fo... - It's not yet clear that people prefer tags to folders. After all, monkeys don't expect a banana to be in two boxes at once. I don't want to label anything with keywords. I want semantic search. 6. Excellent point. Changing GUIDs out from under us is just lame. Having had to resort to http://github.com/rgrove/larch http://github.com/rgrove/larch recently to move a decade of email onto Google Apps because Thunderbird was putting new IDs on each moved message, I fully agree this problem is evil. 7. Seems we're stuck with MIME until we get rid of every legacy email server in the world, or control what servers our recipients use. Between the new server and new clients, sure. But the server will have to know it to send it. Granted, IMAP is for receiving, and SMTP is for sending, but something in the chain has to know MIME. 8. "Call" an HTTP endpoint that "returns" when messages arrive? Just nomenclature, since IDLE is a "call" that "returns" when state changes. Either way, the socket is held open, so neither of these is "push". Maybe Gabor meant "posts back" instead of "returns"? In any case, under the hood, iPhone push and Microsoft ActiveSync are exploiting out-of-band channel such as SMS to notify the phone it should poll (or in case of ActiveSync, maybe doing a "long pull"). See http://msexchangeteam.com/archive/2004/04/26/120520.aspx http://msexchangeteam.com/archive/2004/04/26/120520.aspx versus http://tools.ietf.org/internet-drafts/draft-maes-lemonade-p-imap-12.txt http://tools.ietf.org/internet-drafts/draft-maes-lemonade-p-... 9. Will the RFC specify a search results algorithm, or will the same search return different results depending on the implementation of mail server the client is connected to? We manipulate a "client" tool to perform the search and access the email, so our natural mental model is of the client at hand, not of the remote server. This lets the brain wrap itself around Gmail's web search versus Apple Mail's search box vs OS X Spotlight vs Gabor's reMail all returning different results. Practially speaking, search at the server does make far more sense from both a data retention and security standpoint. I have 509 MB of index in Gabor's "reMail" app on my iPhone, and would rather not. Thanks for brainstorming this, and thanks for a kickass iPhone app.

dlsspy 17y ago

> All data that's ever sent to or received from the server would be in JSON format. JSON is much more human-readable than XML. I don't need my protocols to be human readable. I need my computers to read them and know that I've got good, bug-free, reusable, efficient, and easy-to understand implementations. This isn't an argument for XML over JSON -- I've made good use of both, but I don't want to consider how easy it is to type up communication protocols when I'm defining stuff the replacement for something I expect to take over the world. It's by far easier to implement the memcached binary protocol as a client or a server than it is to implement the text protocol. Easier to get right, more regular, and just generally more pleasant.

regularfry 17y ago
As far as I'm concerned, the argument for JSON over XML is that the parser should be much lighter. No bloody entities to worry about, for a start.

viraptor 17y ago

There are some good points, but: 1. HTTP / HTTPS - please don't... There are stupid/broken transparent proxies in many places that will effectively block this protocol. Also custom TLS connection gives some hope for domain-specific certificates instead of ip-specific - there is enough problem with them in HTTPS as it is. 3. People still don't know how to set the encoding (or what the encoding is) and will happily send you Greek in latin1 or something equally silly. 4. Would it really work for mailing lists where conversations naturally fork into many branches? 5. IMAP supports labels... even if it's crazy number-based labelling where every client needs to set its own names for the labels. We'd just need to make it not suck this time ;) So it's not all that bad in IMAP - there are many areas where we can do worse by accident. But yes - it would be nice to have server-side text searching...

pyre 17y ago
> 4. Would it really work for mailing lists where conversations naturally fork into many branches? I agree with this. The idea that messages must be accessed by conversation is a bit much. Support for conversation group could be nice, but ignoring that feature would have to be allowed so that clients could implement their own views. For example, the following structure would be crappy to force on clients: Label `-> conversation 1 `-> msg0 `-> msg1 `-> conversation 2 `-> msg0 `-> msg1 Needing to iterate through every conversation to pull tease out the full list of messages (in order to use a different conversation-grouping method) would be ridiculous. Not to mention the question of whether or not conversations should span labels and how that should work.

cturner 17y ago

You could implement this as a service on top of couchdb. The oreilly couchdb book that has just been released shows examples of developing apps using the javascript engine that's inside couchdb. You'd get easy access to some functionality by doing this - tagging, email versioning.