8 ms·
How IMAP works under the hood
- steeeeeve 2y agoThis is the kind of thing I would have expected to read in 2600 back in the day. And why I _always_ looked for 2600 at the bookstore.
- therein 2y agoInteresting no attempt has been made to make it at least be less heavy on networked bytes. Especially since it is old and was meant to be used on a connection with no compression or encryption. HasChildren could have been Parent, HasNoChildren could have been Leaf or Child. And so many more things.
- philipwhiuk 2y agoThe protocol has ossified and been entrenched. In general more efficient usage of IMAP relies on extensions to the protocol. A modern replacement (JMAP) hasn't been adopted by major providers. If you really cared about data transfer size you'd use something like Protobuf.
- jgalt212 2y agoIMAP COMPRESS?
- FuriouslyAdrift 2y agoJMAP is fully supported (since 2019) for Fastmail https://www.fastmail.com/dev/ https://www.fastmail.com/dev/
- n3storm 2y agoJmap was developed by Fastmail, meanwhile all primary FLOSS mail servers do not support it and just new projects like Stalwart are starting to bring it up.
- FuriouslyAdrift 1y agoCyrus supports it. https://www.cyrusimap.org/imap/developer/jmap.html https://www.cyrusimap.org/imap/developer/jmap.html
- calvinmorrison 2y agoWell and you could of course server JMAP over something like Protobuf, JSON is used for ease of use but it's not like thats required feature. The bigger benefit of JMAP is it's well designed for dealing with deltas and changes and syncing data requiring less roundtripping than IMAP.
- Mailtemi 2y agoCan confirm that (implemented JMAP). Deltas and asking for the entire mailbox instead of folder by folder is really good. And as a side effect: 1. Less client-side logic for sync. 2. There are many JSON parsers compared to IMAP, making it way easier to use. For example, in C++, you only need a JSON headers-only library, whereas IMAP is meh—only Linux, or clunky usage. Btw, I made one from scratch (IMAP Parser). 3. Goodies like offloading HTTP to the mobile network stack, which supports TLS 1.3 and offline (background) sync, compared to manually extracting certificates from an IMAP connection to validate against the device keychain. It's just 10x easier.
- calvinmorrison 2y agoIt's designed well, from the ground up with a lot of lessons learned from some pretty serious email junkies. Anyone hemming and hawing about JSON being selected as the transport encoding or whatever is just raising silly points. You could do JMAP over protobuf too probably if you cared, but the whole point (IMO) is to get the hell away from unstructured protocols.
- Mailtemi 2y agoYep, there's always someone ranting about JSON inefficiencies, which I don't observe at all :)
- calvinmorrison 2y agooh i mean i rant about it too but its like, a byproduct of the real goal, and since most clients are running some type of webapp it makes sense. you could definitely use something like messagepack or protobuf if you had an extension that negotiates it.
- Aloisius 2y agoIMAP 4rev2 merged in a most of the common extensions into the base protocol. Sadly they left out the THREAD, QRESYNC and OBJECTID, though they reference them. Of course, major providers like Gmail don't support that as well.
- avar 2y agoNetworked bytes generally don't matter, networked packets do. Would your proposed change move the needle on that?
- shiandow 2y agoIt could have, but for stuff you only do once per session that seems excessive. Better to have names that need no explanation, especially for stuff that I think is completely optional.
- zaik 2y agoDespite people always complaining about their perceived inefficiencies, standard protocols like IMAP or XMPP always seem to work on a crappy connection, when most of the modern web doesn't.
- andrewshadura 2y agoIs this a joke? Both XMPP and IMAP are terrible over an unreliable connection.
- Loudergood 2y agoBack in the day when I worked at a WISP with 2.4Ghz mountain top antennas the one thing that would work when we were having massive icing issues was email. It was very impressive how it would get through even with massive packet loss and transfer rates measuring in bytes per second.
- nirui 2y agoProbably wrong context, but the more code I wrote, more I like the these `Has`+Noun style naming than just Noun. Reading `HasChildren` will give you a clearer expectation of what the function would do and return, while `Parent` gives far weaker indication. Maybe they thought the same when they were designing the protocol. Also, in the context of email, given the size of each mail (including headers and body), these bytes "waste" maybe insignificant.
- throw0101d 2y ago> Interesting no attempt has been made to make it at least be less heavy on networked bytes. Kind of a surprising observation given the first spec (IMAP2) was released in 1998, when dial-up was still a thing: * https://datatracker.ietf.org/doc/html/rfc1064 https://datatracker.ietf.org/doc/html/rfc1064 IMAP4 was in 1994: * https://datatracker.ietf.org/doc/html/rfc1730 https://datatracker.ietf.org/doc/html/rfc1730 ITU V.32 gave us 9.6 kbit/s in 1998, and V.34 was 28.8 kbit/s in 1994: * https://en.wikipedia.org/wiki/List_of_ITU-T_V-series_recommendations https://en.wikipedia.org/wiki/List_of_ITU-T_V-series_recomme...
- throw0101d 2y agos/1998/1988/
- ocdtrekkie 2y agoWhen every new Gmail client ships with an entire web browser embedded to load their hundred megabytes of JavaScript, I think we've long jumped the shark on caring about brevity in the length of information in the protocol itself. It might have mattered back then but now it would be less than a rounding error.
- lotharcable2 2y agoIMAP had its day in the sun, but the advent of big webmail providers (especially gmail) has killed off the advancement of email clients. Now all major development is focused on trying to recreate Gmail to varying degrees of success. It all ends up internal to one or another corporation so they are just all endlessly reinventing the wheel with IMAP just being relegated to a afterthought front end to some sort of search-based backend. Actually having a email client software running on your machine is extremely nitch and is mostly in the realm of self-hosters and legacy holdouts that won't let their clients go. A most advanced modern approach is to just use POP3 to download your emails to a local Maildir and have them indexed there non-destructively. And then sync between your various machines that you want your email accessible using some sort of file sync or P2P solution. I use notmuch for this. It automatically indexes and tags emails and thus enables much more advanced email management solutions then what can be offered over something like IMAP. The main advantage of this is that 'folders' are managed virtually. There is no shuffling or copying or editing of emails done normally. I only have to worry about backing up my emails and notmuch config as all the rest can be regenerated relatively quickly. This is more or less replicating what Gmail and other webmail providers do server side. Where as the traditional approach shuffling and moving and deleting of emails on some imap server is fairly dangerous and expensive operation. Mistakes can lead to data loss and are often very difficult to reverse.
- csb6 2y agoThere is an extension that adds compression. [0] I think most widely-used servers implement it (at least Gmail and Dovecot do) [0] https://www.rfc-editor.org/rfc/rfc4978.html https://www.rfc-editor.org/rfc/rfc4978.html
- rmccue 2y agoI started writing a guide to IMAP back when I was working on an email client: https://github.com/rmccue/griffin/tree/master/docs/imap https://github.com/rmccue/griffin/tree/master/docs/imap Pulling large amounts of data for things like threading can be difficult on certain servers; my preferred approach ended up being to pull every ID and thread ID to maintain an in-memory tree. (This was, iirc, partially because Gmail’s implementation was slightly crippled with relation to threading.) (I never finished the guide because I stopped on the project, alas - if IMAP were easier to work with, I might have finished it! And sadly, no JMAP support on Gmail, and the gateway was broken.)
- jeffbee 2y ago> Gmail’s implementation was slightly crippled Gmail is not "crippled". A tiny but vocal community of old nerds have a petrified mental model of email that they associate with unix IMAP software from the 1990's, but those concepts do not appear in the IMAP standards anywhere.
- dbcurtis 2y agoThat is an immature view on how real products and real standards work. The standard document may say one thing, but what people do in the real world is the real standard. For context: I spent 11 years at Intel managing pre-silicon and post-silicon processor validation. No processor that does only and exactly what the Programmers Reference Manual says, and takes the phrase "undefined behavior" seriously, will be successful. Google would do well to adjust their philosophy.
- jeffbee 2y agoIf an x86 implementation was imperfectly compatible with Intel CPUs, nobody would buy it. Gmail, on the other hand, is a massive market success. It is those who shout that IMAP must be exactly and only whatever mutt+uw was doing in 1997 who are on the wrong side of history.
- 2y ago
- camgunz 2y agoI've been working on some email stuff and I think probably four things are vexing about IMAP: - The grammar is hard. I built a parser using lpeg and I'm incredibly glad I did--doing it ad hoc won't lead to good results. - It's an asynchronous protocol. You can send lots of requests to a server and you have to tag them so you can match them up with responses later. You don't generally want to do that in a client; i.e. you don't want to do these transactional things over an async connection and track state across all of it. You want to like, block on deleting things, renaming things, sending things, etc. - IMAP is multi-user--it's built around the idea of multiple clients accessing the same mailbox at the same time and streaming updates. Another thing you really don't want to deal with when building an email client. - There's functionality that you basically shouldn't use; the big one is search. Even the specs more or less say "good luck using this". You can group all this under the a heading of "we thought people would use this over telnet", but attachments and non-plain-text email I think made that non-viable. I think this all means probably every non-web email client treats IMAP like POP and keeps its own store. I haven't done a survey or anything, but I'd be surprised if that weren't true.
- userbinator 2y agoYou can group all this under the a heading of "we thought people would use this over telnet" No, the grammar is too convoluted for that. It's more like "we thought text-based protocols are better". POP3 and SMTP are relatively easy to use manually. IMAP is not.
- mr_mitm 2y ago> I think this all means probably every non-web email client treats IMAP like POP and keeps its own store. I haven't done a survey or anything, but I'd be surprised if that weren't true. Pretty sure mutt doesn't. It only caches the headers.
- bpev 2y agoFor the extra interested… reminded me of this one: https://explained-from-first-principles.com/email/ https://explained-from-first-principles.com/email/
- nashashmi 2y agoCan IMAP be used as a file server system? Outlook has this functionality, where files can be stored directly outside of emails.
- SSLy 2y agomost servers would reject entries without any headers as malformed
- thesuitonym 2y agoThis is one of those questions where the answer is technically `yes', but for all practical purposes should be considered `no'.
- superkuh 2y agoOf course these days the mega-corp walled garden email providers don't really follow standards like IMAP. IMAP will not work with, say, Google's gmail or Microsoft office365, or AT&T ISP email, etc, etc. They have each implemented their own proprietery out-of-band authentication system that only works over HTTPS using the OAuth2.0 toolkit to build it. Any email client that does not explicitly design for each particular OAuth2.0 implementation (each megacorp's is slightly different) will not be able to connect over IMAP (unless they login via HTTPS using a web browser and set up "app passwords" for google, or similar for others).
- Avamander 2y ago> Of course these days the mega-corp walled garden email providers don't really follow standards like IMAP. Not really true. It's usually the client implementations that violate the standard in some way or another, like Outlook. But there are way more bespoke rare clients that have poor implementations. > They have each implemented their own proprietery out-of-band authentication system that only works over HTTPS using the OAuth2.0 toolkit to build it. Well, no. They have implemented OAuth and that's not proprietary. They do it because plain login has massive downsides.
- calvinmorrison 2y agoWhen I worked at fastmail there was of special fix code. You see it with firefox and chrome too, oh this popular site is breaking lets put a hardcoded if statement in. I specifically remember magic fixes for iCal.
- jeffbee 2y agoStruggling to think of a way in which "IMAP will not work with gmail". Please explain.
- Aloisius 2y agoIt can, but it does require doing a lot of Google-specific things (set up a google cloud account, create a consent screen, get a security review, justify your usage of the IMAP API instead of the web APIs to them, find the right scopes, etc) or instruct users to go through multiple screens in their google settings to create an app password. Google really doesn't want you to use IMAP. They're trying to push everyone to their neutered web apis instead.
- tiahura 2y agoI've been looking to migrate from Exchange-Outlook, but there really aren't any options. There just isn't an open source solution to have an integrated email / tasks / events / contacts, with consistent labels across item types and reminders. With Outlook, i can use a custom view to see every item in a category flagged for follow-up. I can also set a reminder on a contact, or drag an email or contact onto my calendar to create an event.
- beagle3 2y agoThunderbird has it all. I don’t like the way thunderbird does it, but I like outlook even less… so thunderbird it is.
- deleted 2y ago[deleted]
- tiahura 2y agoThunderbird doesn’t do most of this. It uses tags for emails and categories for events. Also, no way to view all items with same category. Can’t drag contact to calendar or task list. Can’t set specific reminder times. …
- azhenley 2y agoI've been trying to get approval from Google for the sensitive scopes to use IMAP, and they classified us as needing "CASA Tier 3 Security Assessment". It looks like it is going to be a long, tedious, opaque, and expensive process.
- isaachinman 2y agoWhat are you building?
- azhenley 2y agoAn email client for Gmail.
- isaachinman 2y agoInteresting. I take it you've seen Notion's new product? Are you aiming to directly compete with Notion/Superhuman? Edit: I see you're an academic, perhaps my questions are irrelevant
- accrual 2y agoThat was a pretty interesting read. I didn't realize one could interact directly with an IMAP server like we can with FTP and telnet + HTTP.
- chasil 2y agoPOP is a little simpler, but IMAP is designed for concurrent access by multiple clients to the same account. What this article doesn't address is OAUTH, which is required more often now.
- kazinator 2y agoThat's a hugely important use case, especially nowadays. People have mail clients running at the same time from multiple devices; e.g. desktop and mobile. Also multiple browser tabs with an instance of their webmail.