5 ms·
Ten Commandments of How to Write an IMAP Client (2006)
- mjevans 7y agoI don't know how many of these Outlook (any version ever that I've touched) violates, but I do know that it AT LEAST violates #9 by default. It irks me to no end, and un-checking that accursed options box in the IMAP folders dialog is something no user should ever need to do to find their "hidden folders".
- taneq 7y agoObviously there's some significant benefits to Outlook from an IT administration side (or it turns into a completely different beast when used with Exchange instead of IMAP) because as far as I can tell, it's otherwise awful on any number of axes.
- a2tech 7y agoOutlook is a good email client only when used against Exchange. It should never be used as a stand alone client because while it CAN work as a POP3 and IMAP client, its terrible at it. Its not even a good email client when used against Exchange but its 'good enough'
- yrro 7y agoDoes #1 mean that there are IMAP servers where my phone and my desktop and my laptop (etc.) can't all read my mail simultaneously? #7 makes me sad. Evolution supports a 'virtual trash folder' perfectly, but since I also delete mails in other clients, my mailstore is polluted by a malingering selection of folders with names like Trash, Wastebasket, Deleted Items and so on. The same remark applies to a 'virtual junk folder' and malingering Spam, Junk folders. The 'All About Calendars' file is a wealth of interesting tidbits about time: https://www.washington.edu/imap/documentation/calendar.txt.html https://www.washington.edu/imap/documentation/calendar.txt.h...
- dqv 7y ago>Does #1 mean that there are IMAP servers where my phone and my desktop and my laptop (etc.) can't all read my mail simultaneously? I think they are talking about one particular client instance. In other words, you should have some data structure where you can look up a connection by mailbox and use that existing connection rather than opening a new one.
- ts4z 7y ago> Does #1 mean that there are IMAP servers where my phone and my desktop and my laptop (etc.) can't all read my mail simultaneously? [edited] I believe that's the case. The ancient and venerable UW implementation of the ancient, venerable, and awful Unix "mbox" format from the UW server has a lot of these sorts of problems. The UW server did not have this problem with some of the other formats. It suffered a lot from supporting a lot of different formats in borderline hostile conditions (i.e., users were assumed to have filesystem access to the server so no improvement to locking or caching was practical). None of the current commercial implementations suffer from this, but I suppose I can say that the lowest common denominator is still pretty low. I may be overstating this because what I'm describing would also outlaw message delivery, which I don't think is the case. I think one of the formats did have problems with trivial concurrent access but I was never a UW-imapd expert. I am sure that the mbox format has problems with EXPUNGE (removing messages flagged deleted) as well as many other issues. I guess it's also worth reiterating that STATUSing a SELECTed mailbox (one that's open in another connection) is a pretty big waste of server resources, as someone else noted.
- deleted 7y ago[deleted]
- FroshKiller 7y agoTo malinger is to lie about being sick to get out of work. I think you just mean "lingering."
- yrro 7y agoToday I Learned, thanks!
- dqv 7y agoThis is kind of related... So I'm working on an app that uses IMAP. I need to know which emails I've already processed. Has anyone ever used the message ID header to identify an email as unique/seen? Or do you hash the email in its entirety? Or is there some other way to do it? I guess what I'm still confused about is determining how to handle a UID validity change. So I'm afraid to use UIDs.
- stevekemp 7y agoI wrote a console-based mail-client (think "mutt", but with Lua used for all scripting and user-interface work). MessageID headers are often missing, even though that is "invalid". I've also got thousands of examples of messages with identical MessageID values. (Largely from automated systems that use some trivial/broken template to generate the emails.)
- ts4z 7y agoMessage-ID is not unique per message, or per folder or user. The trivial counterexample is a message that goes through a mailing list may exist from the list and in a Sent folder with the same Message-ID but possibly different content, among other (less convincing) reasons. A UIDVALIDITY change should blow your whole cache. In practice they are rare.
- dqv 7y ago>Message-ID is not unique per message Thank you for saving me from the heartbreak of finding this out later. Seriously you just saved me a ton of time. >A UIDVALIDITY change should blow your whole cache. In practice they are rare. So even in a huge pile of mail, everything would have to be reprocessed. Since it's so rare, I'm definitely going to set up metrics to monitor when it does happen. Do you have any insight on how people store the mail? Do they actually use the UID to reference it or is it used as more of a cursor?
- ts4z 7y agoMost clients want to use UID, and it’s a good key for the same reason a SQL row number is, and inconvenient for servers for the same reason consistent hashing schemes don’t have it. I’m not sure what you mean about a cursor. I’m sure clients use UIDs as cursors. They’re good for that.
- Felz 7y agoIMAP was an okay protocol for its time, but in my experience it's basically mindbogglingly bad by current standards. The root of the reason why is the whole concept of "message sequence numbers", which as far as I know are extremely difficult to implement in a scalable and performant way. Basically, all messages in a mailbox are numbered from 1 to N, and you can reference a message by number, and when a message is deleted all message numbers above it shift down by one. But since this could confuse clients that are sending inflight requests, you can't start translating from the new shifted-down-by-one numbers until you've told your clients about it, which could take a theoretically unbounded amount of time. One implementation of this I've seen is to get a list of every UID in a mailbox at time of selection, and then update that as clients are informed of new or deleted mail. Needless to say, that's a bit problematic with short-lived or multiple connections. So if your mailbox gets too big, too bad. This ends up complicating the client: probably influencing rules 1, 3, part of 5, and 8 listed here. (1) You can't have concurrent mailbox access because it might mess up MSNs reified to disk, (3) you can't have connections live too long because they passively eat up RAM with queued updates and a stored MSN list, (5) there are perfectly good UIDs to reference messages with that you have to learn alongside message sequence numbers, and (8) because opening a new session can be heavy.
- pjc50 7y agoIt was bad even 20 years ago, I remember a discussion when we were trying to implement something coining the phrase "University of Washington brain damage". It seems to have been extremely tightly tied to the UW-IMAP implementations. The need for a better protocol is why JMAP exists: https://jmap.io/spec-core.html https://jmap.io/spec-core.html Although I see their point about non-REST, I think there's still a case for having a REST flavour available even if it may not be the most efficient. I wonder if anyone's tried mail delivery over git repo yet.
- hyperrail 7y ago> University of Washington brain damage This phrase made me laugh, because UW imapd was largely written by Mark Crispin, who invented the IMAP protocol! It seems that in about 2007, Crispin forked UW IMAP to create his own IMAP server, "Panda IMAP" [1], which at least one website claims to be "fully compliant" with the IMAP RFCs while UW IMAP is not. [2] [1] https://github.com/jonabbey/panda-imap https://github.com/jonabbey/panda-imap [2] https://imapwiki.org/ImapTest/ServerStatus https://imapwiki.org/ImapTest/ServerStatus
- robin_reala 7y agoWhat’s the current status of the JMAP spec? Are there any available clients or servers?
- earthboundkid 7y agoFastMail uses it internally, but I don't know if they've gotten any external adoption yet.
- codr7 7y agoI've dealt a lot with email along the way, trying to use it as an async transport layer for higher level messaging etc. At one point I had wrappers around Python's IMAP/SMTP implementations that I excuted from Go, since at least Python had 20 years to make theirs work properly. Any other implementation I tried failed somewhere along the way, especially as soon as you start moving messages around in folders or anything else beyond simply fetching messages. SMTP is simpler from my experience, but then so is the work it performs. I even played around with the idea of cooking my own to get out of this loop, but reading the specifications soon killed that idea. It would take a life time, probably be deprecated by the time, and still wouldn't work properly. The new JSON-stuff honestly doesn't look much better from what I've seen. It seems to still be doing IMAP, only using JSON instead, which wasn't the problem. I'm not holding my breath about it becoming standard enough to depend on. It's a mess, so much that I'm thinking it would probably be better to start over entirely.
- afiori 7y ago> The new JSON-stuff honestly doesn't look much better from what I've seen. It seems to still be doing IMAP, only using JSON instead, regarding JMAP that should not be the case. From their FAQ the intent is to make it easier to use already common workflows.
- codr7 7y agoI do hope you're right, convenient protocol access to email would change the game.
- jcranmer 7y agoJMAP fixes the big problems with IMAP: no MSN bookkeeping that no one uses anymore; no UIDVALIDITY issues; you can listen to events from multiple folders; etc. Essentially, the single biggest issue with IMAP is that most clients nowadays maintain an offline shadow copy of the database, and so what they need to do is use IMAP to synchronize these two databases. And IMAP is a lousy synchronization protocol--most of your IDs are all ephemeral. Extensions to IMAP make it better, but you can't rely on servers implementing them (or clients using them!). Incidentally, that's also why most library implementations of IMAP suck: you can't build a good interface that works around server deficiencies without baking in the integration into your database model, and that heavy opinionation makes it hard to use the IMAP code outside of the full email application.
- billpg 7y ago"Hello IMAP server. I'd like to deal with the message with this UID." "The UIDVALIDITY has changed since then." "Can you tell me the new UID from my old UID perhaps?" "No." (Okay, I last worked with IMAP back in the 90s. Has modern IMAP since added a persistent ID that will stay with a message?)
- throw0101a 7y agoOBJECTID extension: * https://tools.ietf.org/html/rfc8474 https://tools.ietf.org/html/rfc8474 Though if your mail server loses the UID-to-message mapping index and has to rev the UIDVALIDITY, that's a more a problem with the implementation IMHO.
- billpg 7y agoI'm an IMAP client and my user has clicked the delete-message button. Accordingly, I've sent a command to make the change and I'm waiting for the response from the server. Oh no! Before the response came back, my connection to the server died. I don't know if my request arrived and was accepted. Time passed and I've just now managed to reconnect. I need to know if the delete happened or not.
- throw0101a 7y agoCheck to see if the \Deleted flag was added to the message.
- billpg 7y agoWhich message? I have a UID that's no longer valid and the message IDs have all moved.
- moonbug 7y agoThat is unreadable.