4 ms·

This is kind of related... So I'm working on an app that uses IMAP. I need to know which emails I've already processed. Has anyone ever used the message ID hea

by dqv 7y ago

This is kind of related... So I'm working on an app that uses IMAP. I need to know which emails I've already processed. Has anyone ever used the message ID header to identify an email as unique/seen? Or do you hash the email in its entirety? Or is there some other way to do it? I guess what I'm still confused about is determining how to handle a UID validity change. So I'm afraid to use UIDs.

stevekemp 7y ago
I wrote a console-based mail-client (think "mutt", but with Lua used for all scripting and user-interface work). MessageID headers are often missing, even though that is "invalid". I've also got thousands of examples of messages with identical MessageID values. (Largely from automated systems that use some trivial/broken template to generate the emails.)
ts4z 7y ago
Message-ID is not unique per message, or per folder or user. The trivial counterexample is a message that goes through a mailing list may exist from the list and in a Sent folder with the same Message-ID but possibly different content, among other (less convincing) reasons. A UIDVALIDITY change should blow your whole cache. In practice they are rare.
- dqv 7y ago
  >Message-ID is not unique per message Thank you for saving me from the heartbreak of finding this out later. Seriously you just saved me a ton of time. >A UIDVALIDITY change should blow your whole cache. In practice they are rare. So even in a huge pile of mail, everything would have to be reprocessed. Since it's so rare, I'm definitely going to set up metrics to monitor when it does happen. Do you have any insight on how people store the mail? Do they actually use the UID to reference it or is it used as more of a cursor?
  - ts4z 7y ago
    Most clients want to use UID, and it’s a good key for the same reason a SQL row number is, and inconvenient for servers for the same reason consistent hashing schemes don’t have it. I’m not sure what you mean about a cursor. I’m sure clients use UIDs as cursors. They’re good for that.
skshetry 7y ago
How about adding a flag to the emails that are processed? It's fairly well supported. I needed this when i was working on my engineering project, and added 'processed' flags for already processed items. This way i could query all unprocessed items with "UNSEEN UNKEYWORD <flag>".
- dqv 7y ago
  Nice! I'll take a look. Thanks for the advice.
  - ts4z 7y ago
    This is a good plan, probably a notch better than using UIDs. You can use UIDs as a cursor to the server as a hint to speed up the operation if that’s what you meant in the other thread, and that’s a great thing to do.
throw0101a 7y ago
IMAP Flags? (\Seen, \Answered, \Flagged) * https://tools.ietf.org/html/rfc3501#section-2.3.2 https://tools.ietf.org/html/rfc3501#section-2.3.2 UIDS are more than simple 'message numbers'; they are part of a tuple (Folder, UIDVALIDITY, UID). It may be easy to think of UIDVALIDITY as a kind of 'generation number': for a given folder, as long as the UIDVALIDITY has not changed/incremented, the UID-to-messege has to be the same per the spec. Though recently the OBJECTID extension has been published: * https://tools.ietf.org/html/rfc8474 https://tools.ietf.org/html/rfc8474

by dqv 7y ago

stevekemp 7y ago

I wrote a console-based mail-client (think "mutt", but with Lua used for all scripting and user-interface work). MessageID headers are often missing, even though that is "invalid". I've also got thousands of examples of messages with identical MessageID values. (Largely from automated systems that use some trivial/broken template to generate the emails.)

ts4z 7y ago

Message-ID is not unique per message, or per folder or user. The trivial counterexample is a message that goes through a mailing list may exist from the list and in a Sent folder with the same Message-ID but possibly different content, among other (less convincing) reasons. A UIDVALIDITY change should blow your whole cache. In practice they are rare.

dqv 7y ago
>Message-ID is not unique per message Thank you for saving me from the heartbreak of finding this out later. Seriously you just saved me a ton of time. >A UIDVALIDITY change should blow your whole cache. In practice they are rare. So even in a huge pile of mail, everything would have to be reprocessed. Since it's so rare, I'm definitely going to set up metrics to monitor when it does happen. Do you have any insight on how people store the mail? Do they actually use the UID to reference it or is it used as more of a cursor?
- ts4z 7y ago
  Most clients want to use UID, and it’s a good key for the same reason a SQL row number is, and inconvenient for servers for the same reason consistent hashing schemes don’t have it. I’m not sure what you mean about a cursor. I’m sure clients use UIDs as cursors. They’re good for that.

skshetry 7y ago

How about adding a flag to the emails that are processed? It's fairly well supported. I needed this when i was working on my engineering project, and added 'processed' flags for already processed items. This way i could query all unprocessed items with "UNSEEN UNKEYWORD <flag>".

dqv 7y ago
Nice! I'll take a look. Thanks for the advice.
- ts4z 7y ago
  This is a good plan, probably a notch better than using UIDs. You can use UIDs as a cursor to the server as a hint to speed up the operation if that’s what you meant in the other thread, and that’s a great thing to do.

throw0101a 7y ago

IMAP Flags? (\Seen, \Answered, \Flagged) * https://tools.ietf.org/html/rfc3501#section-2.3.2 https://tools.ietf.org/html/rfc3501#section-2.3.2 UIDS are more than simple 'message numbers'; they are part of a tuple (Folder, UIDVALIDITY, UID). It may be easy to think of UIDVALIDITY as a kind of 'generation number': for a given folder, as long as the UIDVALIDITY has not changed/incremented, the UID-to-messege has to be the same per the spec. Though recently the OBJECTID extension has been published: * https://tools.ietf.org/html/rfc8474 https://tools.ietf.org/html/rfc8474