3 ms·

> Basically, all messages in a mailbox are numbered from 1 to N, and you can reference a message by number, and when a message is deleted all message numbers ab

> Basically, all messages in a mailbox are numbered from 1 to N, and you can reference a message by number, and when a message is deleted all message numbers above it shift down by one. But since this could confuse clients that are sending inflight requests, you can't start translating from the new shifted-down-by-one numbers until you've told your clients about it, which could take a theoretically unbounded amount of time. Message ids are monotonically increasing, but not necessarily consecutive. You may need to renumber when you hit the 32-bit ceiling, but that is much less frequent than what you describe. Worst case scenario, you can disconnect unresponsive clients when renumbering, clients behave well on disconnects. This invalidates your conclusions. > (1) You can't have concurrent mailbox access because it might mess up MSNs reified to disk You can. Renumbering is not frequent. > (3) you can't have connections live too long because they passively eat up RAM with queued updates and a stored MSN list My dovecot IMAP processes use an average of 5MB reserved memory, of which 4MB is shared memory, so a server with 100GB memory can hold close to 100k connections. > (8) because opening a new session can be heavy. Monitoring tells me that on loaded production servers, session open + select + list + logout occurs comfortably in less than 100ms. That's not heavy. Dovecot starts to fail on mailboxes with about a million messages. It's a limit I can live with. Users have learned to archive messages by date.

fanf2 7y ago
IMAP has two ways of numbering messages. It is important to be clear whether you are discussing message sequence numbers (which are dense, and compacted on EXPUNGE) or unique IDs (which are not compacted). GP was complaining about MSNs, your reply is about UIDs.
Felz 7y ago
As fanf2 said, MSNs != UIDs. Also, I didn't mean to imply that MSNs are an insurmountable limitation for performance, they just complicate things a lot. I'm not familiar with Dovecot's code or very good at reading C, but I think part of how they handle it is here: https://github.com/dovecot/core/blob/81b5b188c478ec36bea8bda8fcad1e5f32ac612b/src/lib-index/mail-index-private.h https://github.com/dovecot/core/blob/81b5b188c478ec36bea8bda... Basically, building a mail index for UID<->MSN translation that can serialize to disk, which would explain how it doesn't necessarily need absurd amounts of RAM and can select fast, probably at heavy cost when EXPUNGeing old messages. I'm sure there were worse implementations that did just maintain it all in RAM way back in the day, and still some that do (like the one I ran into).
- edoceo 7y ago
  Dovecot puts those index files in the maildir too, so one can read those directly if wanted.
- mehrdadn 7y ago
  The cost of deletions is heavy, and I wouldn't have done it this way, but note that where it really matters is if you have to do this repeatedly in a loop. But you don't have to do it repeatedly -- you can just delete a ton of messages at once, at the same cost as deleting a single once. As long as user perceptions go, the network latency is probably going to dominate shifting down even a million integers (< 2 ms on my machine), which I suspect most people don't even have.
- kbenson 7y ago
  > Basically, building a mail index for UID<->MSN translation that can serialize to disk, which would explain how it doesn't necessarily need absurd amounts of RAM and can select fast That does explain why indexes can get corrupted and need to be cleared out, and why load balancing IMAP can get tricky.

fanf2 7y ago

IMAP has two ways of numbering messages. It is important to be clear whether you are discussing message sequence numbers (which are dense, and compacted on EXPUNGE) or unique IDs (which are not compacted). GP was complaining about MSNs, your reply is about UIDs.

Felz 7y ago

As fanf2 said, MSNs != UIDs. Also, I didn't mean to imply that MSNs are an insurmountable limitation for performance, they just complicate things a lot. I'm not familiar with Dovecot's code or very good at reading C, but I think part of how they handle it is here: https://github.com/dovecot/core/blob/81b5b188c478ec36bea8bda8fcad1e5f32ac612b/src/lib-index/mail-index-private.h https://github.com/dovecot/core/blob/81b5b188c478ec36bea8bda... Basically, building a mail index for UID<->MSN translation that can serialize to disk, which would explain how it doesn't necessarily need absurd amounts of RAM and can select fast, probably at heavy cost when EXPUNGeing old messages. I'm sure there were worse implementations that did just maintain it all in RAM way back in the day, and still some that do (like the one I ran into).

edoceo 7y ago
Dovecot puts those index files in the maildir too, so one can read those directly if wanted.
mehrdadn 7y ago
The cost of deletions is heavy, and I wouldn't have done it this way, but note that where it really matters is if you have to do this repeatedly in a loop. But you don't have to do it repeatedly -- you can just delete a ton of messages at once, at the same cost as deleting a single once. As long as user perceptions go, the network latency is probably going to dominate shifting down even a million integers (< 2 ms on my machine), which I suspect most people don't even have.
kbenson 7y ago
> Basically, building a mail index for UID<->MSN translation that can serialize to disk, which would explain how it doesn't necessarily need absurd amounts of RAM and can select fast That does explain why indexes can get corrupted and need to be cleared out, and why load balancing IMAP can get tricky.