6 ms·

> IMAP clients typically refer to messages by their “sequence number”, the message’s position in a mailbox. The first message has the sequence number “1”, the s

> IMAP clients typically refer to messages by their “sequence number”, the message’s position in a mailbox. The first message has the sequence number “1”, the second “2”, and so on. If a client wants to mark a message as “read”, it will send a command to the server such as “mark message 5 as read”. But what if another client deleted the fourth message in the mailbox? The sequence numbers of all messages after the deleted message will be shifted down by one; the client that sent the “mark message 5 as read” command now refers to a different message than it intended. > IMAP servers (which include applications such as the Proton Mail Bridge) need to be able to handle this situation. When one client moves messages into or out of a mailbox, the server needs to notify all other clients of the changes so that they can update their own view of the mailbox. And until the clients have received the update, the server needs to remember what each client thinks the mailbox looks like to correctly interpret the client’s commands. Holy F what a clusterfuck

amelius 4y ago
I'm wondering why they don't give every mail a UUID. Probably because someone wanted to save a few bytes.
- c0l0 4y ago
  They do, on a per-session basis - but it's an optional feature (that's supported by ~100% of all imap server implementations).
  - KRAKRISMOTT 4y ago
    What's the name of this extension?
    - epilys 4y ago
      It's actually in the original IMAP4rev1 RFC: https://www.rfc-editor.org/rfc/rfc3501#section-2.3.1.1 https://www.rfc-editor.org/rfc/rfc3501#section-2.3.1.1 Using UIDs is opt-in, by prefixing `UID ` in front of some commands. There are indeed extensions that add more `UID ` commands to the protocol. I've implemented IMAP clients and never used sequence numbers at all.
      - vidarh 4y ago
        UID was present already in RFC 1730 (IMAPv4): https://www.rfc-editor.org/rfc/rfc1730#section-6.4.9 https://www.rfc-editor.org/rfc/rfc1730#section-6.4.9
- jcranmer 4y ago
  One of the extensions to IMAP is the UID extension, which gives every email something akin to a UUID. It's technically not mandatory, but virtually every email client is going to try to use it if possible, and likely some email clients don't even attempt to use message sequence numbers. So you don't need to maintain message sequence numbers, except for the few places in the protocol where only the message sequence number is reported (e.g., the unsolicited message deleted alerts). In practice, I would actually expect that trying to use message sequence numbers would be buggier than not using them. But servers still have to support them!
- Mailtemi 4y ago
  Yep, just a few bites :) Especially if you're on vacation. IMAP IDs can change, requiring the email client to re-download all emails.
evmar 4y ago
A long time ago I wrote a toy IMAP client, and ran into similarly confusing issues about how the protocol is supposed to work: https://github.com/evmar/go-imap/blob/master/imap/doc.go https://github.com/evmar/go-imap/blob/master/imap/doc.go
calvinmorrison 4y ago
> Holy F what a clusterfuck And that's why when people say "JMAP is pointless", just go back and read some of the email specs!
- Avamander 4y ago
  Until things like Thunderbird support it, its adoption is unfortunately going to remain at the level it is currently at.
  - tracker1 4y ago
    Which sucks as it's a massive chicken/egg problem... more clients and hosts need to support it. Of course, from what I understand is there are JMAP/IMAP gateway options, that should ease availability... assuming these things actually gain any traction ever.
    - calvinmorrison 4y ago
      Well email moves slowly for one, and adding another protocol is usually baked into a client and servers approach (imap, pop, smtp, etc) so its not impossible
  - e12e 4y ago
    I'm guessing thunderbird will be the first mainstream gui mua to support jmap.
rglover 4y ago
Stuff like this makes the idea of an "Email 2" protocol being worthwhile. Shocking how over-complicated it is to send text/html between two parties.
- Avamander 4y ago
  People have called things email killers several times. An intelligent approach would be to gradually rebuild parts on top of better foundations and push towards that. The amount of deadlocks in adopting things like JMAP, IMAP NOTIFY, OAuth2 (+ Dynamic Client Registration) and so on and on is immense. Won't because others don't, too few do, we see no benefit yet etc. it's really hindering.
throw0101c 4y ago
>> IMAP clients typically refer to messages by their “sequence number” […] > Holy F what a clusterfuck Or you could use the unique identifier of the message (UID) instead of the sequence number: > Messages in IMAP4rev1 are accessed by one of two numbers; the unique identifier or the message sequence number. * https://www.rfc-editor.org/rfc/rfc3501#section-2.3.1 https://www.rfc-editor.org/rfc/rfc3501#section-2.3.1 > A 32-bit value assigned to each message, which when used with the unique identifier validity value (see below) forms a 64-bit value that MUST NOT refer to any other message in the mailbox or any subsequent mailbox with the same name forever. * https://www.rfc-editor.org/rfc/rfc3501#section-2.3.1.1 https://www.rfc-editor.org/rfc/rfc3501#section-2.3.1.1 The designers of the IMAP protocol aren't as dumb as the above quotation would have you to believe and multi-client operations were thought of.
- meghan_rain 4y ago
  The UID capability is a newer, hail-mary attempt to fix the issue, and thus it's optional, not all IMAP servers support it
  - tedivm 4y ago
    I wrote a popular open source IMAP library in 2009 and even then the UID capability was there and widely adopted. My library relied on it 100%, completely ignoring sequence numbers, and I haven't received a single issue related to this. It's not that new, it's not a hail mary, and it is very well supported. * https://packagist.org/packages/tedivm/fetch https://packagist.org/packages/tedivm/fetch * https://github.com/tedious/Fetch https://github.com/tedious/Fetch
    - lsh123 4y ago
      Last time I was writing mail client code was around 2004-2005 and even around that time every large IMAP provider (AOL, Yahoo, …) had UID support.
  - citrin_ru 4y ago
    Newer as in only ~30 years old? It is likely some obscure IMAP servers doesn't support them but AFAIK all popular ones do.
ketralnis 4y ago
Pretty much no client or server uses it that way though. The UID extension is more or less required these days. But IMAP is pretty twisted in other ways too. IMAP is _not_ a request-response protocol like you might think from other protocols. It looks like it at first because you can say things like (from memory, not the real protocol): > cmd1 LIST (here cmd1 is a request/response identifier chosen by the client; the server sends responses prefixed by this identifier) < cmd1 LIST RESPONSE... But you can also multiplex many simultaneous commands like > cmd1 LIST folder1 > cmd2 LIST folder2 > cmd3 LIST folder3 < cmd3 LIST folder3 RESPONSE... < cmd1 LIST folder1 RESPONSE... < cmd2 LIST folder2 RESPONSE... (note that they can come out of order) But it gets weirder because you can receive _unsolicited_ responses. Sometimes you want that like > cmd1 WATCH folder1 (starts a watch which may have responses but not immediately) > (do other stuff) < * NEW MESSAGE folder1 DATA... (some time later) < * FLAGS CHANGE folder1 DATA... (much more time later) (the * means that you didn't ask for this so it's not prefixed by an ID that you gave) Here it's sort of solicited or at least something you did on purpose, but there are other more complicated cases where you'll receive data that you never asked for. Again from loose memory but an example might be > cmd1 OPEN folder1 > (do stuff with folder1) (much later) < * CLOSING folder1 WAS DELETED That means that your client needs to always be listening for data on the socket and reading it out even if the client is sitting in the background, and that if you send a command you might receive responses to other previous or unsolicited commands before the response to the command you just sent. It's a cool way to do what it wants to do for GUI clients in particular because IMAP and GUIs have the same event-oriented model. But it makes implementing IMAP as a library a little more difficult because you need bidirectional hooks into the containing application. And you can't easily just have a `if POP3 then... else if IMAP4 then...` abstraction layer without some work to make it happen. Heck even just having a function that returns the list of messages requires a background thread and a little bit of lying about what's happening underneath. Leaving data on the socket unconsumed can get you disconnected by keepalive checks so the background task is required if you want to avoid that (which by observing mail.app it seems to just accept the disconnection and reconnect when it happens). Python has an imap library in the stdlib which "avoids" this by being so low-level that it sidesteps some of the issues by making you implement request/response on top of it, and its model also makes perfect IDLE implementation difficult (maybe impossible? but I didn't put enough time into it to prove this). Because of the complication many GUI clients don't actually support the liveness built in to the protocol. They just ignore any reponses they didn't ask for, don't use WATCH/IDLE, and do the 15 minute polling cycle as if they were POP3. The protocol has all of the tools to have the liveness of Google Docs but I'm not aware of any GUIs that use it right.
globular-toast 4y ago
It's easy to take things like REST for granted. Until you learn about the stuff that came before it, like this!
- jbverschoor 4y ago
  Most protocols were clear text and session based. Simply connect to a pier and start talking English to the server Kind of like Siri lol
  - globular-toast 4y ago
    I remember randomly trying to connect to an FTP server with telnet and being shocked that it actually worked. I was even able to figure out the commands although couldn't figure out how downloads would work (until I read about how FTP works).

amelius 4y ago

I'm wondering why they don't give every mail a UUID. Probably because someone wanted to save a few bytes.

c0l0 4y ago
They do, on a per-session basis - but it's an optional feature (that's supported by ~100% of all imap server implementations).
- KRAKRISMOTT 4y ago
  What's the name of this extension?
  - epilys 4y ago
    It's actually in the original IMAP4rev1 RFC: https://www.rfc-editor.org/rfc/rfc3501#section-2.3.1.1 https://www.rfc-editor.org/rfc/rfc3501#section-2.3.1.1 Using UIDs is opt-in, by prefixing `UID ` in front of some commands. There are indeed extensions that add more `UID ` commands to the protocol. I've implemented IMAP clients and never used sequence numbers at all.
    - vidarh 4y ago
      UID was present already in RFC 1730 (IMAPv4): https://www.rfc-editor.org/rfc/rfc1730#section-6.4.9 https://www.rfc-editor.org/rfc/rfc1730#section-6.4.9
jcranmer 4y ago
One of the extensions to IMAP is the UID extension, which gives every email something akin to a UUID. It's technically not mandatory, but virtually every email client is going to try to use it if possible, and likely some email clients don't even attempt to use message sequence numbers. So you don't need to maintain message sequence numbers, except for the few places in the protocol where only the message sequence number is reported (e.g., the unsolicited message deleted alerts). In practice, I would actually expect that trying to use message sequence numbers would be buggier than not using them. But servers still have to support them!
Mailtemi 4y ago
Yep, just a few bites :) Especially if you're on vacation. IMAP IDs can change, requiring the email client to re-download all emails.

evmar 4y ago

A long time ago I wrote a toy IMAP client, and ran into similarly confusing issues about how the protocol is supposed to work: https://github.com/evmar/go-imap/blob/master/imap/doc.go https://github.com/evmar/go-imap/blob/master/imap/doc.go

calvinmorrison 4y ago

> Holy F what a clusterfuck And that's why when people say "JMAP is pointless", just go back and read some of the email specs!

Avamander 4y ago
Until things like Thunderbird support it, its adoption is unfortunately going to remain at the level it is currently at.
- tracker1 4y ago
  Which sucks as it's a massive chicken/egg problem... more clients and hosts need to support it. Of course, from what I understand is there are JMAP/IMAP gateway options, that should ease availability... assuming these things actually gain any traction ever.
  - calvinmorrison 4y ago
    Well email moves slowly for one, and adding another protocol is usually baked into a client and servers approach (imap, pop, smtp, etc) so its not impossible
- e12e 4y ago
  I'm guessing thunderbird will be the first mainstream gui mua to support jmap.

rglover 4y ago

Stuff like this makes the idea of an "Email 2" protocol being worthwhile. Shocking how over-complicated it is to send text/html between two parties.

Avamander 4y ago
People have called things email killers several times. An intelligent approach would be to gradually rebuild parts on top of better foundations and push towards that. The amount of deadlocks in adopting things like JMAP, IMAP NOTIFY, OAuth2 (+ Dynamic Client Registration) and so on and on is immense. Won't because others don't, too few do, we see no benefit yet etc. it's really hindering.

throw0101c 4y ago

>> IMAP clients typically refer to messages by their “sequence number” […] > Holy F what a clusterfuck Or you could use the unique identifier of the message (UID) instead of the sequence number: > Messages in IMAP4rev1 are accessed by one of two numbers; the unique identifier or the message sequence number. * https://www.rfc-editor.org/rfc/rfc3501#section-2.3.1 https://www.rfc-editor.org/rfc/rfc3501#section-2.3.1 > A 32-bit value assigned to each message, which when used with the unique identifier validity value (see below) forms a 64-bit value that MUST NOT refer to any other message in the mailbox or any subsequent mailbox with the same name forever. * https://www.rfc-editor.org/rfc/rfc3501#section-2.3.1.1 https://www.rfc-editor.org/rfc/rfc3501#section-2.3.1.1 The designers of the IMAP protocol aren't as dumb as the above quotation would have you to believe and multi-client operations were thought of.

meghan_rain 4y ago
The UID capability is a newer, hail-mary attempt to fix the issue, and thus it's optional, not all IMAP servers support it
- tedivm 4y ago
  I wrote a popular open source IMAP library in 2009 and even then the UID capability was there and widely adopted. My library relied on it 100%, completely ignoring sequence numbers, and I haven't received a single issue related to this. It's not that new, it's not a hail mary, and it is very well supported. * https://packagist.org/packages/tedivm/fetch https://packagist.org/packages/tedivm/fetch * https://github.com/tedious/Fetch https://github.com/tedious/Fetch
  - lsh123 4y ago
    Last time I was writing mail client code was around 2004-2005 and even around that time every large IMAP provider (AOL, Yahoo, …) had UID support.
- citrin_ru 4y ago
  Newer as in only ~30 years old? It is likely some obscure IMAP servers doesn't support them but AFAIK all popular ones do.

ketralnis 4y ago

Pretty much no client or server uses it that way though. The UID extension is more or less required these days. But IMAP is pretty twisted in other ways too. IMAP is _not_ a request-response protocol like you might think from other protocols. It looks like it at first because you can say things like (from memory, not the real protocol): > cmd1 LIST (here cmd1 is a request/response identifier chosen by the client; the server sends responses prefixed by this identifier) < cmd1 LIST RESPONSE... But you can also multiplex many simultaneous commands like > cmd1 LIST folder1 > cmd2 LIST folder2 > cmd3 LIST folder3 < cmd3 LIST folder3 RESPONSE... < cmd1 LIST folder1 RESPONSE... < cmd2 LIST folder2 RESPONSE... (note that they can come out of order) But it gets weirder because you can receive _unsolicited_ responses. Sometimes you want that like > cmd1 WATCH folder1 (starts a watch which may have responses but not immediately) > (do other stuff) < * NEW MESSAGE folder1 DATA... (some time later) < * FLAGS CHANGE folder1 DATA... (much more time later) (the * means that you didn't ask for this so it's not prefixed by an ID that you gave) Here it's sort of solicited or at least something you did on purpose, but there are other more complicated cases where you'll receive data that you never asked for. Again from loose memory but an example might be > cmd1 OPEN folder1 > (do stuff with folder1) (much later) < * CLOSING folder1 WAS DELETED That means that your client needs to always be listening for data on the socket and reading it out even if the client is sitting in the background, and that if you send a command you might receive responses to other previous or unsolicited commands before the response to the command you just sent. It's a cool way to do what it wants to do for GUI clients in particular because IMAP and GUIs have the same event-oriented model. But it makes implementing IMAP as a library a little more difficult because you need bidirectional hooks into the containing application. And you can't easily just have a `if POP3 then... else if IMAP4 then...` abstraction layer without some work to make it happen. Heck even just having a function that returns the list of messages requires a background thread and a little bit of lying about what's happening underneath. Leaving data on the socket unconsumed can get you disconnected by keepalive checks so the background task is required if you want to avoid that (which by observing mail.app it seems to just accept the disconnection and reconnect when it happens). Python has an imap library in the stdlib which "avoids" this by being so low-level that it sidesteps some of the issues by making you implement request/response on top of it, and its model also makes perfect IDLE implementation difficult (maybe impossible? but I didn't put enough time into it to prove this). Because of the complication many GUI clients don't actually support the liveness built in to the protocol. They just ignore any reponses they didn't ask for, don't use WATCH/IDLE, and do the 15 minute polling cycle as if they were POP3. The protocol has all of the tools to have the liveness of Google Docs but I'm not aware of any GUIs that use it right.

globular-toast 4y ago

It's easy to take things like REST for granted. Until you learn about the stuff that came before it, like this!

jbverschoor 4y ago
Most protocols were clear text and session based. Simply connect to a pier and start talking English to the server Kind of like Siri lol
- globular-toast 4y ago
  I remember randomly trying to connect to an FTP server with telnet and being shocked that it actually worked. I was even able to figure out the commands although couldn't figure out how downloads would work (until I read about how FTP works).