3 ms·

You're very welcome to join the EXTRA working group at IETF and propose the extensions that you think would make IMAP better. Apart from working on JMAP, I hav

You're very welcome to join the EXTRA working group at IETF and propose the extensions that you think would make IMAP better. Apart from working on JMAP, I have also written an extension to IMAP and may write another one next year bringing some of the push stuff from JMAP back to IMAP as well. But JMAP is based on many years of working on servers, middleware and (at least on the web) clients. We did actually extent IMAP with some experimental stuff first (cross folder threading and thread collapsing for example) but it never solved the batching and proxying difficulties and the cost of instantiating message numbers per-session that are un-avoidable parts of the IMAP model. By the way, engineers who work on other servers have said similar things to us - IMAP is costly to implement in terms of server resources when you're doing true concurrency, high availability, etc.

jorangreef 7y ago
> IMAP is costly to implement in terms of server resources when you're doing true concurrency, high availability Yes, IMAP requires expensive coordination among multi-master servers to generate UIDs in such a way that won't mess with client synchronization. In other words, IMAP is not a fully distributed design, it assumes a centralized control plane (think Raft or Paxos), i.e. a single central server with many distributed clients, which is a shame for new systems building on CRDTs. How does JMAP address this achilles heel of IMAP? Merkle tree sync? More detail on the problem: https://news.ycombinator.com/item?id=20479011 https://news.ycombinator.com/item?id=20479011
- brongondwana 7y ago
  Individual unique IDs and /changes lists all the IDs that might have changed. One good thing about IMAP is that there's a fair bit of data which is immutable, and JMAP retains that - emails with the same ID must have the same content, so there's a limited amount of metadata that you need to resync. I wrote something about how you fix UID clashes in a split brain while staying within the IMAP data model many years ago... https://lists.andrew.cmu.edu/pipermail/cyrus-devel/2010-July/001500.html https://lists.andrew.cmu.edu/pipermail/cyrus-devel/2010-July...
  - jorangreef 7y ago
    > /changes lists all the IDs that might have changed. How does it do this in a distributed setting? When clients might be connecting to multiple servers with different states?
    - brongondwana 7y ago
      Well, it depends on how you architect your system - you can embed whatever you like in your state. I'd probably do something cheap and nasty like embed a modseq counter for each of the different servers into the state string, and if any of them went backwards I'd reject the /changes with a cannotCalculateChanges error. I mean, you can do other things - but at the point where you're switching between multiple masters in a split brain from your client, all bets are off - so you're going to need to fall back resyncing any mutable data.
      - jorangreef 7y ago
        > but at the point where you're switching between multiple masters in a split brain from your client, all bets are off CRDTs handle split brain fine. They eventually converge. You just don't want your clients flapping in the meantime by resyncing tons of data. That's where a distributed sync algorithm would help keep things working in the interim.
  - jorangreef 7y ago
    > I wrote something about how you fix UID clashes in a split brain while staying within the IMAP data model many years ago... Thanks for this. It was a good read and promoting UIDs instead of bumping UIDVALIDITY is a great idea. Surely, it would be even better if IMAP had support for distributed Merkle Tree synchronization, so that clients can sync with different server replicas without the server replicas having to coordinate or promote UIDs at all in the first place? I don't know of any algorithm other than a Merkle Tree that will do this correctly and simply and efficiently in a distributed setting?
    - brongondwana 7y ago
- throw0101a 7y ago
  > Yes, IMAP requires expensive coordination among multi-master servers to generate UIDs in such a way that won't mess with client synchronization. In other words, IMAP is not a fully distributed design Just use NNTP! /s :)

jorangreef 7y ago

> IMAP is costly to implement in terms of server resources when you're doing true concurrency, high availability Yes, IMAP requires expensive coordination among multi-master servers to generate UIDs in such a way that won't mess with client synchronization. In other words, IMAP is not a fully distributed design, it assumes a centralized control plane (think Raft or Paxos), i.e. a single central server with many distributed clients, which is a shame for new systems building on CRDTs. How does JMAP address this achilles heel of IMAP? Merkle tree sync? More detail on the problem: https://news.ycombinator.com/item?id=20479011 https://news.ycombinator.com/item?id=20479011

brongondwana 7y ago
Individual unique IDs and /changes lists all the IDs that might have changed. One good thing about IMAP is that there's a fair bit of data which is immutable, and JMAP retains that - emails with the same ID must have the same content, so there's a limited amount of metadata that you need to resync. I wrote something about how you fix UID clashes in a split brain while staying within the IMAP data model many years ago... https://lists.andrew.cmu.edu/pipermail/cyrus-devel/2010-July/001500.html https://lists.andrew.cmu.edu/pipermail/cyrus-devel/2010-July...
- jorangreef 7y ago
  > /changes lists all the IDs that might have changed. How does it do this in a distributed setting? When clients might be connecting to multiple servers with different states?
  - brongondwana 7y ago
    Well, it depends on how you architect your system - you can embed whatever you like in your state. I'd probably do something cheap and nasty like embed a modseq counter for each of the different servers into the state string, and if any of them went backwards I'd reject the /changes with a cannotCalculateChanges error. I mean, you can do other things - but at the point where you're switching between multiple masters in a split brain from your client, all bets are off - so you're going to need to fall back resyncing any mutable data.
    - jorangreef 7y ago
      > but at the point where you're switching between multiple masters in a split brain from your client, all bets are off CRDTs handle split brain fine. They eventually converge. You just don't want your clients flapping in the meantime by resyncing tons of data. That's where a distributed sync algorithm would help keep things working in the interim.
- jorangreef 7y ago
  > I wrote something about how you fix UID clashes in a split brain while staying within the IMAP data model many years ago... Thanks for this. It was a good read and promoting UIDs instead of bumping UIDVALIDITY is a great idea. Surely, it would be even better if IMAP had support for distributed Merkle Tree synchronization, so that clients can sync with different server replicas without the server replicas having to coordinate or promote UIDs at all in the first place? I don't know of any algorithm other than a Merkle Tree that will do this correctly and simply and efficiently in a distributed setting?
  - brongondwana 7y ago
throw0101a 7y ago
> Yes, IMAP requires expensive coordination among multi-master servers to generate UIDs in such a way that won't mess with client synchronization. In other words, IMAP is not a fully distributed design Just use NNTP! /s :)