4 ms·

That would be a pretty massive change to the entire data model of IMAP. It's never going to happen[tm]. IMAP4rev2 is much more conservative than that. https:

That would be a pretty massive change to the entire data model of IMAP. It's never going to happen[tm]. IMAP4rev2 is much more conservative than that. https://datatracker.ietf.org/doc/draft-ietf-extra-imap4rev2/ https://datatracker.ietf.org/doc/draft-ietf-extra-imap4rev2/ Merkle tree would is great for full history - I built something like that with digital signing for tracking clinical data over 15 years ago that would have been fine for email as well - but it's way over-expensive for email. Which leads back to the key design principle I try to use when working on protocols, which is something like "make the weird edge cases sand and possible, but don't optimise them!" You don't want to build the Merkle tree complexity into a client and have all the servers be completely independent, because it makes the protocol a pain to use for the simple case. (What we use for IMAP replication in Cyrus is much dumber than all this - it uses a per-message CRC and XORs them together for the whole mailbox to create the sync_crc value. If those don't match, it does a resync. Fixes up split brain just fine while being cheap and dumb for the common case.

jorangreef 7y ago
> Merkle tree would is great for full history - [...] but it's way over-expensive for email. Does JMAP then not support syncing full history efficiently as a key design priority? > That would be a pretty massive change to the entire data model of IMAP. It's never going to happen[tm]. Sure, and that's why I had hoped JMAP would, especially in a distributed master setting, since this was something IMAP overlooked. I guess it's because JMAP still needs to be backwards-compatible with the strict consistency master-slave restriction of IMAP? > What we use for IMAP replication in Cyrus is much dumber than all this - it uses a per-message CRC and XORs them together for the whole mailbox to create the sync_crc value. If those don't match, it does a resync. Fixes up split brain just fine while being cheap and dumb for the common case. You're implementing the first layer of an incremental Merkle tree right there. How many bits are those CRCs just out of interest?
- brongondwana 7y ago
  Not full history as in "you can rewind back in time and see every possible past state", no. If I wanted one of those, I'd probably use git as the substrate. The thing JMAP doesn't have, that I think you're trying to ask for, is the ability to talk to divergent endpoints and resolve the state on the client. JMAP expects to talk to a server which is merging the state into a linear world for it. That linear world view can be different from different servers. For example, if you implemented your server as a series of git commits, complete with merges whenever you'd had changes on two different servers, then you could just use a commit hash as the state string - and if you got an oldState which was a commit that didn't happen on your server you could still use git to calculate the series of changes on your branch since the fork point plus changes on the remote branch after that commit - and you'd be able to issue a full set of changes. If you DON'T have at least that commit and its predecessors locally, you can't calculate the diff, so you have to return cannotCalculateChanges. I think what you're wanting here is for the server to get from the client what the last fork point was, and give the client just the changes it knows about since that fork point (like a git fetch) and then the client calculates the merged changes. That's not how JMAP is designed to work. It's an interesting idea, but it does lead to very high client complexity, so it's not what we wanted JMAP to be. The CRCs are 32 bits, but they're not the only signal. There's also modseq and uidnext, so the CRC doesn't need to be cryptographically strong - if somebody deliberately creates a replica state that comes up with the same CRC but has different content then they could make servers believe they are in sync, but in that attack vector they probably also have enough control to just make the replication layer lie. We're also using sha1 for message content integrity, which I'd like to move away from eventually. There are plans, but we have other things to finish first. We do insert some randomness into a header on message delivery which makes it much harder for someone to calculate what will be injected when sending somebody an email via SMTP!
  - deleted 7y ago
    [deleted]
  - jorangreef 7y ago
    > Not full history as in "you can rewind back in time and see every possible past state", no. If I wanted one of those, I'd probably use git as the substrate. No, not "history" in the git sense, just "history" in the simpler email sense, i.e. "email history", all past emails in their final converged state.

jorangreef 7y ago

> Merkle tree would is great for full history - [...] but it's way over-expensive for email. Does JMAP then not support syncing full history efficiently as a key design priority? > That would be a pretty massive change to the entire data model of IMAP. It's never going to happen[tm]. Sure, and that's why I had hoped JMAP would, especially in a distributed master setting, since this was something IMAP overlooked. I guess it's because JMAP still needs to be backwards-compatible with the strict consistency master-slave restriction of IMAP? > What we use for IMAP replication in Cyrus is much dumber than all this - it uses a per-message CRC and XORs them together for the whole mailbox to create the sync_crc value. If those don't match, it does a resync. Fixes up split brain just fine while being cheap and dumb for the common case. You're implementing the first layer of an incremental Merkle tree right there. How many bits are those CRCs just out of interest?

brongondwana 7y ago
Not full history as in "you can rewind back in time and see every possible past state", no. If I wanted one of those, I'd probably use git as the substrate. The thing JMAP doesn't have, that I think you're trying to ask for, is the ability to talk to divergent endpoints and resolve the state on the client. JMAP expects to talk to a server which is merging the state into a linear world for it. That linear world view can be different from different servers. For example, if you implemented your server as a series of git commits, complete with merges whenever you'd had changes on two different servers, then you could just use a commit hash as the state string - and if you got an oldState which was a commit that didn't happen on your server you could still use git to calculate the series of changes on your branch since the fork point plus changes on the remote branch after that commit - and you'd be able to issue a full set of changes. If you DON'T have at least that commit and its predecessors locally, you can't calculate the diff, so you have to return cannotCalculateChanges. I think what you're wanting here is for the server to get from the client what the last fork point was, and give the client just the changes it knows about since that fork point (like a git fetch) and then the client calculates the merged changes. That's not how JMAP is designed to work. It's an interesting idea, but it does lead to very high client complexity, so it's not what we wanted JMAP to be. The CRCs are 32 bits, but they're not the only signal. There's also modseq and uidnext, so the CRC doesn't need to be cryptographically strong - if somebody deliberately creates a replica state that comes up with the same CRC but has different content then they could make servers believe they are in sync, but in that attack vector they probably also have enough control to just make the replication layer lie. We're also using sha1 for message content integrity, which I'd like to move away from eventually. There are plans, but we have other things to finish first. We do insert some randomness into a header on message delivery which makes it much harder for someone to calculate what will be injected when sending somebody an email via SMTP!
- deleted 7y ago
  [deleted]
- jorangreef 7y ago
  > Not full history as in "you can rewind back in time and see every possible past state", no. If I wanted one of those, I'd probably use git as the substrate. No, not "history" in the git sense, just "history" in the simpler email sense, i.e. "email history", all past emails in their final converged state.