4 ms·

You're very welcome to comment on the IETF mailing list if you think you can make a compelling argument for having getMailboxes also take a filter allowing you

You're very welcome to comment on the IETF mailing list if you think you can make a compelling argument for having getMailboxes also take a filter allowing you to build a tree with "parentId": [null] and then use the IDs from each of those responses to create a "parentIds": [x, y, z] query. Of course nothing stops you storing thousands of mailboxes in a non-heirarchical layout and making IMAP have large LIST fetches too. (I do wonder - did your solution via IMAP still fetch everything each time, because in that case all you're doing is spreading that megabyte out over multiple round trips)

mike-cardwell 9y ago
> You're very welcome to comment on the IETF mailing list if you think you can make a compelling argument for having getMailboxes also take a filter allowing you to build a tree with "parentId": [null] and then use the IDs from each of those responses to create a "parentIds": [x, y, z] query. I'm not invested enough to sign up to a mailing list and try to convince people to change a protocol. I'm only invested enough to fire off a comment on a forum I am already signed up to. I explained my use case above. Whether or not it is compelling is up to the reader to decide. Maybe I'm the only person in the World who has dealt with large hierarchies of mailboxes. > Of course nothing stops you storing thousands of mailboxes in a non-heirarchical layout and making IMAP have large LIST fetches too. Yes. I would issue the LIST command, and as the IMAP server started streaming the results to the client, I would immediately process each mailbox as soon as it was down, rather than waiting for them all to be retrieved. I fear that with a JSON based protocol, most server and client implementations wouldn't deal with streaming (although they technically could), and would just deal with completed blobs. Does the Cyrus implementation construct the entire blob of JSON before beginning sending it to the client, or does it start constructing and streaming the JSON as soon as possible, before it has even fetched the entire list of mailboxes from whatever the backend data store is? The JMAP streaming solution seems like a more complicated solution for developers to implement than the IMAP streaming solution. > (I do wonder - did your solution via IMAP still fetch everything each time, because in that case all you're doing is spreading that megabyte out over multiple round trips) People were accessing this mailbox with many different clients. How each of those clients worked under the hood, I do not know. I did build a web frontend, and that web frontend worked efficiently, fetching only the folders that were being displayed to the user, and their immediate children. A user clicked a folder to expand it, and they would immediately see the pre-fetched child folders, and a new request to fetch those folders children was then triggered.
- brongondwana 9y ago
  Sadly protocols are made by those who show up and put the effort in. I'll pass a link to this comment on to the list and see if others chime in. The downside of allowing filtered getMailboxes is more protocol complexity, but maybe it's worth it. On to the next topic: Cyrus creates the entire blob. Even when it's streaming a LIST reply it still batches a lot in memory. You'll discover pretty fast if you don't do that, you might wind up holding a lock for a LONG time if you're trying to build consistent replies and the remote end is on a slow network link or just consuming slowly. So we batch all replies right now. If I was building a brand new product I might use an engine which supported MVCC (like Postgres database for example) and hence have flexibility. Having said that - even then you might find that a server talking to slow clients would want to buffer to temp file and free up memory and locks while it streamed the data. With HTTP we get around this by having nginx handle all the mess, so I guess that's what will happen with JMAP too, nginx buffering large bodies to disk.
  - mike-cardwell 9y ago
    > Sadly protocols are made by those who show up and put the effort in. Yes. And like everybody, I have my own priorities, and will not apologise for JMAP not being one of them. Somebody like yourself who is actually invested in making JMAP a success should welcome people pointing out real life use cases where the protocol you're trying to improve upon actually works better than the one you've come up with. I would hope you're spending significant effort trying to get people to look over your spec and point out flaws.
    - jsmthrowaway 9y ago
      > Somebody like yourself who is actually invested in making JMAP a success should welcome people pointing out real life use cases where the protocol you're trying to improve upon actually works better I'm pretty sure it was welcomed, and you were invited to contribute, and even after you declined to do so your comment was still acted upon. You're not coming off super well in this thread, by the way: "I have feedback and time to engage a HN subthread, but getting involved with the actual thing I'm criticizing is a burden I choose not to bear, and this is your problem and you should change to accommodate me." (I don't think that's what you intend, but that was what I read.) There's a bunch of snark in this thread I don't comprehend. It seems like a fine exchange? Everybody's friends?

mike-cardwell 9y ago

> You're very welcome to comment on the IETF mailing list if you think you can make a compelling argument for having getMailboxes also take a filter allowing you to build a tree with "parentId": [null] and then use the IDs from each of those responses to create a "parentIds": [x, y, z] query. I'm not invested enough to sign up to a mailing list and try to convince people to change a protocol. I'm only invested enough to fire off a comment on a forum I am already signed up to. I explained my use case above. Whether or not it is compelling is up to the reader to decide. Maybe I'm the only person in the World who has dealt with large hierarchies of mailboxes. > Of course nothing stops you storing thousands of mailboxes in a non-heirarchical layout and making IMAP have large LIST fetches too. Yes. I would issue the LIST command, and as the IMAP server started streaming the results to the client, I would immediately process each mailbox as soon as it was down, rather than waiting for them all to be retrieved. I fear that with a JSON based protocol, most server and client implementations wouldn't deal with streaming (although they technically could), and would just deal with completed blobs. Does the Cyrus implementation construct the entire blob of JSON before beginning sending it to the client, or does it start constructing and streaming the JSON as soon as possible, before it has even fetched the entire list of mailboxes from whatever the backend data store is? The JMAP streaming solution seems like a more complicated solution for developers to implement than the IMAP streaming solution. > (I do wonder - did your solution via IMAP still fetch everything each time, because in that case all you're doing is spreading that megabyte out over multiple round trips) People were accessing this mailbox with many different clients. How each of those clients worked under the hood, I do not know. I did build a web frontend, and that web frontend worked efficiently, fetching only the folders that were being displayed to the user, and their immediate children. A user clicked a folder to expand it, and they would immediately see the pre-fetched child folders, and a new request to fetch those folders children was then triggered.

brongondwana 9y ago
Sadly protocols are made by those who show up and put the effort in. I'll pass a link to this comment on to the list and see if others chime in. The downside of allowing filtered getMailboxes is more protocol complexity, but maybe it's worth it. On to the next topic: Cyrus creates the entire blob. Even when it's streaming a LIST reply it still batches a lot in memory. You'll discover pretty fast if you don't do that, you might wind up holding a lock for a LONG time if you're trying to build consistent replies and the remote end is on a slow network link or just consuming slowly. So we batch all replies right now. If I was building a brand new product I might use an engine which supported MVCC (like Postgres database for example) and hence have flexibility. Having said that - even then you might find that a server talking to slow clients would want to buffer to temp file and free up memory and locks while it streamed the data. With HTTP we get around this by having nginx handle all the mess, so I guess that's what will happen with JMAP too, nginx buffering large bodies to disk.
- mike-cardwell 9y ago
  > Sadly protocols are made by those who show up and put the effort in. Yes. And like everybody, I have my own priorities, and will not apologise for JMAP not being one of them. Somebody like yourself who is actually invested in making JMAP a success should welcome people pointing out real life use cases where the protocol you're trying to improve upon actually works better than the one you've come up with. I would hope you're spending significant effort trying to get people to look over your spec and point out flaws.
  - jsmthrowaway 9y ago
    > Somebody like yourself who is actually invested in making JMAP a success should welcome people pointing out real life use cases where the protocol you're trying to improve upon actually works better I'm pretty sure it was welcomed, and you were invited to contribute, and even after you declined to do so your comment was still acted upon. You're not coming off super well in this thread, by the way: "I have feedback and time to engage a HN subthread, but getting involved with the actual thing I'm criticizing is a burden I choose not to bear, and this is your problem and you should change to accommodate me." (I don't think that's what you intend, but that was what I read.) There's a bunch of snark in this thread I don't comprehend. It seems like a fine exchange? Everybody's friends?