5 ms·

So bummed. Encryption is missing again. > Users expect to be able to search their whole archive, so either you need all the data in the client, or the server n

So bummed. Encryption is missing again. > Users expect to be able to search their whole archive, so either you need all the data in the client, or the server needs to have access to the data... ...JMAP is therefore not introducing any new measures to address end-to-end encryption You don't need to transfer gigs of emails to a client and manually search character-by-character through a massive trove of data. That isn't how search functions. You can use probabilistic filters (like bloomfilters) so a client can actually run a search on thousands of emails using only a few hundred kilobytes (or maybe megabytes) of actual encrypted data. Basically, less than a modern webpage. Each time a new email is downloaded, parse it, add it to an index segment, encrypt it, and store it back on the email server. Different clients can reuse the same indexes.

eviks 3y ago
Exactly, the explanation doesn't hold water, and it's a pity such and important issue is sidestepped in this way
Szpadel 3y ago
from technical point of view this sounds interesting, do you have any example of such implementation for text search anywhere? I can see multiple challenges in implementing something like this and I would like to educate myself
- Xeoncross 3y ago
  https://www.stavros.io/posts/bloom-filter-search-engine/ https://www.stavros.io/posts/bloom-filter-search-engine/
  - Szpadel 3y ago
    Either I didn't understand something or this method would still require server to have way to decrypt data to create that index? Or this could be computed on client side but that's not much different than fetching all emails locally
    - Xeoncross 3y ago
      Clients already fetch all emails locally over the years you use the email. You don't have to build the index in one pass. As you read/download each new email, you add it to the index segments. The same way you don't have to download the internet to create your searchable browser history.
      - j16sdiz 3y ago
        It don't help with my existing multi-GB email archive.
    - eviks 3y ago
      You could also have the bigger desktop client fetch everything for the purposes of building an index (and also having a backup of your email), then your smaller phone doesn't have to fetch everything
solarkraft 3y ago
I don't know, Tutanota does client side search and I don't like it. But they do full text searches. Do bloom filters deliver the same accuracy?
- klysm 3y ago
  No, they are probabilistic data structures and it’s not immediately obvious to me how to use them effectively for search
  - dataangel 3y ago
    You use bloom filtering to massively reduce the number of possible matching messages, then you do a regular text search on the remaining set.
    - klysm 3y ago
      But how do you actually index text with a single bloom filter?
      - j16sdiz 3y ago
        You don't. You put mail into buckets (using some hashes), and hands out the bloom filter for each bucket. It is better than downloading all mail, but it is nowhere near the speed I want my email search to be.
        klysm 3y ago
        And there is a bucket for each word? That seems like a lot of bloom filters
joelthelion 3y ago
How do you keep your bloom filter up to date if the server doesn't have access to the data?
tiffanyh 3y ago
The proposal for encryption extension to JMAP is already on it's 3rd draft for review. https://www.ietf.org/id/draft-ietf-jmap-smime-sender-extensions-03.html https://www.ietf.org/id/draft-ietf-jmap-smime-sender-extensi...
- Avamander 3y ago
  Eugh, that might fly in some enterprise contexts but it violates a lot of principles what S/MIME is meant to be used for. Is there a draft for strict JMAP transport security?
  - chrismorgan 3y ago
    JMAP is already TLS-only <https://datatracker.ietf.org/doc/html/rfc8620#section-8.1 https://datatracker.ietf.org/doc/html/rfc8620#section-8.1>: > To ensure the confidentiality and integrity of data sent and received via JMAP, all requests MUST use TLS 1.2 or later, following the recommendations in RFC 7525. Servers SHOULD support TLS 1.3 or later. > Clients MUST validate TLS certificate chains to protect against man-in-the-middle attacks.
    - Avamander 3y ago
      That's not "MUST prevent the end-user from trivially accepting the invalid chain" though, which is also an issue with current MUAs and IMAP/SMTP.
crazygringo 3y ago
That still requires the client to have downloaded all emails in the past in order to generate the Bloom filters. Those filters can't be provided by the server. Which is maybe fine if you're starting an e-mail account from scratch, but it is going to take a long time to set up email search on a new phone. Also e-mails are a pretty bad use case for Bloom filters because you need to construct an entire filter per-email for search. Since many emails are short (a sentence or two) you'll find each Bloom filter may actually be larger than the e-mail itself. I don't know if there are other types of probabilistic filters more suited to e-mail search however.
- Xeoncross 3y ago
  > That still requires the client to have downloaded all emails in the past in order to generate the Bloom filters. That's generally not an issue. When people first create an email there is only 1 welcome email in the account. The client web/mobile/desktop downloads that one email and creates the search index. If you get a new client/phone/desktop you can still reuse the encrypted index files. It's not like you have to download your last 5 years emails on each new device.
  - SahAssar 3y ago
    In that scenario would each device to publish their index files and the others using the same account download and use them? Are there good merging methods for this or would it just be latest wins?
    - Xeoncross 3y ago
      They would reuse a few large fragments. Maybe one each month or year. There would need to be some sort of locking mech to keep clients from clobbering each other, or maybe not since the final state of the filter would probably be the same anyway.
  - willhackett 3y ago
    You can possibly generate these filters on the server-side and serialize them for the client. I like the idea of only needing to store some pointers and metadata locally; that said I do prefer all my mail is available offline.
slaymaker1907 3y ago
The only way I could see this working would be if there was some way for an email client to store its own encrypted data on the server that is not an email/attachment and is tied to a particular email client. With such an API, clients could just do the indexing themselves and then store the index on the server. You'd probably want something like oblivious-RAM so that clients could avoid reading and writing the entire index all at once. Ok, the more I think this out, the more I like the idea of email servers providing ORAM for email clients. The oblivious aspect could actually be optional as well since the server only needs to provide random access and/or a K/V store. Maybe this could be implemented currently using a fake email draft? Blobs are apparently cleaned up transparently when they are no longer referenced so you could just make sure blobs are small and treat them as immutable blocks. The remaining problem would be encryption of the actual emails, but there is already PGP/GPG for that.
- Avamander 3y ago
  The receiving mail server could encrypt everything for the mailbox owner's S/MIME keypair, that would be really rather seamless with already existing protocols and clients. This "only" leaves the problem of key management to the user, which in my opinion is far out of scope for JMAP as well. Search will also become difficult but it can be done. At least most mail clients could display all your mail if you have the key.
  - slaymaker1907 3y ago
    The thing I wanted to cover with my idea is how to avoid rebuilding the index on each client as well as not having to store the entire index on the client. In my opinion, large indexes are probably a requirement for broad adoption by users since accurate search is very important for email (besides user desires, this could even be a legal requirement for discovery requests during a lawsuit).

eviks 3y ago

Exactly, the explanation doesn't hold water, and it's a pity such and important issue is sidestepped in this way

Szpadel 3y ago

from technical point of view this sounds interesting, do you have any example of such implementation for text search anywhere? I can see multiple challenges in implementing something like this and I would like to educate myself

Xeoncross 3y ago
https://www.stavros.io/posts/bloom-filter-search-engine/ https://www.stavros.io/posts/bloom-filter-search-engine/
- Szpadel 3y ago
  Either I didn't understand something or this method would still require server to have way to decrypt data to create that index? Or this could be computed on client side but that's not much different than fetching all emails locally
  - Xeoncross 3y ago
    Clients already fetch all emails locally over the years you use the email. You don't have to build the index in one pass. As you read/download each new email, you add it to the index segments. The same way you don't have to download the internet to create your searchable browser history.
    - j16sdiz 3y ago
      It don't help with my existing multi-GB email archive.
  - eviks 3y ago
    You could also have the bigger desktop client fetch everything for the purposes of building an index (and also having a backup of your email), then your smaller phone doesn't have to fetch everything

solarkraft 3y ago

I don't know, Tutanota does client side search and I don't like it. But they do full text searches. Do bloom filters deliver the same accuracy?

klysm 3y ago
No, they are probabilistic data structures and it’s not immediately obvious to me how to use them effectively for search
- dataangel 3y ago
  You use bloom filtering to massively reduce the number of possible matching messages, then you do a regular text search on the remaining set.
  - klysm 3y ago
    But how do you actually index text with a single bloom filter?
    - j16sdiz 3y ago
      You don't. You put mail into buckets (using some hashes), and hands out the bloom filter for each bucket. It is better than downloading all mail, but it is nowhere near the speed I want my email search to be.
      - klysm 3y ago
        And there is a bucket for each word? That seems like a lot of bloom filters

joelthelion 3y ago

How do you keep your bloom filter up to date if the server doesn't have access to the data?

tiffanyh 3y ago

The proposal for encryption extension to JMAP is already on it's 3rd draft for review. https://www.ietf.org/id/draft-ietf-jmap-smime-sender-extensions-03.html https://www.ietf.org/id/draft-ietf-jmap-smime-sender-extensi...

Avamander 3y ago
Eugh, that might fly in some enterprise contexts but it violates a lot of principles what S/MIME is meant to be used for. Is there a draft for strict JMAP transport security?
- chrismorgan 3y ago
  JMAP is already TLS-only <https://datatracker.ietf.org/doc/html/rfc8620#section-8.1 https://datatracker.ietf.org/doc/html/rfc8620#section-8.1>: > To ensure the confidentiality and integrity of data sent and received via JMAP, all requests MUST use TLS 1.2 or later, following the recommendations in RFC 7525. Servers SHOULD support TLS 1.3 or later. > Clients MUST validate TLS certificate chains to protect against man-in-the-middle attacks.
  - Avamander 3y ago
    That's not "MUST prevent the end-user from trivially accepting the invalid chain" though, which is also an issue with current MUAs and IMAP/SMTP.

crazygringo 3y ago

That still requires the client to have downloaded all emails in the past in order to generate the Bloom filters. Those filters can't be provided by the server. Which is maybe fine if you're starting an e-mail account from scratch, but it is going to take a long time to set up email search on a new phone. Also e-mails are a pretty bad use case for Bloom filters because you need to construct an entire filter per-email for search. Since many emails are short (a sentence or two) you'll find each Bloom filter may actually be larger than the e-mail itself. I don't know if there are other types of probabilistic filters more suited to e-mail search however.

Xeoncross 3y ago
> That still requires the client to have downloaded all emails in the past in order to generate the Bloom filters. That's generally not an issue. When people first create an email there is only 1 welcome email in the account. The client web/mobile/desktop downloads that one email and creates the search index. If you get a new client/phone/desktop you can still reuse the encrypted index files. It's not like you have to download your last 5 years emails on each new device.
- SahAssar 3y ago
  In that scenario would each device to publish their index files and the others using the same account download and use them? Are there good merging methods for this or would it just be latest wins?
  - Xeoncross 3y ago
    They would reuse a few large fragments. Maybe one each month or year. There would need to be some sort of locking mech to keep clients from clobbering each other, or maybe not since the final state of the filter would probably be the same anyway.
- willhackett 3y ago
  You can possibly generate these filters on the server-side and serialize them for the client. I like the idea of only needing to store some pointers and metadata locally; that said I do prefer all my mail is available offline.

slaymaker1907 3y ago

The only way I could see this working would be if there was some way for an email client to store its own encrypted data on the server that is not an email/attachment and is tied to a particular email client. With such an API, clients could just do the indexing themselves and then store the index on the server. You'd probably want something like oblivious-RAM so that clients could avoid reading and writing the entire index all at once. Ok, the more I think this out, the more I like the idea of email servers providing ORAM for email clients. The oblivious aspect could actually be optional as well since the server only needs to provide random access and/or a K/V store. Maybe this could be implemented currently using a fake email draft? Blobs are apparently cleaned up transparently when they are no longer referenced so you could just make sure blobs are small and treat them as immutable blocks. The remaining problem would be encryption of the actual emails, but there is already PGP/GPG for that.

Avamander 3y ago
The receiving mail server could encrypt everything for the mailbox owner's S/MIME keypair, that would be really rather seamless with already existing protocols and clients. This "only" leaves the problem of key management to the user, which in my opinion is far out of scope for JMAP as well. Search will also become difficult but it can be done. At least most mail clients could display all your mail if you have the key.
- slaymaker1907 3y ago
  The thing I wanted to cover with my idea is how to avoid rebuilding the index on each client as well as not having to store the entire index on the client. In my opinion, large indexes are probably a requirement for broad adoption by users since accurate search is very important for email (besides user desires, this could even be a legal requirement for discovery requests during a lawsuit).