5 ms·

Bah - file storage. Yep, it certainly is. And to a large degree, because it's a log. Once you email something as an attachment (even to yourself) it's like a

Bah - file storage. Yep, it certainly is. And to a large degree, because it's a log. Once you email something as an attachment (even to yourself) it's like a version control commit. That copy is locked in, and won't change. You can find it and refer back to it later. It has a timestamp on it, and it's probably stored in the cloud somewhere with backups and such. Hence "memory" - because that's how people use it :)

dgudkov 9y ago
>And to a large degree, because it's a log. I agree. Although, it's more than just a log. For instance, it would be great to have the ability to see a collection of all files sent by a particular sender, with file versions (with dates and links to according emails) stacked under one file name. Another suggestion -- view a collection of files in all emails with particular label. Finally, in some cases it can be helpful to have a local read-only copy of files in a particular collection. It's like one-way read-only Dropbox. This would make automated processing of email attachments so much simpler. (BTW, a product idea, anyone?!).
- brongondwana 9y ago
  So one of the smartass login messages in our internal slack is "JMAP will fix it". Interestingly enough, JMAP will make this kind of thing much easier - and we have thought about it in the design! Our Cyrus server already internally keeps a blobId which is a digest of the binary content of each MIME part. Which means you can easily tell that two attachments are identical without reading the contents or guessing based on size and name. And it's easy to search for messages with file attachments and fetch just the file attachement details via JMAP, making this kind of interface relatively easy to write. The usual caveats about "we haven't got a particular timeframe in mind" apply - but once JMAP is standard, anyone can write a client to that standard!
  - pdfernhout 9y ago
    Hi Bron. JMAP is a great design for what it is (a much better IMAP). Thanks for creating it. I can see JMAP becoming a great standard for email-oriented client/server communications. That said, I'd still encourage you to think bigger to make a more general system. Of course, the risk of thinking bigger is you may end up with either nothing or a monstrosity no one wants to use. And also you'd likely have to adapt your business model eventually if the bigger picture changes through your actions. So, it's fair to think smaller too. From a related post I made to the Thunderbird Planning list in April 2017 on what I see as the bigger picture: https://mail.mozilla.org/pipermail/tb-planning/2017-April/005533.html https://mail.mozilla.org/pipermail/tb-planning/2017-April/00... "My perspective is from wanting to create tools to support a "social semantic desktop" of which email is a subset of the items. Other types of items include news messages, RSS feed messages, chat messages, notification messages, IoT messages, blog posts, pingbacks, saved web pages, commands constructing complex shared documents, VR world update messages, code snippets, notes, and more. Email is a special case of a much larger world of messages. So, for me, a good assumption is that people will eventually have billions of message items comprising terabytes of data in their local (shared-with-email) store. Eventually each user would be part of a global federation of users that together have billions of times that number of messages as a federated semantic web. I understand that such a vision of messaging on that scale is not yet the goal of most people in this discussion right now. But I'd still urge people to think a little beyond redoing Thunderbird as-it-is and try to imagine a personal messaging system that processes, archives, indexes, and recalls vast numbers of items every day in interaction with a user and automated systems as part of a larger federation of similar systems using shared semantic standards." I've been beating my head against that wall for years with very limited progress (as have others) -- so I can't say it isn't a high risk endeavor with a low probability of success. But the benefit for humanity could potentially be huge if such an effort succeeded. Maybe it might just take a few tweaks to JMAP to generalize it enough to realize some of that big picture?
    - brongondwana 9y ago
      JMAP is pretty generalizable to other datatypes! It's a large part of why we split it into Core and Mail. We also have Calendar and Contacts modules in a reasonable state already. We also use JMAP-style APIs for Files, for Notes, and for all the extra Preferences and group management stuff in both FastMail and Topicbox.
lloeki 9y ago
> it's like a version control commit. As an alternative to mbox/maildir/whatevs I've been toying with a prototype of email storage that works in a content-addressed way with objects very much like how git operates internally. > Our Cyrus server already internally keeps a blobId which is a digest of the binary content of each MIME part. Which means you can easily tell that two attachments are identical without reading the contents or guessing based on size and name. This deduplication property then comes naturally from the above design, and not just for attachments but also for email headers and bodies. Many other things come easily such as rsync "just working" for fetching, or making use of git packs[0] [0]: https://git-scm.com/docs/pack-format https://git-scm.com/docs/pack-format
- brongondwana 9y ago
  You're not the first person to think of using git-like storage for email, and you won't be the last! Sadly deduplication isn't quite that easy - while the binary formats might be the same, MIME is very un-roundtrip-friendly when you start from the MIME structure. There are many ways to encode the same bytes, so if you need to reconstruct the raw bytes (FETCH RFC822) then you need a way to recover those original bytes. I've started work on that a few times, but it's never been worth the effort to make it stable.

dgudkov 9y ago

>And to a large degree, because it's a log. I agree. Although, it's more than just a log. For instance, it would be great to have the ability to see a collection of all files sent by a particular sender, with file versions (with dates and links to according emails) stacked under one file name. Another suggestion -- view a collection of files in all emails with particular label. Finally, in some cases it can be helpful to have a local read-only copy of files in a particular collection. It's like one-way read-only Dropbox. This would make automated processing of email attachments so much simpler. (BTW, a product idea, anyone?!).

brongondwana 9y ago
So one of the smartass login messages in our internal slack is "JMAP will fix it". Interestingly enough, JMAP will make this kind of thing much easier - and we have thought about it in the design! Our Cyrus server already internally keeps a blobId which is a digest of the binary content of each MIME part. Which means you can easily tell that two attachments are identical without reading the contents or guessing based on size and name. And it's easy to search for messages with file attachments and fetch just the file attachement details via JMAP, making this kind of interface relatively easy to write. The usual caveats about "we haven't got a particular timeframe in mind" apply - but once JMAP is standard, anyone can write a client to that standard!
- pdfernhout 9y ago
  Hi Bron. JMAP is a great design for what it is (a much better IMAP). Thanks for creating it. I can see JMAP becoming a great standard for email-oriented client/server communications. That said, I'd still encourage you to think bigger to make a more general system. Of course, the risk of thinking bigger is you may end up with either nothing or a monstrosity no one wants to use. And also you'd likely have to adapt your business model eventually if the bigger picture changes through your actions. So, it's fair to think smaller too. From a related post I made to the Thunderbird Planning list in April 2017 on what I see as the bigger picture: https://mail.mozilla.org/pipermail/tb-planning/2017-April/005533.html https://mail.mozilla.org/pipermail/tb-planning/2017-April/00... "My perspective is from wanting to create tools to support a "social semantic desktop" of which email is a subset of the items. Other types of items include news messages, RSS feed messages, chat messages, notification messages, IoT messages, blog posts, pingbacks, saved web pages, commands constructing complex shared documents, VR world update messages, code snippets, notes, and more. Email is a special case of a much larger world of messages. So, for me, a good assumption is that people will eventually have billions of message items comprising terabytes of data in their local (shared-with-email) store. Eventually each user would be part of a global federation of users that together have billions of times that number of messages as a federated semantic web. I understand that such a vision of messaging on that scale is not yet the goal of most people in this discussion right now. But I'd still urge people to think a little beyond redoing Thunderbird as-it-is and try to imagine a personal messaging system that processes, archives, indexes, and recalls vast numbers of items every day in interaction with a user and automated systems as part of a larger federation of similar systems using shared semantic standards." I've been beating my head against that wall for years with very limited progress (as have others) -- so I can't say it isn't a high risk endeavor with a low probability of success. But the benefit for humanity could potentially be huge if such an effort succeeded. Maybe it might just take a few tweaks to JMAP to generalize it enough to realize some of that big picture?
  - brongondwana 9y ago
    JMAP is pretty generalizable to other datatypes! It's a large part of why we split it into Core and Mail. We also have Calendar and Contacts modules in a reasonable state already. We also use JMAP-style APIs for Files, for Notes, and for all the extra Preferences and group management stuff in both FastMail and Topicbox.

lloeki 9y ago

> it's like a version control commit. As an alternative to mbox/maildir/whatevs I've been toying with a prototype of email storage that works in a content-addressed way with objects very much like how git operates internally. > Our Cyrus server already internally keeps a blobId which is a digest of the binary content of each MIME part. Which means you can easily tell that two attachments are identical without reading the contents or guessing based on size and name. This deduplication property then comes naturally from the above design, and not just for attachments but also for email headers and bodies. Many other things come easily such as rsync "just working" for fetching, or making use of git packs[0] [0]: https://git-scm.com/docs/pack-format https://git-scm.com/docs/pack-format

brongondwana 9y ago
You're not the first person to think of using git-like storage for email, and you won't be the last! Sadly deduplication isn't quite that easy - while the binary formats might be the same, MIME is very un-roundtrip-friendly when you start from the MIME structure. There are many ways to encode the same bytes, so if you need to reconstruct the raw bytes (FETCH RFC822) then you need a way to recover those original bytes. I've started work on that a few times, but it's never been worth the effort to make it stable.