4 ms·

I don't agree with you. Like with anything else, less abstraction = more performance. Creating a specific/targeted way to index, store and manage email data has

by ksk 10y ago

I don't agree with you. Like with anything else, less abstraction = more performance. Creating a specific/targeted way to index, store and manage email data has its benefits too. I'd hazard a guess that most proprietary formats are similar to the maildir format. Its just that the individual files are opqaue at the filesystem level. If you create thousands of tiny files then you are hoping that the file system design - which like any other design can only be optimized for specific workloads - can handle it better than a database type format specifically designed for email. Then again a generic full text search system (which probably won't even know that they're email files) over those thousands of files would have to be better than a targeted design for email data. You would miss certain optimizations like.. (making one up now) extracting and storing just the email headers in a separate region to speed up indexing and having some indirection built in to lookup that specific email.

gary_bernhardt 10y ago
There's nothing stopping a mail client from maintaining arbitrary indexes, caches, etc. in addition to the maildir.
- ksk 10y ago
  Sure, every time you deviate from a targeted design you leave performance on the table. Whats acceptable to you, may not be to others.
  - gary_bernhardt 10y ago
    Then keep exactly your preferred high-performance database design, but derive its data from maildir, such that all mail is persisted in maildir first.
    - ksk 10y ago
      Yes, that would be a good way to go.
  - 616c 10y ago
    Performance doing what? Email has several operations and none of these clients are performant at scale. You know why mutt can be for me? I have a second program (isync/mbsync) sync the maildirs, a third to index it, and mutt is just for reading and sorting. When you have one monolith doing all 3 at its will all the time, you get calls from users that Thunderbird and Mail.app are janky and the computer is hosed. It is interesting as FTS came to Thunderbird and they ran the indexer all the time, until they intelligently designed to have better controls around its execution as it destroyed general usability. Mail.app is better, but I see executive running it for days and it becomes a misbehaving pile of mess. I am asking in a respectful tone and not to make light of your position, but what do you use? I want to get a better look into your view and I think this might help me understand you better, if you are willing to indulge me. UPDATE: And when I say several things at once, ironically this is not by design! Email has changed a lot since the sendmail days (and that is cutting edge with respect to many veterans here) and thus there is well-known and accepted design I can think of beyond layers of addition on clients to meet definitions and changes in protocol, routinely made without any discussion of client operation beyond standards for file structures like mbox and Maildir, even they are formal standards, I would have to check,
    - ksk 10y ago
      >Performance doing what? Email has several operations and none of these clients are performant at scale. Well OK, so the most important for me is search, and the second most important is being able to quickly apply labels based on filters on a bulk-scale. I have found mutt to not really be performant and the workflow that I achieved was rather brittle (complicated by other factors like having to deal with HTML email). I'm currently using Gmail, but I am experimenting with outlook and I quite like the performance. I don't have a ton of email, maybe around 3-4GB, 25K emails. Bulk operations like 'select all' - do X on several thousand emails takes seconds (which is still slow IMHO FWIW). I haven't seen such performance with other clients. Granted, I haven't broken down every single client and benchmarked different workloads, so I'm only speaking in generalities. And don't worry about offending me. I was just trying to stir up some conversation. I don't take offense if someone disagrees with me on the internet.
      - 10y ago
- 616c 10y ago
  I am glad you made this point; I forgot to.
nine_k 10y ago
If a filesystem is a poor match for an email database, I'd appreciate the use of a well-established, open, and documented database format. That is, a format not only your unique email client can understand. Something with an independent manipulation tool. In the perfect case, something that can be interoperably used by several different email clients (not in parallel, of course) and / or third-party indexing / filtering / other processing tools. SQLite is used in many cases to store smaller things. It's likely not a good fit for a few gigabytes of email with binary attachments, though. I wonder what would be? NoSQL document-storing databases seem to have been created for a purpose like this. Is any of these used by any remotely popular client?
- ksk 10y ago
  >I wonder what would be? Well, maybe the people writing email clients have already thought quite a bit on this topic, likely created several prototypes, benchmarked them and chosen one that works the best given their constraints and features. Now all that's left is to open the format specification :)
- 616c 10y ago
  Again, shoving everything into one file is mbox, which is a recognized file format (if not database), and it does not help things that much. I have wondered the Sqlite thing myself beyond my one criticism. If anything, you would love K9 Mail on Android, which does this out of necessity. And some people have made SQL and noSQL email stores. http://nosql.mypopescu.com/post/3083070885/couchdb-as-email-store http://nosql.mypopescu.com/post/3083070885/couchdb-as-email-... https://www.npmjs.com/package/haraka-nosql https://www.npmjs.com/package/haraka-nosql http://www.dbmail.org/ http://www.dbmail.org/
616c 10y ago
As far as I can tell from playing with many of them: - Almost all (open source and proprietary) prefer mbox-style - If not mbox, then it is monolithic file per folder, very few if any store email per file like maildir, despite it having many benefits [0] (if you are smarter than Bernstein on such things, please do write on these things; no one gives a shit to explain email out because it is not sexy) - The index system always stores data separate of the email store; I have not seen a counter example, perhaps for Outlook, which I presume uses the Windows/OSX OS-level indexing - To get back to my point re generic FTS, there it is, keep the OS ones so people can search in one place (I use recoll and notmuch for email specifically; it's annoying) - What optimizations are there for indexing beyond header info? Again, good FTS systems can be extended (for what I define as good)[1] I do not mean to criticize your points, but I do support work and all of these clients do good for non-technical users, but are ticking times bombs that shit themslves and require resyncing all data when they corrupt massive multi-gig files, let alone re-indexing it all. Using them is one thing, having to fix them is what informs my opinions on this and why I sync to maildirs, use mutt-kz, and webmail depending on mood and time. [0] https://cr.yp.to/proto/maildir.html https://cr.yp.to/proto/maildir.html [1] https://bitbucket.org/medoc/recoll/wiki/IndexMailHeader https://bitbucket.org/medoc/recoll/wiki/IndexMailHeader apparently they have improved, perhaps I shall ditch notmuch out of notmuch discipline for email! Haha
- ksk 10y ago
  Well you're right that we can have a generic FTS system with a general OS based search with specific handlers for mailbox types. I guess you're trying to say that the file formats have become too complex where the clients themselves can't implement them bug-free. About that, perhaps another way to tackle that is not to go back to a more "primitive" format but to improve the quality of email clients. Personally, I never have IT problems, so I can't comment on the support burden. But I agree with you that if I had to re-prioritize, then reducing the support costs by trading performance and going back to a simpler format is probably worth it.

by ksk 10y ago

gary_bernhardt 10y ago

There's nothing stopping a mail client from maintaining arbitrary indexes, caches, etc. in addition to the maildir.

ksk 10y ago
Sure, every time you deviate from a targeted design you leave performance on the table. Whats acceptable to you, may not be to others.
- gary_bernhardt 10y ago
  Then keep exactly your preferred high-performance database design, but derive its data from maildir, such that all mail is persisted in maildir first.
  - ksk 10y ago
    Yes, that would be a good way to go.
- 616c 10y ago
  Performance doing what? Email has several operations and none of these clients are performant at scale. You know why mutt can be for me? I have a second program (isync/mbsync) sync the maildirs, a third to index it, and mutt is just for reading and sorting. When you have one monolith doing all 3 at its will all the time, you get calls from users that Thunderbird and Mail.app are janky and the computer is hosed. It is interesting as FTS came to Thunderbird and they ran the indexer all the time, until they intelligently designed to have better controls around its execution as it destroyed general usability. Mail.app is better, but I see executive running it for days and it becomes a misbehaving pile of mess. I am asking in a respectful tone and not to make light of your position, but what do you use? I want to get a better look into your view and I think this might help me understand you better, if you are willing to indulge me. UPDATE: And when I say several things at once, ironically this is not by design! Email has changed a lot since the sendmail days (and that is cutting edge with respect to many veterans here) and thus there is well-known and accepted design I can think of beyond layers of addition on clients to meet definitions and changes in protocol, routinely made without any discussion of client operation beyond standards for file structures like mbox and Maildir, even they are formal standards, I would have to check,
  - ksk 10y ago
    >Performance doing what? Email has several operations and none of these clients are performant at scale. Well OK, so the most important for me is search, and the second most important is being able to quickly apply labels based on filters on a bulk-scale. I have found mutt to not really be performant and the workflow that I achieved was rather brittle (complicated by other factors like having to deal with HTML email). I'm currently using Gmail, but I am experimenting with outlook and I quite like the performance. I don't have a ton of email, maybe around 3-4GB, 25K emails. Bulk operations like 'select all' - do X on several thousand emails takes seconds (which is still slow IMHO FWIW). I haven't seen such performance with other clients. Granted, I haven't broken down every single client and benchmarked different workloads, so I'm only speaking in generalities. And don't worry about offending me. I was just trying to stir up some conversation. I don't take offense if someone disagrees with me on the internet.
    - 10y ago
616c 10y ago
I am glad you made this point; I forgot to.

nine_k 10y ago

If a filesystem is a poor match for an email database, I'd appreciate the use of a well-established, open, and documented database format. That is, a format not only your unique email client can understand. Something with an independent manipulation tool. In the perfect case, something that can be interoperably used by several different email clients (not in parallel, of course) and / or third-party indexing / filtering / other processing tools. SQLite is used in many cases to store smaller things. It's likely not a good fit for a few gigabytes of email with binary attachments, though. I wonder what would be? NoSQL document-storing databases seem to have been created for a purpose like this. Is any of these used by any remotely popular client?

ksk 10y ago
>I wonder what would be? Well, maybe the people writing email clients have already thought quite a bit on this topic, likely created several prototypes, benchmarked them and chosen one that works the best given their constraints and features. Now all that's left is to open the format specification :)
616c 10y ago
Again, shoving everything into one file is mbox, which is a recognized file format (if not database), and it does not help things that much. I have wondered the Sqlite thing myself beyond my one criticism. If anything, you would love K9 Mail on Android, which does this out of necessity. And some people have made SQL and noSQL email stores. http://nosql.mypopescu.com/post/3083070885/couchdb-as-email-store http://nosql.mypopescu.com/post/3083070885/couchdb-as-email-... https://www.npmjs.com/package/haraka-nosql https://www.npmjs.com/package/haraka-nosql http://www.dbmail.org/ http://www.dbmail.org/

616c 10y ago

As far as I can tell from playing with many of them: - Almost all (open source and proprietary) prefer mbox-style - If not mbox, then it is monolithic file per folder, very few if any store email per file like maildir, despite it having many benefits [0] (if you are smarter than Bernstein on such things, please do write on these things; no one gives a shit to explain email out because it is not sexy) - The index system always stores data separate of the email store; I have not seen a counter example, perhaps for Outlook, which I presume uses the Windows/OSX OS-level indexing - To get back to my point re generic FTS, there it is, keep the OS ones so people can search in one place (I use recoll and notmuch for email specifically; it's annoying) - What optimizations are there for indexing beyond header info? Again, good FTS systems can be extended (for what I define as good)[1] I do not mean to criticize your points, but I do support work and all of these clients do good for non-technical users, but are ticking times bombs that shit themslves and require resyncing all data when they corrupt massive multi-gig files, let alone re-indexing it all. Using them is one thing, having to fix them is what informs my opinions on this and why I sync to maildirs, use mutt-kz, and webmail depending on mood and time. [0] https://cr.yp.to/proto/maildir.html https://cr.yp.to/proto/maildir.html [1] https://bitbucket.org/medoc/recoll/wiki/IndexMailHeader https://bitbucket.org/medoc/recoll/wiki/IndexMailHeader apparently they have improved, perhaps I shall ditch notmuch out of notmuch discipline for email! Haha

ksk 10y ago
Well you're right that we can have a generic FTS system with a general OS based search with specific handlers for mailbox types. I guess you're trying to say that the file formats have become too complex where the clients themselves can't implement them bug-free. About that, perhaps another way to tackle that is not to go back to a more "primitive" format but to improve the quality of email clients. Personally, I never have IT problems, so I can't comment on the support burden. But I agree with you that if I had to re-prioritize, then reducing the support costs by trading performance and going back to a simpler format is probably worth it.