5 ms·

A side question: what's the best way to build an email message store which stores millions of messages (and adding tens of thousands on average each day) on dis

• beza1e1 15y ago
  You could look at http://notmuchmail.org/ http://notmuchmail.org/ which builds on top of maildir.
• gwillen 15y ago
  If you only ever want to append messages, and never want to mutate or remove them, I would do this scheme in two parts: 1) an append-only message store; 2) a set of indices stored alongside (1). To append thousands of messages per second, I would probably want a system with multiple writers appending to multiple files (mbox) or directories (maildir), and cycling/archiving the mailboxes as they get large. As for (2), I know a lot less about indexing messages than I do about storing them, but I would try out an off the shelf SQL or noSQL system and measure the performance, before trying to build anything fancy.
• JoshTriplett 15y ago
  Maildir (or a very similar one-message-per-file store) plus an index of some kind. If not for the continuous addition of messages, I'd suggest notmuch (http://notmuchmail.org/ http://notmuchmail.org/), which does a great job of indexing in any way you could want. Unfortunately, as far as I know notmuch does not have an "online" indexing mechanism; someone wrote a notmuch-deliver that adds a message and indexes it, but I doubt it would meet your performance requirements. You could probably make notmuch do what you want, but you might want a new indexing mechanism instead. I still think you want something very similar to Maildir underneath though.
  • riobard 15y ago
    I was using the one-message-per-file approach, but keeping millions of small files in a filesystem is quite a challenge: I have to worry about available inodes, and listing/manipulating these files takes minutes!
    • JoshTriplett 15y ago
      On a relatively small ext4 filesystem here, I have tens of millions of inodes available, and I haven't done anything to tweak that. On a larger filesystem, with a slightly bumped number of inodes, I'd expect several hundred million inodes, more than enough to store all the messages that'll fit in the available space. As for listing and manipulating, I'd suggest not ever listing all the files. Access them by name only, which should require near-constant time no matter how many files you have. Manipulating the files should take similarly little time. On the occasions that you do need to list all the files, you might find https://www.olark.com/spw/2011/08/you-can-list-a-directory-with-8-million-files-but-not-with-ls/ https://www.olark.com/spw/2011/08/you-can-list-a-directory-w... interesting. A pile of RAM helps, to keep as many directory entries in the dcache as possible.
      • mike-cardwell 15y ago
        Email can use up inodes quickly. A thousand users with a thousand emails each is a million files in Maildir. Plus three directories (cur/new/tmp) for each mail folder.
        • JoshTriplett 15y ago
          Folders seem unlikely to use up a significant number of inodes compared to the messages. As for messages, how many users do you expect to serve from each mail server? I think you'll hit the limits of whatever service you want to provide before you hit the limit of inodes in your mail store. (Or you could always encrypt each email and throw it at an object storage system similar to S3. ;) )
    • mike-cardwell 15y ago
• djcb 15y ago
  There are some systems based on the Xapian full-text indexer, including my own 'mu'[1] which might be able to do this. It expects the messages in a Maildir, and it can handle > 150 messages per sec, on fairly slow hardware - so after a peak of thousands of new messages, it would take a few seconds to catch up. Tens-of-thousands per day would be no problem. There have been people using [1] for a few million of messages without any problem, but I'm not sure how well it scales beyond that. [1] http://www.djcbsoftware.nl/code/mu http://www.djcbsoftware.nl/code/mu, or 'maildir-utils' in debian/ubuntu