3 ms·

I was using the one-message-per-file approach, but keeping millions of small files in a filesystem is quite a challenge: I have to worry about available inodes,

by riobard 15y ago

I was using the one-message-per-file approach, but keeping millions of small files in a filesystem is quite a challenge: I have to worry about available inodes, and listing/manipulating these files takes minutes!

JoshTriplett 15y ago
On a relatively small ext4 filesystem here, I have tens of millions of inodes available, and I haven't done anything to tweak that. On a larger filesystem, with a slightly bumped number of inodes, I'd expect several hundred million inodes, more than enough to store all the messages that'll fit in the available space. As for listing and manipulating, I'd suggest not ever listing all the files. Access them by name only, which should require near-constant time no matter how many files you have. Manipulating the files should take similarly little time. On the occasions that you do need to list all the files, you might find https://www.olark.com/spw/2011/08/you-can-list-a-directory-with-8-million-files-but-not-with-ls/ https://www.olark.com/spw/2011/08/you-can-list-a-directory-w... interesting. A pile of RAM helps, to keep as many directory entries in the dcache as possible.
- mike-cardwell 15y ago
  Email can use up inodes quickly. A thousand users with a thousand emails each is a million files in Maildir. Plus three directories (cur/new/tmp) for each mail folder.
  - JoshTriplett 15y ago
    Folders seem unlikely to use up a significant number of inodes compared to the messages. As for messages, how many users do you expect to serve from each mail server? I think you'll hit the limits of whatever service you want to provide before you hit the limit of inodes in your mail store. (Or you could always encrypt each email and throw it at an object storage system similar to S3. ;) )
mike-cardwell 15y ago
I think the best system would be one inbetween mbox and maildir. You specify a filesize limit of say "10MB". Emails are repeatedly added to the same file, until it hits 10MB, then a new file is created. Emails larger than 10MB would not be split over multiple files. An index is kept of all of the messages. When a message is deleted, rather than immediately removing it from the file and blocking like most mbox implementations work, it should just be flagged as deleted in the index. A background process can clean up these messages when the file is not in use.
- ams6110 15y ago
  What you describe is essentially a database for email. You might have a look at http://www.dbmail.org/ http://www.dbmail.org/
  - mike-cardwell 15y ago
    This is true. Maildir and mbox are also essentially databases for email.
    - ams6110 15y ago
      OK. Maybe I should have said DBMS? I'm speaking specifically about the management of the back-end storage files, e.g. splitting storage over multiple size-limited files, tracking and re-using free space, etc. are all things that DBMSs provide, without a need to reinvent such things for the purposes of storing email. If there's a better term to describe this I'd be glad to learn it.
- JoshTriplett 15y ago
  Seems like in such a system you've gone to a lot of work to recreate the purpose of a filesystem: to keep the data of different files separate from each other. Why do you want to keep emails together in 10MB chunks?
  - mike-cardwell 15y ago
vidarh 15y ago
I did millions upon millions of small e-mails using Maildir in a filesystem in 2000 (accounts for 1.7 million users on a single filesystem). It's only a challenge if you pick a filesystem with pathological behaviour for directories with many files and/or with small and hard limits for number of inodes (in our case we used ReiserFS 3). The biggest problem is read bandwidth for your disks. Write IO is likely to be constrained by available network bandwidth unless you do something stupid. Read bandwidth constraints for e-mail are generally easily mitigated by creating an index (so you only need to open or stat the message files when opening individual messages, not to, say, list message sizes in a POP3 server or present an index view on a web interface). If you need to list/manipulate those files other than writing them once and reading them when a user actually opens that specific message, or deleting them when the user actually deletes the message, then you're doing something wrong (e.g. you didn't create a sufficiently flexible index).

by riobard 15y ago

JoshTriplett 15y ago

On a relatively small ext4 filesystem here, I have tens of millions of inodes available, and I haven't done anything to tweak that. On a larger filesystem, with a slightly bumped number of inodes, I'd expect several hundred million inodes, more than enough to store all the messages that'll fit in the available space. As for listing and manipulating, I'd suggest not ever listing all the files. Access them by name only, which should require near-constant time no matter how many files you have. Manipulating the files should take similarly little time. On the occasions that you do need to list all the files, you might find https://www.olark.com/spw/2011/08/you-can-list-a-directory-with-8-million-files-but-not-with-ls/ https://www.olark.com/spw/2011/08/you-can-list-a-directory-w... interesting. A pile of RAM helps, to keep as many directory entries in the dcache as possible.

mike-cardwell 15y ago
Email can use up inodes quickly. A thousand users with a thousand emails each is a million files in Maildir. Plus three directories (cur/new/tmp) for each mail folder.
- JoshTriplett 15y ago
  Folders seem unlikely to use up a significant number of inodes compared to the messages. As for messages, how many users do you expect to serve from each mail server? I think you'll hit the limits of whatever service you want to provide before you hit the limit of inodes in your mail store. (Or you could always encrypt each email and throw it at an object storage system similar to S3. ;) )

mike-cardwell 15y ago

I think the best system would be one inbetween mbox and maildir. You specify a filesize limit of say "10MB". Emails are repeatedly added to the same file, until it hits 10MB, then a new file is created. Emails larger than 10MB would not be split over multiple files. An index is kept of all of the messages. When a message is deleted, rather than immediately removing it from the file and blocking like most mbox implementations work, it should just be flagged as deleted in the index. A background process can clean up these messages when the file is not in use.

ams6110 15y ago
What you describe is essentially a database for email. You might have a look at http://www.dbmail.org/ http://www.dbmail.org/
- mike-cardwell 15y ago
  This is true. Maildir and mbox are also essentially databases for email.
  - ams6110 15y ago
    OK. Maybe I should have said DBMS? I'm speaking specifically about the management of the back-end storage files, e.g. splitting storage over multiple size-limited files, tracking and re-using free space, etc. are all things that DBMSs provide, without a need to reinvent such things for the purposes of storing email. If there's a better term to describe this I'd be glad to learn it.
JoshTriplett 15y ago
Seems like in such a system you've gone to a lot of work to recreate the purpose of a filesystem: to keep the data of different files separate from each other. Why do you want to keep emails together in 10MB chunks?
- mike-cardwell 15y ago

vidarh 15y ago

I did millions upon millions of small e-mails using Maildir in a filesystem in 2000 (accounts for 1.7 million users on a single filesystem). It's only a challenge if you pick a filesystem with pathological behaviour for directories with many files and/or with small and hard limits for number of inodes (in our case we used ReiserFS 3). The biggest problem is read bandwidth for your disks. Write IO is likely to be constrained by available network bandwidth unless you do something stupid. Read bandwidth constraints for e-mail are generally easily mitigated by creating an index (so you only need to open or stat the message files when opening individual messages, not to, say, list message sizes in a POP3 server or present an index view on a web interface). If you need to list/manipulate those files other than writing them once and reading them when a user actually opens that specific message, or deleting them when the user actually deletes the message, then you're doing something wrong (e.g. you didn't create a sufficiently flexible index).