3 ms·

There seems to be nothing about performance and how to deal with file count within a directory. Anyone who tried to naively store millions of files in a file s

There seems to be nothing about performance and how to deal with file count within a directory. Anyone who tried to naively store millions of files in a file system folder realizes at some point that listing files becomes horrible, there's no GUI tool that will handle that gracefully, and even on the CLI this is a very serious roadblock. It's still fine for accessing files straight by name, and there must be ways to read each file sequentially, but the concept of folder merely becomes an arbitrary namespace and not something to handle a whole group. The other obvious option is to shard the mail folders to ensure there's no more than X files in each folders, but that becomes pretty complex IMHO. At the end of the day, a database is needed somewhere down the line.

aidenn0 2y ago
> The other obvious option is to shard the mail folders to ensure there's no more than X files in each folders, but that becomes pretty complex IMHO. I mean it depends on the volume of e-mail you get, but: INBOX Archive/2004 Archive/2005 ... Archive/2023 Isn't something I would call complex, and works out to about 10k files per directory for me.
tomatocracy 2y ago
I've been pretty happy with Dovecot's mdbox format. This stores multiple emails per file and multiple files per mailbox which alleviates the problem of ending up with millions of files in one directory on filesystems which aren't tuned for this. Metadata is stored (only) in a separate index file which at first feels a bit "brittle", but this also alleviates a lot of the downsides of "indexed Maildir" type solutions (no more need to check for consistency between index and filesystem etc). Breaking with the idea that the entire email is stored in a single file (whether with other emails as with mbox or on its own as with Maildir etc) then also lets you do things like deduplicating attachments between emails (this was my primary motivation for exploring it). The main downside for me is that you can only really access the mailbox using dovecot/IMAP as tools like mutt don't support it.

aidenn0 2y ago

> The other obvious option is to shard the mail folders to ensure there's no more than X files in each folders, but that becomes pretty complex IMHO. I mean it depends on the volume of e-mail you get, but: INBOX Archive/2004 Archive/2005 ... Archive/2023 Isn't something I would call complex, and works out to about 10k files per directory for me.

tomatocracy 2y ago

I've been pretty happy with Dovecot's mdbox format. This stores multiple emails per file and multiple files per mailbox which alleviates the problem of ending up with millions of files in one directory on filesystems which aren't tuned for this. Metadata is stored (only) in a separate index file which at first feels a bit "brittle", but this also alleviates a lot of the downsides of "indexed Maildir" type solutions (no more need to check for consistency between index and filesystem etc). Breaking with the idea that the entire email is stored in a single file (whether with other emails as with mbox or on its own as with Maildir etc) then also lets you do things like deduplicating attachments between emails (this was my primary motivation for exploring it). The main downside for me is that you can only really access the mailbox using dovecot/IMAP as tools like mutt don't support it.