5 ms·

It's a somewhat interesting idea... I've had similar ideas in the past regarding maildir replacement without resorting to a db file. I like the idea of having

It's a somewhat interesting idea... I've had similar ideas in the past regarding maildir replacement without resorting to a db file. I like the idea of having directories representing email dir/folders, you generally will want some level of aggregation and/or search... I've thought that having separate eml (header + body) along with a .meta.json file for additional tagging/details (deleted flag, tags, etc). Search is a very different story, you wouldn't want to have to do a full directory scan for text based search. So some level of indexing would be useful for a client mail service. Similarly, I've thought it would be really cool if Cloudflare offered a TCP worker option, you could to a simple mail service backed by R2. The web ui/ux could be pretty awesome and geo distributed.

arp242 2y ago
> Search is a very different story, you wouldn't want to have to do a full directory scan for text based search. So some level of indexing would be useful for a client mail service. I don't know; my ~/code directory has tons of stuff and searching with ripgrep doesn't seem too slow: % time rg HelloWorld | wc -l 4 rg HelloWorld > /dev/null 0.13s user 0.12s system 99% cpu 0.251 total % time rg string | wc -l 57813 rg string > /dev/null 0.20s user 0.14s system 99% cpu 0.339 total Rough estimate of files that rg will search: % scc ─────────────────────────────────────────────────────────────────────────────── Language Files Lines Blanks Comments Code … Total 11024 1864982 175565 208777 1480640 ─────────────────────────────────────────────────────────────────────────────── Finding close to 60k matches in 11k files/1.7M lines in about 0.3 seconds isn't too bad. It should be said I ran a few commands on that directory before the above results, so there's probably some filesystem caching going on, but I can't be bothered to reboot. For many (not all, obviously) cases I think you may be able to get away without a index. Most people aren't subscribed to tons of email lists and get maybe a few emails a day at the most. I'd consider anything below ~3 seconds to be fine for search, so this scales to about 100k files/emails. At 10 emails/day on average that's about a decade. Most people do not get 10 emails/day on average. And you can even do some "poor man indexing" by just making a new directory every five or ten years. Most of the time you want just emails from the last year or so.
- Arelius 2y ago
  > Most people do not get 10 emails/day on average. I'd like to see the stats, but I seem to average around > 40 emails a day, (most are unactionable) but always considered my email load quite light. For people like my wife who do much of their work communication over email, it appears to be much higher.
  - deleted 2y ago
    [deleted]
  - rakoo 2y ago
    Ran some stats on my mails of the last 4 years, here are the daily characteristics: N = 24k Min = 1 Max = 211 Median = 11 Avg = 16.023907 Stddev = 18.312062 A lot of them are actually chat messages through DeltaChat so not representative of usual mail activity. When I remove them I get this: N = 16k Min = 1 Max = 56 Median = 10 Avg = 11.378933 Stddev = 8.1572529
    - Arelius 2y ago
      Sorry, I had meant by "stats" anything about the average among users, since I suspect that you might actually be an outlier on the lower-end among people who work professionally in or with technology.
- tracker1 2y ago
  I'm also considering a Server/Service that has a web ui component, where it's shared server resources... yeah, running a search on a local ssd/nvme is crazy fast... now do it when there are 100k other users on that filesystem.
- aidenn0 2y ago
  I get about 32 per day: notmuch count date:-100d..-1d 3248 And I have more than 100k emails: notmuch count 267584
rakoo 2y ago
> Search is a very different story, you wouldn't want to have to do a full directory scan for text based search. So some level of indexing would be useful for a client mail service. While notmuch and mu exist, I myself use the mblaze suite (https://github.com/leahneukirchen/mblaze https://github.com/leahneukirchen/mblaze) and it's more than enough for me. As a totally unscientific benchmark, it takes 300 ms to find 7 mails out of 24k when searching in headers, 4 seconds when searching in the body. I myself use a different way: I convert the entire (all 24k of them) list of emails to 1-lines with Sender, Subject, Date, Folder and feed it to fzf which gives me preview as well. The search is then instant; on the given fields only, but I never need more than that. This is my full MUA: https://sr.ht/~rakoo/omail/ https://sr.ht/~rakoo/omail/
mxuribe 2y ago
@tracker1 If i'm not mistaken i think thunderbird and other email clients who support conventional maildir often include a local db (such as sqlite) whose purpose tends to be mostly for helping indexing content to ease some aspects of search. That being said, as others have noted, search mostly tends to be fast enough at the filesystem level. ;-)
- tracker1 2y ago
  Not in an instance where the file system is remote, and used by thousands of users. Such as via S3/R2 or similar.
  - mxuribe 2y ago
    Good point! I was only thinking from the client side/perspective, and not server side. :-)
geek_at 2y ago
not 100% related but I have build OpenTrashmail [1] which gives you the emails in 3 variants. As folders on disk (no DB used), as RSS feed or as JSON feed. Which satisfied my needs for local management of emails [1] https://github.com/HaschekSolutions/opentrashmail https://github.com/HaschekSolutions/opentrashmail
zimpenfish 2y ago
> separate Then you start running into race conditions when updating / reading. The great thing about Maildir is that the updates can be done atomically - saves you a lot of locking and complexity.

arp242 2y ago

> Search is a very different story, you wouldn't want to have to do a full directory scan for text based search. So some level of indexing would be useful for a client mail service. I don't know; my ~/code directory has tons of stuff and searching with ripgrep doesn't seem too slow: % time rg HelloWorld | wc -l 4 rg HelloWorld > /dev/null 0.13s user 0.12s system 99% cpu 0.251 total % time rg string | wc -l 57813 rg string > /dev/null 0.20s user 0.14s system 99% cpu 0.339 total Rough estimate of files that rg will search: % scc ─────────────────────────────────────────────────────────────────────────────── Language Files Lines Blanks Comments Code … Total 11024 1864982 175565 208777 1480640 ─────────────────────────────────────────────────────────────────────────────── Finding close to 60k matches in 11k files/1.7M lines in about 0.3 seconds isn't too bad. It should be said I ran a few commands on that directory before the above results, so there's probably some filesystem caching going on, but I can't be bothered to reboot. For many (not all, obviously) cases I think you may be able to get away without a index. Most people aren't subscribed to tons of email lists and get maybe a few emails a day at the most. I'd consider anything below ~3 seconds to be fine for search, so this scales to about 100k files/emails. At 10 emails/day on average that's about a decade. Most people do not get 10 emails/day on average. And you can even do some "poor man indexing" by just making a new directory every five or ten years. Most of the time you want just emails from the last year or so.

Arelius 2y ago
> Most people do not get 10 emails/day on average. I'd like to see the stats, but I seem to average around > 40 emails a day, (most are unactionable) but always considered my email load quite light. For people like my wife who do much of their work communication over email, it appears to be much higher.
- deleted 2y ago
  [deleted]
- rakoo 2y ago
  Ran some stats on my mails of the last 4 years, here are the daily characteristics: N = 24k Min = 1 Max = 211 Median = 11 Avg = 16.023907 Stddev = 18.312062 A lot of them are actually chat messages through DeltaChat so not representative of usual mail activity. When I remove them I get this: N = 16k Min = 1 Max = 56 Median = 10 Avg = 11.378933 Stddev = 8.1572529
  - Arelius 2y ago
    Sorry, I had meant by "stats" anything about the average among users, since I suspect that you might actually be an outlier on the lower-end among people who work professionally in or with technology.
tracker1 2y ago
I'm also considering a Server/Service that has a web ui component, where it's shared server resources... yeah, running a search on a local ssd/nvme is crazy fast... now do it when there are 100k other users on that filesystem.
aidenn0 2y ago
I get about 32 per day: notmuch count date:-100d..-1d 3248 And I have more than 100k emails: notmuch count 267584

rakoo 2y ago

> Search is a very different story, you wouldn't want to have to do a full directory scan for text based search. So some level of indexing would be useful for a client mail service. While notmuch and mu exist, I myself use the mblaze suite (https://github.com/leahneukirchen/mblaze https://github.com/leahneukirchen/mblaze) and it's more than enough for me. As a totally unscientific benchmark, it takes 300 ms to find 7 mails out of 24k when searching in headers, 4 seconds when searching in the body. I myself use a different way: I convert the entire (all 24k of them) list of emails to 1-lines with Sender, Subject, Date, Folder and feed it to fzf which gives me preview as well. The search is then instant; on the given fields only, but I never need more than that. This is my full MUA: https://sr.ht/~rakoo/omail/ https://sr.ht/~rakoo/omail/

mxuribe 2y ago

@tracker1 If i'm not mistaken i think thunderbird and other email clients who support conventional maildir often include a local db (such as sqlite) whose purpose tends to be mostly for helping indexing content to ease some aspects of search. That being said, as others have noted, search mostly tends to be fast enough at the filesystem level. ;-)

tracker1 2y ago
Not in an instance where the file system is remote, and used by thousands of users. Such as via S3/R2 or similar.
- mxuribe 2y ago
  Good point! I was only thinking from the client side/perspective, and not server side. :-)

geek_at 2y ago

not 100% related but I have build OpenTrashmail [1] which gives you the emails in 3 variants. As folders on disk (no DB used), as RSS feed or as JSON feed. Which satisfied my needs for local management of emails [1] https://github.com/HaschekSolutions/opentrashmail https://github.com/HaschekSolutions/opentrashmail

zimpenfish 2y ago

> separate Then you start running into race conditions when updating / reading. The great thing about Maildir is that the updates can be done atomically - saves you a lot of locking and complexity.