5 ms·

Make thunderbird supports a local database with 100k emails with proper search ! Make us pay for that optimization if needed. Email is a big tool of communicati

Make thunderbird supports a local database with 100k emails with proper search ! Make us pay for that optimization if needed. Email is a big tool of communication for all businesses, Pros who make money daily through emails need to handle tons of emails, we’re ready to pay for that

Avamander 2y ago
I mentioned the same problem in one other subthread as well. Current hardware is certainly performant enough not to become this sluggish at just 100 000 or so emails. There's actually no reason it shouldn't work well with say a million emails in one inbox.
- xnx 2y ago
  I haven't used Thunderbird in a long time, but regularly used Outlook with multi-gigabye .pst files. Surely sqlite on an SSD would be up to the task of handling at least million emails of average size.
  - mmooss 2y ago
    > Outlook with multi-gigabye .pst files What has been your experience? Mine in trying to use and support it is that Outlook is an Exchange client; PSTs are hacks to meet demand, though they work well enough in limited circumstances. Especially PSTs over a LAN connection are a disaster.
    - xnx 2y ago
      The Exchange server hardware was so underpowered (or the software so ill-designed for large mailboxes) that Exchange powered searches would fail, but ones run on the local pst would complete successfully (if slowly). This was on an HDD. SSD would be much faster.
      - mmooss 2y ago
        > This was on an HDD. SSD would be much faster. OT but is that right? SSDs have many advantages but sequential read isn't necessarily one of them. SSDs seek is much faster, but this is ~one file. Throughput can be much faster due to the better interfaces, but is throughput the bottleneck for this kind of search?
        xnx 2y ago
        Good question. Property benchmarking would be required to know for sure. It's probably rare that a multi-gigabyte file would be contiguous on disk, so lots of seeking would probably be required anyway.
        toast0 2y ago
        SSDs are usually better at sequential read as well as seeks. Depending on how outlook organizes the file (and how it gets organized in the file system) there's probably a mix of seeking and sequential reads anyway.
  - Spooky23 2y ago
    Apple Mail.app is 10x better and right there.
    - DrillShopper 2y ago
      I'm primarily a Linux user, but Mail.app is probably the best graphical email client I've ever had the pleasure of using (you can pry mutt from my cold, dead hands). Yes, I know it's also an GNUStep application
  - Merad 2y ago
    I did an internship in IT 20 years ago where we were building/maintaining desktops and other general helpdesk type stuff. I'm pretty sure I remember us having a handful of users with multi-gig pst files, running on 2005 hardware.
- kjkjadksj 2y ago
  Tell that to apple mail. Makes no sense how an app seemingly unchanged since the tiger days when I started using it could still be as performant as it always was on far better hardware. In fact I frequently find it to be the culprit when I wonder what the hell could be spinning my fans on this m3 pro just churning over the database. Iphone version is arguably worse because it also has performance issues but doesn’t support inbox rules. Then again those inbox rules often fail to filter emails anyhow.
  - milch 2y ago
    I just checked mine and I have about 250k emails sitting in my personal laptop's mailbox, no issues there. It might be dependent on the provider - I know at work where we use Exchange I get occasional slowdowns but I'm not sure whether that's due to Mail.app, due to Exchange, due to our dear endpoint security software taking its sweet time checking whatever I'm doing, or any combination of these. I probably have a couple million emails in my work laptop's mailbox though. Often the "Rebuild Mailbox" function fixes any problems for me, at least for a while. The iPhone one regularly just doesn't search properly for me though. I'll search for the exact subject or contents of a message and just won't be able to find it, then when I go to my laptop and type in the exact same terms it finds it instantly.
isaachinman 2y ago
We're building what you want. https://marcoapp.io https://marcoapp.io
- Avamander 2y ago
  Your link does not work?
  - isaachinman 2y ago
    Apologies, on mobile. Fixed.
- mmooss 2y ago
  I'd love it - email could use serious tools and refinement - but so many questions: Is it local or hosted? What is the story with privacy? Do you use an existing application (like a Thunderbird fork) or something you created? Can you / will you integrate other messaging such as SMS, even WhatsApp, etc.? RSS?
  - isaachinman 2y ago
    Great questions. 1. It is _both_ local and hosted. The client itself is fully offline-capable, including proper full-text search (single digit ms), writing drafts – anything you would expect an email client to do. The "hosted" bit is to ensure rapid synchronisation across multiple clients (ie your desktop and mobile). 2. Some metadata is hosted in pg to facilitate cross-platform synchronisation, as mentioned. This is encrypted at rest on a provider with SOC 2 Type I certification. Further symmetric encryption (AES-256) of sensitive columns is also done. We're well aware that security is the most important aspect of this product and is our primary focus. 3. We've not forked Thunderbird. Marco has been built from the ground up, both on the FE and BE, and has been a monumental task. 4. We have no immediate plans to add SMS/WhatsApp/RSS. If those interest you, you might have a look at Missive. We understand that storing email metadata is potentially a turn-off to some, but is actually the key driver to an entirely new email experience. It means that a Marco client itself is virtually stateless (save for some lightweight metadata) and syncs instantly across N number of clients – it runs on web/OSX/Windows/Android/etc, and changes propagate between them instantly. New client setup happens via Marco in a proprietary way on the order of seconds and doesn't take hours to sync via IMAP. We're building this for ourselves. Thunderbird is "alright". Apple Mail is "alright". Superhuman is decent, but ridiculously expensive and Google/Microsoft only. Missive is fairly decent (and also stores metadata), but is built for team collaboration, not individual use.
Koffiepoeder 2y ago
I have a (non-published) plugin that I'm using that is capable of using elasticsearch for indexing & search from within Thunderbird. I never bothered publishing it, since I never really wanted to maintain it/build a business out of it. Would this be something you are interested in, potentially for a small fee?
- ttoinou 2y ago
  It might not work for me for various reasons, but pay a fee to release the source code in the wild for anyone (me or others) to pick it up, yes why not ! Safer if you put a way to contact you on your profile
  - Koffiepoeder 2y ago
    Updated contact details ;)
- xiconfjs 2y ago
  Yes please, would love to try it out!
  - Koffiepoeder 2y ago
    I'll see what I can cobble together somewhere in the course of the weekend/next week ;) Probably need to file off some rough edges and hardcoded stuff.
- packetlost 2y ago
  Elasticsearch is a pretty bad choice for what really needs to be an embedded database. There are other FTS engines out there of varying quality that would be better suited to this particular case. Off the top of my head, Meillisearch and sqlite3's FTS5 would both be highly embed-able but have other tradeoffs (such as storage overhead).
  - Koffiepoeder 2y ago
    I do agree that it is a bad choice, though the reason for this is historical; after quite some testing and comparisons, the best JS-based index I could find at the time was elasticlunr [0]. It may very well be the case that there are better wasm alternatives available currently. Over time I then added a non-JS, external index. Since I already had an ES cluster running elsewhere anyway and the querying of elasticlunr and ElasticSearch is forwards compatible, I decided to just opt for re-using my ES cluster. In short, the decisionmaking was a mix of historical and compatibility/ease-of-maintenance reasons. [0]: http://elasticlunr.com/ http://elasticlunr.com/
RachelF 2y ago
Use Betterbird. They upgrade Thunderbird and fix bugs.
miles 2y ago
> Make thunderbird supports a local database with 100k emails with proper search Currently working with a Thunderbird database which contains over 300,000 messages and search works quite reliably (once in a blue moon have to switch from "Search Messages..." to "Global Search"), though the emails are stored in Maildir format rather than the default mbox: https://tinyapps.org/blog/202207100700_thunderbird_mbox_to_maildir.html https://tinyapps.org/blog/202207100700_thunderbird_mbox_to_m... .
- accoil 2y ago
  How decent is the maildir support? I looked into it a few years ago, and it seemed to still be experimental. My goal was to have other mail clients use the same maildir store, but I didn't feel like it would work at the time.
- meonkeys 2y ago
  Woah, how? My global search indexer seems to poop out when the sqlite db gets too big or something. https://www.reddit.com/r/Thunderbird/comments/1jjzb6b/global_search_db_size_or_mailbox_size_limit/ https://www.reddit.com/r/Thunderbird/comments/1jjzb6b/global... On another note, Thunderbird feels quite snappy for me. Fast and responsive, especially global search. Even so I think I prefer Recoll now that I've got it working. That thing is amazing.
- nh2 2y ago
  I have ~100k mails in Thunderbird. The GUI always feels a bit sluggish: Sometimes spinners don't spin, reactions to clicks take ~500 ms, when I switch from Inbox to Calendar for the first time, I can see how the buttons in the top row render one after the other in ~100ms. (I don't think a human should _ever_ see buttons render!) Sliding around the size of panels renders at 10 FPS, not so cool. Opening "Account settings" first produces a full white-flash, then a grey-flash (dark mode), and then renders the UI element. Startup takes ~5 seconds till the GUI fully shows. Then it hangs at "Opening folder INBOX..." for 60 seconds. Not sure why that sync takes so long when there are no new emails. So it works acceptably but doesn't feel great. Searching for e.g. "horse" in the Ctrl+K global search and hitting ender takes 5 seconds for full-text search to produce results. I think that part is OK. I mostly use the "Quick Filter" == "Filter messages" == Shift+Ctrl+K instead to search only subjects and correspondents. I have ~100 IMAP folders (from +suffix emails). Unfortunately Thunderbird doesn't notice when a new folder gets spawned by a new +suffix email, I have to restart it Thunderbird to ever get to see that email. RAM usage is 900 MB RES on Linux. (I could not check if that's glibc's fault as so often, because Thunderbird crashes when jemalloc is preloaded.) When I move the mouse cursor around anywhere in the GUI, that causes 90% CPU usage. For comparison, in Sublime Text, moving the mouse cursor around causes 10% CPU usage over the text buffer and 25% over tabs.
plzdotheneedful 2y ago
Why tackle problems like search when you can redesign the UI/UX half a dozen times instead? I don't blame the developers; they do whatever they're paid to do. Mozilla has terrible leadership and no vision. It's the worst aspects of directionless, corporate software masquerading as an open source project.
deleted 2y ago
[deleted]

Avamander 2y ago

I mentioned the same problem in one other subthread as well. Current hardware is certainly performant enough not to become this sluggish at just 100 000 or so emails. There's actually no reason it shouldn't work well with say a million emails in one inbox.

xnx 2y ago
I haven't used Thunderbird in a long time, but regularly used Outlook with multi-gigabye .pst files. Surely sqlite on an SSD would be up to the task of handling at least million emails of average size.
- mmooss 2y ago
  > Outlook with multi-gigabye .pst files What has been your experience? Mine in trying to use and support it is that Outlook is an Exchange client; PSTs are hacks to meet demand, though they work well enough in limited circumstances. Especially PSTs over a LAN connection are a disaster.
  - xnx 2y ago
    The Exchange server hardware was so underpowered (or the software so ill-designed for large mailboxes) that Exchange powered searches would fail, but ones run on the local pst would complete successfully (if slowly). This was on an HDD. SSD would be much faster.
    - mmooss 2y ago
      > This was on an HDD. SSD would be much faster. OT but is that right? SSDs have many advantages but sequential read isn't necessarily one of them. SSDs seek is much faster, but this is ~one file. Throughput can be much faster due to the better interfaces, but is throughput the bottleneck for this kind of search?
      - xnx 2y ago
        Good question. Property benchmarking would be required to know for sure. It's probably rare that a multi-gigabyte file would be contiguous on disk, so lots of seeking would probably be required anyway.
      - toast0 2y ago
        SSDs are usually better at sequential read as well as seeks. Depending on how outlook organizes the file (and how it gets organized in the file system) there's probably a mix of seeking and sequential reads anyway.
- Spooky23 2y ago
  Apple Mail.app is 10x better and right there.
  - DrillShopper 2y ago
    I'm primarily a Linux user, but Mail.app is probably the best graphical email client I've ever had the pleasure of using (you can pry mutt from my cold, dead hands). Yes, I know it's also an GNUStep application
- Merad 2y ago
  I did an internship in IT 20 years ago where we were building/maintaining desktops and other general helpdesk type stuff. I'm pretty sure I remember us having a handful of users with multi-gig pst files, running on 2005 hardware.
kjkjadksj 2y ago
Tell that to apple mail. Makes no sense how an app seemingly unchanged since the tiger days when I started using it could still be as performant as it always was on far better hardware. In fact I frequently find it to be the culprit when I wonder what the hell could be spinning my fans on this m3 pro just churning over the database. Iphone version is arguably worse because it also has performance issues but doesn’t support inbox rules. Then again those inbox rules often fail to filter emails anyhow.
- milch 2y ago
  I just checked mine and I have about 250k emails sitting in my personal laptop's mailbox, no issues there. It might be dependent on the provider - I know at work where we use Exchange I get occasional slowdowns but I'm not sure whether that's due to Mail.app, due to Exchange, due to our dear endpoint security software taking its sweet time checking whatever I'm doing, or any combination of these. I probably have a couple million emails in my work laptop's mailbox though. Often the "Rebuild Mailbox" function fixes any problems for me, at least for a while. The iPhone one regularly just doesn't search properly for me though. I'll search for the exact subject or contents of a message and just won't be able to find it, then when I go to my laptop and type in the exact same terms it finds it instantly.

isaachinman 2y ago

We're building what you want. https://marcoapp.io https://marcoapp.io

Avamander 2y ago
Your link does not work?
- isaachinman 2y ago
  Apologies, on mobile. Fixed.
mmooss 2y ago
I'd love it - email could use serious tools and refinement - but so many questions: Is it local or hosted? What is the story with privacy? Do you use an existing application (like a Thunderbird fork) or something you created? Can you / will you integrate other messaging such as SMS, even WhatsApp, etc.? RSS?
- isaachinman 2y ago
  Great questions. 1. It is _both_ local and hosted. The client itself is fully offline-capable, including proper full-text search (single digit ms), writing drafts – anything you would expect an email client to do. The "hosted" bit is to ensure rapid synchronisation across multiple clients (ie your desktop and mobile). 2. Some metadata is hosted in pg to facilitate cross-platform synchronisation, as mentioned. This is encrypted at rest on a provider with SOC 2 Type I certification. Further symmetric encryption (AES-256) of sensitive columns is also done. We're well aware that security is the most important aspect of this product and is our primary focus. 3. We've not forked Thunderbird. Marco has been built from the ground up, both on the FE and BE, and has been a monumental task. 4. We have no immediate plans to add SMS/WhatsApp/RSS. If those interest you, you might have a look at Missive. We understand that storing email metadata is potentially a turn-off to some, but is actually the key driver to an entirely new email experience. It means that a Marco client itself is virtually stateless (save for some lightweight metadata) and syncs instantly across N number of clients – it runs on web/OSX/Windows/Android/etc, and changes propagate between them instantly. New client setup happens via Marco in a proprietary way on the order of seconds and doesn't take hours to sync via IMAP. We're building this for ourselves. Thunderbird is "alright". Apple Mail is "alright". Superhuman is decent, but ridiculously expensive and Google/Microsoft only. Missive is fairly decent (and also stores metadata), but is built for team collaboration, not individual use.

Koffiepoeder 2y ago

I have a (non-published) plugin that I'm using that is capable of using elasticsearch for indexing & search from within Thunderbird. I never bothered publishing it, since I never really wanted to maintain it/build a business out of it. Would this be something you are interested in, potentially for a small fee?

ttoinou 2y ago
It might not work for me for various reasons, but pay a fee to release the source code in the wild for anyone (me or others) to pick it up, yes why not ! Safer if you put a way to contact you on your profile
- Koffiepoeder 2y ago
  Updated contact details ;)
xiconfjs 2y ago
Yes please, would love to try it out!
- Koffiepoeder 2y ago
  I'll see what I can cobble together somewhere in the course of the weekend/next week ;) Probably need to file off some rough edges and hardcoded stuff.
packetlost 2y ago
Elasticsearch is a pretty bad choice for what really needs to be an embedded database. There are other FTS engines out there of varying quality that would be better suited to this particular case. Off the top of my head, Meillisearch and sqlite3's FTS5 would both be highly embed-able but have other tradeoffs (such as storage overhead).
- Koffiepoeder 2y ago
  I do agree that it is a bad choice, though the reason for this is historical; after quite some testing and comparisons, the best JS-based index I could find at the time was elasticlunr [0]. It may very well be the case that there are better wasm alternatives available currently. Over time I then added a non-JS, external index. Since I already had an ES cluster running elsewhere anyway and the querying of elasticlunr and ElasticSearch is forwards compatible, I decided to just opt for re-using my ES cluster. In short, the decisionmaking was a mix of historical and compatibility/ease-of-maintenance reasons. [0]: http://elasticlunr.com/ http://elasticlunr.com/

RachelF 2y ago

Use Betterbird. They upgrade Thunderbird and fix bugs.

miles 2y ago

> Make thunderbird supports a local database with 100k emails with proper search Currently working with a Thunderbird database which contains over 300,000 messages and search works quite reliably (once in a blue moon have to switch from "Search Messages..." to "Global Search"), though the emails are stored in Maildir format rather than the default mbox: https://tinyapps.org/blog/202207100700_thunderbird_mbox_to_maildir.html https://tinyapps.org/blog/202207100700_thunderbird_mbox_to_m... .

accoil 2y ago
How decent is the maildir support? I looked into it a few years ago, and it seemed to still be experimental. My goal was to have other mail clients use the same maildir store, but I didn't feel like it would work at the time.
meonkeys 2y ago
Woah, how? My global search indexer seems to poop out when the sqlite db gets too big or something. https://www.reddit.com/r/Thunderbird/comments/1jjzb6b/global_search_db_size_or_mailbox_size_limit/ https://www.reddit.com/r/Thunderbird/comments/1jjzb6b/global... On another note, Thunderbird feels quite snappy for me. Fast and responsive, especially global search. Even so I think I prefer Recoll now that I've got it working. That thing is amazing.
nh2 2y ago
I have ~100k mails in Thunderbird. The GUI always feels a bit sluggish: Sometimes spinners don't spin, reactions to clicks take ~500 ms, when I switch from Inbox to Calendar for the first time, I can see how the buttons in the top row render one after the other in ~100ms. (I don't think a human should _ever_ see buttons render!) Sliding around the size of panels renders at 10 FPS, not so cool. Opening "Account settings" first produces a full white-flash, then a grey-flash (dark mode), and then renders the UI element. Startup takes ~5 seconds till the GUI fully shows. Then it hangs at "Opening folder INBOX..." for 60 seconds. Not sure why that sync takes so long when there are no new emails. So it works acceptably but doesn't feel great. Searching for e.g. "horse" in the Ctrl+K global search and hitting ender takes 5 seconds for full-text search to produce results. I think that part is OK. I mostly use the "Quick Filter" == "Filter messages" == Shift+Ctrl+K instead to search only subjects and correspondents. I have ~100 IMAP folders (from +suffix emails). Unfortunately Thunderbird doesn't notice when a new folder gets spawned by a new +suffix email, I have to restart it Thunderbird to ever get to see that email. RAM usage is 900 MB RES on Linux. (I could not check if that's glibc's fault as so often, because Thunderbird crashes when jemalloc is preloaded.) When I move the mouse cursor around anywhere in the GUI, that causes 90% CPU usage. For comparison, in Sublime Text, moving the mouse cursor around causes 10% CPU usage over the text buffer and 25% over tabs.

plzdotheneedful 2y ago

Why tackle problems like search when you can redesign the UI/UX half a dozen times instead? I don't blame the developers; they do whatever they're paid to do. Mozilla has terrible leadership and no vision. It's the worst aspects of directionless, corporate software masquerading as an open source project.

deleted 2y ago

[deleted]