5 ms·

Well, part of this is on us. Let's think on the reasons Gmail is so popular: 1. It's very easy to get to. 2. It has incredibly fast search that has 0 setup.

Well, part of this is on us. Let's think on the reasons Gmail is so popular: 1. It's very easy to get to. 2. It has incredibly fast search that has 0 setup. We have never really even tried to address problem 1 as an open source community. Networks, name lookup, and VPNs remain incredibly complex topics that beginners cannot hope to wrestle with. The best we have is .mdns which either works magically or perversely refuses to work. Similarly for free text search, the software world simply hasn't delivered a lego-like solution for email search. You CAN rig up any number of open source projects but it is neither easy nor instant. And even other professional products like Apple Mail struggle with a mere gigabyte of email. Despite the fact that it's 2021 and every successful email provider aggressively solves these problems, the open source world still debates about the utility of ubiquitous search or pretends that local networking isn't a pressing problem.

sumtechguy 6y ago
Another issue that needs to be solved is 'bad actor'. Basically someone gaming the system and overloading it with spam. There is no real nice neat way to fix that. Instead it is a mishmash of blacklists/whitelists/blocklists and sorta intelligent alg filtering. Getting that all filled and working is not trivial either. Oh its 'doable' but kind of a pain for even someone with decent experience at it.
- ywei3410 6y ago
  To add to that; with re-assigned IP's, widespread port 25 blocking /most/ ISP connections won't allow you to run an email server on a residential connection. Spam/botnets have a lot to answer for.
creeble 6y ago
Agree 100% on both of these, and #2 is seldom listed as a big issue. Had IMAP addressed search in a better way, I think it would have made a huge difference. Of course, it's not simple to set up a mail server that stays clean of spam RBLs, or that is Gmail-acceptable out the gate. But that's just the bar that got set as people went to Gmail because of the lack of other good alternatives. Question is, is Gmail hegemony fixable now?
- secabeen 6y ago
  Yeah, none of the IMAP server authors really thought of search as a first-class feature, nor did they seem to want to do the development to build the backend support. Resource constraints were also an issue back then, as well as a client-first approach. Most large mail servers only had enough resources to run the IMAP and POP services themselves; adding search index maintenance would have overloaded their CPUs. (Not that they couldn't have added more CPU, but this was all pre-cloud.)
- thayne 6y ago
  I think the best approach to breaking the Gmail hegemony is to replace email with something better. Some features I think such a system should have are: 1. End to end encryption 2. search as a first-class feature 3. decentralized/federated 4. standard minimal rich text format (maybe something like markdown), instead of an inconsistent subset of html 5. Fix some of the legacy limitations of email, like having to be ascii safe, line limit of 70 characters etc. 6. Possibly make it easier to have an identity that isn't tied to your service provider The big questions are how would you get the general population to switch from email, and what to do about spam.
  - anthk 6y ago
    1> GPG. 2> Your MUA does that. Get one that doesn't suck. 3> Impossible and/or dangerous. 4> No. HTML is NOT part of a MIME standard, even. 5> That's an issue of netiquette, often you have to post lines over 70 chars in News servers, too. 6> That's a different issue.
    - thayne 6y ago
      1. GPG has a lot of problems. Perhaps the biggest of which is that there isn't a critical mass of people who use it, and it is pretty difficult to us at all, much less to use correctly. 2. I have tried several MUAs and haven't found one that IMHO "doesn't suck." Although search isn't always the problem. (The biggest problem I've run into is actually getting them to work with logging in using an SSO method like SAML or OIDC). And this point was specifically in response to the parents statment that "Had IMAP addressed search in a better way, I think it would have made a huge difference." 3. Not sure what you mean by that since email _is_ decentralized and federated. Meaning, if you use gmail as your email provider you can still send messages to people using a different email provider. As opposed to say something like Signal or Whatsapp, where you can only message people using the same service as you. 4. Well, some subset of html has become the de facto standard for rich text in emails. Although what subset that is depends on the client... 6. Different issue from what? It is a very real complaint I have with email. Changing your email address is a pretty big pain, possibly more so than changing your physical address or phone number. For many people, myself included, the thought of having to change my email address is a pretty good motivation for staying with the service I currently use. Now it is possible to use your own domain with many email providers. But doing so is beyond the technical skills of most people, and costs more for essentially insurance in case you stop liking your current email provider some time in the future. Unfortunately, I don't know of a good solution for this. At least not one that is scalable.
teloli 6y ago
I do agree with you, but let's add: 3. dealing with spam is a massive PITA
- Avamander 6y ago
  Most of spam problems could be solved if mail software came with sane defaults. Sane rate-limits, sane IP reputation, sane security rules, some backscatter protection. Something. Most setups are so widely open because good configuration is arcane knowledge.
varjag 6y ago
I use offlineimap/mu4e over Exchange account at work, in parallel to a personal Gmail account. Would not say now that Gmail has a perceptible edge there. Seriously, mu4e search on modern hw is incredibly fast (and I have a 10 year work email archive).
throwaway201103 6y ago
For the search problem, notmuch is pretty easy and has better search capabilities than gmail. https://notmuchmail.org/ https://notmuchmail.org/
thayne 6y ago
3. A whole lot of free storage, so most people don't have to worry about needing to delete old emails.
jerf 6y ago
"We have never really even tried to address problem 1 as an open source community. Networks, name lookup, and VPNs remain incredibly complex topics that beginners cannot hope to wrestle with." I kinda disagree. It's probably easier than ever to set up your own mail server, in some abstract sense. You can get a virtual machine, use docker, heck, someone can hand you a complete image that you just have to bring up and set up with some config. The problem is, it literally doesn't matter how much the 'open source' community comes together, it simply can not provide a turn key solution as good as Desired email account: [________]@gmail.com Password: [________] Verify Password: [________] [X] I agree to have all my data used in arbitrary ways It's not possible. There is no way to set up a server that easily, even in principle. Or at least, not in a sane way. I can set up a site where you feed me your credit card number and pick a domain name, and I set up your AWS account for you, register your DNS name for you, configure DNS, and stand up everything you need and set it all up... but then we've got a split ownership interest. I can hand it all back to you, but you don't understand the setup. I can give you root on the system, but when you change anything, my automation stops working.
- ryukafalz 6y ago
  >It's not possible. There is no way to set up a server that easily, even in principle. I partially agree, but I think we could get a lot closer than we are now. It feels like the main reason this isn't possible is because you need to go through a registrar to get a DNS name, and that's tricky to do as part of a FOSS project. Maybe you could integrate with the APIs of a few registrars, but... it's not ideal. As far as the "run thing on server" side of it goes, though, projects like Sandstorm[0] have gotten really far re: making it a simple process. I stood up this instance of Etherpad with a few clicks on a web UI, for example: https://sandstorm.terracrypt.net/shared/aR2HXaoLSkLuXLhhAQono_89zWJkE7bHnyCYp5YoQRY https://sandstorm.terracrypt.net/shared/aR2HXaoLSkLuXLhhAQon... Sandstorm in particular doesn't quite work for mail servers just because the software is heavily oriented towards webapps, but there's no reason a similar system couldn't work in principle. [0] https://sandstorm.io/ https://sandstorm.io/
  - Avamander 6y ago
michaelmrose 6y ago
What about mu plus mu4e Thunderbird plus global search Recoll + client of choice https://www.monperrus.net/martin/recoll-maildir-thunderbird https://www.monperrus.net/martin/recoll-maildir-thunderbird

sumtechguy 6y ago

Another issue that needs to be solved is 'bad actor'. Basically someone gaming the system and overloading it with spam. There is no real nice neat way to fix that. Instead it is a mishmash of blacklists/whitelists/blocklists and sorta intelligent alg filtering. Getting that all filled and working is not trivial either. Oh its 'doable' but kind of a pain for even someone with decent experience at it.

ywei3410 6y ago
To add to that; with re-assigned IP's, widespread port 25 blocking /most/ ISP connections won't allow you to run an email server on a residential connection. Spam/botnets have a lot to answer for.

creeble 6y ago

Agree 100% on both of these, and #2 is seldom listed as a big issue. Had IMAP addressed search in a better way, I think it would have made a huge difference. Of course, it's not simple to set up a mail server that stays clean of spam RBLs, or that is Gmail-acceptable out the gate. But that's just the bar that got set as people went to Gmail because of the lack of other good alternatives. Question is, is Gmail hegemony fixable now?

secabeen 6y ago
Yeah, none of the IMAP server authors really thought of search as a first-class feature, nor did they seem to want to do the development to build the backend support. Resource constraints were also an issue back then, as well as a client-first approach. Most large mail servers only had enough resources to run the IMAP and POP services themselves; adding search index maintenance would have overloaded their CPUs. (Not that they couldn't have added more CPU, but this was all pre-cloud.)
thayne 6y ago
I think the best approach to breaking the Gmail hegemony is to replace email with something better. Some features I think such a system should have are: 1. End to end encryption 2. search as a first-class feature 3. decentralized/federated 4. standard minimal rich text format (maybe something like markdown), instead of an inconsistent subset of html 5. Fix some of the legacy limitations of email, like having to be ascii safe, line limit of 70 characters etc. 6. Possibly make it easier to have an identity that isn't tied to your service provider The big questions are how would you get the general population to switch from email, and what to do about spam.
- anthk 6y ago
  1> GPG. 2> Your MUA does that. Get one that doesn't suck. 3> Impossible and/or dangerous. 4> No. HTML is NOT part of a MIME standard, even. 5> That's an issue of netiquette, often you have to post lines over 70 chars in News servers, too. 6> That's a different issue.
  - thayne 6y ago
    1. GPG has a lot of problems. Perhaps the biggest of which is that there isn't a critical mass of people who use it, and it is pretty difficult to us at all, much less to use correctly. 2. I have tried several MUAs and haven't found one that IMHO "doesn't suck." Although search isn't always the problem. (The biggest problem I've run into is actually getting them to work with logging in using an SSO method like SAML or OIDC). And this point was specifically in response to the parents statment that "Had IMAP addressed search in a better way, I think it would have made a huge difference." 3. Not sure what you mean by that since email _is_ decentralized and federated. Meaning, if you use gmail as your email provider you can still send messages to people using a different email provider. As opposed to say something like Signal or Whatsapp, where you can only message people using the same service as you. 4. Well, some subset of html has become the de facto standard for rich text in emails. Although what subset that is depends on the client... 6. Different issue from what? It is a very real complaint I have with email. Changing your email address is a pretty big pain, possibly more so than changing your physical address or phone number. For many people, myself included, the thought of having to change my email address is a pretty good motivation for staying with the service I currently use. Now it is possible to use your own domain with many email providers. But doing so is beyond the technical skills of most people, and costs more for essentially insurance in case you stop liking your current email provider some time in the future. Unfortunately, I don't know of a good solution for this. At least not one that is scalable.

teloli 6y ago

I do agree with you, but let's add: 3. dealing with spam is a massive PITA

Avamander 6y ago
Most of spam problems could be solved if mail software came with sane defaults. Sane rate-limits, sane IP reputation, sane security rules, some backscatter protection. Something. Most setups are so widely open because good configuration is arcane knowledge.

varjag 6y ago

I use offlineimap/mu4e over Exchange account at work, in parallel to a personal Gmail account. Would not say now that Gmail has a perceptible edge there. Seriously, mu4e search on modern hw is incredibly fast (and I have a 10 year work email archive).

throwaway201103 6y ago

For the search problem, notmuch is pretty easy and has better search capabilities than gmail. https://notmuchmail.org/ https://notmuchmail.org/

thayne 6y ago

3. A whole lot of free storage, so most people don't have to worry about needing to delete old emails.

jerf 6y ago

"We have never really even tried to address problem 1 as an open source community. Networks, name lookup, and VPNs remain incredibly complex topics that beginners cannot hope to wrestle with." I kinda disagree. It's probably easier than ever to set up your own mail server, in some abstract sense. You can get a virtual machine, use docker, heck, someone can hand you a complete image that you just have to bring up and set up with some config. The problem is, it literally doesn't matter how much the 'open source' community comes together, it simply can not provide a turn key solution as good as Desired email account: [________]@gmail.com Password: [________] Verify Password: [________] [X] I agree to have all my data used in arbitrary ways It's not possible. There is no way to set up a server that easily, even in principle. Or at least, not in a sane way. I can set up a site where you feed me your credit card number and pick a domain name, and I set up your AWS account for you, register your DNS name for you, configure DNS, and stand up everything you need and set it all up... but then we've got a split ownership interest. I can hand it all back to you, but you don't understand the setup. I can give you root on the system, but when you change anything, my automation stops working.

ryukafalz 6y ago
>It's not possible. There is no way to set up a server that easily, even in principle. I partially agree, but I think we could get a lot closer than we are now. It feels like the main reason this isn't possible is because you need to go through a registrar to get a DNS name, and that's tricky to do as part of a FOSS project. Maybe you could integrate with the APIs of a few registrars, but... it's not ideal. As far as the "run thing on server" side of it goes, though, projects like Sandstorm[0] have gotten really far re: making it a simple process. I stood up this instance of Etherpad with a few clicks on a web UI, for example: https://sandstorm.terracrypt.net/shared/aR2HXaoLSkLuXLhhAQono_89zWJkE7bHnyCYp5YoQRY https://sandstorm.terracrypt.net/shared/aR2HXaoLSkLuXLhhAQon... Sandstorm in particular doesn't quite work for mail servers just because the software is heavily oriented towards webapps, but there's no reason a similar system couldn't work in principle. [0] https://sandstorm.io/ https://sandstorm.io/
- Avamander 6y ago

michaelmrose 6y ago

What about mu plus mu4e Thunderbird plus global search Recoll + client of choice https://www.monperrus.net/martin/recoll-maildir-thunderbird https://www.monperrus.net/martin/recoll-maildir-thunderbird