4 ms·

The problem is that email has been taken over by a handful of providers, all of which have "growth & engagement" as their business model and would rather not im

The problem is that email has been taken over by a handful of providers, all of which have "growth & engagement" as their business model and would rather not implement any open protocol at all. IMAP remains supported for backwards-compatibility, but I'm sure its shortcomings are seen as a feature to push users towards the official clients so there is no incentive to implement another open protocol to address them.

stonogo 3y ago
This is a problem, but the other problem is the tools to implement the service. You can use Cyrus, Apache James, or Stalwart, and that's it. These tools do not integrate with anything; in the case of Stalwart (the only JMAP server not in "experimental" status) you can't just install a JMAP server attached to an existing email service; you have to hand it complete control of everything, even unto the on-disk storage of mail. This is a complete non-starter for any email service outside of hobby/personal hosting. In other words, even if someone wanted to make JMAP available to their users, they'd have to tear the whole thing down and rebuild it in JMAP's image, and the tools available to do so are not fully-featured or integratable enough to do so.
- StalwartLabs 3y ago
  > These tools do not integrate with anything; in the case of Stalwart (the only JMAP server not in "experimental" status) you can't just install a JMAP server attached to an existing email service; you have to hand it complete control of everything, even unto the on-disk storage of mail. This point you mention is being improved. The next release of Stalwart JMAP (expected in one or two months) will delegate user management to either a SQL database or an LDAP directory. Messages will be stored in either Maildir or MinIO/S3. And all other information will be in either SQLite or FoundationDB. All these features were already implemented except MinIO/S3. Development progress can be tracked at https://github.com/stalwartlabs/mail-server/tree/main/crates/jmap https://github.com/stalwartlabs/mail-server/tree/main/crates...
- josephg 3y ago
  Cyrus is what fastmail use for their own mail servers. And even the fastmail web ui uses jmap. I don’t know where it says jmap support is experimental, but I think it’s pretty stable and well supported in practice. The main maintainers of the project depend on it utterly.
  - stonogo 3y ago
    They don't use the word "experimental" any more but they call it a "work in progress": https://www.cyrusimap.org/imap/developer/jmap.html https://www.cyrusimap.org/imap/developer/jmap.html bottom of the page.
- Avamander 3y ago
  The file storage approach is unfortunately super common and makes a bunch of really basic things difficult and slow. Need to phase it out more heavily, especially from new implementations.
  - stonogo 3y ago
    Telling people who have decades of perfectly serviceable email solutions that they need to 'phase out' what works is a great example of an unconvincing argument. What you say might be true from a developer's perspective but maildir is a battle-tested, accessible, well-understood and interoperable storage format. Bundling everything up behind some bespoke SQL schema might ease performance hurdles but it sucks for everything else.
    - Avamander 3y ago
      Does it work though? Nobody runs such systems at scale, for very good reasons, because maildir just sucks in so many ways. Horses have also been battle-tested, they're well-understood, but they don't transport tonnes of cargo and getting kicked in your face because they don't like you isn't pleasant. So if it takes a custom schema in a (No)SQL database, so be it, it's simply so much more performant, scalable, reliable and usable. It's simply idiotic how much maildir-based software starts "dying" at around 100000 emails in a folder, in 2023!
nousermane 3y ago
Yep. If you go and register a new gmail account today, there will likely be no option to enable IMAP access for that account, altogether.
- jeffbee 3y ago
  Why would you post a baseless conspiracy theory that is easily disproven by anyone in less than a minute?
  - rudy6912 3y ago
    FYI, the above is true for "child accounts", as my son painfully discovered the other day, after his school shoved Gmail down his throat. No IMAP in the settings, or anywhere else, for the next generation of users, not until they are 16+ in my country.
    - jeffbee 3y ago
      What makes you think this is for youth accounts, and is not simply the preference of the workspace administrator for your school?
- IceWreck 3y ago
  Why would you need to enable IMAP ? It works out of the box. Every non-android platform uses IMAP to access your account (k9 mail, outlook, thunderbird, the iOS/mac email client, etc).
  - jeffbee 3y ago
    You do have to go into the gmail settings to enable IMAP, FWIW.
  - philipwhiuk 3y ago
    As a former K9 Mail developer I will say that the Google extensions to IMAP for Labels and OAUTH (neither of which are standards) are a pain in the backside that no email platform would have dealt with were it not for the fact they are huge. (I did a spike development for OAuth and a separate investigation into Labels when I had some free time). So they are definitely not nice IMAP citizens. Oh and then there's the issues with Labels that their own documentation apparently gets wrong...
ilyt 3y ago
Case in point: MS making it more and more annoying to use IMAP. Hell, now you need to create oauth2 app just to login from non-approved client, as they recently forced OAUTH2 auth on IMAP clients. Sure thunderbird works but most lesser known clients require a lot of fuckery to make work (and possibly admin permissions for domain, not sure).
- ggm 3y ago
  https://github.com/simonrob/email-oauth2-proxy https://github.com/simonrob/email-oauth2-proxy "Just works" I run it with mbsync at the command line. You do need a client id and some interaction with your O365 admin.
  - ilyt 3y ago
    This looks promising, thanks. I did the song and dance for claws-mail and one day it just... stopped working ;/
- jojobas 3y ago
  I would be migrating away (to anything) if my email provider pulled that on me.
  - ilyt 3y ago
    It's entirely idiotic history. We had our own e-mail, but we had some problems that were generally caused by management wanting @example1.com and @example2.com to be entirely equivalent, not just an alias, but still land on the same account. which was fine for the email, as it can just have custom rules, but any related services like calendars bugged, as it saw different domains as different users. We tried to hack around it but it was still buggy in places. Boss finally got pissed off and decided to migrate to O365, all while IT was screaming nooo. "Because our clients use it too, it will be easier to collaborate" ...and it turned out that what he wanted isn't possible in O365 *fucking anyway* so all e-mails are under @example1.com. And we replaced some weird quirks with many more quirks we can't even hope to fix. We waste more time dealing with this shit than we did on running our own mail servers. Which we didn't like, coz running mail servers sucks, but O365 sucks harder.
Avamander 3y ago
It really really hasn't been. Email is just an enormously massive ecosystem with tremendous amount of variance that anything new will take a long time to adopt and a lot of convincing and proving. Just look at how far the deployment of something as essential as DMARC is, it's sad. That's not some big vendor's fault, it's all the small legacy cruft nobody wants to clean out as it might break things.

stonogo 3y ago

This is a problem, but the other problem is the tools to implement the service. You can use Cyrus, Apache James, or Stalwart, and that's it. These tools do not integrate with anything; in the case of Stalwart (the only JMAP server not in "experimental" status) you can't just install a JMAP server attached to an existing email service; you have to hand it complete control of everything, even unto the on-disk storage of mail. This is a complete non-starter for any email service outside of hobby/personal hosting. In other words, even if someone wanted to make JMAP available to their users, they'd have to tear the whole thing down and rebuild it in JMAP's image, and the tools available to do so are not fully-featured or integratable enough to do so.

StalwartLabs 3y ago
> These tools do not integrate with anything; in the case of Stalwart (the only JMAP server not in "experimental" status) you can't just install a JMAP server attached to an existing email service; you have to hand it complete control of everything, even unto the on-disk storage of mail. This point you mention is being improved. The next release of Stalwart JMAP (expected in one or two months) will delegate user management to either a SQL database or an LDAP directory. Messages will be stored in either Maildir or MinIO/S3. And all other information will be in either SQLite or FoundationDB. All these features were already implemented except MinIO/S3. Development progress can be tracked at https://github.com/stalwartlabs/mail-server/tree/main/crates/jmap https://github.com/stalwartlabs/mail-server/tree/main/crates...
josephg 3y ago
Cyrus is what fastmail use for their own mail servers. And even the fastmail web ui uses jmap. I don’t know where it says jmap support is experimental, but I think it’s pretty stable and well supported in practice. The main maintainers of the project depend on it utterly.
- stonogo 3y ago
  They don't use the word "experimental" any more but they call it a "work in progress": https://www.cyrusimap.org/imap/developer/jmap.html https://www.cyrusimap.org/imap/developer/jmap.html bottom of the page.
Avamander 3y ago
The file storage approach is unfortunately super common and makes a bunch of really basic things difficult and slow. Need to phase it out more heavily, especially from new implementations.
- stonogo 3y ago
  Telling people who have decades of perfectly serviceable email solutions that they need to 'phase out' what works is a great example of an unconvincing argument. What you say might be true from a developer's perspective but maildir is a battle-tested, accessible, well-understood and interoperable storage format. Bundling everything up behind some bespoke SQL schema might ease performance hurdles but it sucks for everything else.
  - Avamander 3y ago
    Does it work though? Nobody runs such systems at scale, for very good reasons, because maildir just sucks in so many ways. Horses have also been battle-tested, they're well-understood, but they don't transport tonnes of cargo and getting kicked in your face because they don't like you isn't pleasant. So if it takes a custom schema in a (No)SQL database, so be it, it's simply so much more performant, scalable, reliable and usable. It's simply idiotic how much maildir-based software starts "dying" at around 100000 emails in a folder, in 2023!

nousermane 3y ago

Yep. If you go and register a new gmail account today, there will likely be no option to enable IMAP access for that account, altogether.

jeffbee 3y ago
Why would you post a baseless conspiracy theory that is easily disproven by anyone in less than a minute?
- rudy6912 3y ago
  FYI, the above is true for "child accounts", as my son painfully discovered the other day, after his school shoved Gmail down his throat. No IMAP in the settings, or anywhere else, for the next generation of users, not until they are 16+ in my country.
  - jeffbee 3y ago
    What makes you think this is for youth accounts, and is not simply the preference of the workspace administrator for your school?
IceWreck 3y ago
Why would you need to enable IMAP ? It works out of the box. Every non-android platform uses IMAP to access your account (k9 mail, outlook, thunderbird, the iOS/mac email client, etc).
- jeffbee 3y ago
  You do have to go into the gmail settings to enable IMAP, FWIW.
- philipwhiuk 3y ago
  As a former K9 Mail developer I will say that the Google extensions to IMAP for Labels and OAUTH (neither of which are standards) are a pain in the backside that no email platform would have dealt with were it not for the fact they are huge. (I did a spike development for OAuth and a separate investigation into Labels when I had some free time). So they are definitely not nice IMAP citizens. Oh and then there's the issues with Labels that their own documentation apparently gets wrong...

ilyt 3y ago

Case in point: MS making it more and more annoying to use IMAP. Hell, now you need to create oauth2 app just to login from non-approved client, as they recently forced OAUTH2 auth on IMAP clients. Sure thunderbird works but most lesser known clients require a lot of fuckery to make work (and possibly admin permissions for domain, not sure).

ggm 3y ago
https://github.com/simonrob/email-oauth2-proxy https://github.com/simonrob/email-oauth2-proxy "Just works" I run it with mbsync at the command line. You do need a client id and some interaction with your O365 admin.
- ilyt 3y ago
  This looks promising, thanks. I did the song and dance for claws-mail and one day it just... stopped working ;/
jojobas 3y ago
I would be migrating away (to anything) if my email provider pulled that on me.
- ilyt 3y ago
  It's entirely idiotic history. We had our own e-mail, but we had some problems that were generally caused by management wanting @example1.com and @example2.com to be entirely equivalent, not just an alias, but still land on the same account. which was fine for the email, as it can just have custom rules, but any related services like calendars bugged, as it saw different domains as different users. We tried to hack around it but it was still buggy in places. Boss finally got pissed off and decided to migrate to O365, all while IT was screaming nooo. "Because our clients use it too, it will be easier to collaborate" ...and it turned out that what he wanted isn't possible in O365 *fucking anyway* so all e-mails are under @example1.com. And we replaced some weird quirks with many more quirks we can't even hope to fix. We waste more time dealing with this shit than we did on running our own mail servers. Which we didn't like, coz running mail servers sucks, but O365 sucks harder.

Avamander 3y ago

It really really hasn't been. Email is just an enormously massive ecosystem with tremendous amount of variance that anything new will take a long time to adopt and a lot of convincing and proving. Just look at how far the deployment of something as essential as DMARC is, it's sad. That's not some big vendor's fault, it's all the small legacy cruft nobody wants to clean out as it might break things.