7 ms·

I run one of the largest email sending services on the internet. I have been living the “mess” of internet email for over 20 years. Here’s the thing: despite t

by ttul 2y ago

I run one of the largest email sending services on the internet. I have been living the “mess” of internet email for over 20 years. Here’s the thing: despite the internet’s email system being complex and confusing and riddled with problems, it is universally adopted and interoperable. No other open communication tool can boast of email’s massive interoperability. Any new system will face the impossible burden of winning hearts and minds while the old system continues to chug along with billions of annual R&D spending and dozens of conferences full of smart people working on solving problems. Even if “MX2” can peacefully coexist in the DNS, why would anyone spend the millions of dollars in engineering effort to move while their teams are busy building the latest layer that’s been invented to patch over the current system? By all means if you want to propose a new email system, show up at IETF or M3AAWG and make a bold proposal. Someone will buy you a beer while they explain why you are much better off getting into the mud pit with the rest of us and working on the next pragmatic fix to keep things rolling along.

gjsman-1000 2y ago
https://xkcd.com/2347/ https://xkcd.com/2347/
lxgr 2y ago
I think it all depends on the right pitch: As a replacement for current email only, you're probably right in that nobody would care enough to do the significant investments necessary in a reasonable timeframe, similarly to IPv6 (and the pain there is arguably even greater than for email). As an alternative to something with the semantics of what companies currently use SMS for (i.e. having ostensibly higher reliability and security), I think you'd see a lot of interest.
deskr 2y ago
> universally adopted and interoperable No. Big email server all over the place might drop your emails for whatever reason is and they won't tell you.
- 1vuio0pswjnm7 2y ago
  Question is whether a new or updated protocol could force users such as "email providers" to change their behaviour. Here is something to downvote, a series of questions: What if email were "pickup only" _from the sender_. Not from some intermediary recipient, e.g., an "email provider". What if the the recipient had to identify acceptable senders before they could "send" mail to the recipient. Arguably this already happens every time an email recpient gives out their email address to some email sender. What if the sender did not "send" mail but instead uploaded it to host run by the sender and accessible by the recipient. Then the recipient retrieves the mail from the sender. In the past, one large email provider had an RSS feed for a user's inbox. What if the RSS feed is not provided by some third party "email provider" but by _senders_. What if the feed indicates whether there is new mail waiting to be retrieved by the recipient. Arguably, something like this already happens, albeit using third party intermediaries, for example as millions of people use non-public webpages on third party websites to communicate with each other, instead of using email. Recipients check these pages for comments or messages from "senders". A simpler idea that requires no changes to any email protocol, which I have tested successfully on home network, is for sender and recipient to be on a peer-to-peer overlay and run their own SMTP servers, like the original internet. The sender SMTP server communicates directly with the receiver's SMTP server, not a third party SMTP server run by an "email provider". Obviously, sender and receiver should not invite anyone onto this network who they do not know and from which they do not want to receive email. The fundamental problem with email is that personal, non-commercial email is mixed with commercial email, mail that is selling something. That's beneficial to marketers, but not email users. Any change to email that threatens to exclude the unsolicited, commercial email will be opposed fervently.
  - deleted 2y ago
    [deleted]
  - donut 2y ago
    The pickup idea sounds similar to how newsgroups work.
    - 2y ago
kilburn 2y ago
Also, they would school them on actual-world problems in the process: - You can't wait until you receive the entire body to be able to compute a signature to then validate the sender as your first line of defense. It is just too expensive and opens you to DDOS attacks. People use IP reputation as a first line of defense because it is cheap, not because it is good. - You cannot enforce people's behavior through RFCs. I can assure you that random guy next desk will not care about your "this is a top-posting-thread" header and bottom post there. Even if she has to manually copy/paste things around. - Likewise, auto-generated plain-text versions of HTML (or other rich-text formats) are no better than what screen readers can achieve. Most poeple won't bother writing that alternate version, meaning the obligatory alt text is now less useful than when it was optional and only people who cared inculded it. - Your largest client may not update their e-mail infrastructure to comply with the latest standards. If that happens, you don't tell them to update or otherwise you won't be answering them because their e-mails go to spam. You do whatever is necessary to ensure that their e-mails don't go to spam. Business always comes first.
- gjsman-1000 2y ago
  Hypothetically, while I’m no expert: 1. Could a future protocol require an immediate initial message (a “hello”) stating exactly how much content will be sent, and until the “hello” is sent, it’s limited to, say, 128KB before the connection is immediately terminated? (And of course, if the content exceeds the declaration, termination and immediate IP temporary ban, safe to do as this is an obvious violation of a new spec?) 2. The goal is to make it easier for the email client which by itself will encourage good behavior. There’s also no requirement for the messages to all be in one massive blob. 3. The goal is that it would be automatically created by the client. For personal emails, this is easy. For enhanced HTML emails, that is where the requirement comes in. Email providers can come up with their own ways of enforcement from there (I.e. “if it’s only one sentence, you obviously didn’t do it”), though I get your point and that would become messy unofficial spec again. 4. Could a future emails system have versioning, allowing the server to clearly communicate (“Hello, I implement MX2 v3.1.”)? In addition, a business can obviously make their own settings that original email alerts do not go to Junk in their business mailboxes - but they do know they’d better get on it or their messages to clients might go to Junk.
gerdesj 2y ago
I run several of the smallest email sending services on the internet. Been doing it for 26 years or so. {rest of your comment ... verbatim!} Email doesn't need fixing or a v2. I've run GroupWise, Exchange, Lotus (various flavours but never peppermint) and others too. For me, for little systems: Dovecot (IMAP) + Exim (MTA) is a golden combo. Email_v1 itself is just complicated enough without being too funky. You bolt on TLS/SSL and the rest. MX records are a simplified form of SRV records (I think MX came first but the point holds) and remove the requirement for gateway load balancers and clustering technologies if you don't want to do all that stuff. Nothing is perfect but email does deliver a lot more significant messages than any other method from before or since its inception.
- naasking 2y ago
  If email didn't need fixing, spam wouldn't exist. There's more spam than there is legitimate email traversing the internet. That is a problem worth solving. You could solve it with existing infrastructure to some extent, eg. your email address is actually a cryptographically generated guid rather than something easily guessed or harvested. If you combine that with a background handshake procedure for introductions, so that all of your contacts get their own guid alias mapped to your canonical one, then you can revoke any of those if they get compromised at any time. Spam is effectively solved. This is basically like the web of trust, but for email.
  - felsokning 2y ago
    > ...your email address is actually a cryptographically generated guid rather than something easily guessed or harvested. If you combine that with a background handshake procedure for introductions, so that all of your contacts get their own guid alias mapped to your canonical one, then you can revoke any of those if they get compromised at any time... Here, you're kicking the problem further down the road, though, to another known attack vector: Directory Harvest Attack[1]. In this case, though, the directory (presumably) contains the guid mapping (which - by definition - would have to be a different guid than the object) and would have to process parsing these guids against the users. (This already occurs on recipient receive for some SMTP servers [just before BDATA/DATA] via the email address). What would one bad email to an email guid do? Would it force rotation of the guid[s] throughout the entire forest? If so, how would that be communicated externally? How would you communicate it for just the one address, if you just changed the one guid? Would you, instead, have to keep a guid history to check against -- or lose all of the email between possible compromise and the sender's database update? Would you just keep it in the Transport Queue, until manual intervention could check out email between the possible compromise of the guid and new mail would be received for the new guid? That wouldn't scale for large enterprises. Keep in mind that nothing has to be sent for recipient validation to occur. The SMTP Server[s] just respond[s] to the recipient block with the next step -- but the caller doesn't have to complete the SMTP negotiation from this point, they already have validation if the addresses (even these proposed guids) are valid. Tarpitting is somewhat of a viable option, here, but it isn't foolproof. [1] - https://en.wikipedia.org/wiki/Directory_Harvest_Attack https://en.wikipedia.org/wiki/Directory_Harvest_Attack
naasking 2y ago
> why would anyone spend the millions of dollars in engineering effort to move Why would it take millions of dollars to move?
- ttul 2y ago
  The Gmail engineering team is perhaps 500 strong based on various estimates that try to exclude team members who work on Google Workspace more broadly. At Google’s median pay of $300K, the cost is very approximately $150M/yr to engineer the stuff that makes Gmail work. That’s before accounting for infrastructure specifically required for engineering efforts, such as training models. Let’s speculate that migrating to a whole new email standard would eat up 10% of the team for a couple of years, giving time to build and adapt a software stack that can run at Google scale on the new standard. That’s a $30M spend. While this figure is peanuts for Google, consider that during the span of time that engineers would be transitioning to “email 2.0,” the full Gmail team would continue to work on just maintaining “email 1.0” and adapting all the new standards that continue to make the existing system improve. Why would anyone at Google want this disruption unless email 1.0 was so broken that it was hopeless?
  - naasking 2y ago
    From the article, the new software stack is just a slight modification of the existing stack. I just don't see how it would take a few years and 10% of the workforce to add a few new steps to the existing pipeline. Once implemented in some open source mail agents, everyone can just upgrade on their usual schedule, so it's not like every single person needs to spend millions of dollars to transition.
schneems 2y ago
I’m not in that space but it sounds familiar. In my experience there are a few phases in fixing a problem: - Noticing the problem - Finding something better - Finding a way to migrate between the problem and the fix The hardest is the last bullet. It is easy to say “things would be better if X” it’s much harder to build a roadmap and buy-in on how to get to X.
xnx 2y ago
Email is amazing and underappreciated. The federated social network that everyone keeps trying to invent has been there all along. If more people controlled their own email address, it would be nearly perfect.
brightball 2y ago
I got to present at M3AAWG several years back and it was a cool experience...but that is not a conference you get to just show up to. The only reason I was there is because I was invited and memberships are pricey.
lubujackson 2y ago
It seems a lot like trying to fight against Javascript on the web. They are eternally linked at this point, and the only way to replace it is to envelop with some sort of language polyfill for bckwards compatibility - which is just yet another layer of abstaction...
winternett 2y ago
The next phase of email should be for Federal and state governments to host it. We need to move it off of private infrastructure for essential services like payments and recordkeeping. There should be better penalties for misuse of email services and scams now that it handles so many important things, including authentication to many sites everyone uses. I wrote a whole rant on this a while back on my blog, despite objections, email can well replace the outdated physical mail delivery systems currently in place around the world, vital communication in the age of corporate surveillance and scams needs to be regulated if not run, regulated, and secured by more accountable government resources. We have already seen how our personal information can be harvested and weaponized. Leaving important email services to strategically opportunistic companies like Google will not be wise into the future.

by ttul 2y ago

gjsman-1000 2y ago

https://xkcd.com/2347/ https://xkcd.com/2347/

lxgr 2y ago

I think it all depends on the right pitch: As a replacement for current email only, you're probably right in that nobody would care enough to do the significant investments necessary in a reasonable timeframe, similarly to IPv6 (and the pain there is arguably even greater than for email). As an alternative to something with the semantics of what companies currently use SMS for (i.e. having ostensibly higher reliability and security), I think you'd see a lot of interest.

deskr 2y ago

> universally adopted and interoperable No. Big email server all over the place might drop your emails for whatever reason is and they won't tell you.

1vuio0pswjnm7 2y ago
Question is whether a new or updated protocol could force users such as "email providers" to change their behaviour. Here is something to downvote, a series of questions: What if email were "pickup only" _from the sender_. Not from some intermediary recipient, e.g., an "email provider". What if the the recipient had to identify acceptable senders before they could "send" mail to the recipient. Arguably this already happens every time an email recpient gives out their email address to some email sender. What if the sender did not "send" mail but instead uploaded it to host run by the sender and accessible by the recipient. Then the recipient retrieves the mail from the sender. In the past, one large email provider had an RSS feed for a user's inbox. What if the RSS feed is not provided by some third party "email provider" but by _senders_. What if the feed indicates whether there is new mail waiting to be retrieved by the recipient. Arguably, something like this already happens, albeit using third party intermediaries, for example as millions of people use non-public webpages on third party websites to communicate with each other, instead of using email. Recipients check these pages for comments or messages from "senders". A simpler idea that requires no changes to any email protocol, which I have tested successfully on home network, is for sender and recipient to be on a peer-to-peer overlay and run their own SMTP servers, like the original internet. The sender SMTP server communicates directly with the receiver's SMTP server, not a third party SMTP server run by an "email provider". Obviously, sender and receiver should not invite anyone onto this network who they do not know and from which they do not want to receive email. The fundamental problem with email is that personal, non-commercial email is mixed with commercial email, mail that is selling something. That's beneficial to marketers, but not email users. Any change to email that threatens to exclude the unsolicited, commercial email will be opposed fervently.
- deleted 2y ago
  [deleted]
- donut 2y ago
  The pickup idea sounds similar to how newsgroups work.
  - 2y ago

kilburn 2y ago

Also, they would school them on actual-world problems in the process: - You can't wait until you receive the entire body to be able to compute a signature to then validate the sender as your first line of defense. It is just too expensive and opens you to DDOS attacks. People use IP reputation as a first line of defense because it is cheap, not because it is good. - You cannot enforce people's behavior through RFCs. I can assure you that random guy next desk will not care about your "this is a top-posting-thread" header and bottom post there. Even if she has to manually copy/paste things around. - Likewise, auto-generated plain-text versions of HTML (or other rich-text formats) are no better than what screen readers can achieve. Most poeple won't bother writing that alternate version, meaning the obligatory alt text is now less useful than when it was optional and only people who cared inculded it. - Your largest client may not update their e-mail infrastructure to comply with the latest standards. If that happens, you don't tell them to update or otherwise you won't be answering them because their e-mails go to spam. You do whatever is necessary to ensure that their e-mails don't go to spam. Business always comes first.

gjsman-1000 2y ago
Hypothetically, while I’m no expert: 1. Could a future protocol require an immediate initial message (a “hello”) stating exactly how much content will be sent, and until the “hello” is sent, it’s limited to, say, 128KB before the connection is immediately terminated? (And of course, if the content exceeds the declaration, termination and immediate IP temporary ban, safe to do as this is an obvious violation of a new spec?) 2. The goal is to make it easier for the email client which by itself will encourage good behavior. There’s also no requirement for the messages to all be in one massive blob. 3. The goal is that it would be automatically created by the client. For personal emails, this is easy. For enhanced HTML emails, that is where the requirement comes in. Email providers can come up with their own ways of enforcement from there (I.e. “if it’s only one sentence, you obviously didn’t do it”), though I get your point and that would become messy unofficial spec again. 4. Could a future emails system have versioning, allowing the server to clearly communicate (“Hello, I implement MX2 v3.1.”)? In addition, a business can obviously make their own settings that original email alerts do not go to Junk in their business mailboxes - but they do know they’d better get on it or their messages to clients might go to Junk.

gerdesj 2y ago

I run several of the smallest email sending services on the internet. Been doing it for 26 years or so. {rest of your comment ... verbatim!} Email doesn't need fixing or a v2. I've run GroupWise, Exchange, Lotus (various flavours but never peppermint) and others too. For me, for little systems: Dovecot (IMAP) + Exim (MTA) is a golden combo. Email_v1 itself is just complicated enough without being too funky. You bolt on TLS/SSL and the rest. MX records are a simplified form of SRV records (I think MX came first but the point holds) and remove the requirement for gateway load balancers and clustering technologies if you don't want to do all that stuff. Nothing is perfect but email does deliver a lot more significant messages than any other method from before or since its inception.

naasking 2y ago
If email didn't need fixing, spam wouldn't exist. There's more spam than there is legitimate email traversing the internet. That is a problem worth solving. You could solve it with existing infrastructure to some extent, eg. your email address is actually a cryptographically generated guid rather than something easily guessed or harvested. If you combine that with a background handshake procedure for introductions, so that all of your contacts get their own guid alias mapped to your canonical one, then you can revoke any of those if they get compromised at any time. Spam is effectively solved. This is basically like the web of trust, but for email.
- felsokning 2y ago
  > ...your email address is actually a cryptographically generated guid rather than something easily guessed or harvested. If you combine that with a background handshake procedure for introductions, so that all of your contacts get their own guid alias mapped to your canonical one, then you can revoke any of those if they get compromised at any time... Here, you're kicking the problem further down the road, though, to another known attack vector: Directory Harvest Attack[1]. In this case, though, the directory (presumably) contains the guid mapping (which - by definition - would have to be a different guid than the object) and would have to process parsing these guids against the users. (This already occurs on recipient receive for some SMTP servers [just before BDATA/DATA] via the email address). What would one bad email to an email guid do? Would it force rotation of the guid[s] throughout the entire forest? If so, how would that be communicated externally? How would you communicate it for just the one address, if you just changed the one guid? Would you, instead, have to keep a guid history to check against -- or lose all of the email between possible compromise and the sender's database update? Would you just keep it in the Transport Queue, until manual intervention could check out email between the possible compromise of the guid and new mail would be received for the new guid? That wouldn't scale for large enterprises. Keep in mind that nothing has to be sent for recipient validation to occur. The SMTP Server[s] just respond[s] to the recipient block with the next step -- but the caller doesn't have to complete the SMTP negotiation from this point, they already have validation if the addresses (even these proposed guids) are valid. Tarpitting is somewhat of a viable option, here, but it isn't foolproof. [1] - https://en.wikipedia.org/wiki/Directory_Harvest_Attack https://en.wikipedia.org/wiki/Directory_Harvest_Attack

naasking 2y ago

> why would anyone spend the millions of dollars in engineering effort to move Why would it take millions of dollars to move?

ttul 2y ago
The Gmail engineering team is perhaps 500 strong based on various estimates that try to exclude team members who work on Google Workspace more broadly. At Google’s median pay of $300K, the cost is very approximately $150M/yr to engineer the stuff that makes Gmail work. That’s before accounting for infrastructure specifically required for engineering efforts, such as training models. Let’s speculate that migrating to a whole new email standard would eat up 10% of the team for a couple of years, giving time to build and adapt a software stack that can run at Google scale on the new standard. That’s a $30M spend. While this figure is peanuts for Google, consider that during the span of time that engineers would be transitioning to “email 2.0,” the full Gmail team would continue to work on just maintaining “email 1.0” and adapting all the new standards that continue to make the existing system improve. Why would anyone at Google want this disruption unless email 1.0 was so broken that it was hopeless?
- naasking 2y ago
  From the article, the new software stack is just a slight modification of the existing stack. I just don't see how it would take a few years and 10% of the workforce to add a few new steps to the existing pipeline. Once implemented in some open source mail agents, everyone can just upgrade on their usual schedule, so it's not like every single person needs to spend millions of dollars to transition.

schneems 2y ago

I’m not in that space but it sounds familiar. In my experience there are a few phases in fixing a problem: - Noticing the problem - Finding something better - Finding a way to migrate between the problem and the fix The hardest is the last bullet. It is easy to say “things would be better if X” it’s much harder to build a roadmap and buy-in on how to get to X.

xnx 2y ago

Email is amazing and underappreciated. The federated social network that everyone keeps trying to invent has been there all along. If more people controlled their own email address, it would be nearly perfect.

brightball 2y ago

I got to present at M3AAWG several years back and it was a cool experience...but that is not a conference you get to just show up to. The only reason I was there is because I was invited and memberships are pricey.

lubujackson 2y ago

It seems a lot like trying to fight against Javascript on the web. They are eternally linked at this point, and the only way to replace it is to envelop with some sort of language polyfill for bckwards compatibility - which is just yet another layer of abstaction...

winternett 2y ago

The next phase of email should be for Federal and state governments to host it. We need to move it off of private infrastructure for essential services like payments and recordkeeping. There should be better penalties for misuse of email services and scams now that it handles so many important things, including authentication to many sites everyone uses. I wrote a whole rant on this a while back on my blog, despite objections, email can well replace the outdated physical mail delivery systems currently in place around the world, vital communication in the age of corporate surveillance and scams needs to be regulated if not run, regulated, and secured by more accountable government resources. We have already seen how our personal information can be harvested and weaponized. Leaving important email services to strategically opportunistic companies like Google will not be wise into the future.