7 ms·

I really want to see some innovation in the email space. The landscape is like a sea of false promises and dashed dreams. SMTP is one of the bread-and-butters o

I really want to see some innovation in the email space. The landscape is like a sea of false promises and dashed dreams. SMTP is one of the bread-and-butters of the internet, yet it just doesn't seem to be moving forward (maybe that's for the best), and no one's building extensions on top of it. Maybe I'm naive in thinking it was possible but we could have avoided this whole "make an account on X messenger so we can talk", if someone had just jammed XMPP and SMTP together. Every few years I'm tempted to try and write a SMTP server (MX) myself but then realize that postfix has been around so long and is the go to choice for a reason. I don't have 20 years of figuring out how X mail server interprets SMTP to be as reliable as postfix. I settled for trying to wrap it[0]. [0]: https://gitlab.com/postmgr/postmgr https://gitlab.com/postmgr/postmgr

amanzi 8y ago
"if someone had just jammed XMPP and SMTP together" - wasn't that kind of what Google Wave was aiming for?
- lucideer 8y ago
  Yes. And the vision for Google Wave was great. The execution was just abominably bad (primarily on the frontend/UI side), and Google's failures in execution carried over reputationally to the—now dead—Apache incubator.
  - vidarh 8y ago
    Not just the UI side. The underlying protocol was also pointlessly over-engineered. SMTP is still around because it is simple. You can get a basic SMTP client running in an hour or two. You can get a basic server running in a similar amount of time, at least in a higher level language. That matters, not because anyone will have anything usable in that time, but because it lets you start to experiment very, very easily. Wave suffered from a UI that was a mess layered on top of a protocol that made it really hard for people to get started on experimenting with alternative frontends, and without a user-base giving people a reason to persist figuring out how to interoperate with it. Had the protocol been simpler, the UI mess might not have mattered so much - people might have come up with their own ideas.
    - arethuza 8y ago
      You don't even need a client - connecting with telnet and sending emails is pretty straightforward. I guess the modern equivalent is checking if a REST API is usable just with curl.
    - sjwright 8y ago
      The S in SMTP literally stands for Simple. And it was developed in 1982. So it was considered simple in 1982. That gives you some idea how simple it is. https://tools.ietf.org/html/rfc821 https://tools.ietf.org/html/rfc821
      - PaulHoule 8y ago
        It is simple today, except for the authentication. Of course, if the authentication doesn't work, nothing works!
- hardwaresofton 8y ago
  Yes, but no -- I wasn't one of the Google Wave faithful so I don't know the ins and outs of it, but I what I wanted was the ability to extend SMTP all together to include the possibility of negotiating XMPP -- almost like what's happened with HTTP 1.1/2/3 evolving to support new connection types and content types/etc.
- rakoo 8y ago
  Further than that, it seems to me SMTP is functionally included inside XMPP so there's no need in mixing them. Perhaps having an SMTP gateway that feeds into an XMPP server at most.
raimue 8y ago
Yes, postfix has been around for so long and it is very solid. However, to run a mail server nowadays, it still needs a lot of configuration work and even additional daemons to add SPF checks, forwarding with SRS, and DKIM signatures. All of this is possible to be added though, which speaks for the flexibility of the postfix design.
- hardwaresofton 8y ago
  This is exactly what I was hoping to add to it with my postmgr project. There's projects like Mailu[0] which are fantastic as well. [0]: https://github.com/Mailu/mailu https://github.com/Mailu/mailu
pjc50 8y ago
It was sometimes possible to hold realtime conversations over email, before the amount of required spam processing became too high. There's no protocol reason why you can't do that between servers, although getting it into a client requires either IMAP push or something more modern. > make an account You are required to make an account somewhere so your account can be banned if you spam, basically.
pjmlp 8y ago
Emails are perfectly fine as they are. We also don't keep re-inventing forks, umbrellas, hammers and such.
- vidarh 8y ago
  But we do. The "re-inventions" just tends to be increasingly subtle, because the high level structure continues to make sense and the overall designs are simple to begin with, which makes it hard to come up with revolutionary new ideas in the space. Umbrella's for example see continuous evolution in means of making better fully collapsible versions, making the deployment more automated, reducing weight, etc. From Wikipedia, illustrating both that there's continued substantial effort in coming up with new ideas, and at the same time that it is hard to come up with something genuinely new: > Umbrellas continue to be actively developed. In the US, so many umbrella-related patents are being filed that the U.S. Patent Office employs four full-time examiners to assess them. As of 2008, the office registered 3000 active patents on umbrella-related inventions. Nonetheless, Totes, the largest American umbrella producer, has stopped accepting unsolicited proposals. Its director of umbrella development was reported as saying that while umbrellas are so ordinary that everyone thinks about them, "it's difficult to come up with an umbrella idea that hasn’t already been done."[36] (https://en.wikipedia.org/wiki/Umbrella https://en.wikipedia.org/wiki/Umbrella)
  - pjmlp 8y ago
    The ones being sold at the shops I regulary buy haven't changed a bit, besides their colours.
    - puszczyk 8y ago
      So you know that “we” don’t reinvent umbrellas, because “you” buy the same all the time?
      - pjmlp 8y ago
        I don't imagine who would waste their money in re-invented umbrellas, maybe it is an hipster thing.
PaulHoule 8y ago
Email is "good enough". The real hole is "instant messaging". Somehow every few years we got from ICQ to AIM to Paltalk to Skype to Facebook Messenger to Slack to ... (at least Altassian had the decency to take HipChat behind the woodshed and shoot it) The strange thing is that these services dry up, blow away, get replaced, but they don't seem to improve on what came before. There is a standard, XMPP, but the only people who care about it are firefighters, cops, and soldiers. For all the anger at internet giants these days, I can't see for the life of me why people aren't pushing for open instant messenging.
- hardwaresofton 8y ago
  I didn't explain it fully, but the alternate future I was imagining was one where SMTP could evolve/instantiate a connection for "ASAP email" -- AKA instant messaging. Kinda like how HTTP can upgrade to websockets. There would obviously be a lot to work out, but if something like this was in the SMTP standard (or at least introduced), I think eventually email providers would race to differentiate themselves with support for it, and we might not be where we are today. In my view there's no reason to even require the SMTP servers to serve chat traffic -- as long as they could hand off reasonably, and everything spoke the language as specified in the spec (or at least came close).
  - toomuchtodo 8y ago
    I don't want instant messaging in my email though. If you need synchronous communications, use a chat client.
    - hardwaresofton 8y ago
      This is probably where we differ -- I don't think limiting email to asynchronous communications helps anyone, and it could have been written in an extensible manner to also support synchronous communication. Even if I'm completely wrong and email should be left asynchronous, SMTP could at least have introduced some standard around chat negotiation and a protocol negotiation process for remote chat servers/agents.
wwarner 8y ago
Apache James is a great smtp and imap server that has a lot of really great features, like distributed storage in cassandra and some support for jmap.
- hardwaresofton 8y ago
  This is why I keep coming back to HN -- thanks for recommending Apache James. I thought I searched far and wide for other mail servers, yet only came up with postfix, iredmail and cyrus, I clearly didn't look hard enough
e12e 8y ago
> if someone had just jammed XMPP and SMTP together. Xmpp is/supports federated/ion (user1@server1.com can message user2@server2.com). But the major players (by numbers) wanted silos: Google (talk) and fasebook (first gen of messenger). They basically went lol, screw users (arguably because: spam. But hello, Gmail? And today fb spam...). So thank Google and Facebook for deliberately gimping it so you can't fb message user@gmail.com or gtalk first name.lastname123@facebook.com.
- rakoo 8y ago
  The official reason for Google to give up on this is that there was a real problem of incoming spam from lousy domains, and no major player wanted to play the federation game with them. It's not as simple as you make it sound.
  - e12e 8y ago
    1) they are able (forced to) handle spam for Gmail (and internally for gtalk) - I'm not convinced by the spam argument. 2) there were only two "big" players, Google and Facebook. The benefit of federation would be as with email: an open internet with federation across organisation level services (community/company run servers). The fact that Google didn't implement ssl for federation doesn't mean people "wouldn't" federate with them, it mean Google didn't make a real effort.
elFarto 8y ago
Honestly I'm surprised no one has suggested replacing SMTP with a HTTP POST request. All you would need is an email address -> HTTP URL conversion (which I believe something like WebFinger already has). Then just POST your message to that URL and you're done. ISPs can run proxies that just authenticate that requests are from users and forwards them on.

amanzi 8y ago

"if someone had just jammed XMPP and SMTP together" - wasn't that kind of what Google Wave was aiming for?

lucideer 8y ago
Yes. And the vision for Google Wave was great. The execution was just abominably bad (primarily on the frontend/UI side), and Google's failures in execution carried over reputationally to the—now dead—Apache incubator.
- vidarh 8y ago
  Not just the UI side. The underlying protocol was also pointlessly over-engineered. SMTP is still around because it is simple. You can get a basic SMTP client running in an hour or two. You can get a basic server running in a similar amount of time, at least in a higher level language. That matters, not because anyone will have anything usable in that time, but because it lets you start to experiment very, very easily. Wave suffered from a UI that was a mess layered on top of a protocol that made it really hard for people to get started on experimenting with alternative frontends, and without a user-base giving people a reason to persist figuring out how to interoperate with it. Had the protocol been simpler, the UI mess might not have mattered so much - people might have come up with their own ideas.
  - arethuza 8y ago
    You don't even need a client - connecting with telnet and sending emails is pretty straightforward. I guess the modern equivalent is checking if a REST API is usable just with curl.
  - sjwright 8y ago
    The S in SMTP literally stands for Simple. And it was developed in 1982. So it was considered simple in 1982. That gives you some idea how simple it is. https://tools.ietf.org/html/rfc821 https://tools.ietf.org/html/rfc821
    - PaulHoule 8y ago
      It is simple today, except for the authentication. Of course, if the authentication doesn't work, nothing works!
hardwaresofton 8y ago
Yes, but no -- I wasn't one of the Google Wave faithful so I don't know the ins and outs of it, but I what I wanted was the ability to extend SMTP all together to include the possibility of negotiating XMPP -- almost like what's happened with HTTP 1.1/2/3 evolving to support new connection types and content types/etc.
rakoo 8y ago
Further than that, it seems to me SMTP is functionally included inside XMPP so there's no need in mixing them. Perhaps having an SMTP gateway that feeds into an XMPP server at most.

raimue 8y ago

Yes, postfix has been around for so long and it is very solid. However, to run a mail server nowadays, it still needs a lot of configuration work and even additional daemons to add SPF checks, forwarding with SRS, and DKIM signatures. All of this is possible to be added though, which speaks for the flexibility of the postfix design.

hardwaresofton 8y ago
This is exactly what I was hoping to add to it with my postmgr project. There's projects like Mailu[0] which are fantastic as well. [0]: https://github.com/Mailu/mailu https://github.com/Mailu/mailu

pjc50 8y ago

It was sometimes possible to hold realtime conversations over email, before the amount of required spam processing became too high. There's no protocol reason why you can't do that between servers, although getting it into a client requires either IMAP push or something more modern. > make an account You are required to make an account somewhere so your account can be banned if you spam, basically.

pjmlp 8y ago

Emails are perfectly fine as they are. We also don't keep re-inventing forks, umbrellas, hammers and such.

vidarh 8y ago
But we do. The "re-inventions" just tends to be increasingly subtle, because the high level structure continues to make sense and the overall designs are simple to begin with, which makes it hard to come up with revolutionary new ideas in the space. Umbrella's for example see continuous evolution in means of making better fully collapsible versions, making the deployment more automated, reducing weight, etc. From Wikipedia, illustrating both that there's continued substantial effort in coming up with new ideas, and at the same time that it is hard to come up with something genuinely new: > Umbrellas continue to be actively developed. In the US, so many umbrella-related patents are being filed that the U.S. Patent Office employs four full-time examiners to assess them. As of 2008, the office registered 3000 active patents on umbrella-related inventions. Nonetheless, Totes, the largest American umbrella producer, has stopped accepting unsolicited proposals. Its director of umbrella development was reported as saying that while umbrellas are so ordinary that everyone thinks about them, "it's difficult to come up with an umbrella idea that hasn’t already been done."[36] (https://en.wikipedia.org/wiki/Umbrella https://en.wikipedia.org/wiki/Umbrella)
- pjmlp 8y ago
  The ones being sold at the shops I regulary buy haven't changed a bit, besides their colours.
  - puszczyk 8y ago
    So you know that “we” don’t reinvent umbrellas, because “you” buy the same all the time?
    - pjmlp 8y ago
      I don't imagine who would waste their money in re-invented umbrellas, maybe it is an hipster thing.

PaulHoule 8y ago

Email is "good enough". The real hole is "instant messaging". Somehow every few years we got from ICQ to AIM to Paltalk to Skype to Facebook Messenger to Slack to ... (at least Altassian had the decency to take HipChat behind the woodshed and shoot it) The strange thing is that these services dry up, blow away, get replaced, but they don't seem to improve on what came before. There is a standard, XMPP, but the only people who care about it are firefighters, cops, and soldiers. For all the anger at internet giants these days, I can't see for the life of me why people aren't pushing for open instant messenging.

hardwaresofton 8y ago
I didn't explain it fully, but the alternate future I was imagining was one where SMTP could evolve/instantiate a connection for "ASAP email" -- AKA instant messaging. Kinda like how HTTP can upgrade to websockets. There would obviously be a lot to work out, but if something like this was in the SMTP standard (or at least introduced), I think eventually email providers would race to differentiate themselves with support for it, and we might not be where we are today. In my view there's no reason to even require the SMTP servers to serve chat traffic -- as long as they could hand off reasonably, and everything spoke the language as specified in the spec (or at least came close).
- toomuchtodo 8y ago
  I don't want instant messaging in my email though. If you need synchronous communications, use a chat client.
  - hardwaresofton 8y ago
    This is probably where we differ -- I don't think limiting email to asynchronous communications helps anyone, and it could have been written in an extensible manner to also support synchronous communication. Even if I'm completely wrong and email should be left asynchronous, SMTP could at least have introduced some standard around chat negotiation and a protocol negotiation process for remote chat servers/agents.

wwarner 8y ago

Apache James is a great smtp and imap server that has a lot of really great features, like distributed storage in cassandra and some support for jmap.

hardwaresofton 8y ago
This is why I keep coming back to HN -- thanks for recommending Apache James. I thought I searched far and wide for other mail servers, yet only came up with postfix, iredmail and cyrus, I clearly didn't look hard enough

e12e 8y ago

> if someone had just jammed XMPP and SMTP together. Xmpp is/supports federated/ion (user1@server1.com can message user2@server2.com). But the major players (by numbers) wanted silos: Google (talk) and fasebook (first gen of messenger). They basically went lol, screw users (arguably because: spam. But hello, Gmail? And today fb spam...). So thank Google and Facebook for deliberately gimping it so you can't fb message user@gmail.com or gtalk first name.lastname123@facebook.com.

rakoo 8y ago
The official reason for Google to give up on this is that there was a real problem of incoming spam from lousy domains, and no major player wanted to play the federation game with them. It's not as simple as you make it sound.
- e12e 8y ago
  1) they are able (forced to) handle spam for Gmail (and internally for gtalk) - I'm not convinced by the spam argument. 2) there were only two "big" players, Google and Facebook. The benefit of federation would be as with email: an open internet with federation across organisation level services (community/company run servers). The fact that Google didn't implement ssl for federation doesn't mean people "wouldn't" federate with them, it mean Google didn't make a real effort.

elFarto 8y ago

Honestly I'm surprised no one has suggested replacing SMTP with a HTTP POST request. All you would need is an email address -> HTTP URL conversion (which I believe something like WebFinger already has). Then just POST your message to that URL and you're done. ISPs can run proxies that just authenticate that requests are from users and forwards them on.