4 ms·

Is there any work being done to add to SMTP and other open protocols to address these issues? I had started to look into doing it as a hobby project, but as fa

• toomim 7y ago
  I'm working on such a project, but only indirectly. The core problem I see is that sending a message over SMTP means you push it to someone, and then you can't modify the message afterward. This means (1) you can't add someone to a message. And as a result, if someone's added to a list, they don't automatically get the messages that were sent to the list before, because those were already sent. And from another angle, it also means (2) you need to know all the recipients in advance. You can't send a message to the "public", for instance. There's no natural support for publishing all "channels", because there's nothing to subscribe to. And from a third angle, it means (3) you have to build a separate system to archive a mailing list, because SMTP itself acts as if it forgets a message after it's sent. There's nothing in the protocol to retrieve a past message! The solution I'm working on is a little circuitous, but I think it will work, eventually. I think email needs a synchronization protocol. Where everyone who has received a message will receive updates, in sync. And every email will have a URL that can be referenced in replies. And email servers will serve emails, rather than just firing-and-forgetting them like SMTP. Now for the indirect part: I'm building this new email protocol on top of an extension to HTTP called Braid (https://braid.news https://braid.news). The idea is to first add synchronization to HTTP, in a way that allows data to be federated in a P2P network, like with email. Then you can host any data, anywhere on the web, and any client or server, anywhere else on the web, can synchronize with it. This gives you the same features as email, and then some. Consider that most people use webmail anyway. And mailing lists host message archives on websites. So your email is already on the web. Well, now instead of having a different protocol for sending emails between servers (SMTP) and for getting emails from a server to a client (IMAP/POP/JMAP), and for viewing them in a web browser (HTTP), we can have a single synchronization protocol, which is general enough to share email messages between servers, and from servers to clients, all using the same network messages. And it has the added benefits that (a) you can edit emails after you send them, (b) you can delete emails that you regret having sent (although you can't prevent recipients from caching a sent message), and (c) you can use the same protocol for posting a blog -- just send a message to a special "public" address, and your server's access control will allow any viewer to read it. This last part is because we'll be switching email from a push protocol (SMTP) to a subscribe protocol (HTTP+Braid). With the Braid mail, when you send a message to someone else, you'll really just be pinging their server and saying "I'm hosting some emails that are addressed to you." Then their server will just do a GET /inbox On your server and your server will send them all the outstanding messages that it has access to (anything addressed to `<user>@that_server.com`). And if you send a message to `public`, then anyone who does a GET /inbox will see it. Similar to this work is the JMAP specification in the IETF. (https://jmap.io https://jmap.io) This is a great improvement over IMAP, but it doesn't handle the server-to-server case that SMTP does, and it also has to implement its own particular synchronization mechanism over web push APIs because it doesn't get to benefit from the Braid synchronization extensions to HTTP. Finally — and perhaps most importantly — we can solve email spoofing. This is huge. The prevalence of spoofing is why it's so hard to run your own mail server. Most email providers actually blacklist entire C-blocks of IP addresses, because they have been used to send spam. So the only practical way to send emails these days is to use one of the big mail providers which is already trusted by other mail servers -- and thus email has become centralized, into a few big boys. If we solve spoofing, we can decentralize email again. Spoofing is so easy because SMTP doesn't inherently care who is sending a message, so there's no default way to verify the sender. A multitude of patchy standards (SPF, DKIM, DMARC) have tried to add layers of signatures onto the messages, each trying to patch flaws in the last one, but even altogether they can't solve spoofing. Since SMTP is a forwarding protocol, they have to verify the signatures of every server in between A and B, and this breaks, for instance, when a mail server modifies a message, adding text to the subject, or a footer, or even just a header, and thus breaks the signatures on the email that was sent out. But if emails are served over a synchronized HTTP, rather than pushed with a fire-and-forget SMTP, then it's trivial to verify the sender -- it's already built-in with SSL/TLS certificates. You just GET the message. And because this is HTTP, we can go ever further. We can also add in metadata, like have public/private keys generated for each user, and serve that at a GET /user/<foo>/public-key URL. This is much better than how SPF does it -- by forcing you to hack metadata into your mail server's DNS records. It uses this DNS hack because if they hosted the keys over SMTP... it would basically turn SMTP into HTTP. Similarly, a server can publish its mailing lists at a GET /lists URL. And if you fetch a list, you'll get all the emails in the list. So you don't need to implement a separate system to archive your mailing list -- it's all just built in!
  • em-bee 7y ago
    i agree that email could use server-to-server synchronization. but isn't that what usenet does/did? i think it would be worth to explore the specific differences to usenet to see if there is anything that can be learned from it
    • toomim 7y ago
      Interesting! Good point. I'll look into it.