8 ms·

If you’re going to learn anything, email is a good bet. It’s barely changed in 30 years, and I suspect it will not change much for a long time. I built my own

If you’re going to learn anything, email is a good bet. It’s barely changed in 30 years, and I suspect it will not change much for a long time. I built my own transactional email provider (https://mailpace.com https://mailpace.com), I wish I had found this link before I started. I would also recommend reading the email RFCs, they’re not that difficult to understand and the history explains a lot. Email aside we can learn a lot from this kind of distributed, decentralised system design in the future.

klabb3 4y ago
> Email aside we can learn a lot from this kind of distributed, decentralised system design in the future. Nit for other people curious about email: email isn’t decentralized by the popularized use of the term, both in theory and more so in practice. Email relies on DNS which is federated. That doesn’t mean that it’s worse, federation is a proven-to-work trade off which provides many of the benefits of decentralization, (arguably the most important parts). As for practice, email relies on opaque reputation systems to prevent abuse, which acts as a centralizing force. I’m sure you know (better than me) that the barrier of entry to sending email on your own is quite a bit above zero.
- codetrotter 4y ago
  > Email relies on DNS which is federated To a great extent, yes. If I give my public email address to someone and expect them to be able to send me email from anywhere at anytime, they will almost certainly have to rely on the DNS that exists like we know it. But the situation can be slightly different in various ways: - If our mail servers have globally routable IP-addresses permanently assigned to them, and are set up to accept mails where the IP address is used directly, we could use the IP addresses instead of domain names. But that's still similar to the situation with domain names, since it's still ICANN at the top for that too. - In a LAN or a VPN we could exchange email between hosts using either DNS that is completely under our control, or we could use mDNS on it which is also under our control, or in the case of either really small networks or a short time span we could even rely on the IP addresses our mail servers have on said LAN or VPN and send mail and again that would be completely under our own control as well. - Since Tor offers a base for TCP connections, every TCP based protocol can operate on top of it. So if we both use Tor connected email servers then we could exchange emails that way. Our email clients could connect to our email servers over clearnet without having to be aware of Tor, and we could use email addresses like bob@fastrcl5totos3vekjbqcmgpnias5qytxnaj7gpxtxhubdcnfrkapqad.onion and alice@mysitenameyx4fi3l6x2gyzmtmgxjyqyorj9qsb5r543izcwymle.onion In these ways, e-mail could still be regarded as decentralized. As well as in a simpler way that as opposed to for example Facebook where there is one company that owns the whole system and oversees and decides who gets to have an account on the system or not, you and I can always find another e-mail provider to give us a new account and we can communicate with others from those new accounts. And in fact, if we were willing to do it, we could change how we set up our e-mail servers so that we have the server accept mail from any other e-mail server but with the condition that the mail was encrypted with a known PGP key. Now imagine that you and I set up our mail servers like this and we exchange PGP keys. Then even if I lose access to my original email account that I had told you about, I could mail you from a different account with another provider using the same PGP key, and your server would recognise that the mail was from me and accept it and then you could respond back. But what if we both lose access to our accounts at the same time? If we plan ahead we could make not just one account each but a number of accounts on a number of different mail servers that we both tell each other about. That would give us quite a lot of wiggle room.
- dwheeler 4y ago
  Few things are perfectly anything, and email is way more distributed than (say) Twitter messages. In particular, I own my own domain (which costs little), and then set up where my email goes. If I want to change email providers, I can do it without having to track down everyone who knew my old email address. Similarly, if I don't like my domain registrar, I can change that too. Being able to control my own destiny, and not being completely under the control of someone else, is in my opinion the real goal. Distributed & decentralized systems at least hold out the promise of meeting that goal, and where they do, I'm delighted. It's not perfect, but it beats many other systems.
  - account42 4y ago
    To add to that, while DNS is centralized at the root, that centralization is unavoidable if you want to have human-readable addresses that consistently resolve to the correct recipient. And each TLD is (at least the ccTLDs and the old gTLDs) managed by a government or international organization acting on behalf of governments, which in turn (at least in theory) act on your behalf. While this system is not perfect (see e.g. the US governments history of seizing domains) it is almost infinitely better than most other communication mediums on the internet where the single company operating it can kick you out more or less for arbitrary reasons and without recourse. That alone makes this system precious enough that any pains from running your own mail server are IMHO worth it if it means that we can preserve Email as we know it - as a federated system not under the control of a single or a small number of corporations. Unfortunately, Email is at risk. For a lot of people it is already synonymous with Gmail and while Gmail is hardly the worst when it comes to deliverability (IME that crown goes to Microsoft) they do control enough accounts that they can pretty much nuke your little server if they want to.
- lisper 4y ago
  > the barrier of entry to sending email on your own is quite a bit above zero I can attest to this. I ran my own email server with no problem for fifteen years. Then my provider was acquired, and I decided to switch to a different provider, and so the IP address of my server changed for the first time in 15 years. I have not been able to reliably send email since, and I finally threw in the towel and signed up with Fastmail to handle my outgoing email. The biggest problem is people banning entire blocks of IP addresses. This can give you a bad reputation even if your behavior is stellar. All it takes is for someone using the same hosting provider to be a spammer to get yourself blacklisted. The whole situation sucks big fat honking weenies.
  - klabb3 4y ago
    Thanks for the data point. That sucks. I guess generally speaking, federation works well for DNS itself but not so well for email, since the malicious stuff you can do with DNS is greatly limited. Abuse/spam is everywhere, not just email, and the sad part is that providers not only went and rolled their own anti abuse systems, but they also went to great lengths to keep them secret. Without solving the spam problem, we won’t see decentralized or even federated systems widely popularized for any products which has these incentives. This includes mastodon, Nostr etc. The worst part is that projects can become pretty big before the spam starts, so it kills projects at the worst possible time. I wish the FOSS community came together and focused on finding solutions that work, because decent abuse prevention today is neither easy nor cheap to get right. And going without one is a ticking time bomb.
    - lisper 4y ago
      No, it's not a fundamental problem. It's just laziness on the part of many email providers. My email server has proper SPF records, and so anyone whose email server is configured properly can easily verify that my server is authorized to send email on behalf of my domain(s). But email admins just subscribe to blacklists and follow them blindly. There are a few big players, including Microsoft and AT&T, who just don't care if a few small players get blocked by false positives. (Ironically, despite their tolerance for false positives, they still let through an enormous amount of spam. I know this because my parents have their email on AT&T.)
- hackernews1134 4y ago
  For those who did not immediately grasp the parents claim that DNS is federated and not decentralized; here [0] is a nice article I found that talks about network architecture terminology definitions used in various famous research papers. It is a very nice, terse, compare and contrast. [0] https://networkcultures.org/unlikeus/resources/articles/what-is-a-federated-network/ https://networkcultures.org/unlikeus/resources/articles/what... Thank you for sending me down the rabbit hole klabb3! :)
NetOpWibby 4y ago
Thanks for sharing, I think I'll use this over Postmark. Just spent ~10 minutes looking over the site and the blog. Seems super solid.
OJFord 4y ago
Seconded, especially re RFCs. I built/am building on the client/personal user side, and agree they're really accessible. Of course what I've done is still riddled with bugs, and my hours spent on it are few and far between, but it's fun to build something to such a common protocol; have it work and interact with others. I was finally pushed into getting it off the ground when Fastmail stopped working the way I relied on it (niche/awkward/not documented as explicitly working) a week before my renewal - so I spent long nights getting something usable off the ground with that as a deadline. Extremely satisfying. (Then Fastmail started working as it had again, a day or so before my time was up! >_<)
- KasparEtter 4y ago
  Out of curiosity: What's the "feature" you needed; and what for?
  - OJFord 4y ago
    I can't remember exactly what aspect broke - though I think I have notes on it somewhere because I intended to blog a bit about it (not to hate on Fastmail, just to explain what I was doing and how I fixed it) - but it was something to do with relaying email, i.e. delivering it 'envelope to' my Fastmail address, but addressed to another.

klabb3 4y ago

> Email aside we can learn a lot from this kind of distributed, decentralised system design in the future. Nit for other people curious about email: email isn’t decentralized by the popularized use of the term, both in theory and more so in practice. Email relies on DNS which is federated. That doesn’t mean that it’s worse, federation is a proven-to-work trade off which provides many of the benefits of decentralization, (arguably the most important parts). As for practice, email relies on opaque reputation systems to prevent abuse, which acts as a centralizing force. I’m sure you know (better than me) that the barrier of entry to sending email on your own is quite a bit above zero.

codetrotter 4y ago
> Email relies on DNS which is federated To a great extent, yes. If I give my public email address to someone and expect them to be able to send me email from anywhere at anytime, they will almost certainly have to rely on the DNS that exists like we know it. But the situation can be slightly different in various ways: - If our mail servers have globally routable IP-addresses permanently assigned to them, and are set up to accept mails where the IP address is used directly, we could use the IP addresses instead of domain names. But that's still similar to the situation with domain names, since it's still ICANN at the top for that too. - In a LAN or a VPN we could exchange email between hosts using either DNS that is completely under our control, or we could use mDNS on it which is also under our control, or in the case of either really small networks or a short time span we could even rely on the IP addresses our mail servers have on said LAN or VPN and send mail and again that would be completely under our own control as well. - Since Tor offers a base for TCP connections, every TCP based protocol can operate on top of it. So if we both use Tor connected email servers then we could exchange emails that way. Our email clients could connect to our email servers over clearnet without having to be aware of Tor, and we could use email addresses like bob@fastrcl5totos3vekjbqcmgpnias5qytxnaj7gpxtxhubdcnfrkapqad.onion and alice@mysitenameyx4fi3l6x2gyzmtmgxjyqyorj9qsb5r543izcwymle.onion In these ways, e-mail could still be regarded as decentralized. As well as in a simpler way that as opposed to for example Facebook where there is one company that owns the whole system and oversees and decides who gets to have an account on the system or not, you and I can always find another e-mail provider to give us a new account and we can communicate with others from those new accounts. And in fact, if we were willing to do it, we could change how we set up our e-mail servers so that we have the server accept mail from any other e-mail server but with the condition that the mail was encrypted with a known PGP key. Now imagine that you and I set up our mail servers like this and we exchange PGP keys. Then even if I lose access to my original email account that I had told you about, I could mail you from a different account with another provider using the same PGP key, and your server would recognise that the mail was from me and accept it and then you could respond back. But what if we both lose access to our accounts at the same time? If we plan ahead we could make not just one account each but a number of accounts on a number of different mail servers that we both tell each other about. That would give us quite a lot of wiggle room.
dwheeler 4y ago
Few things are perfectly anything, and email is way more distributed than (say) Twitter messages. In particular, I own my own domain (which costs little), and then set up where my email goes. If I want to change email providers, I can do it without having to track down everyone who knew my old email address. Similarly, if I don't like my domain registrar, I can change that too. Being able to control my own destiny, and not being completely under the control of someone else, is in my opinion the real goal. Distributed & decentralized systems at least hold out the promise of meeting that goal, and where they do, I'm delighted. It's not perfect, but it beats many other systems.
- account42 4y ago
  To add to that, while DNS is centralized at the root, that centralization is unavoidable if you want to have human-readable addresses that consistently resolve to the correct recipient. And each TLD is (at least the ccTLDs and the old gTLDs) managed by a government or international organization acting on behalf of governments, which in turn (at least in theory) act on your behalf. While this system is not perfect (see e.g. the US governments history of seizing domains) it is almost infinitely better than most other communication mediums on the internet where the single company operating it can kick you out more or less for arbitrary reasons and without recourse. That alone makes this system precious enough that any pains from running your own mail server are IMHO worth it if it means that we can preserve Email as we know it - as a federated system not under the control of a single or a small number of corporations. Unfortunately, Email is at risk. For a lot of people it is already synonymous with Gmail and while Gmail is hardly the worst when it comes to deliverability (IME that crown goes to Microsoft) they do control enough accounts that they can pretty much nuke your little server if they want to.
lisper 4y ago
> the barrier of entry to sending email on your own is quite a bit above zero I can attest to this. I ran my own email server with no problem for fifteen years. Then my provider was acquired, and I decided to switch to a different provider, and so the IP address of my server changed for the first time in 15 years. I have not been able to reliably send email since, and I finally threw in the towel and signed up with Fastmail to handle my outgoing email. The biggest problem is people banning entire blocks of IP addresses. This can give you a bad reputation even if your behavior is stellar. All it takes is for someone using the same hosting provider to be a spammer to get yourself blacklisted. The whole situation sucks big fat honking weenies.
- klabb3 4y ago
  Thanks for the data point. That sucks. I guess generally speaking, federation works well for DNS itself but not so well for email, since the malicious stuff you can do with DNS is greatly limited. Abuse/spam is everywhere, not just email, and the sad part is that providers not only went and rolled their own anti abuse systems, but they also went to great lengths to keep them secret. Without solving the spam problem, we won’t see decentralized or even federated systems widely popularized for any products which has these incentives. This includes mastodon, Nostr etc. The worst part is that projects can become pretty big before the spam starts, so it kills projects at the worst possible time. I wish the FOSS community came together and focused on finding solutions that work, because decent abuse prevention today is neither easy nor cheap to get right. And going without one is a ticking time bomb.
  - lisper 4y ago
    No, it's not a fundamental problem. It's just laziness on the part of many email providers. My email server has proper SPF records, and so anyone whose email server is configured properly can easily verify that my server is authorized to send email on behalf of my domain(s). But email admins just subscribe to blacklists and follow them blindly. There are a few big players, including Microsoft and AT&T, who just don't care if a few small players get blocked by false positives. (Ironically, despite their tolerance for false positives, they still let through an enormous amount of spam. I know this because my parents have their email on AT&T.)
hackernews1134 4y ago
For those who did not immediately grasp the parents claim that DNS is federated and not decentralized; here [0] is a nice article I found that talks about network architecture terminology definitions used in various famous research papers. It is a very nice, terse, compare and contrast. [0] https://networkcultures.org/unlikeus/resources/articles/what-is-a-federated-network/ https://networkcultures.org/unlikeus/resources/articles/what... Thank you for sending me down the rabbit hole klabb3! :)

NetOpWibby 4y ago

Thanks for sharing, I think I'll use this over Postmark. Just spent ~10 minutes looking over the site and the blog. Seems super solid.

OJFord 4y ago

Seconded, especially re RFCs. I built/am building on the client/personal user side, and agree they're really accessible. Of course what I've done is still riddled with bugs, and my hours spent on it are few and far between, but it's fun to build something to such a common protocol; have it work and interact with others. I was finally pushed into getting it off the ground when Fastmail stopped working the way I relied on it (niche/awkward/not documented as explicitly working) a week before my renewal - so I spent long nights getting something usable off the ground with that as a deadline. Extremely satisfying. (Then Fastmail started working as it had again, a day or so before my time was up! >_<)

KasparEtter 4y ago
Out of curiosity: What's the "feature" you needed; and what for?
- OJFord 4y ago
  I can't remember exactly what aspect broke - though I think I have notes on it somewhere because I intended to blog a bit about it (not to hate on Fastmail, just to explain what I was doing and how I fixed it) - but it was something to do with relaying email, i.e. delivering it 'envelope to' my Fastmail address, but addressed to another.