5 ms·

This post [1,2] from last October discusses a major problem with hosting your own email, which is that the big email providers pretty much reject your messages

by kardos 10y ago

This post [1,2] from last October discusses a major problem with hosting your own email, which is that the big email providers pretty much reject your messages until your "reputation" is good. The reputation concept is purportedly a spam fighting effort, but a more cynical view is to see it as an effort by the big providers to sabotage self-hosted email. More discussion here [3] and here [4]. [1] Original post, now defunct: http://liminality.xyz/the-hostile-email-landscape/ http://liminality.xyz/the-hostile-email-landscape/ [2] Thank you archive.org: https://web.archive.org/web/20151121132739/http://liminality.xyz/the-hostile-email-landscape/ https://web.archive.org/web/20151121132739/http://liminality...? [3] https://news.ycombinator.com/item?id=10405681 https://news.ycombinator.com/item?id=10405681 [4] https://lobste.rs/s/ckfyqd/the_hostile_email_landscape https://lobste.rs/s/ckfyqd/the_hostile_email_landscape

danieltillett 10y ago
Self-hosting is dead unless you are massive business.
- moray 10y ago
  I have a very small (on the side) hosting business that has been running for almost 6 years now and growing little by little. I have direct contact with my clients and they have it with me in case of need (if they need technical advice or support not necessarily directly related to the services). And this is exactly the kind of service they were looking for, very difficult to have with a massive business company (have you ever opened a ticket with OVH?) Then of course, the technical challenges to setup and maintain such services are high and a control-panel all-included package or a docker with everything in it is just a very small part of the work required.
- pestaa 10y ago
  Self-hosting email or self-hosting in general?
  - danieltillett 10y ago
    Email, but I would not be surprised if self-hosting in general become in practice impossible - self-host and your site only shows up on page 100 in Google.
thalev 10y ago
Well this seems to be a dealbreaker, I can't have an email server that is likely to get blacklisted.
- kardos 10y ago
  I posted that in part hoping that someone would chime in with a workaround/solution figured out in the 6-7 months since that discussion.
- jasode 10y ago
  I happened to post comments in GP's cited thread and also another email server thread. You can read both of them to get a deeper understanding that personal email servers are hampered by "trust" issues more so than technical challenges. https://news.ycombinator.com/item?id=10405945 https://news.ycombinator.com/item?id=10405945 https://news.ycombinator.com/item?id=10498988 https://news.ycombinator.com/item?id=10498988 Basically, you could conceivably get past all the technical set up. There may even be Docker containers or virtual images to jumpstart most of the set up. Or you can follow some step-by-step instructions.[1] However, the problem is that SMTP servers originating from residential IP addresses by default are treated as home computers that have been hijacked by a malware spam bot. Using Bayesian reasoning, this is a rational filter because the number of zombie home computers sending unwanted spam vastly outnumbers any tiny amount of "good" people setting up legitimate SMTP servers. Given today's realities of distrust-by-default (unknown SMTP server is guilty until proven innocent), it looks like the best strategy for a "personal" SMTP server is to pay for a virtual machine at a hosting provider (Rackspace, etc) whose IP address ranges are not blacklisted. You could then run encrypted email storage there so Rackspace has no visibility into private emails. [1]http://arstechnica.com/information-technology/2014/02/how-to-run-your-own-e-mail-server-with-your-own-domain-part-1/ http://arstechnica.com/information-technology/2014/02/how-to...
- toomanythings2 10y ago
  I run email for most of my web clients. We have never been blacklisted and my email servers were set up without any help but the docs from day one.
moray 10y ago
There are a number of things that can (and should) be done to prevent outgoing emails from your SMTP server not to be rejected or flagged as SPAM (or worse your IP address blocked). A part from the obvious basics (IP reverse lookup working, TLS support, SMTP auth for your users..) you can also set up DKIM [1] and SPF [2] that help a lot and are well supported by big providers like gmail. But more importantly not ending up in a blacklist is important, so if you have users using your service you should monitor the volume of outgoing emails to avoid someone using your server to spam. There are a few tools that you can use to check your SMTP server MXToolbox and CheckTLS are the two I often use. [3] I agree though that managing an SMTP server is challenging, I had a couple of issues I had to work on that I am sure would never have happened to a big provider. [1] http://www.dkim.org http://www.dkim.org [2] http://www.openspf.org http://www.openspf.org [3] http://mxtoolbox.com http://mxtoolbox.com http://www.checktls.com/testreceiver.html http://www.checktls.com/testreceiver.html
- danieltillett 10y ago
  You can do all of these things and still a certain percentage of your emails will disappear into the ether. The game is rigged and the little guy has lost.
  - lj3 10y ago
    I'm willing to accept email is rigged. Who uses it for actual communication these days anyway? I use mine primarily to get lifecycle emails from webapps I tried once and to reset passwords. What's next, then? There have been lots of replacements for all of the old school protocols like IM and IRC. Even newsgroups have been (adequately?) replaced by forums and link aggregates and facebook. But, I've seen no attempt to replace email. I've seen lots of attempts to enhance email, but none to replace it completely.
    - mixedCase 10y ago
      >Who uses it for actual communication these days anyway? People outside of tech bubbles.
reasonattlm 10y ago
What this means in practice is that you have to set up with DKIM and SPF at minimum. E.g.: https://www.exratione.com/2016/05/a-mailserver-on-ubuntu-16-04-postfix-dovecot-mysql/ https://www.exratione.com/2016/05/a-mailserver-on-ubuntu-16-... But: you must then also use a host that is going to give you a decent chance at a non-polluted IP address. The host of the server appears to be a big factor in the hidden part of the anti-spam ecosystem used by the big email providers like Gmail and Hotmail. This means that trial and error and experience in the ever-changing field is the only way to make a good choice. Expect to be moving around a bit as you figure things out. AWS EC2 is the best choice that I know of. Digital Ocean is the worse choice that I know of. A possible approach is to set up your mailserver so that it relays outbound messages via AWS SES. SES has all of the advantages of a big service that puts in time to ensure deliverability, with few of the downsides, such as large cost. For calendar apps, you might look at Horde (see an old setup example here https://www.exratione.com/2012/05/a-mailserver-on-ubuntu-1204-postfix-dovecot-mysql/ https://www.exratione.com/2012/05/a-mailserver-on-ubuntu-120...) - I don't know how it matches up with other options. I moved away from it because I really didn't need a calendar as it turned out, and Horde is something like 10% application and 90% configuration system. It was a lot of work to get it set up and comfortable.
btym 10y ago
This was the problem I ran into when trying to self-host. Getting postfix etc set up was a pain in the ass, but I was getting flat-out refusal responses from big mail providers. Ended up pointing my MX records at google and paying them five buck a month to host it for me.
ajnin 10y ago
I don't have the same experience, I run my own mail server (that I wrote in python) on a personal domain and all it took was adding the correct SPF records to have my mail accepted by Google & co.
edoceo 10y ago
I've been running my own for years. On a VPS now. With SPF and DKIM my delivery rate is awesome. I host domain mail and send two large-ish newsletters (~4k each). Postfix, OpenDKIM, Dovecot in Linked. Granted, takes a bit to set up. Up-side is full control. I have a robust black-list too, keeps loads of spam from even entering the syatem

by kardos 10y ago

danieltillett 10y ago

Self-hosting is dead unless you are massive business.

moray 10y ago
I have a very small (on the side) hosting business that has been running for almost 6 years now and growing little by little. I have direct contact with my clients and they have it with me in case of need (if they need technical advice or support not necessarily directly related to the services). And this is exactly the kind of service they were looking for, very difficult to have with a massive business company (have you ever opened a ticket with OVH?) Then of course, the technical challenges to setup and maintain such services are high and a control-panel all-included package or a docker with everything in it is just a very small part of the work required.
pestaa 10y ago
Self-hosting email or self-hosting in general?
- danieltillett 10y ago
  Email, but I would not be surprised if self-hosting in general become in practice impossible - self-host and your site only shows up on page 100 in Google.

thalev 10y ago

Well this seems to be a dealbreaker, I can't have an email server that is likely to get blacklisted.

kardos 10y ago
I posted that in part hoping that someone would chime in with a workaround/solution figured out in the 6-7 months since that discussion.
jasode 10y ago
I happened to post comments in GP's cited thread and also another email server thread. You can read both of them to get a deeper understanding that personal email servers are hampered by "trust" issues more so than technical challenges. https://news.ycombinator.com/item?id=10405945 https://news.ycombinator.com/item?id=10405945 https://news.ycombinator.com/item?id=10498988 https://news.ycombinator.com/item?id=10498988 Basically, you could conceivably get past all the technical set up. There may even be Docker containers or virtual images to jumpstart most of the set up. Or you can follow some step-by-step instructions.[1] However, the problem is that SMTP servers originating from residential IP addresses by default are treated as home computers that have been hijacked by a malware spam bot. Using Bayesian reasoning, this is a rational filter because the number of zombie home computers sending unwanted spam vastly outnumbers any tiny amount of "good" people setting up legitimate SMTP servers. Given today's realities of distrust-by-default (unknown SMTP server is guilty until proven innocent), it looks like the best strategy for a "personal" SMTP server is to pay for a virtual machine at a hosting provider (Rackspace, etc) whose IP address ranges are not blacklisted. You could then run encrypted email storage there so Rackspace has no visibility into private emails. [1]http://arstechnica.com/information-technology/2014/02/how-to-run-your-own-e-mail-server-with-your-own-domain-part-1/ http://arstechnica.com/information-technology/2014/02/how-to...
toomanythings2 10y ago
I run email for most of my web clients. We have never been blacklisted and my email servers were set up without any help but the docs from day one.

moray 10y ago

There are a number of things that can (and should) be done to prevent outgoing emails from your SMTP server not to be rejected or flagged as SPAM (or worse your IP address blocked). A part from the obvious basics (IP reverse lookup working, TLS support, SMTP auth for your users..) you can also set up DKIM [1] and SPF [2] that help a lot and are well supported by big providers like gmail. But more importantly not ending up in a blacklist is important, so if you have users using your service you should monitor the volume of outgoing emails to avoid someone using your server to spam. There are a few tools that you can use to check your SMTP server MXToolbox and CheckTLS are the two I often use. [3] I agree though that managing an SMTP server is challenging, I had a couple of issues I had to work on that I am sure would never have happened to a big provider. [1] http://www.dkim.org http://www.dkim.org [2] http://www.openspf.org http://www.openspf.org [3] http://mxtoolbox.com http://mxtoolbox.com http://www.checktls.com/testreceiver.html http://www.checktls.com/testreceiver.html

danieltillett 10y ago
You can do all of these things and still a certain percentage of your emails will disappear into the ether. The game is rigged and the little guy has lost.
- lj3 10y ago
  I'm willing to accept email is rigged. Who uses it for actual communication these days anyway? I use mine primarily to get lifecycle emails from webapps I tried once and to reset passwords. What's next, then? There have been lots of replacements for all of the old school protocols like IM and IRC. Even newsgroups have been (adequately?) replaced by forums and link aggregates and facebook. But, I've seen no attempt to replace email. I've seen lots of attempts to enhance email, but none to replace it completely.
  - mixedCase 10y ago
    >Who uses it for actual communication these days anyway? People outside of tech bubbles.

reasonattlm 10y ago

What this means in practice is that you have to set up with DKIM and SPF at minimum. E.g.: https://www.exratione.com/2016/05/a-mailserver-on-ubuntu-16-04-postfix-dovecot-mysql/ https://www.exratione.com/2016/05/a-mailserver-on-ubuntu-16-... But: you must then also use a host that is going to give you a decent chance at a non-polluted IP address. The host of the server appears to be a big factor in the hidden part of the anti-spam ecosystem used by the big email providers like Gmail and Hotmail. This means that trial and error and experience in the ever-changing field is the only way to make a good choice. Expect to be moving around a bit as you figure things out. AWS EC2 is the best choice that I know of. Digital Ocean is the worse choice that I know of. A possible approach is to set up your mailserver so that it relays outbound messages via AWS SES. SES has all of the advantages of a big service that puts in time to ensure deliverability, with few of the downsides, such as large cost. For calendar apps, you might look at Horde (see an old setup example here https://www.exratione.com/2012/05/a-mailserver-on-ubuntu-1204-postfix-dovecot-mysql/ https://www.exratione.com/2012/05/a-mailserver-on-ubuntu-120...) - I don't know how it matches up with other options. I moved away from it because I really didn't need a calendar as it turned out, and Horde is something like 10% application and 90% configuration system. It was a lot of work to get it set up and comfortable.

btym 10y ago

This was the problem I ran into when trying to self-host. Getting postfix etc set up was a pain in the ass, but I was getting flat-out refusal responses from big mail providers. Ended up pointing my MX records at google and paying them five buck a month to host it for me.

ajnin 10y ago

I don't have the same experience, I run my own mail server (that I wrote in python) on a personal domain and all it took was adding the correct SPF records to have my mail accepted by Google & co.

edoceo 10y ago

I've been running my own for years. On a VPS now. With SPF and DKIM my delivery rate is awesome. I host domain mail and send two large-ish newsletters (~4k each). Postfix, OpenDKIM, Dovecot in Linked. Granted, takes a bit to set up. Up-side is full control. I have a robust black-list too, keeps loads of spam from even entering the syatem