6 ms·

Over the last year, I ended a 20-plus year long run of hosting my own email (and email for a handful of other people and businesses I had a relationship with),

Over the last year, I ended a 20-plus year long run of hosting my own email (and email for a handful of other people and businesses I had a relationship with), entirely because of Gmail's behavior. People here saying "it's spam, not Gmail" are being distracted from the numerous issues that independent mail services do have with Gmail. Gmail is extremely uncooperative at accepting email from services that aren't Gmail, Comcast (sometimes), or Microsoft. You can have everything configured correctly, on an IP you've owned for years, and aggressively manage any outbound spam, and Gmail will still hate your guts and bounce your email or file it in the recipient's Junk folder. Before Gmail got huge, email service providers typically offered an avenue for addressing false-positives in their filtering systems. Gmail really pioneered the "nah, screw you" approach to this. Meanwhile, Gmail is itself a huge source of spam. I (maybe perversely) loved hosting my email and email for a handful of other people. It's fun. Gmail took all the fun out of it and turned it into a seething hatred.

mattw2121 3y ago
Exactly this. I enjoyed the technical aspects of running my own mail server. I hated having to constantly manage black lists and dealing with "postmasters" to get my email to be consistently delivered. Some spammer manages to hack a server on the same IP block as you equals trouble for months. I just couldn't justify the time spent and the non-delivery any longer.
- croutonwagon 3y ago
  Even google is aggressive about IP bans on other services. There was a time my home IP on Comcast was on some blacklist with good. Every video was run through captcha. Searches too. Comcast was useless to give me an IP in a new block. What solved it? I signed up for gsuite free at the time and moved my email (from a colo) from “on prem” to them. Suddenly my home IP that’s used to access their services is cherry and no longer suspicious. Note: that doesn’t seem to work anymore though. I regularly proxy some traffic through a linode and google does the same thing. Everything behind a captcha that’s stupid difficult to clearly 60% of the time. I hate traffic lights.
jimmaswell 3y ago
I just recently started self hosting email. I settled on relaying through gmail to all hosts except a whitelist of hosts I know won't blackhole me - mostly just a few niche domains of friends who do the same thing or small organizations they're a part of. Have you considered this? It's not optimal but it's a practical compromise. You get to keep your From: field as your domain too if you set it up right. I couldn't make it into Microsoft inboxes at all before I set up the relay but now I can. - Make a gmail account just for your email server, which forwards anything incoming to your host in case someone emails it directly (you'll be able to discover this gmail address if you dig through delivered emails' headers but it won't be in the From: field) - Let Gmail authenticate with your SMTP server in Gmail's advanced options, and make sure the options are checked to retain the From: headers of relayed emails - Generate an App Password and set up your mail server software to use Google's SMTP relay with the email and app password There are other services to do this too I'm sure, but I'm happy with Gmail for now. And you can always transparently switch it out to another service if Google pulls something. I plan to write a blog post on the whole process of setting up a mail server to configuring it with a gmail relay like this.
- thaumaturgy 3y ago
  I did consider that as an option, and I tried routing email through a number of other delivery services, but ultimately I rejected it for a few reasons: I was hosting email not just for myself but for a few other people, and that gets tricky to route through Gmail; Gmail could change their policies at any time and give me a really bad day, potentially when I can't respond to it in a timely manner; If routing email in this way ever triggers Google's abuse mechanisms, then potentially I'm losing access to a lot of the Google network, and while I don't use it for anything personally, sometimes I have to work with companies that do; After spending tons of hours dealing with Gmail-related headaches despite not using Gmail myself, relying on them to get mail routed felt like a deal with the devil. I've instead helped a bunch of people get set up with Fastmail. I love Fastmail, they're great, I miss hosting my own email but they're the next best thing. Fastmail must be handling enough traffic that they're hovering above Gmail's piss-off threshold, and really my experience with them has been extraordinarily good. Everyone I've set up over there has been happy with them too, save for one person who got told by the next IT hat-wearer that "everyone's using Google Workspace and you should be too" (and then immediately ran into a problem during setup that snowballed into a big hairy mess).
arbitrage 3y ago
I doggedly still run my 20+ year email platform for self-use. I ran into the same problems with Google/Gmail. I am ashamed to say that the solution I settled on was to just give up and pay them for smarthosting. I still host my own servers, but my exit pipe is through gmail. My life got easier. The rule is pretty much every interaction you have in the Googleverse is easier if you just pay them something. It's pay to play internet, and yeah, it's a problem.
- dave_walko 3y ago
  Can you explain why? I looked this up and still am clueless as to why.
- jabagonuts 3y ago
  First of a > It's pay to play internet, and yeah, it's a problem. 1. How much do they charge? I'm genuinely curious. I don't self host, but use a 3rd party (fastmail). I send very few emails to people I don't know, so personally, I don't run into issues with having my email sent to spam. 2. I don't think paying in itself is the real problem. I think it's more a matter of who you pay and why you pay. - You have to pay to register a domain name. - You have to pay to host your own server (whether your using a hosting service or hosting from your basement) - You have to pay to have gmail not mark your email as spam - ok, I'll admit, this is a little silly, but you also have to pay (via a stamp) to have USPS send letters to their recipient 3. Perhaps because so many people use and trust (whether they should or not is another question) gmail, it makes sense to pay in some scenarios? But obviously, for personal mail servers, I agree, asking to pay to play is a bit of a stretch.
nine_k 3y ago
I suppose that the only real problem is sending, not receiving, nor setting up software. If I were to set up my own mail server, that would send outbound email through Sendgrid, AWS SES, or maybe some other established and trusted sender. Building trusts with behemoths like GMail or outlook.com is too involved an affair to be worth it in the general case, and not even through some malice on the side of the latter; it's just the reality of a medium not protected from spam.
- thaumaturgy 3y ago
  I tried both Sendgrid and Mailgun and couldn't reliably get mail to Gmail through either one. FWIW, I used to get a fair bit of inbound spam from Sendgrid IPs, so they floated in and out of my greylist on the regular and I wasn't too surprised when they didn't turn out to be a good option for my outbounds. I was surprised about Mailgun; I don't recall receiving any bad traffic from them ever, and the couple of times I've used them, they seem to focus a lot of energy on not being a source of trouble.
  - jjav 3y ago
    > I tried both Sendgrid and Mailgun and couldn't reliably get mail to Gmail through either one. I mean, you can't reliably send email to gmail from gmail. It's not really the origin that's the issue, it is that gmail spam classification is quite bad. Good emails from known recipients you've corresponded actively for years will continue going to spam in gmail even as you mark them not-spam for the thousandth time. So yes, sending to anyone @gmail.com has decents odds of ending up in spam for no reason. That's just how gmail is. But no, sending it from your own email server doesn't make things worse. I run all my own email infrastructure and sending to gmail addresses works just as well from my server as it does from gmail itself.
- takeda 3y ago
  > Building trusts with behemoths like GMail or outlook.com is too involved There's nothing like that actually. I have e-mail server running since 2000s. When gmail appeared, everything continued to work as expected and it did for many years. Around 2018 or 2019 I noticed that my e-mails started going to spam folder. I didn't send any bulk e-mails, I was not on any RBLs, I was not compromised (I actually was very careful to block outbound SMTP from any user except the one running MTA), and of course had SPF & DKIM set up. It looks like their spam filtering is just arbitrary and feels like done on purpose to discourage running personal mail servers (and looking at comments it works really well). Honestly, I think some kind of campaign is needed to put them in place, like starting blocking e-mails from gmail (maybe responding with a message encouraging to switch account). I remember in the past steps like that were done, but feels like today people are more acceptable of centralization.
heleninboodler 3y ago
> Gmail really pioneered the "nah, screw you" approach to this. From my POV, Yahoo pioneered this way before Gmail. 10 years ago when I was managing outgoing mail infra for a company that probably sent 100k emails per day (mostly purchase receipts and subscription notifications), I was intimately aware of deliverability characteristics for Gmail and Yahoo, which were our two biggest destinations by a large margin (if I recall, Gmail was about 60% and Yahoo was about 30%). Gmail, despite being the overwhelming majority of our traffic, was hardly any problem at all, whereas Yahoo would regularly give us the dreaded 451 response that indicated that your mail was going to be held for 2-24 hours and then randomly rejected or accepted. BTW, if I were to run my own mail infrastructure again, I'd definitely use someone like Sendgrid with a dedicated outgoing IP with a clean reputation. It makes a very big difference. We eventually ended up with our infrastructure split across multiple IPs by function (e.g. receipt emails were the only thing ever sent from one specific IP because we wanted to ensure deliverability).
- tzs 3y ago
  The trick to get reliably mail delivery to Yahoo back in the day was to be spending several thousand dollars a month buying ads through Yahoo. Then when your mails to Yahoo got repeatedly blocked by Yahoo and you got tired of having to repeatedly report the false positives and and get unblocked only to get blocked again a few weeks later you have the person in charge of your ad spending call up your ad sales rep at Yahoo. You asked the ad sales rep "Why should we keep spending $X/month on Yahoo ads to try to acquire customers, when Yahoo mail keeps blocking the receipt emails, setup instruction emails, and response emails to support questions we try to send to those customers which angers those customers and they cancel (or even try chargebacks)???". What happened then is the sales rep puts your ad manager on hold for a few moments, then comes back with someone high up in Yahoo IT management conferenced in, and explains the situation. The high IT manager puts you all on hold, and comes back in a few minutes with someone who actually deals with maintaining the spam filters conferenced in, explains the situation, and tells the maintainer to add your domain to a special whitelist of mail that is always accepted.
stjohnswarts 3y ago
If google can't automate it then it doesn't exist for the most part. They bend over backwards not to deal with actual humans, and they're so big now they could not care less if they lose -1 human or business, it simply doesn't matter unless you are rich enough to know someone in the 1st tier executive team.

mattw2121 3y ago

Exactly this. I enjoyed the technical aspects of running my own mail server. I hated having to constantly manage black lists and dealing with "postmasters" to get my email to be consistently delivered. Some spammer manages to hack a server on the same IP block as you equals trouble for months. I just couldn't justify the time spent and the non-delivery any longer.

croutonwagon 3y ago
Even google is aggressive about IP bans on other services. There was a time my home IP on Comcast was on some blacklist with good. Every video was run through captcha. Searches too. Comcast was useless to give me an IP in a new block. What solved it? I signed up for gsuite free at the time and moved my email (from a colo) from “on prem” to them. Suddenly my home IP that’s used to access their services is cherry and no longer suspicious. Note: that doesn’t seem to work anymore though. I regularly proxy some traffic through a linode and google does the same thing. Everything behind a captcha that’s stupid difficult to clearly 60% of the time. I hate traffic lights.

jimmaswell 3y ago

I just recently started self hosting email. I settled on relaying through gmail to all hosts except a whitelist of hosts I know won't blackhole me - mostly just a few niche domains of friends who do the same thing or small organizations they're a part of. Have you considered this? It's not optimal but it's a practical compromise. You get to keep your From: field as your domain too if you set it up right. I couldn't make it into Microsoft inboxes at all before I set up the relay but now I can. - Make a gmail account just for your email server, which forwards anything incoming to your host in case someone emails it directly (you'll be able to discover this gmail address if you dig through delivered emails' headers but it won't be in the From: field) - Let Gmail authenticate with your SMTP server in Gmail's advanced options, and make sure the options are checked to retain the From: headers of relayed emails - Generate an App Password and set up your mail server software to use Google's SMTP relay with the email and app password There are other services to do this too I'm sure, but I'm happy with Gmail for now. And you can always transparently switch it out to another service if Google pulls something. I plan to write a blog post on the whole process of setting up a mail server to configuring it with a gmail relay like this.

thaumaturgy 3y ago
I did consider that as an option, and I tried routing email through a number of other delivery services, but ultimately I rejected it for a few reasons: I was hosting email not just for myself but for a few other people, and that gets tricky to route through Gmail; Gmail could change their policies at any time and give me a really bad day, potentially when I can't respond to it in a timely manner; If routing email in this way ever triggers Google's abuse mechanisms, then potentially I'm losing access to a lot of the Google network, and while I don't use it for anything personally, sometimes I have to work with companies that do; After spending tons of hours dealing with Gmail-related headaches despite not using Gmail myself, relying on them to get mail routed felt like a deal with the devil. I've instead helped a bunch of people get set up with Fastmail. I love Fastmail, they're great, I miss hosting my own email but they're the next best thing. Fastmail must be handling enough traffic that they're hovering above Gmail's piss-off threshold, and really my experience with them has been extraordinarily good. Everyone I've set up over there has been happy with them too, save for one person who got told by the next IT hat-wearer that "everyone's using Google Workspace and you should be too" (and then immediately ran into a problem during setup that snowballed into a big hairy mess).

arbitrage 3y ago

I doggedly still run my 20+ year email platform for self-use. I ran into the same problems with Google/Gmail. I am ashamed to say that the solution I settled on was to just give up and pay them for smarthosting. I still host my own servers, but my exit pipe is through gmail. My life got easier. The rule is pretty much every interaction you have in the Googleverse is easier if you just pay them something. It's pay to play internet, and yeah, it's a problem.

dave_walko 3y ago
Can you explain why? I looked this up and still am clueless as to why.
jabagonuts 3y ago
First of a > It's pay to play internet, and yeah, it's a problem. 1. How much do they charge? I'm genuinely curious. I don't self host, but use a 3rd party (fastmail). I send very few emails to people I don't know, so personally, I don't run into issues with having my email sent to spam. 2. I don't think paying in itself is the real problem. I think it's more a matter of who you pay and why you pay. - You have to pay to register a domain name. - You have to pay to host your own server (whether your using a hosting service or hosting from your basement) - You have to pay to have gmail not mark your email as spam - ok, I'll admit, this is a little silly, but you also have to pay (via a stamp) to have USPS send letters to their recipient 3. Perhaps because so many people use and trust (whether they should or not is another question) gmail, it makes sense to pay in some scenarios? But obviously, for personal mail servers, I agree, asking to pay to play is a bit of a stretch.

nine_k 3y ago

I suppose that the only real problem is sending, not receiving, nor setting up software. If I were to set up my own mail server, that would send outbound email through Sendgrid, AWS SES, or maybe some other established and trusted sender. Building trusts with behemoths like GMail or outlook.com is too involved an affair to be worth it in the general case, and not even through some malice on the side of the latter; it's just the reality of a medium not protected from spam.

thaumaturgy 3y ago
I tried both Sendgrid and Mailgun and couldn't reliably get mail to Gmail through either one. FWIW, I used to get a fair bit of inbound spam from Sendgrid IPs, so they floated in and out of my greylist on the regular and I wasn't too surprised when they didn't turn out to be a good option for my outbounds. I was surprised about Mailgun; I don't recall receiving any bad traffic from them ever, and the couple of times I've used them, they seem to focus a lot of energy on not being a source of trouble.
- jjav 3y ago
  > I tried both Sendgrid and Mailgun and couldn't reliably get mail to Gmail through either one. I mean, you can't reliably send email to gmail from gmail. It's not really the origin that's the issue, it is that gmail spam classification is quite bad. Good emails from known recipients you've corresponded actively for years will continue going to spam in gmail even as you mark them not-spam for the thousandth time. So yes, sending to anyone @gmail.com has decents odds of ending up in spam for no reason. That's just how gmail is. But no, sending it from your own email server doesn't make things worse. I run all my own email infrastructure and sending to gmail addresses works just as well from my server as it does from gmail itself.
takeda 3y ago
> Building trusts with behemoths like GMail or outlook.com is too involved There's nothing like that actually. I have e-mail server running since 2000s. When gmail appeared, everything continued to work as expected and it did for many years. Around 2018 or 2019 I noticed that my e-mails started going to spam folder. I didn't send any bulk e-mails, I was not on any RBLs, I was not compromised (I actually was very careful to block outbound SMTP from any user except the one running MTA), and of course had SPF & DKIM set up. It looks like their spam filtering is just arbitrary and feels like done on purpose to discourage running personal mail servers (and looking at comments it works really well). Honestly, I think some kind of campaign is needed to put them in place, like starting blocking e-mails from gmail (maybe responding with a message encouraging to switch account). I remember in the past steps like that were done, but feels like today people are more acceptable of centralization.

heleninboodler 3y ago

> Gmail really pioneered the "nah, screw you" approach to this. From my POV, Yahoo pioneered this way before Gmail. 10 years ago when I was managing outgoing mail infra for a company that probably sent 100k emails per day (mostly purchase receipts and subscription notifications), I was intimately aware of deliverability characteristics for Gmail and Yahoo, which were our two biggest destinations by a large margin (if I recall, Gmail was about 60% and Yahoo was about 30%). Gmail, despite being the overwhelming majority of our traffic, was hardly any problem at all, whereas Yahoo would regularly give us the dreaded 451 response that indicated that your mail was going to be held for 2-24 hours and then randomly rejected or accepted. BTW, if I were to run my own mail infrastructure again, I'd definitely use someone like Sendgrid with a dedicated outgoing IP with a clean reputation. It makes a very big difference. We eventually ended up with our infrastructure split across multiple IPs by function (e.g. receipt emails were the only thing ever sent from one specific IP because we wanted to ensure deliverability).

tzs 3y ago
The trick to get reliably mail delivery to Yahoo back in the day was to be spending several thousand dollars a month buying ads through Yahoo. Then when your mails to Yahoo got repeatedly blocked by Yahoo and you got tired of having to repeatedly report the false positives and and get unblocked only to get blocked again a few weeks later you have the person in charge of your ad spending call up your ad sales rep at Yahoo. You asked the ad sales rep "Why should we keep spending $X/month on Yahoo ads to try to acquire customers, when Yahoo mail keeps blocking the receipt emails, setup instruction emails, and response emails to support questions we try to send to those customers which angers those customers and they cancel (or even try chargebacks)???". What happened then is the sales rep puts your ad manager on hold for a few moments, then comes back with someone high up in Yahoo IT management conferenced in, and explains the situation. The high IT manager puts you all on hold, and comes back in a few minutes with someone who actually deals with maintaining the spam filters conferenced in, explains the situation, and tells the maintainer to add your domain to a special whitelist of mail that is always accepted.

stjohnswarts 3y ago

If google can't automate it then it doesn't exist for the most part. They bend over backwards not to deal with actual humans, and they're so big now they could not care less if they lose -1 human or business, it simply doesn't matter unless you are rich enough to know someone in the 1st tier executive team.