9 ms·

I run a few moderately sized corporate e-mail systems. SPF, DKIM, DMARC are all necessities, but unfortunately they don't guarantee delivery. We have much bette

I run a few moderately sized corporate e-mail systems. SPF, DKIM, DMARC are all necessities, but unfortunately they don't guarantee delivery. We have much better success sending e-mail out via Mailgun or Sendgrid. Since Verizon swallowed up Yahoo Mail and @aol.com, it's very easy to get on their shadow ban list, and nearly impossible to get off. Good luck telling all the employees they can't email @yahoo.com and @aol.com for at least 2 weeks. I see e-mail as another Internet service that is slowly centralizing, and if you operate outside of the big tech companies- for example, if you don't use Google/Yahoo/M365 e-mail, you're probably going to be in for a headache, even if you run perfect e-mail infrastructure.

sebmellen 6y ago
Sadly I've found this to be the case both with perfectly configured personal email servers and when using privacy-preserving email providers like Tutanota. Do you think there's anything that can be done to counter this centralizing force?
- da_chicken 6y ago
  Now? Almost certainly not. On the side of centralization is a market worth unimagined billions, and perhaps seven providers that already routinely lobby to get what they want. On the side of decentralization? Little guys who are already being bullied and don't have the resources. Politicians who have other extremely dangerous problems currently, including and especially those in their own industry. Don't expect their help for a long time. Smaller service providers who only want the chance to switch sides. And large independent organizations (businesses, schools, governments) who are really only interested in a cheaper bill for the same service quality. Look, we have struggled to fix the MITM/plain text problem and spam/origin problem for well over 20 years, and the Internet Age is maybe 30? It's extremely likely that the only way to fix the core design problems with email is to have the service consolidate to a small consortium of services, have them improve the base design, and then fracture the monopolies. That's going to take decades. And mail is still such a pain in the butt for a user document database. And the user experience is a joke! Asynchronous delivery, best effort failure? I mean, what? The average web page is orders of magnitude larger now! People complain about FAX and FTP.
  - nirui 6y ago
    The three protocols for email (SMTP/POP/IMAP) are all outdated really badly. People today needs more efficient, multiplexed and real time message RX/TX, and those tree just not going to cut it. A new, chat-styled mail protocol can be designed to address all the problems above while reduce the risk of abuse. I wonder why nobody standing up to do it.
    - marcbradshaw 6y ago
      SMTP is going to be with us for a long while yet, but some folks are standing up to do something about IMAP - https://jmap.io/ https://jmap.io/
- jabroni_salad 6y ago
  These configurements do not signal "This person sends good emails", they signal "This server is what it says it is". I can load up barracuda right now and show you plenty of pornbots that have figured out how to put in a passing SPF record... as well as legitimate businesses that don't validate email addresses put into their marketing funnels. Unfortunately, we are back to the age old question. How can we have an ecosystem where anybody can email anybody without it being taken over by spam?
  - trhway 6y ago
    Dint proof of work originated as a way of fighting spam? I wonder though why it didn't become widespread. Instead of shadowbanning/blocking the providers like VRZ could just increase the required POW difficulty.
    - marcbradshaw 6y ago
      and of course spammers would never have access to vast botnets to do that work. POW is (imho) a terrible idea.
- belorn 6y ago
  I am actually a bit more optimistic that the iron grip over email might be attacked from an unexpected direction. A lot of political parities over the world depend on email to operate internally, and the power demonstrated recently by cloud providers in the last few months are making people scared. It only take some publicity and some badly timed bans/lockouts from google before that risk seems much more real again.
endgame 6y ago
I moved my mail away from Google last year, and can no longer accept people's calendar invites. It seems that the "Yes"/"No"/"Maybe" links in the email body of a calendar invite cause Google's (or whoever's) servers to spoof an email in my name, which predictably fails if my SPF is set strictly. I have not been able to find a good workaround for this.
- fjni 6y ago
  I had something similar so perhaps this helps: If you still have a google account that has the “calendar” feature enabled, this will lead to google sending invite acceptance emails. Especially if you have a business account (which was my experience,) where you verified the domain. The solution for me was to disable “calendar” as a service on the entire account. Realize that “google calendar” and “google mail,” are technically distinct services. However that doesn’t seem to stop “calendar” from sending emails on your behalf.
- behringer 6y ago
  couldn't you just add google as an approved sender for your domain? https://dmarcly.com/blog/spf-dkim-dmarc-set-up-guide-for-g-suite-gmail-for-business https://dmarcly.com/blog/spf-dkim-dmarc-set-up-guide-for-g-s... I doubt google will send out spam on your domain.
  - chalst 6y ago
    I added them as neutral (the '?' character) rather than an allowed domain: I don't want to have to understand Google's mailing policy when assessing the risk of my site being downgraded.
briHass 6y ago
Unfortunately, I agree 100% Even using SendGrid, it will be a week or two and a few hundred emails to 'warm up' one of their mailserver's IP addresses and get reliable delivery. Like you, I found VZ to be the worst at trusting new servers. The days of firing off an email from any ol' SMTP server are completely gone.
- tomxor 6y ago
  > The days of firing off an email from any ol' SMTP server are completely gone. I'm not sure... I agree with the general vibe in this thread however my experience (within the last year) is that you can fire up a fresh one and send email to gmail at least, without SPF, DKIM or DMARC. However as soon as you start doing any mass mailing of any kind you need all three of those things, SPF to not get outright blocked and get into the spam, and the other two to have their spam detection filters to even bother parsing your email content for the chance of getting into the inbox. This is my experience from setting up systems that send automated status emails not marketing, so fairly low volume but frequent enough with same message content to be immediately classified as mass emailing.
  - roblabla 6y ago
    I've never had issues with gmail, though I know a few people who did. But I think setting up SPF and DKIM will, for the most part, do the trick to allow communicating with gmail users. The problem is the other provider. Yahoo, AOL, Microsoft. They're all a pain in the neck and much, much stricter. And sadly, they still amount to a not insignificant portion of the market.
bobflorian 6y ago
I actively work with and set up ~100 domains in SendGrid, and Yahoo and AOL are still, have have been, THE BANE OF MY EXISTENCE since I started working at this job 8 years ago. It's always been a black hole into any type of support. I always get the same answer back... "If you don't want to be seen as a spam bot, don't act like one".... OK, I'll just tell my client they can't send their 20k emails when they want (with a non insignificant going to Yahoo/AOL still). Honestly I don't know what else we can do better. We don't send newsletters and such, we only send transnational emails.
- _nickwhite 6y ago
  The struggle is real. I found Mailgun to be pretty reliable for yahoo/aol/verizon domains, but my volume isn’t crazy big.
  - throwawayboise 6y ago
    I tried Mailgun about 5 years ago. I had a short term, low-volume need, but found that their free tier servers had poor reputations and I remember that Yahoo in particular would not deliver their email. Maybe that's to be expected: any free email sending service is going to be abused by spammers. But it wasn't a great first impression.
- dkdk8283 6y ago
  PSA: Don’t accept job as mail sysadmin.
  - vishnugupta 6y ago
    We are a small SaaS start-up with 4 engineers in total, I'm one of them. Being a senior person, I've to deal with all kinds of crap; SFP, DKIM, domain validation and what not. Just when I relax content that things are in control I get a folder full of files which contain spaces in the file names that need to be processed.
- darylteo 6y ago
  Mandrill has been reliable, but more expensive, when it comes to AOL Yahoo Hotmail and Outlook. (I work at an agency, we recently migrated off sendgrid to mandrill for all our clients due to deliverability issues)
acidburnNSA 6y ago
Same. I run a 10/10 (mail-tester.com) mail server for myself personally on a VPS from Digital Ocean. Its IP address is on Charter's permanent blocklist and has been for years so I cannot email my mom. I set up a SMTP relay via a free-tier sendgrid and it worked for a while but then that relay got added to spamlists and I could basically only email my mom. Presumably the paid sendgrid setups allow a bit more resilience against getting on those lists.
- pbhjpbhj 6y ago
  I reckon your mom is reporting you as a spammer. /jk
  - bombcar 6y ago
    My sister obviously thinks the “spam” button is the same as “delete email” button. Every once in awhile I get the Google apps for domains notification “tons of spam has been marked”.
- walrus01 6y ago
  Any high volume vps, vm, dedicated server, Colo hosting company is going to have this problem with ip space reputation. The only place to fully self host your own email these days is on a more boutique isp where it's not possible for anyone with a credit card and a pulse to sign up as a customer in a fully automated process.
thaumaturgy 6y ago
Just this last weekend I started moving clients off of the mail hosting services I've managed for years and onto Fastmail. I have self-hosted my email since around 2005, and had my first email address around 1994 or thereabouts. It has never been more broken than it is now, and I mean that without even the slightest hyperbole. The first RBL showed up around 1998, operated by Paul Vixie of DNS fame. I was using a small ISP in the East Bay at the time and they hated the MAPS RBL, especially the idea that they had to deal with some third party to resolve mail transport issues. But, from 1998 until the early 2000s, RBLs proliferated and lots of mail services subscribed to one or more. The one thing they had going for them was that they were easy to query, so it took no effort to find out if you were on the list, and if you were, there was usually a living, breathing human being somewhere that you could contact to get the matter resolved. It was a pain in the butt, but it could be handled. During this time, the responsibility for ensuring that a message was received shifted from the recipient to the sender. Beleaguered systems administrators soon found themselves in a situation where they were supposed to be responsible for both ensuring that no spam reached their customers and that all of their customers' email reached the intended recipients. Gmail came along and decided that, because they were operating "at scale", they didn't need to play in the same ecosystem. Over the years, ensuring that a message lands in a Gmail user's inbox has turned in to an infuriating game of trial-and-error. Gmail can do this because they now manage between 40% and 60% of the internet's email traffic. AT&T/SBCGlobal/Yahoo/whoever they are now seem to have recently penalized all of Linode's and DigitalOcean's IP space. I deployed several mail exchanges and didn't have any luck reaching any addresses managed by AT&T's network. And, again, there's nobody I can kibbutz with to resolve it. AOL has been such a hot tire fire that I ended up blackholing any outbound traffic to them. I know that sounds drastic, but anytime a single AOL user clicked the "spam" button on an email from one of my customers -- who weren't spamming, newslettering, or anything else remotely skeezy -- it would generate an automated complaint from AOL to Linode, and Linode would threaten to suspend my account. I'd have to dig the relevant traffic out of the logs and respond back to Linode with a polite "AOL's full of shit, please stop listening to them". I explained the situation to the few people that were impacted by it and everybody got on with their lives. Microsoft's outlookprotection.com filter has been a gigantic pain recently too. It's annoyingly capricious and, again, the tools just aren't there to resolve it. Email delivery has been a bit tricky for several years, but the last year especially it has become impossible for small services. You have SPF, DKIM, DMARC, great, but it turns out that Gmail also dings you if you don't have ipv6 records arranged right or if you aren't transporting mail over ssl or or or or. Email was designed as a cooperative system but the BigCos have carved it up and are working hard to ensure that if a message doesn't come from one of their networks, then it's immediately suspicious. Sendgrid isn't much better in this regard. My network received a flood of spam from them, with legitimate traffic mixed in. I couldn't block them without complaints and I couldn't not block them without complaints. I wondered if I was the only, so I signed up for their service and routed some mail through them for a few days to see what it was like as one of their subscribers. Turns out that Comcast and half a dozen other service providers hate them just as much and deliverability was around 79%. The things you and others are experiencing, and that I experienced, are going to keep getting worse. The bigger networks are going to keep squeezing customers out of the smaller ones, breaking email bit by bit in the process. I hope Fastmail is able to grow quickly enough to keep sitting at the table.
karmicthreat 6y ago
Also, get a private IP from your email service. Don't use their public ips. Or you will cry as the IP they have you on will randomly get put on SpamCop. Even with my private IPs I've had random anti-spam services just decide to list all Sendgrid IPs.
- iamacyborg 6y ago
  Yeah, no. Unless you're sending sufficiently high volume (ie millions of emails a week) or are sending B2B to exchange servers, getting a private IP is a waste of money and could negatively affect your sender reputation more than using a shared IP. Spam filters have significantly advanced beyond simple IP checks.
  - karmicthreat 6y ago
    We can agree to disagree here. You have no recourse when one of the Sendgrid shared IPs you are using gets put on Spamcop. Sendgrid might get around to working with them to remove it in a couple weeks. You also have no way to eliminate that IP from your pool of IPs. So in my case about 40% of reports I was sending out were getting blocked at the email server of the receiver. Buying a private IP is your only recourse. Unless your emails don't matter, in which case it might be fine to lose 40% of them.
    - yread 6y ago
      This is my experience as well. Submitted a support request, after a week got told "we're working on it" while my users were not receiving email address confirmation emails. Moved away from SendGrid very quickly after that.
      - karmicthreat 6y ago
        I have some legacy IoT devices in the field that directly talk to sendgrid. Would be a big pain if I had to recredential them. So I stayed on sendgrid. My newer devices get their email proxied through a proxy I host. It was a dumb mistake on my part to tie myself to so tightly to a SaaS vendor.
throwawayboise 6y ago
Not centralizing -- these providers are colluding in a oligopoly. I think there is an argument to be made that these large providers are creating a situation where they set up barriers of trust that only allow email from each other to pass. Small independent senders are locked out simply by not being part of the trusted club, even if their email is valid and passes all of the standard hurdles (SPF, DKIM, etc).
- jjeaff 6y ago
  They are doing that. But it doesn't seem to be in order to capture market share. It is because of the extreme spam abuse coming from so many unknown domains.
fogihujy 6y ago
The sad reality is that you should expect to be delivered into the recipients' spam boxes until you have managed to convince Google/Microsoft/Yahoo that you're not a spammer. As far as I can tell, it simply is not possible to buy a new domain, pay for a email service and have mails delivered straight into people's inboxes. After all, if you would be able to do that then so could the spammers. Ramp up volumes slowly. Make sure the recipients actually want the mails you send, and for all that is holy, educate your users that the "Junk" and "Trashcan" buttons are two very different things. People who repeatedly mark your mails as spam need to be blacklisted, no matter if they're paying customers. Period. Oh, and handle abuse complaints right away. Don't be afraid to tell people they're not supposed to mark your mails as spam and to explain the consequences.
- chillfox 6y ago
  The biggest difference comes from actually owning the block of IP addresses that you use instead of renting them.
- avereveard 6y ago
  > educate your users that the "Junk" and "Trashcan" buttons are two very different things. we have 97% reputation on sendgrid, and the only mail we send are a) customer requested password reset and b) customer requested notification for completed workflow jobs. I don't know who that 3% is, but it is enough to get sometime filtered and I don't think we can ever win this
  - fogihujy 6y ago
    Sendgrid itself is a major source of spam, and simply using them can result in mails getting filtered. There are even Spamassassin rules (3rd party) that specifically add spam score to mails, if they are sent through Sendgrid. The 3% may very well be ignorant users who are using the Junk button as a way to remove the mails from the inbox (yeah, people actually do that), but for good measure, things like signup forms and password reset forms really should be protected with a captcha, as any form able to send any form of email whatsoever will be abused by bots trying to send spam.
jjav 6y ago
Which is why it is so vital for anyone with at least minimal tech know-how to run their own email infrastructure. The Internet is only a meaningful concept if it consists of peers running standard decentralized protocols. If all we had was a few giant corporations with proprietary apps and proprietary protocols, that's not the Internet. As I post every time on these email discussions, the difficulty of running your own email infrastructure is vastly overstated on HN. The more people do it, the better off the Internet world is for all of us.
- roblabla 6y ago
  I have run my own email infrastructure for 3 years, for personal use (I was the only user). I have since moved to using fastmail due to my mail failing to be delivered to any microsoft email. Microsoft would accept the mail, but it would never appear in the mailbox, and neither would it arrive in spam. The mail just disappeared somewhere in their system. I have spent many days debugging the issue, to no avail. Every other operator accepts my email, and my domain was never used to send out spam (I set up DMARC to make sure I am aware of all outgoing email). I imagine the IP might have been the issue, but I cycled through a few hosts without success. Everything was configured correctly AFAICT, I could send email to all major providers, except microsoft. SPF, DKIM, DMARC were all set up. I tried many different configuration testers, and they all returned green on my domain. Since migrating to fastmail (keeping the same domain), I haven't had any problem. The thing is, I agree with you, the world would be a better place if everyone could host their own email. But at the end of the day, I need the ability to contact users using the major email providers, and so do I suspect many HN users.
- nor-and-or-not 6y ago
  I completely agree! I'm running an email server for myself since 1998 and in addition some low traffic email servers for a few small companies that I administrate and we have had no delivery problems so far, not even with Google, although I have heard from other people that run their own email servers, that Google occasionally puts delivered emails in the Spam folder. I always keep an eye if some of our IPs may appear on any of the well known abuse lists, but so far that never has happened. I would guess if once there's coming spam from your mail server IP, it is negatively branded forever.
human 6y ago
I feel you 100%. The timing of this post is near perfect with what is happening in my life. I run my own web server with email hosting for clients. I’ve spent the last few years doing everything I can on the deliverability side but with no success. All the checkboxes have been checked but I still hit the spam folders. Just yesterday I gave up. I migrated all my clients to Office 365 and I did it knowing I was killing the soul of the Internet. But I was just too tired of dealing with clients who don’t understand why their emails end up in spam folders all the time. These clients also don’t have the same “values” when it comes to decentralization. They run small businesses and cannot afford this spam issue. I feel like there should be laws againsts what the tech giants are doing. Their spam filter rules are killing competition and the free internet.

sebmellen 6y ago

Sadly I've found this to be the case both with perfectly configured personal email servers and when using privacy-preserving email providers like Tutanota. Do you think there's anything that can be done to counter this centralizing force?

da_chicken 6y ago
Now? Almost certainly not. On the side of centralization is a market worth unimagined billions, and perhaps seven providers that already routinely lobby to get what they want. On the side of decentralization? Little guys who are already being bullied and don't have the resources. Politicians who have other extremely dangerous problems currently, including and especially those in their own industry. Don't expect their help for a long time. Smaller service providers who only want the chance to switch sides. And large independent organizations (businesses, schools, governments) who are really only interested in a cheaper bill for the same service quality. Look, we have struggled to fix the MITM/plain text problem and spam/origin problem for well over 20 years, and the Internet Age is maybe 30? It's extremely likely that the only way to fix the core design problems with email is to have the service consolidate to a small consortium of services, have them improve the base design, and then fracture the monopolies. That's going to take decades. And mail is still such a pain in the butt for a user document database. And the user experience is a joke! Asynchronous delivery, best effort failure? I mean, what? The average web page is orders of magnitude larger now! People complain about FAX and FTP.
- nirui 6y ago
  The three protocols for email (SMTP/POP/IMAP) are all outdated really badly. People today needs more efficient, multiplexed and real time message RX/TX, and those tree just not going to cut it. A new, chat-styled mail protocol can be designed to address all the problems above while reduce the risk of abuse. I wonder why nobody standing up to do it.
  - marcbradshaw 6y ago
    SMTP is going to be with us for a long while yet, but some folks are standing up to do something about IMAP - https://jmap.io/ https://jmap.io/
jabroni_salad 6y ago
These configurements do not signal "This person sends good emails", they signal "This server is what it says it is". I can load up barracuda right now and show you plenty of pornbots that have figured out how to put in a passing SPF record... as well as legitimate businesses that don't validate email addresses put into their marketing funnels. Unfortunately, we are back to the age old question. How can we have an ecosystem where anybody can email anybody without it being taken over by spam?
- trhway 6y ago
  Dint proof of work originated as a way of fighting spam? I wonder though why it didn't become widespread. Instead of shadowbanning/blocking the providers like VRZ could just increase the required POW difficulty.
  - marcbradshaw 6y ago
    and of course spammers would never have access to vast botnets to do that work. POW is (imho) a terrible idea.
belorn 6y ago
I am actually a bit more optimistic that the iron grip over email might be attacked from an unexpected direction. A lot of political parities over the world depend on email to operate internally, and the power demonstrated recently by cloud providers in the last few months are making people scared. It only take some publicity and some badly timed bans/lockouts from google before that risk seems much more real again.

endgame 6y ago

I moved my mail away from Google last year, and can no longer accept people's calendar invites. It seems that the "Yes"/"No"/"Maybe" links in the email body of a calendar invite cause Google's (or whoever's) servers to spoof an email in my name, which predictably fails if my SPF is set strictly. I have not been able to find a good workaround for this.

fjni 6y ago
I had something similar so perhaps this helps: If you still have a google account that has the “calendar” feature enabled, this will lead to google sending invite acceptance emails. Especially if you have a business account (which was my experience,) where you verified the domain. The solution for me was to disable “calendar” as a service on the entire account. Realize that “google calendar” and “google mail,” are technically distinct services. However that doesn’t seem to stop “calendar” from sending emails on your behalf.
behringer 6y ago
couldn't you just add google as an approved sender for your domain? https://dmarcly.com/blog/spf-dkim-dmarc-set-up-guide-for-g-suite-gmail-for-business https://dmarcly.com/blog/spf-dkim-dmarc-set-up-guide-for-g-s... I doubt google will send out spam on your domain.
- chalst 6y ago
  I added them as neutral (the '?' character) rather than an allowed domain: I don't want to have to understand Google's mailing policy when assessing the risk of my site being downgraded.

briHass 6y ago

Unfortunately, I agree 100% Even using SendGrid, it will be a week or two and a few hundred emails to 'warm up' one of their mailserver's IP addresses and get reliable delivery. Like you, I found VZ to be the worst at trusting new servers. The days of firing off an email from any ol' SMTP server are completely gone.

tomxor 6y ago
> The days of firing off an email from any ol' SMTP server are completely gone. I'm not sure... I agree with the general vibe in this thread however my experience (within the last year) is that you can fire up a fresh one and send email to gmail at least, without SPF, DKIM or DMARC. However as soon as you start doing any mass mailing of any kind you need all three of those things, SPF to not get outright blocked and get into the spam, and the other two to have their spam detection filters to even bother parsing your email content for the chance of getting into the inbox. This is my experience from setting up systems that send automated status emails not marketing, so fairly low volume but frequent enough with same message content to be immediately classified as mass emailing.
- roblabla 6y ago
  I've never had issues with gmail, though I know a few people who did. But I think setting up SPF and DKIM will, for the most part, do the trick to allow communicating with gmail users. The problem is the other provider. Yahoo, AOL, Microsoft. They're all a pain in the neck and much, much stricter. And sadly, they still amount to a not insignificant portion of the market.

bobflorian 6y ago

I actively work with and set up ~100 domains in SendGrid, and Yahoo and AOL are still, have have been, THE BANE OF MY EXISTENCE since I started working at this job 8 years ago. It's always been a black hole into any type of support. I always get the same answer back... "If you don't want to be seen as a spam bot, don't act like one".... OK, I'll just tell my client they can't send their 20k emails when they want (with a non insignificant going to Yahoo/AOL still). Honestly I don't know what else we can do better. We don't send newsletters and such, we only send transnational emails.

_nickwhite 6y ago
The struggle is real. I found Mailgun to be pretty reliable for yahoo/aol/verizon domains, but my volume isn’t crazy big.
- throwawayboise 6y ago
  I tried Mailgun about 5 years ago. I had a short term, low-volume need, but found that their free tier servers had poor reputations and I remember that Yahoo in particular would not deliver their email. Maybe that's to be expected: any free email sending service is going to be abused by spammers. But it wasn't a great first impression.
dkdk8283 6y ago
PSA: Don’t accept job as mail sysadmin.
- vishnugupta 6y ago
  We are a small SaaS start-up with 4 engineers in total, I'm one of them. Being a senior person, I've to deal with all kinds of crap; SFP, DKIM, domain validation and what not. Just when I relax content that things are in control I get a folder full of files which contain spaces in the file names that need to be processed.
darylteo 6y ago
Mandrill has been reliable, but more expensive, when it comes to AOL Yahoo Hotmail and Outlook. (I work at an agency, we recently migrated off sendgrid to mandrill for all our clients due to deliverability issues)

acidburnNSA 6y ago

Same. I run a 10/10 (mail-tester.com) mail server for myself personally on a VPS from Digital Ocean. Its IP address is on Charter's permanent blocklist and has been for years so I cannot email my mom. I set up a SMTP relay via a free-tier sendgrid and it worked for a while but then that relay got added to spamlists and I could basically only email my mom. Presumably the paid sendgrid setups allow a bit more resilience against getting on those lists.

pbhjpbhj 6y ago
I reckon your mom is reporting you as a spammer. /jk
- bombcar 6y ago
  My sister obviously thinks the “spam” button is the same as “delete email” button. Every once in awhile I get the Google apps for domains notification “tons of spam has been marked”.
walrus01 6y ago
Any high volume vps, vm, dedicated server, Colo hosting company is going to have this problem with ip space reputation. The only place to fully self host your own email these days is on a more boutique isp where it's not possible for anyone with a credit card and a pulse to sign up as a customer in a fully automated process.

thaumaturgy 6y ago

Just this last weekend I started moving clients off of the mail hosting services I've managed for years and onto Fastmail. I have self-hosted my email since around 2005, and had my first email address around 1994 or thereabouts. It has never been more broken than it is now, and I mean that without even the slightest hyperbole. The first RBL showed up around 1998, operated by Paul Vixie of DNS fame. I was using a small ISP in the East Bay at the time and they hated the MAPS RBL, especially the idea that they had to deal with some third party to resolve mail transport issues. But, from 1998 until the early 2000s, RBLs proliferated and lots of mail services subscribed to one or more. The one thing they had going for them was that they were easy to query, so it took no effort to find out if you were on the list, and if you were, there was usually a living, breathing human being somewhere that you could contact to get the matter resolved. It was a pain in the butt, but it could be handled. During this time, the responsibility for ensuring that a message was received shifted from the recipient to the sender. Beleaguered systems administrators soon found themselves in a situation where they were supposed to be responsible for both ensuring that no spam reached their customers and that all of their customers' email reached the intended recipients. Gmail came along and decided that, because they were operating "at scale", they didn't need to play in the same ecosystem. Over the years, ensuring that a message lands in a Gmail user's inbox has turned in to an infuriating game of trial-and-error. Gmail can do this because they now manage between 40% and 60% of the internet's email traffic. AT&T/SBCGlobal/Yahoo/whoever they are now seem to have recently penalized all of Linode's and DigitalOcean's IP space. I deployed several mail exchanges and didn't have any luck reaching any addresses managed by AT&T's network. And, again, there's nobody I can kibbutz with to resolve it. AOL has been such a hot tire fire that I ended up blackholing any outbound traffic to them. I know that sounds drastic, but anytime a single AOL user clicked the "spam" button on an email from one of my customers -- who weren't spamming, newslettering, or anything else remotely skeezy -- it would generate an automated complaint from AOL to Linode, and Linode would threaten to suspend my account. I'd have to dig the relevant traffic out of the logs and respond back to Linode with a polite "AOL's full of shit, please stop listening to them". I explained the situation to the few people that were impacted by it and everybody got on with their lives. Microsoft's outlookprotection.com filter has been a gigantic pain recently too. It's annoyingly capricious and, again, the tools just aren't there to resolve it. Email delivery has been a bit tricky for several years, but the last year especially it has become impossible for small services. You have SPF, DKIM, DMARC, great, but it turns out that Gmail also dings you if you don't have ipv6 records arranged right or if you aren't transporting mail over ssl or or or or. Email was designed as a cooperative system but the BigCos have carved it up and are working hard to ensure that if a message doesn't come from one of their networks, then it's immediately suspicious. Sendgrid isn't much better in this regard. My network received a flood of spam from them, with legitimate traffic mixed in. I couldn't block them without complaints and I couldn't not block them without complaints. I wondered if I was the only, so I signed up for their service and routed some mail through them for a few days to see what it was like as one of their subscribers. Turns out that Comcast and half a dozen other service providers hate them just as much and deliverability was around 79%. The things you and others are experiencing, and that I experienced, are going to keep getting worse. The bigger networks are going to keep squeezing customers out of the smaller ones, breaking email bit by bit in the process. I hope Fastmail is able to grow quickly enough to keep sitting at the table.

karmicthreat 6y ago

Also, get a private IP from your email service. Don't use their public ips. Or you will cry as the IP they have you on will randomly get put on SpamCop. Even with my private IPs I've had random anti-spam services just decide to list all Sendgrid IPs.

iamacyborg 6y ago
Yeah, no. Unless you're sending sufficiently high volume (ie millions of emails a week) or are sending B2B to exchange servers, getting a private IP is a waste of money and could negatively affect your sender reputation more than using a shared IP. Spam filters have significantly advanced beyond simple IP checks.
- karmicthreat 6y ago
  We can agree to disagree here. You have no recourse when one of the Sendgrid shared IPs you are using gets put on Spamcop. Sendgrid might get around to working with them to remove it in a couple weeks. You also have no way to eliminate that IP from your pool of IPs. So in my case about 40% of reports I was sending out were getting blocked at the email server of the receiver. Buying a private IP is your only recourse. Unless your emails don't matter, in which case it might be fine to lose 40% of them.
  - yread 6y ago
    This is my experience as well. Submitted a support request, after a week got told "we're working on it" while my users were not receiving email address confirmation emails. Moved away from SendGrid very quickly after that.
    - karmicthreat 6y ago
      I have some legacy IoT devices in the field that directly talk to sendgrid. Would be a big pain if I had to recredential them. So I stayed on sendgrid. My newer devices get their email proxied through a proxy I host. It was a dumb mistake on my part to tie myself to so tightly to a SaaS vendor.

throwawayboise 6y ago

Not centralizing -- these providers are colluding in a oligopoly. I think there is an argument to be made that these large providers are creating a situation where they set up barriers of trust that only allow email from each other to pass. Small independent senders are locked out simply by not being part of the trusted club, even if their email is valid and passes all of the standard hurdles (SPF, DKIM, etc).

jjeaff 6y ago
They are doing that. But it doesn't seem to be in order to capture market share. It is because of the extreme spam abuse coming from so many unknown domains.

fogihujy 6y ago

The sad reality is that you should expect to be delivered into the recipients' spam boxes until you have managed to convince Google/Microsoft/Yahoo that you're not a spammer. As far as I can tell, it simply is not possible to buy a new domain, pay for a email service and have mails delivered straight into people's inboxes. After all, if you would be able to do that then so could the spammers. Ramp up volumes slowly. Make sure the recipients actually want the mails you send, and for all that is holy, educate your users that the "Junk" and "Trashcan" buttons are two very different things. People who repeatedly mark your mails as spam need to be blacklisted, no matter if they're paying customers. Period. Oh, and handle abuse complaints right away. Don't be afraid to tell people they're not supposed to mark your mails as spam and to explain the consequences.

chillfox 6y ago
The biggest difference comes from actually owning the block of IP addresses that you use instead of renting them.
avereveard 6y ago
> educate your users that the "Junk" and "Trashcan" buttons are two very different things. we have 97% reputation on sendgrid, and the only mail we send are a) customer requested password reset and b) customer requested notification for completed workflow jobs. I don't know who that 3% is, but it is enough to get sometime filtered and I don't think we can ever win this
- fogihujy 6y ago
  Sendgrid itself is a major source of spam, and simply using them can result in mails getting filtered. There are even Spamassassin rules (3rd party) that specifically add spam score to mails, if they are sent through Sendgrid. The 3% may very well be ignorant users who are using the Junk button as a way to remove the mails from the inbox (yeah, people actually do that), but for good measure, things like signup forms and password reset forms really should be protected with a captcha, as any form able to send any form of email whatsoever will be abused by bots trying to send spam.

jjav 6y ago

Which is why it is so vital for anyone with at least minimal tech know-how to run their own email infrastructure. The Internet is only a meaningful concept if it consists of peers running standard decentralized protocols. If all we had was a few giant corporations with proprietary apps and proprietary protocols, that's not the Internet. As I post every time on these email discussions, the difficulty of running your own email infrastructure is vastly overstated on HN. The more people do it, the better off the Internet world is for all of us.

roblabla 6y ago
I have run my own email infrastructure for 3 years, for personal use (I was the only user). I have since moved to using fastmail due to my mail failing to be delivered to any microsoft email. Microsoft would accept the mail, but it would never appear in the mailbox, and neither would it arrive in spam. The mail just disappeared somewhere in their system. I have spent many days debugging the issue, to no avail. Every other operator accepts my email, and my domain was never used to send out spam (I set up DMARC to make sure I am aware of all outgoing email). I imagine the IP might have been the issue, but I cycled through a few hosts without success. Everything was configured correctly AFAICT, I could send email to all major providers, except microsoft. SPF, DKIM, DMARC were all set up. I tried many different configuration testers, and they all returned green on my domain. Since migrating to fastmail (keeping the same domain), I haven't had any problem. The thing is, I agree with you, the world would be a better place if everyone could host their own email. But at the end of the day, I need the ability to contact users using the major email providers, and so do I suspect many HN users.
nor-and-or-not 6y ago
I completely agree! I'm running an email server for myself since 1998 and in addition some low traffic email servers for a few small companies that I administrate and we have had no delivery problems so far, not even with Google, although I have heard from other people that run their own email servers, that Google occasionally puts delivered emails in the Spam folder. I always keep an eye if some of our IPs may appear on any of the well known abuse lists, but so far that never has happened. I would guess if once there's coming spam from your mail server IP, it is negatively branded forever.

human 6y ago

I feel you 100%. The timing of this post is near perfect with what is happening in my life. I run my own web server with email hosting for clients. I’ve spent the last few years doing everything I can on the deliverability side but with no success. All the checkboxes have been checked but I still hit the spam folders. Just yesterday I gave up. I migrated all my clients to Office 365 and I did it knowing I was killing the soul of the Internet. But I was just too tired of dealing with clients who don’t understand why their emails end up in spam folders all the time. These clients also don’t have the same “values” when it comes to decentralization. They run small businesses and cannot afford this spam issue. I feel like there should be laws againsts what the tech giants are doing. Their spam filter rules are killing competition and the free internet.