7 ms·

Maybe slightly unrelated to the main topic, but: > If you’re worried about mail deliverability issues, don’t be — it’s more or less a myth in $CURRENTYEAR. If

Maybe slightly unrelated to the main topic, but: > If you’re worried about mail deliverability issues, don’t be — it’s more or less a myth in $CURRENTYEAR. If you set up DKIM properly and unlist your IP address from the DNSBLs (a simple process), then your mails will get through. Does this match most people's experience? Or is he just talking about the deliverability of smaller providers like Migadu? I run my own mail server and can never send emails to people on Gmail/Microsoft 365/etc unless it's in reply to an email that I received. I've checked DKIM and the DNSBLs, and everything seems correct...is it just a matter of not having enough volume for email providers to trust me?

ddevault 6y ago
Send me an email: sir@cmpwn.com I'll see if I notice any configuration issues and report back to this thread.
- alex_studer 6y ago
  Done, thanks!
  - ddevault 6y ago
    Copying my reply to HN: Some feedback: you should add a PTR record (reverse DNS) for studer.dev and mail.studer.dev. I also noted that your DMARC record has p=quarantine, which depending on the details of your outgoing setup, may cause your emails to be placed in the spam folder if DMARC fails for any reason. I generally advise against using DMARC with any policy other than `v=DMARC1; p=none` or else you're liable to run into issues, it's not a very good standard. Otherwise, your configuration looks good to me. Also try https://www.mail-tester.com https://www.mail-tester.com to see what they think.
    - alex_studer 6y ago
      In case anyone is curious: I checked and the PTR record for my mail server (168.62.161.121 -> mail.studer.dev) is valid. mail-tester.com also gives me a 9.9/10, but I just tried sending an email to a Gmail account and it went straight to spam :( I suspect it's probably a volume thing? As in I'm not sending enough email for Gmail to make any judgements about my mail server, since I'm only an individual. Of course there's not really a great way to resolve that...
      - smartbit 6y ago
        For gmail, try adding an IPv6 address with proper reverse resolving. Tip from HN some time ago.
- Nux 6y ago
  There's a site for that: https://www.mail-tester.com/ https://www.mail-tester.com/
  - looperhacks 6y ago
    Weird, for my server spam assassin tells me my dkim signature is invalid but the dkim check in the next category says that it's valid.
    - alex_studer 6y ago
      Same here!
jacquesm 6y ago
No, it doesn't match my experience at all. Microsoft, Apple and Google all tend to randomly drop messages from addresses that were perfectly good the day before without any change in the sending mail servers configuration. It is super annoying because it actually forces me to use one of those to send email and that actually rewards those companies for creating this problem in the first place. I'd rather deal with spam than random delivery issues without any knowledge about it. Oh, and I get more and more false positives in gmails spam folder as well, and since they're deleted after 30 days I now have one more 'inbox' to monitor.
- elric 6y ago
  Yeah that's my experience as well. And it's impossible to find anyone to talk to at Google/MS/Apple about an email issue -- unless you send really large volumes, then suddenly all kinds of support and tools become available. IMO it's a deliberate attempt to get more people to switch to gmail. But maybe I'm a cynic.
  - thu2111 6y ago
    The problem is that providing email server debugging services is very expensive because not many people really understand the whole SMTP stack these days. You can't find anyone who will talk to you at these firms because there's probably only about 20 people in the company who could genuinely help. Additionally the world is filled with people who would like to run a whole email server for a stream of personal mail who would get it wrong and demand debugging services. That's why they spend their time building automated tools. The problem the article discusses about needing to send lots of mail to qualify for these tools is because the systems don't scale infinitely - data storage still costs these firms money - and botnets mean basically every single IP on the internet will try to abuse your service at some point or another. So tracking deliverability data without any thresholding means tracking it for every connected machine in the world, which is hugely expensive, all to satisfy the tiny minority of people who feel passionately that they should run their own email server. The only possible justification for that would be recruiting related, but they don't have challenges recruiting.
jnwatson 6y ago
Yeah, that's not been our experience, and this is just for password reset/logins. You pretty much have to use a dedicated provider for that.
anonydsfsfs 6y ago
> Does this match most people's experience? It does not. I used to work for a non-profit that ran our own SMTP server, and we had issues with our subscriber e-mails getting marked as spam, mainly for Office 365 users. We set up DKIM, DMARC, and SPF, but it still happened. We ended up having to move to using Google's SMTP servers just so our e-mails would go through.
noja 6y ago
No it doesn't. My mail server is massively rate limited even a year later.
kureikain 6y ago
It does but only for IPv4. If you have those: - reverse ip. origin ip -> dns -> resolve to ip - SPF - DKIM - DMARC - Make sure ip are not on DNSBL - Message ID - Try to connect to gmail mx server using TLS Then you have very low chance to get in spam unless you are sending spam and get blacklist. Time to time an gmail mx server will reject you, rate limiting, just try again with another and you will be ok. Source: I run https://hanami.run https://hanami.run and have to deal with this a lot due to nature of an email forwarding service that people usually use as one-off email (anything@domain.com for their one-off service) so it attract a lot of spam which I have to filter and make sure it won't go to gmail/microsft I would say gmail is the best among big provide: Microsoft/Apple(iclouds) are garbage mail server where they just blindly trust DNSBL and block you no matter what. Gmail was way better. However, if you use IPv6, it's a bit harder to avoid being flag as spam. It's really random and I don't know why yet and I contacted them and now "Waiting for 2 weeks to get a response"
- jacquesm 6y ago
  Even replying to gmail messages or forwarding an older address to gmail will randomly fail. I've had some pretty important messages (for instance: invoices) randomly blackholed while all other mail arrived fine. It is super frustrating to answer someone's request for an invoice and to see that invoice then disappear without any notice that it was spam trapped.
  - kureikain 6y ago
    Yes, I heard you. To gmail point of view, they don't care about forwarding or replying to an email someone send you at all. All they care seems is about the message follow standard best practice(SPF/DKIM/DMARC/PTR) and send from a good IP. Sometime your IP can be flagged as spam wrongly on DNSBL. But it's very easy to get remove from DNSBL. Just submit a removal request, explain your situation and it's usually remove within a few days. Also, the reason while your important emails are marked as spam is depended on gmail mx server you connected to. I don't know the detail but I have seen gmail mx server rejected/spammed my email, but on subsequent try if I connect to different one(by resolving other MX record), the email went through just fine. If you send me an email vinh@hanami.run I can try to spot if anything may randomly flagged your email. Also, did you have multiple SMTP servers?
throwanem 6y ago
Deliverability issues for personal servers are emphatically not a myth. One reason I moved off self-hosting mail a couple of months ago, after 17 uninterrupted years of doing so, was because of perennial problems with deliverability - and Gmail, while bad, wasn't even the worst; some large providers (glaring daggers at you, Microsoft) make it effectively impossible even to understand why mail is getting blackholed, much less do anything about it. (Fastmail, in case anyone is wondering. $50 a year, and there's nothing I want that they don't do.)
bluecalm 6y ago
Not at all my experience. I was sending paying customers emails with their license key and a link to our starting page for years and it was fine. Recently our emails randomly land in spam v even though: -we have never sent one unsolicited email -100% of those emails are opened and read (I know because customers are activating their licenses). It's hard to imagine anyone mark the email with license they paid several hundred dollars for as spam -we get tons of engagement, we answer around 30-40 emails a day Something is terribly wrong with Gmail spam filter recently. Some of the emails from my customers land in my spam folder as well. They more often than not don't contain any links or anything out of the ordinary. Just a simple question or two. It started happening a few months ago. It's a huge pain. All Google support tells us is that their magical box they can't disclose thinks our emails are spam even though there is nothing suspicious about either our website, the link or the content of the email. They can't tell us what triggers the filter cause it will make the job of the spammers easier. I am now left to play the game against spammers: they try to imitate legit emails, if they get close enough then my emails are similar to spam, I have to modify them in random way, then spammers catch up etc. Yay for machine learning I guess. Doesn't work so well even you have an adversary trying to fool you. Btw, we use Gmail and it's Gmail spam filter which is causing problems. Google analytics is able to guess our monthly revenue and number of licenses pretty well. We send exactly as many emails. We are paying customer of Google Suite. It's still not their problem that their magic box puts our emails in spam but it's all on us.
nr2x 6y ago
yeah, that's a 100% bullshit claim.
syntheticnature 6y ago
As someone who runs a discussion e-list server for a small volunteer org, Gmail has not been an issue with a good config, generally. The other big providers are annoying, to varying degrees, often silently dropping email. ISP email tends to be a pain in the neck, putting one on a blacklist if you exchange email with two different addresses they provide service to within one hour, though at least they tend to be noisy about it.
Cu3PO42 6y ago
It absolutely does not. I tried setting up mail on my own server and by all accounts I did everything right. Mail-Tester and similar services found no issues, I had all the optional records including DKIM, SPF, etc. and yet I couldn't rely on my mail going through. Microsoft (Outlook.com) was particularly bad: mails were silently dropped without notice to me or the recipient even when I was replying to a mail they sent. I tried adding my e-mail address on my own servers to the trusted senders list on a MS account of my own and even then not all mail would go through. Gmail ocasionally put me in spam but at least always delivered my messages. At that point I gave up since I consider email to be too essential to be flaky and paid for Migadu. I have been very happy with their service and appreciate their pricing strategy (charging for number of mails sent, not the number of domains or mailboxes added). That said, I still dislike paying for hosting that I had sufficient capacity for on my own servers.
joana035 6y ago
Yes, it works for me. SPF, DKIM and proper DNS records and that is it.
sildur 6y ago
It does match my experience. I have a personal mail server with every bell and whistle working and my emails arrive to gmail addresses. My IP address was used by a spammer in the past, and took me some time cleaning it up in a bunch of dnsbl lists. But it reliably works now.
superluserdo 6y ago
It doesn't match my experience but in fairness I was running the server off my home IP. I think it was the lack of reverse DNS that was tripping it.
baobabKoodaa 6y ago
Delivering email from your own little server is theoretically possible, but extremely difficult to do in practice. I wrote about this at length here: https://www.attejuvonen.fi/dont-send-email-from-your-own-server/ https://www.attejuvonen.fi/dont-send-email-from-your-own-ser...
- ZWoz 6y ago
  I have run small servers something like 18 years, my own personal something like 15-16 years. I still don't have much trouble. Only issue I had few years ago was spamhaus suddenly listing my IP as residental, that was annoying few hours.
  - baobabKoodaa 6y ago
    How did you measure your deliverability?
    - ZWoz 6y ago
      Well, for personal matters I don't write usually letters without needing feedback. When I get answer, then my previous mail is clearly arrived. My friend has forwarding to gmail address, so he can compare both ends. Over years there isn't been more issues than mentioned spamhaus incident.
      - baobabKoodaa 6y ago
        Ok, so you actually haven't measured your deliverability in any way. Yes, you can probably get emails delivered to your friends who have already emailed you in the past. Spam filtering is more of an issue when you try to email someone who has not engaged with you. If you actually go out and run some tests, you will probably find out a good portion of your email is blackholed by Google and Microsoft.
        ZWoz 6y ago
        How to you really test that? Any provider is autonomous, you really don't verify delivery. You have few technical options, like DSN or MDNs, but servers or clients don't need honor them. I didn't mentioned clients, I had few of them too. Some used internal mailing list style things, group addresses forwarded some outside service, like hotmail or gmail. I didn't had any complaints from them. Now talking about personal experience, your assumption is wrong. I don't change mail only with my family or friends. When you write merchant or government agency there is even more initiative to notice missing answer.
shock 6y ago
Is your mail server IPv6 enabled?
- smartbit 6y ago
  IPv6 magically makes email go to the inbox of gmail. A tip I read here in HN some time ago.
  - WGH_ 6y ago
    It really depends. In our case, Gmail rejects our IPv6, but accepts IPv4. The IPs are from university network and they don't have any history other than being used by us. SPF, DKIM and PTR records are in place. > Our system has detected that this message is likely suspicious due to the very low reputation of sending domain. To best protect our users from spam, the message has been blocked. Please visit https://support.google.com/mail/answer/188131 https://support.google.com/mail/answer/188131 for more information. This message is extra confusing because mail from said "sending domain" is accepted by IPv4.
jcelerier 6y ago
> Does this match most people's experience? Or is he just talking about the deliverability of smaller providers like Migadu? I run my own mail server and can never send emails to people on Gmail/Microsoft 365/etc unless it's in reply to an email that I received. I use a gmail address and all my friends's self hosted mail have always ended up in spam by default

ddevault 6y ago

Send me an email: sir@cmpwn.com I'll see if I notice any configuration issues and report back to this thread.

alex_studer 6y ago
Done, thanks!
- ddevault 6y ago
  Copying my reply to HN: Some feedback: you should add a PTR record (reverse DNS) for studer.dev and mail.studer.dev. I also noted that your DMARC record has p=quarantine, which depending on the details of your outgoing setup, may cause your emails to be placed in the spam folder if DMARC fails for any reason. I generally advise against using DMARC with any policy other than `v=DMARC1; p=none` or else you're liable to run into issues, it's not a very good standard. Otherwise, your configuration looks good to me. Also try https://www.mail-tester.com https://www.mail-tester.com to see what they think.
  - alex_studer 6y ago
    In case anyone is curious: I checked and the PTR record for my mail server (168.62.161.121 -> mail.studer.dev) is valid. mail-tester.com also gives me a 9.9/10, but I just tried sending an email to a Gmail account and it went straight to spam :( I suspect it's probably a volume thing? As in I'm not sending enough email for Gmail to make any judgements about my mail server, since I'm only an individual. Of course there's not really a great way to resolve that...
    - smartbit 6y ago
      For gmail, try adding an IPv6 address with proper reverse resolving. Tip from HN some time ago.
Nux 6y ago
There's a site for that: https://www.mail-tester.com/ https://www.mail-tester.com/
- looperhacks 6y ago
  Weird, for my server spam assassin tells me my dkim signature is invalid but the dkim check in the next category says that it's valid.
  - alex_studer 6y ago
    Same here!

jacquesm 6y ago

No, it doesn't match my experience at all. Microsoft, Apple and Google all tend to randomly drop messages from addresses that were perfectly good the day before without any change in the sending mail servers configuration. It is super annoying because it actually forces me to use one of those to send email and that actually rewards those companies for creating this problem in the first place. I'd rather deal with spam than random delivery issues without any knowledge about it. Oh, and I get more and more false positives in gmails spam folder as well, and since they're deleted after 30 days I now have one more 'inbox' to monitor.

elric 6y ago
Yeah that's my experience as well. And it's impossible to find anyone to talk to at Google/MS/Apple about an email issue -- unless you send really large volumes, then suddenly all kinds of support and tools become available. IMO it's a deliberate attempt to get more people to switch to gmail. But maybe I'm a cynic.
- thu2111 6y ago
  The problem is that providing email server debugging services is very expensive because not many people really understand the whole SMTP stack these days. You can't find anyone who will talk to you at these firms because there's probably only about 20 people in the company who could genuinely help. Additionally the world is filled with people who would like to run a whole email server for a stream of personal mail who would get it wrong and demand debugging services. That's why they spend their time building automated tools. The problem the article discusses about needing to send lots of mail to qualify for these tools is because the systems don't scale infinitely - data storage still costs these firms money - and botnets mean basically every single IP on the internet will try to abuse your service at some point or another. So tracking deliverability data without any thresholding means tracking it for every connected machine in the world, which is hugely expensive, all to satisfy the tiny minority of people who feel passionately that they should run their own email server. The only possible justification for that would be recruiting related, but they don't have challenges recruiting.

jnwatson 6y ago

Yeah, that's not been our experience, and this is just for password reset/logins. You pretty much have to use a dedicated provider for that.

anonydsfsfs 6y ago

> Does this match most people's experience? It does not. I used to work for a non-profit that ran our own SMTP server, and we had issues with our subscriber e-mails getting marked as spam, mainly for Office 365 users. We set up DKIM, DMARC, and SPF, but it still happened. We ended up having to move to using Google's SMTP servers just so our e-mails would go through.

noja 6y ago

No it doesn't. My mail server is massively rate limited even a year later.

kureikain 6y ago

It does but only for IPv4. If you have those: - reverse ip. origin ip -> dns -> resolve to ip - SPF - DKIM - DMARC - Make sure ip are not on DNSBL - Message ID - Try to connect to gmail mx server using TLS Then you have very low chance to get in spam unless you are sending spam and get blacklist. Time to time an gmail mx server will reject you, rate limiting, just try again with another and you will be ok. Source: I run https://hanami.run https://hanami.run and have to deal with this a lot due to nature of an email forwarding service that people usually use as one-off email (anything@domain.com for their one-off service) so it attract a lot of spam which I have to filter and make sure it won't go to gmail/microsft I would say gmail is the best among big provide: Microsoft/Apple(iclouds) are garbage mail server where they just blindly trust DNSBL and block you no matter what. Gmail was way better. However, if you use IPv6, it's a bit harder to avoid being flag as spam. It's really random and I don't know why yet and I contacted them and now "Waiting for 2 weeks to get a response"

jacquesm 6y ago
Even replying to gmail messages or forwarding an older address to gmail will randomly fail. I've had some pretty important messages (for instance: invoices) randomly blackholed while all other mail arrived fine. It is super frustrating to answer someone's request for an invoice and to see that invoice then disappear without any notice that it was spam trapped.
- kureikain 6y ago
  Yes, I heard you. To gmail point of view, they don't care about forwarding or replying to an email someone send you at all. All they care seems is about the message follow standard best practice(SPF/DKIM/DMARC/PTR) and send from a good IP. Sometime your IP can be flagged as spam wrongly on DNSBL. But it's very easy to get remove from DNSBL. Just submit a removal request, explain your situation and it's usually remove within a few days. Also, the reason while your important emails are marked as spam is depended on gmail mx server you connected to. I don't know the detail but I have seen gmail mx server rejected/spammed my email, but on subsequent try if I connect to different one(by resolving other MX record), the email went through just fine. If you send me an email vinh@hanami.run I can try to spot if anything may randomly flagged your email. Also, did you have multiple SMTP servers?

throwanem 6y ago

Deliverability issues for personal servers are emphatically not a myth. One reason I moved off self-hosting mail a couple of months ago, after 17 uninterrupted years of doing so, was because of perennial problems with deliverability - and Gmail, while bad, wasn't even the worst; some large providers (glaring daggers at you, Microsoft) make it effectively impossible even to understand why mail is getting blackholed, much less do anything about it. (Fastmail, in case anyone is wondering. $50 a year, and there's nothing I want that they don't do.)

bluecalm 6y ago

Not at all my experience. I was sending paying customers emails with their license key and a link to our starting page for years and it was fine. Recently our emails randomly land in spam v even though: -we have never sent one unsolicited email -100% of those emails are opened and read (I know because customers are activating their licenses). It's hard to imagine anyone mark the email with license they paid several hundred dollars for as spam -we get tons of engagement, we answer around 30-40 emails a day Something is terribly wrong with Gmail spam filter recently. Some of the emails from my customers land in my spam folder as well. They more often than not don't contain any links or anything out of the ordinary. Just a simple question or two. It started happening a few months ago. It's a huge pain. All Google support tells us is that their magical box they can't disclose thinks our emails are spam even though there is nothing suspicious about either our website, the link or the content of the email. They can't tell us what triggers the filter cause it will make the job of the spammers easier. I am now left to play the game against spammers: they try to imitate legit emails, if they get close enough then my emails are similar to spam, I have to modify them in random way, then spammers catch up etc. Yay for machine learning I guess. Doesn't work so well even you have an adversary trying to fool you. Btw, we use Gmail and it's Gmail spam filter which is causing problems. Google analytics is able to guess our monthly revenue and number of licenses pretty well. We send exactly as many emails. We are paying customer of Google Suite. It's still not their problem that their magic box puts our emails in spam but it's all on us.

nr2x 6y ago

yeah, that's a 100% bullshit claim.

syntheticnature 6y ago

As someone who runs a discussion e-list server for a small volunteer org, Gmail has not been an issue with a good config, generally. The other big providers are annoying, to varying degrees, often silently dropping email. ISP email tends to be a pain in the neck, putting one on a blacklist if you exchange email with two different addresses they provide service to within one hour, though at least they tend to be noisy about it.

Cu3PO42 6y ago

It absolutely does not. I tried setting up mail on my own server and by all accounts I did everything right. Mail-Tester and similar services found no issues, I had all the optional records including DKIM, SPF, etc. and yet I couldn't rely on my mail going through. Microsoft (Outlook.com) was particularly bad: mails were silently dropped without notice to me or the recipient even when I was replying to a mail they sent. I tried adding my e-mail address on my own servers to the trusted senders list on a MS account of my own and even then not all mail would go through. Gmail ocasionally put me in spam but at least always delivered my messages. At that point I gave up since I consider email to be too essential to be flaky and paid for Migadu. I have been very happy with their service and appreciate their pricing strategy (charging for number of mails sent, not the number of domains or mailboxes added). That said, I still dislike paying for hosting that I had sufficient capacity for on my own servers.

joana035 6y ago

Yes, it works for me. SPF, DKIM and proper DNS records and that is it.

sildur 6y ago

It does match my experience. I have a personal mail server with every bell and whistle working and my emails arrive to gmail addresses. My IP address was used by a spammer in the past, and took me some time cleaning it up in a bunch of dnsbl lists. But it reliably works now.

superluserdo 6y ago

It doesn't match my experience but in fairness I was running the server off my home IP. I think it was the lack of reverse DNS that was tripping it.

baobabKoodaa 6y ago

Delivering email from your own little server is theoretically possible, but extremely difficult to do in practice. I wrote about this at length here: https://www.attejuvonen.fi/dont-send-email-from-your-own-server/ https://www.attejuvonen.fi/dont-send-email-from-your-own-ser...

ZWoz 6y ago
I have run small servers something like 18 years, my own personal something like 15-16 years. I still don't have much trouble. Only issue I had few years ago was spamhaus suddenly listing my IP as residental, that was annoying few hours.
- baobabKoodaa 6y ago
  How did you measure your deliverability?
  - ZWoz 6y ago
    Well, for personal matters I don't write usually letters without needing feedback. When I get answer, then my previous mail is clearly arrived. My friend has forwarding to gmail address, so he can compare both ends. Over years there isn't been more issues than mentioned spamhaus incident.
    - baobabKoodaa 6y ago
      Ok, so you actually haven't measured your deliverability in any way. Yes, you can probably get emails delivered to your friends who have already emailed you in the past. Spam filtering is more of an issue when you try to email someone who has not engaged with you. If you actually go out and run some tests, you will probably find out a good portion of your email is blackholed by Google and Microsoft.
      - ZWoz 6y ago
        How to you really test that? Any provider is autonomous, you really don't verify delivery. You have few technical options, like DSN or MDNs, but servers or clients don't need honor them. I didn't mentioned clients, I had few of them too. Some used internal mailing list style things, group addresses forwarded some outside service, like hotmail or gmail. I didn't had any complaints from them. Now talking about personal experience, your assumption is wrong. I don't change mail only with my family or friends. When you write merchant or government agency there is even more initiative to notice missing answer.

shock 6y ago

Is your mail server IPv6 enabled?

smartbit 6y ago
IPv6 magically makes email go to the inbox of gmail. A tip I read here in HN some time ago.
- WGH_ 6y ago
  It really depends. In our case, Gmail rejects our IPv6, but accepts IPv4. The IPs are from university network and they don't have any history other than being used by us. SPF, DKIM and PTR records are in place. > Our system has detected that this message is likely suspicious due to the very low reputation of sending domain. To best protect our users from spam, the message has been blocked. Please visit https://support.google.com/mail/answer/188131 https://support.google.com/mail/answer/188131 for more information. This message is extra confusing because mail from said "sending domain" is accepted by IPv4.

jcelerier 6y ago

> Does this match most people's experience? Or is he just talking about the deliverability of smaller providers like Migadu? I run my own mail server and can never send emails to people on Gmail/Microsoft 365/etc unless it's in reply to an email that I received. I use a gmail address and all my friends's self hosted mail have always ended up in spam by default