18 ms·

Understanding SPF, DKIM, and DMARC: A Simple Guide

443 points by miles 2y ago

XCSme 2y ago
I also wrote a similar post a while ago, after someone was sending emails using my domain name: https://www.uxwizz.com/blog/stop-others-use-your-domain-emails https://www.uxwizz.com/blog/stop-others-use-your-domain-emai...
jprjr 2y ago
What I really want to see is a guide for SPF/DKIM/DMARC oriented towards people writing apps that send email using other people's domains. I have dealt with so many ticketing systems and marketing platforms that do not understand the roles of SPF/DKIM/DMARC at all. Things like, insisting we need to include their SPF record in ours, even going so far as to scan the SPF record for the include, only to find out they use their own domain in the envelope address (which is what I wanted them to do in the first place). Or not distinguishing at all between envelope and header addresses and using our domain in both. Which of course means they're not tracking delayed bounces. It really becomes an issue with larger orgs where everybody wants to use the main domain for brand purposes and subdomains are just totally frowned upon for whatever reason. If you just leave my SPF alone and rely on DKIM, it means you can still pass DMARC and track bounces properly. Hell I'd be fine with making subdomains for the envelope address that lists your infrastructure in the MX records but again, eyes really start to glaze over when you say "envelope address." Basically what I really want is a guide that boils down to: if you're not their primary email provider, then don't touch your client's SPF record.
- jabroni_salad 2y ago
  I recently set up a mailchimp tenant for someone and was surprised that their email authentication wizard pretty much began and ended with DKIM. I'm way too used to b2b solutions that want the equivalent of a chmod+999 before it can run a hello world.
  - TheNewsIsHere 2y ago
    In my experience this approach is becoming more common, but slowly. I still configure my default apex DMARC records (in most scenarios) to enforce strict alignment on both SPF and DKIM, but I’ve been relaxing that on a case by case basis or overriding the apex DMARC policy at a subdomain level and only using DKIM where supported.
    - joveian 2y ago
      There seems to be incorrect information out there about strict alignment requesting more strict checking. Strict vs relaxed alignment is just about the domain being able to send mail from subdomains without extra DNS records on the subdomains (relaxed) or not (strict). The envelope from (for SPF) and d= domain (for DKIM) must match the DNS records used. If you don't use SPF for DMARC you don't use SPF for DMARC and it doesn't matter if the DMARC SPF alignment is strict or relaxed (there is no way in DMARC to require both DKIM and SPF to pass). It is still a good idea to always pass plain (non-aligned) SPF (just based on envelope from) since sometimes this is checked independently from DMARC.
      - TheNewsIsHere 2y ago
        I am slightly confused, and perhaps misunderstanding your framing. I do use DMARC and SPF in the fashion you described. In my environments I typically need to take every measure to ensure only authorized services/servers/senders are sending, via authorized hosts and IPs, and this often changes based on subdomain, so that’s why I use strict alignment. Personally I strive to keep various services strictly separated by (sub)domain.
        joveian 2y ago
        I thought it might just be ambiguous language. It wasn't clear to me what you were trying to say and it could be read as implying what I have seen (incorrectly) stated in some DMARC descriptions elsewhere. Just trying to prevent any misunderstandings from anyone reading the thread and improve my own understading by checking that I got it right before posting. Thanks for clarifying what you ment :).
velcrovan 2y ago
I manage IT at a mid-size business. At least once a month, I get asked to release some incoming email from quarantine that got sent there because the sender's SPF record is wrong or outdated and doesn't include all the email services they actually use. (What this really tells me is how many small businesses are out there running with no in-house IT expertise or support of any kind.) I don't do whitelisting. Instead, I always reach out and offer to help the other party correct their SPF record. It happens often enough that I wrote a script in Racket that will generate the email for me and paste it into the clipboard [1]. The email tells them exactly what they need to change, and links to docs from their current email provider (so they don't have to trust me about edits to their DNS). [1]: https://gist.github.com/otherjoel/6b8bf02f6db6e0c47ba6bca72ecf1c3e https://gist.github.com/otherjoel/6b8bf02f6db6e0c47ba6bca72e...
- luckman212 2y ago
  Neat! I must try this.
- ziddoap 2y ago
  What would you say the normal reception you receive from this email template is? I like the idea, but I would think sending a technical email (with industry-specific acronyms that you don't spell out!) to a business that has no in-house IT would just be ignored in most cases.
  - velcrovan 2y ago
    Well if you read the template, you'll see I start out with a non-technical explanation, advise that they forward the email to an IT type person, and offer to help in any way I can. Then I put in a "More info" heading further down with all the details and instructions. Overall I'm pleased with how well this approach works. When people realize that their email is getting stuck in spam filters because of a problem on their end, they're usually motivated to get it fixed. Sometimes it gets sent to an owner who had barely enough tech mojo to stand up a gmail account at a custom domain, and even then the instructions are usually simple enough for them to follow.
NoboruWataya 2y ago
> If you are invovled in developing, supporting, or maintaining an application that sends emails, this guide is a must read. I would also say a guide like this is helpful whenever you are using a custom domain with an email service, and therefore need to set these records yourself. Okay, you might not need to have an in-depth knowledge of these concepts, but it's certainly helpful to understand why your email provider is telling you to set all these weird DNS records.
lovasoa 2y ago
Just today, someone sent me the link to a great tool to debug dmark issues: https://www.learndmarc.com/ https://www.learndmarc.com/
- paravz 2y ago
  https://www.mailhardener.com https://www.mailhardener.com is another great "validator" and more for a personal email domain
- UberFly 2y ago
  Just tried it - you're right that is really good. Thanks.
- nevster 2y ago
  Fantastic!
- jesterson 2y ago
  Is it just me who finds this tool horrendous? Instead of just giving a report, it makes words fly on the page... Certainly it looks cool and perhaps appealing to some younger generation, but if it is tech analysis, you would expect clear and concise report, not some flying things with explanation like I am 4
  - lovasoa 2y ago
    It's called learndmarc, not testdmarc. As someone who wasn't familiar with the technologies involved, I like that I actually learned something while using it to debug my email issue.
- oriettaxx 2y ago
  uh, you are right, super cool! sysadmins will love it
detourdog 2y ago
When I first implemented the above the next step was going to be ARC. Does anyone have thoughts on ARC? https://www.validity.com/blog/how-to-explain-authenticated-received-chain-arc-in-plain-english/ https://www.validity.com/blog/how-to-explain-authenticated-r...
- jabroni_salad 2y ago
  It's only really needed if you need to robo-forward stuff between domains. For example if you set up a domain but want to receive emails to your usual gmail. I noticed that cloudflare's email forwarder uses an ARC record and it works a treat.
  - lxgr 2y ago
    I recently noticed this when debugging email delivery issues for a family member who have their own TLD, but forward everything to Gmail. Unfortunately, the mail server of the forwarding domain doesn't seem to support ARC, so Gmail frequently throws away everything that doesn't have a DMARC header, since without DMARC the only other option is SPF, which doesn't work for forwarding.
    - jabroni_salad 2y ago
      A lot of small businesses in my area have been bit by that this year. I have to give hostgator/bluehost/godaddy etc kudos for having an email forwarder work reliably for so long but I wish they were more proactive about getting their customers compliant with this. Also it's kinda messed up how much of the small business sector is relying on AOL webmail to operate.
  - detourdog 2y ago
    I'm thinking I will add it as part of next upgrade. My domain has so few users and we are chill enough to work through email delivery issues. My understanding at the time was that the receiver verifies that the sender sent it. My current delivery settings are strict so that only my server can delivery our email. I would think I could implement ARC in less strict manner and tighten it up as it becomes more common. Does that seem reasonable? Any better ideas.
njt 2y ago
On a slightly related note, Michael W. Lucas[1] is working on an upcoming book entitled "Run Your Own Mail Server", that will be published shortly (there's a Kickstarter campaign as well[2]). I attended his tutorial and talk at BSDCan[3] this year and both were excellent. I highly recommend buying the book when it comes out (or supporting the Kickstarter), it will go through all the gory details of setting up and running a mail server, and best practices, including a ton of material on SPF/DKIM/DMARC. (P.S. I have no affiliation with the author or the book in any way.) [1] - https://mwl.io/ https://mwl.io/ [2] - https://www.kickstarter.com/projects/mwlucas/run-your-own-mail-server https://www.kickstarter.com/projects/mwlucas/run-your-own-ma... [3] - https://www.bsdcan.org/2024/ https://www.bsdcan.org/2024/
- whartung 2y ago
  Looking forward to this. First thing I ever ponied up on KS for. I don't even run a mailserver, I'm just hoping it will take a bunch of the guides that have been floating about on the web, consolidate the sharp edges, and make sure its up to date. I also hope it has some discussion on troubleshooting. Like dealing with blacklists and what not, folks always talk about that, but I've never see it documented what is actually done to resolve these problems (Like who do you send an email to, how do you even find out who to send an email to, etc.)
disport 2y ago
Disclaimer: My startup is made to get user domains automatically onboarded to applications, DNS-related things such as SPF/DKIM/DMARC. It's a "Stripe-for-DNS" called cloudvalid.com. I like the quality of this SPF/DKIM/DMARC guide. This is the industry I started out in, and I actually wrote the guides for SendGrid, Amazon SES, and a few other email products. I don't mind saying that author has done a better job at this than me. That being said, I see SPF/DKIM/DMARC guides like this pop up with some regularity, but users continue to have the same level of comprehension as before. I think the nature of this problem is not one that lends itself to being solved by guides. It's the sort of problem that a user is really only faced with once, which means that they're not getting the repetitions in to warrant any long-term comprehension. I'm naturally biased here, but if you're onboarding users with SPF/DKIM/DMARC to your application, it's good if you can just get them setup with automation.
- apitman 2y ago
  > My startup is made to get user domains automatically onboarded to applications, DNS-related things such as SPF/DKIM/DMARC. It's a "Stripe-for-DNS" called cloudvalid.com. Interesting. Not lot of startups out there talking about Domain Connect on their front page. Seems like a decent amount of overlap with my project TakingNames.io[0]. Feel free to reach out if you want to talk shop. [0]: https://takingnames.io/blog/introducing-takingnames-io https://takingnames.io/blog/introducing-takingnames-io
deleted 2y ago
[deleted]
heavyset_go 2y ago
Tangential, but what is the contemporary go-to for standing up a mail server these days? The last time I had to do so was a decade ago. I remember Mail-in-a-Box being popular at one point, wondering if that's still the case.
- jeroenhd 2y ago
  Really depends on what you want to get out of your server. Most easy to use server software I know seems to be geared towards personal use and small organisations. Mail-in-a-box still works, though last time I checked they were on quite an old Ubuntu LTS release. There are a few pre-packaged docker containers too (i.e. docker-mailserver) which seem to be popular. I myself use Mailcow, but that's pretty heavy for "just" a mail server. There's also a mail server that I can't for the life of me remember the name of, which packaged a whole bunch of stuff into one single binary you can run rather than use the classic "every part of the email delivery chain is a separate process" approach. I think it was written in Go?
  - cuu508 2y ago
    > There's also a mail server that I can't for the life of me remember the name of, which packaged a whole bunch of stuff into one single binary you can run rather than use the classic "every part of the email delivery chain is a separate process" approach. I think it was written in Go? maddy or mox?
- bongodongobob 2y ago
  For a business? You don't. For hobby/fun? I still wouldn't. I tried years ago and you're just going to fight blacklisting constantly. If you're on a residential/consumer internet service, trying to set it up in your home lab, forget it.
- cauterize 2y ago
  I’m partial to https://gitlab.com/simple-nixos-mailserver/nixos-mailserver https://gitlab.com/simple-nixos-mailserver/nixos-mailserver Does require some NixOS knowledge, but is nice for upgrades.
- 2y ago
deleted 2y ago
[deleted]
remram 2y ago
My problem with SPF (& co) is redirections. I have email redirected from other domains into my (Gmail) inbox. For it to arrive, I use SRS, so the email is properly aligned and always makes it into my inbox. The problem is that some of that email is malicious. I have a choice of dropping those mails, and I never see a trace of it in my inbox, or forwarding them with SRS, and they look to Gmail like 100% perfectly good mails sent from my own domain (but still potentially malicious). It's annoying.
- alt227 2y ago
  > It's annoying. This seems very much like a problem you have created for yourself.
  - remram 2y ago
    I know, how dare I use other domains than gmail...? Why post a comment at all, if you have no insight. It's useless and insulting.
    - alt227 2y ago
      Why post on a public forum if you dont want, or even get insulted by, other peoples opinions? Its nothing to do with using domains other than gmail, its that you said you are actively relaying all mail into your gmail account and rewriting the sender as yourself, but then its annoying that spam gets marked as valid mail from your domain. Thats nobodies problem other than your own setup, and there are loads of other ways you could do it. But the way you chose to set it up is 'annoying'. Sorry to insult you, but I feel that warrants letting you know you caused your own problem!
      - remram 2y ago
        If your opinion is "you suck" it's called an insult. You could have ignored my question, you could have explained why I'm doing it wrong, hell you could have posted a random setup guide from Google and downvoted me. Instead you post this attack with no content, justification, or insight. You are being an ass. You are allowed to be, but I'm allowed to call it out. And if you think I set my email up in a sub-optimal way, just so that I could ask for advice on HN, with the goal of pissing you off... you are delusional.
betaby 2y ago
We need more email diversity. Use your own email servers as often as you can. Monocultures of Apple/Google/MS is deterioration for the Internet.
- nerdponx 2y ago
  And you don't need to run your own server in order to escape the big players. I have been a happy Fastmail customer for many years now.
  - TheNewsIsHere 2y ago
    I second Fastmail wholeheartedly. I moved multiple self-hosted (Postfix and Dovecot) environments to Fastmail and haven’t looked back. The only thing I lost was the ability to manually configure an address to bounce back. Sending and receiving as a wildcard alias is fantastic, and Fastmail allowing that was the reason I finally moved. I held out on that feature for a long time. (You can also do this in Exchange Online if you want to run a convoluted and officially unsupported configuration.) I only twice had any kind of RBL issues, and one was a motivating factor in moving. I also got tired of worrying about patching and CVEs. I do that enough for my work.
- 0xpgm 2y ago
  One of the reasons I decided to self-host emails for my domain names. There's no reason it should be hard to host your own email in 2024 even if you're just mildly technical. An Ansible playbook that sets up Debian to host email, all the dependencies from the default Debian repositories. https://github.com/programmer-ke/replatform/ https://github.com/programmer-ke/replatform/
deleted 2y ago
[deleted]
dankai 2y ago
This is a great guide but from my experience, even if you configure it 100% correctly, email services like Gmail may still classify your emails as spam for no apparent reason while not being on any IP or domain blacklist. I tried for hundreds of hours to get around it with no avail, and my emails to Gmail always went to spam unless it was a response to an email from a Gmail address. Had to go back to a 3rd party hosted service (iCloud) because of it.
- gerdesj 2y ago
  I have had a Gmail account from the days when it was invitation only. The inbox contains spam and my test emails and nothing else! I've run tiny smtp systems for 25 years or so. It can be done. I am based in the UK but at least one of my domains is a .net jobbie, so nominally American. That one still works fine and it is my (ltd) company domain, so all good. The MX records etc have moved around a bit but always very carefully. It all starts around the IP address you are using. Is it "tainted"? is it in a tainted block? If it is then you need to either go elsewhere or clean it up and that takes a bit of time. By clean it up I mean apply for removal from the usual suspect's blocklists - Spamcop (lol), Spamhaus and all the rest that you can find. Now setup PTR records. That has to be done by your ISP. If they can't do it for you, then find a new ISP. If you can't get PTR records to match A records then you may have to give up. One of the first checks an anti spam system will do is reverse look up an incoming IP address and compare it. Also that should match the HELO/EHLO announced by the SMTP MTA: SMTP connection from IP address 12.13.14.15 HELO (my name is) smtp.example.co.uk Receiver will check: smtp.example.co.uk == 12.13.14.15 AND 15.14.13.12.in-addr.arpa == smtp.example.co.uk. Everyone gets their knickers in a twist about SPF, DKIM and DMARC but if you do not get the prior basics of IP -> A -> HELO -> PTR sorted out first then you will fail sooner or later. I also recommend that you ensure your MX records (receiving) match up too with your sending records. It means you can use mx is SPF, for example. If you have multiple internet connections and IPs then be absolutely certain that your inbound and outbound IPs for SMTP match up. Sorted all that? Cool, now proceed to SPF. Most people fail at the PTR stage. If your ISP will not do PTR for you then you are probably screwed for self hosted SMTP. If you cannot change ISP to one that will, then you are really screwed. Sorry. In that case you will have to engage a service that will route SMTP on your behalf. It won't cost much but you won't own it and you will have to pay someone to do it. Soz.
jgalt212 2y ago
Many of our emails were blocked because our SPF record was correct for our domain, but not correct for the Return-Path domain. There is no mention of Return-Path in the guide, but it's essential to get this right. https://easydmarc.com/blog/how-does-an-spf-record-work/ https://easydmarc.com/blog/how-does-an-spf-record-work/
nubinetwork 2y ago
These kinds of articles pop up on HN all the time... Give me a mail server that can use LE for certificates and I'll gladly give DKIM and DMARC a try...
- yubiox 2y ago
  Sendmail
  - nubinetwork 2y ago
    I can't believe sendmail still exists... worst configuration format ever.
- inejge 2y ago
  You don't need public certificates for DKIM, it uses privately generated ones for as long as you want to keep them. (A security researcher recently found quite a few domains using weak DKIM keys generated by buggy Debian OpenSSL, more than fifteen years ago.)
  - nubinetwork 2y ago
    I could swear at one point you needed one, but I just half-setup opendkim and it generated one without me needing to make one by hand... when I get around to updating the DNS on my personal domains, I guess I'll see how things turn out.
kuon 2y ago
I run my own email server, and I'll add that SFP /DKIM and spam filtering should be done at the connection level during SMTP session. Your MTA should not accept a message for delivery that has been filtered. This ensures that the sender of false positive gets a notification from his MTA and also, in my experience, it works very well to discourage spam senders from trying again as they think your MTA is broken.
oriettaxx 2y ago
I would add https://www.mail-tester.com/ https://www.mail-tester.com/ to the testing tool list the free version of https://eu.dmarcian.com https://eu.dmarcian.com (which is great btw, but costs a bit out of my budget) is good, too this is good too https://www.20i.com/blog/dkim-demystified/ https://www.20i.com/blog/dkim-demystified/
- WhyNotHugo 2y ago
  dmarcian won't load in NL: > On February 20th 2024, the Hague Appellate court instructed dmarcian to block dmarcian.com visitors who originate from Dutch Internet Protocol addresses. Apparently they had some trademark dispute with their local representative. The story on their website seems to be missing some key details to understand the whole picture.
  - oriettaxx 2y ago
    oh my that's odd thanks for telling us
cromulent 2y ago
FWIW: Postmark (a service I like using) offers a free DMARC monitoring service. https://dmarc.postmarkapp.com https://dmarc.postmarkapp.com
BOOSTERHIDROGEN 2y ago
is this good? --- SPF --- RFC5321.MailFrom domain: example2.com Auth Result: PASS DMARC Alignment: PASS --- DKIM --- Domain: example2.com Algorithm: rsa-sha256 Auth Result: PASS DMARC Alignment: PASS --- DMARC --- RFC5322.From domain: example2.com Policy (p=): none SPF: PASS DKIM: PASS DMARC Result: PASS
- kemotep 2y ago
  The DMARC policy is none. Meaning if both SPF and DKIM fail then nothing should be done about the email. Now, many email security gateways and spam filters will just have rules automatically blocking anything that fails SPF regardless but you want to eventually get to `p=reject`. Start with something like `p=quarantine; pct=25` to have 25% of reported DMARC policy failures be marked for quarantine, review the reports after a week and then ramp it up to 50%, 75%, 100% every few days. Then if your domain is not having a significant percentage marked for quarantine in your DMARC reports after a week or two, switch to `p=reject; pct=100%` and continue to monitor the reports to make sure everything is good. DMARC is not bulletproof to people using your domain as spam though because even with a reject policy, if SPF fails but DKIM passes, DMARC will pass or vice versa. It helps curb abuse and takes 15 minutes of effort to set up once but still is not enough to kill spam.
deleted 2y ago
[deleted]
VagabundoP 2y ago
I needed to set up an email for my own hobby domain and spun up a little Ubuntu server and ran this: https://mailinabox.email/ https://mailinabox.email/ Worked a treat and did everything. I was sending and receiving email within a few hours.
JohnFen 2y ago
> For example, a business might send promotional emails to its customers. By implementing SPF, DKIM, and DMARC, the business ensures that its emails are not marked as spam But such emails can very well be (and in my experience usually are) spam. My spam filter will take the lack of implementation of these as a strong signal that the email may be spam, but does not think that just because these are implemented, the email isn't spam. As it should. Just wanted to call this out because the article strongly implies that implementing these will let your email bypass spam filters.
patja 2y ago
I am guessing that Apple is one of the top 10 email providers. Why don't they send DMARC reports? Every little podunk email provider seems to be able to send me DMARC reports, as well as Microsoft, Yahoo!, Gmail, Xfinity, and AOL. Every morning I get a pile of them. But never a peep from Apple. They like to reject our email for inscrutable reasons. We don't send bulk email. We only send email (almost entirely transactional) to about 5,000 paying customers who have accounts on our website. Two days ago I was however delighted to receive a reply to an email I had sent to them on May 1 asking for assistance and clarification on why they block us. They told me that the website of one of my wife's employers (not our website or domain), featured in her email signature, was hosting malicious software. It isn't on any blocklists and I can't find any evidence of this currently being the case. Maybe it did at one point and they cleaned it up? I suspect Apple blocks and never revisits if the reason is still valid.

443 points by miles 2y ago

I also wrote a similar post a while ago, after someone was sending emails using my domain name: https://www.uxwizz.com/blog/stop-others-use-your-domain-emails https://www.uxwizz.com/blog/stop-others-use-your-domain-emai...

jprjr 2y ago

What I really want to see is a guide for SPF/DKIM/DMARC oriented towards people writing apps that send email using other people's domains. I have dealt with so many ticketing systems and marketing platforms that do not understand the roles of SPF/DKIM/DMARC at all. Things like, insisting we need to include their SPF record in ours, even going so far as to scan the SPF record for the include, only to find out they use their own domain in the envelope address (which is what I wanted them to do in the first place). Or not distinguishing at all between envelope and header addresses and using our domain in both. Which of course means they're not tracking delayed bounces. It really becomes an issue with larger orgs where everybody wants to use the main domain for brand purposes and subdomains are just totally frowned upon for whatever reason. If you just leave my SPF alone and rely on DKIM, it means you can still pass DMARC and track bounces properly. Hell I'd be fine with making subdomains for the envelope address that lists your infrastructure in the MX records but again, eyes really start to glaze over when you say "envelope address." Basically what I really want is a guide that boils down to: if you're not their primary email provider, then don't touch your client's SPF record.

jabroni_salad 2y ago
I recently set up a mailchimp tenant for someone and was surprised that their email authentication wizard pretty much began and ended with DKIM. I'm way too used to b2b solutions that want the equivalent of a chmod+999 before it can run a hello world.
- TheNewsIsHere 2y ago
  In my experience this approach is becoming more common, but slowly. I still configure my default apex DMARC records (in most scenarios) to enforce strict alignment on both SPF and DKIM, but I’ve been relaxing that on a case by case basis or overriding the apex DMARC policy at a subdomain level and only using DKIM where supported.
  - joveian 2y ago
    There seems to be incorrect information out there about strict alignment requesting more strict checking. Strict vs relaxed alignment is just about the domain being able to send mail from subdomains without extra DNS records on the subdomains (relaxed) or not (strict). The envelope from (for SPF) and d= domain (for DKIM) must match the DNS records used. If you don't use SPF for DMARC you don't use SPF for DMARC and it doesn't matter if the DMARC SPF alignment is strict or relaxed (there is no way in DMARC to require both DKIM and SPF to pass). It is still a good idea to always pass plain (non-aligned) SPF (just based on envelope from) since sometimes this is checked independently from DMARC.
    - TheNewsIsHere 2y ago
      I am slightly confused, and perhaps misunderstanding your framing. I do use DMARC and SPF in the fashion you described. In my environments I typically need to take every measure to ensure only authorized services/servers/senders are sending, via authorized hosts and IPs, and this often changes based on subdomain, so that’s why I use strict alignment. Personally I strive to keep various services strictly separated by (sub)domain.
      - joveian 2y ago
        I thought it might just be ambiguous language. It wasn't clear to me what you were trying to say and it could be read as implying what I have seen (incorrectly) stated in some DMARC descriptions elsewhere. Just trying to prevent any misunderstandings from anyone reading the thread and improve my own understading by checking that I got it right before posting. Thanks for clarifying what you ment :).

velcrovan 2y ago

I manage IT at a mid-size business. At least once a month, I get asked to release some incoming email from quarantine that got sent there because the sender's SPF record is wrong or outdated and doesn't include all the email services they actually use. (What this really tells me is how many small businesses are out there running with no in-house IT expertise or support of any kind.) I don't do whitelisting. Instead, I always reach out and offer to help the other party correct their SPF record. It happens often enough that I wrote a script in Racket that will generate the email for me and paste it into the clipboard [1]. The email tells them exactly what they need to change, and links to docs from their current email provider (so they don't have to trust me about edits to their DNS). [1]: https://gist.github.com/otherjoel/6b8bf02f6db6e0c47ba6bca72ecf1c3e https://gist.github.com/otherjoel/6b8bf02f6db6e0c47ba6bca72e...

luckman212 2y ago
Neat! I must try this.
ziddoap 2y ago
What would you say the normal reception you receive from this email template is? I like the idea, but I would think sending a technical email (with industry-specific acronyms that you don't spell out!) to a business that has no in-house IT would just be ignored in most cases.
- velcrovan 2y ago
  Well if you read the template, you'll see I start out with a non-technical explanation, advise that they forward the email to an IT type person, and offer to help in any way I can. Then I put in a "More info" heading further down with all the details and instructions. Overall I'm pleased with how well this approach works. When people realize that their email is getting stuck in spam filters because of a problem on their end, they're usually motivated to get it fixed. Sometimes it gets sent to an owner who had barely enough tech mojo to stand up a gmail account at a custom domain, and even then the instructions are usually simple enough for them to follow.

NoboruWataya 2y ago

> If you are invovled in developing, supporting, or maintaining an application that sends emails, this guide is a must read. I would also say a guide like this is helpful whenever you are using a custom domain with an email service, and therefore need to set these records yourself. Okay, you might not need to have an in-depth knowledge of these concepts, but it's certainly helpful to understand why your email provider is telling you to set all these weird DNS records.

lovasoa 2y ago

Just today, someone sent me the link to a great tool to debug dmark issues: https://www.learndmarc.com/ https://www.learndmarc.com/

paravz 2y ago
https://www.mailhardener.com https://www.mailhardener.com is another great "validator" and more for a personal email domain
UberFly 2y ago
Just tried it - you're right that is really good. Thanks.
nevster 2y ago
Fantastic!
jesterson 2y ago
Is it just me who finds this tool horrendous? Instead of just giving a report, it makes words fly on the page... Certainly it looks cool and perhaps appealing to some younger generation, but if it is tech analysis, you would expect clear and concise report, not some flying things with explanation like I am 4
- lovasoa 2y ago
  It's called learndmarc, not testdmarc. As someone who wasn't familiar with the technologies involved, I like that I actually learned something while using it to debug my email issue.
oriettaxx 2y ago
uh, you are right, super cool! sysadmins will love it

detourdog 2y ago

When I first implemented the above the next step was going to be ARC. Does anyone have thoughts on ARC? https://www.validity.com/blog/how-to-explain-authenticated-received-chain-arc-in-plain-english/ https://www.validity.com/blog/how-to-explain-authenticated-r...

jabroni_salad 2y ago
It's only really needed if you need to robo-forward stuff between domains. For example if you set up a domain but want to receive emails to your usual gmail. I noticed that cloudflare's email forwarder uses an ARC record and it works a treat.
- lxgr 2y ago
  I recently noticed this when debugging email delivery issues for a family member who have their own TLD, but forward everything to Gmail. Unfortunately, the mail server of the forwarding domain doesn't seem to support ARC, so Gmail frequently throws away everything that doesn't have a DMARC header, since without DMARC the only other option is SPF, which doesn't work for forwarding.
  - jabroni_salad 2y ago
    A lot of small businesses in my area have been bit by that this year. I have to give hostgator/bluehost/godaddy etc kudos for having an email forwarder work reliably for so long but I wish they were more proactive about getting their customers compliant with this. Also it's kinda messed up how much of the small business sector is relying on AOL webmail to operate.
- detourdog 2y ago
  I'm thinking I will add it as part of next upgrade. My domain has so few users and we are chill enough to work through email delivery issues. My understanding at the time was that the receiver verifies that the sender sent it. My current delivery settings are strict so that only my server can delivery our email. I would think I could implement ARC in less strict manner and tighten it up as it becomes more common. Does that seem reasonable? Any better ideas.

njt 2y ago

On a slightly related note, Michael W. Lucas[1] is working on an upcoming book entitled "Run Your Own Mail Server", that will be published shortly (there's a Kickstarter campaign as well[2]). I attended his tutorial and talk at BSDCan[3] this year and both were excellent. I highly recommend buying the book when it comes out (or supporting the Kickstarter), it will go through all the gory details of setting up and running a mail server, and best practices, including a ton of material on SPF/DKIM/DMARC. (P.S. I have no affiliation with the author or the book in any way.) [1] - https://mwl.io/ https://mwl.io/ [2] - https://www.kickstarter.com/projects/mwlucas/run-your-own-mail-server https://www.kickstarter.com/projects/mwlucas/run-your-own-ma... [3] - https://www.bsdcan.org/2024/ https://www.bsdcan.org/2024/

whartung 2y ago
Looking forward to this. First thing I ever ponied up on KS for. I don't even run a mailserver, I'm just hoping it will take a bunch of the guides that have been floating about on the web, consolidate the sharp edges, and make sure its up to date. I also hope it has some discussion on troubleshooting. Like dealing with blacklists and what not, folks always talk about that, but I've never see it documented what is actually done to resolve these problems (Like who do you send an email to, how do you even find out who to send an email to, etc.)

disport 2y ago

Disclaimer: My startup is made to get user domains automatically onboarded to applications, DNS-related things such as SPF/DKIM/DMARC. It's a "Stripe-for-DNS" called cloudvalid.com. I like the quality of this SPF/DKIM/DMARC guide. This is the industry I started out in, and I actually wrote the guides for SendGrid, Amazon SES, and a few other email products. I don't mind saying that author has done a better job at this than me. That being said, I see SPF/DKIM/DMARC guides like this pop up with some regularity, but users continue to have the same level of comprehension as before. I think the nature of this problem is not one that lends itself to being solved by guides. It's the sort of problem that a user is really only faced with once, which means that they're not getting the repetitions in to warrant any long-term comprehension. I'm naturally biased here, but if you're onboarding users with SPF/DKIM/DMARC to your application, it's good if you can just get them setup with automation.

apitman 2y ago
> My startup is made to get user domains automatically onboarded to applications, DNS-related things such as SPF/DKIM/DMARC. It's a "Stripe-for-DNS" called cloudvalid.com. Interesting. Not lot of startups out there talking about Domain Connect on their front page. Seems like a decent amount of overlap with my project TakingNames.io[0]. Feel free to reach out if you want to talk shop. [0]: https://takingnames.io/blog/introducing-takingnames-io https://takingnames.io/blog/introducing-takingnames-io

deleted 2y ago

[deleted]

heavyset_go 2y ago

Tangential, but what is the contemporary go-to for standing up a mail server these days? The last time I had to do so was a decade ago. I remember Mail-in-a-Box being popular at one point, wondering if that's still the case.

jeroenhd 2y ago
Really depends on what you want to get out of your server. Most easy to use server software I know seems to be geared towards personal use and small organisations. Mail-in-a-box still works, though last time I checked they were on quite an old Ubuntu LTS release. There are a few pre-packaged docker containers too (i.e. docker-mailserver) which seem to be popular. I myself use Mailcow, but that's pretty heavy for "just" a mail server. There's also a mail server that I can't for the life of me remember the name of, which packaged a whole bunch of stuff into one single binary you can run rather than use the classic "every part of the email delivery chain is a separate process" approach. I think it was written in Go?
- cuu508 2y ago
  > There's also a mail server that I can't for the life of me remember the name of, which packaged a whole bunch of stuff into one single binary you can run rather than use the classic "every part of the email delivery chain is a separate process" approach. I think it was written in Go? maddy or mox?
bongodongobob 2y ago
For a business? You don't. For hobby/fun? I still wouldn't. I tried years ago and you're just going to fight blacklisting constantly. If you're on a residential/consumer internet service, trying to set it up in your home lab, forget it.
cauterize 2y ago
I’m partial to https://gitlab.com/simple-nixos-mailserver/nixos-mailserver https://gitlab.com/simple-nixos-mailserver/nixos-mailserver Does require some NixOS knowledge, but is nice for upgrades.
2y ago

deleted 2y ago

[deleted]

remram 2y ago

My problem with SPF (& co) is redirections. I have email redirected from other domains into my (Gmail) inbox. For it to arrive, I use SRS, so the email is properly aligned and always makes it into my inbox. The problem is that some of that email is malicious. I have a choice of dropping those mails, and I never see a trace of it in my inbox, or forwarding them with SRS, and they look to Gmail like 100% perfectly good mails sent from my own domain (but still potentially malicious). It's annoying.

alt227 2y ago
> It's annoying. This seems very much like a problem you have created for yourself.
- remram 2y ago
  I know, how dare I use other domains than gmail...? Why post a comment at all, if you have no insight. It's useless and insulting.
  - alt227 2y ago
    Why post on a public forum if you dont want, or even get insulted by, other peoples opinions? Its nothing to do with using domains other than gmail, its that you said you are actively relaying all mail into your gmail account and rewriting the sender as yourself, but then its annoying that spam gets marked as valid mail from your domain. Thats nobodies problem other than your own setup, and there are loads of other ways you could do it. But the way you chose to set it up is 'annoying'. Sorry to insult you, but I feel that warrants letting you know you caused your own problem!
    - remram 2y ago
      If your opinion is "you suck" it's called an insult. You could have ignored my question, you could have explained why I'm doing it wrong, hell you could have posted a random setup guide from Google and downvoted me. Instead you post this attack with no content, justification, or insight. You are being an ass. You are allowed to be, but I'm allowed to call it out. And if you think I set my email up in a sub-optimal way, just so that I could ask for advice on HN, with the goal of pissing you off... you are delusional.

betaby 2y ago

We need more email diversity. Use your own email servers as often as you can. Monocultures of Apple/Google/MS is deterioration for the Internet.

nerdponx 2y ago
And you don't need to run your own server in order to escape the big players. I have been a happy Fastmail customer for many years now.
- TheNewsIsHere 2y ago
  I second Fastmail wholeheartedly. I moved multiple self-hosted (Postfix and Dovecot) environments to Fastmail and haven’t looked back. The only thing I lost was the ability to manually configure an address to bounce back. Sending and receiving as a wildcard alias is fantastic, and Fastmail allowing that was the reason I finally moved. I held out on that feature for a long time. (You can also do this in Exchange Online if you want to run a convoluted and officially unsupported configuration.) I only twice had any kind of RBL issues, and one was a motivating factor in moving. I also got tired of worrying about patching and CVEs. I do that enough for my work.
0xpgm 2y ago
One of the reasons I decided to self-host emails for my domain names. There's no reason it should be hard to host your own email in 2024 even if you're just mildly technical. An Ansible playbook that sets up Debian to host email, all the dependencies from the default Debian repositories. https://github.com/programmer-ke/replatform/ https://github.com/programmer-ke/replatform/

deleted 2y ago

[deleted]

dankai 2y ago

This is a great guide but from my experience, even if you configure it 100% correctly, email services like Gmail may still classify your emails as spam for no apparent reason while not being on any IP or domain blacklist. I tried for hundreds of hours to get around it with no avail, and my emails to Gmail always went to spam unless it was a response to an email from a Gmail address. Had to go back to a 3rd party hosted service (iCloud) because of it.

gerdesj 2y ago
I have had a Gmail account from the days when it was invitation only. The inbox contains spam and my test emails and nothing else! I've run tiny smtp systems for 25 years or so. It can be done. I am based in the UK but at least one of my domains is a .net jobbie, so nominally American. That one still works fine and it is my (ltd) company domain, so all good. The MX records etc have moved around a bit but always very carefully. It all starts around the IP address you are using. Is it "tainted"? is it in a tainted block? If it is then you need to either go elsewhere or clean it up and that takes a bit of time. By clean it up I mean apply for removal from the usual suspect's blocklists - Spamcop (lol), Spamhaus and all the rest that you can find. Now setup PTR records. That has to be done by your ISP. If they can't do it for you, then find a new ISP. If you can't get PTR records to match A records then you may have to give up. One of the first checks an anti spam system will do is reverse look up an incoming IP address and compare it. Also that should match the HELO/EHLO announced by the SMTP MTA: SMTP connection from IP address 12.13.14.15 HELO (my name is) smtp.example.co.uk Receiver will check: smtp.example.co.uk == 12.13.14.15 AND 15.14.13.12.in-addr.arpa == smtp.example.co.uk. Everyone gets their knickers in a twist about SPF, DKIM and DMARC but if you do not get the prior basics of IP -> A -> HELO -> PTR sorted out first then you will fail sooner or later. I also recommend that you ensure your MX records (receiving) match up too with your sending records. It means you can use mx is SPF, for example. If you have multiple internet connections and IPs then be absolutely certain that your inbound and outbound IPs for SMTP match up. Sorted all that? Cool, now proceed to SPF. Most people fail at the PTR stage. If your ISP will not do PTR for you then you are probably screwed for self hosted SMTP. If you cannot change ISP to one that will, then you are really screwed. Sorry. In that case you will have to engage a service that will route SMTP on your behalf. It won't cost much but you won't own it and you will have to pay someone to do it. Soz.

jgalt212 2y ago

Many of our emails were blocked because our SPF record was correct for our domain, but not correct for the Return-Path domain. There is no mention of Return-Path in the guide, but it's essential to get this right. https://easydmarc.com/blog/how-does-an-spf-record-work/ https://easydmarc.com/blog/how-does-an-spf-record-work/

nubinetwork 2y ago

These kinds of articles pop up on HN all the time... Give me a mail server that can use LE for certificates and I'll gladly give DKIM and DMARC a try...

yubiox 2y ago
Sendmail
- nubinetwork 2y ago
  I can't believe sendmail still exists... worst configuration format ever.
inejge 2y ago
You don't need public certificates for DKIM, it uses privately generated ones for as long as you want to keep them. (A security researcher recently found quite a few domains using weak DKIM keys generated by buggy Debian OpenSSL, more than fifteen years ago.)
- nubinetwork 2y ago
  I could swear at one point you needed one, but I just half-setup opendkim and it generated one without me needing to make one by hand... when I get around to updating the DNS on my personal domains, I guess I'll see how things turn out.

kuon 2y ago

I run my own email server, and I'll add that SFP /DKIM and spam filtering should be done at the connection level during SMTP session. Your MTA should not accept a message for delivery that has been filtered. This ensures that the sender of false positive gets a notification from his MTA and also, in my experience, it works very well to discourage spam senders from trying again as they think your MTA is broken.

oriettaxx 2y ago

I would add https://www.mail-tester.com/ https://www.mail-tester.com/ to the testing tool list the free version of https://eu.dmarcian.com https://eu.dmarcian.com (which is great btw, but costs a bit out of my budget) is good, too this is good too https://www.20i.com/blog/dkim-demystified/ https://www.20i.com/blog/dkim-demystified/

WhyNotHugo 2y ago
dmarcian won't load in NL: > On February 20th 2024, the Hague Appellate court instructed dmarcian to block dmarcian.com visitors who originate from Dutch Internet Protocol addresses. Apparently they had some trademark dispute with their local representative. The story on their website seems to be missing some key details to understand the whole picture.
- oriettaxx 2y ago
  oh my that's odd thanks for telling us

cromulent 2y ago

FWIW: Postmark (a service I like using) offers a free DMARC monitoring service. https://dmarc.postmarkapp.com https://dmarc.postmarkapp.com

BOOSTERHIDROGEN 2y ago

is this good? --- SPF --- RFC5321.MailFrom domain: example2.com Auth Result: PASS DMARC Alignment: PASS --- DKIM --- Domain: example2.com Algorithm: rsa-sha256 Auth Result: PASS DMARC Alignment: PASS --- DMARC --- RFC5322.From domain: example2.com Policy (p=): none SPF: PASS DKIM: PASS DMARC Result: PASS

kemotep 2y ago
The DMARC policy is none. Meaning if both SPF and DKIM fail then nothing should be done about the email. Now, many email security gateways and spam filters will just have rules automatically blocking anything that fails SPF regardless but you want to eventually get to `p=reject`. Start with something like `p=quarantine; pct=25` to have 25% of reported DMARC policy failures be marked for quarantine, review the reports after a week and then ramp it up to 50%, 75%, 100% every few days. Then if your domain is not having a significant percentage marked for quarantine in your DMARC reports after a week or two, switch to `p=reject; pct=100%` and continue to monitor the reports to make sure everything is good. DMARC is not bulletproof to people using your domain as spam though because even with a reject policy, if SPF fails but DKIM passes, DMARC will pass or vice versa. It helps curb abuse and takes 15 minutes of effort to set up once but still is not enough to kill spam.

deleted 2y ago

[deleted]

VagabundoP 2y ago

I needed to set up an email for my own hobby domain and spun up a little Ubuntu server and ran this: https://mailinabox.email/ https://mailinabox.email/ Worked a treat and did everything. I was sending and receiving email within a few hours.

JohnFen 2y ago

> For example, a business might send promotional emails to its customers. By implementing SPF, DKIM, and DMARC, the business ensures that its emails are not marked as spam But such emails can very well be (and in my experience usually are) spam. My spam filter will take the lack of implementation of these as a strong signal that the email may be spam, but does not think that just because these are implemented, the email isn't spam. As it should. Just wanted to call this out because the article strongly implies that implementing these will let your email bypass spam filters.

patja 2y ago

I am guessing that Apple is one of the top 10 email providers. Why don't they send DMARC reports? Every little podunk email provider seems to be able to send me DMARC reports, as well as Microsoft, Yahoo!, Gmail, Xfinity, and AOL. Every morning I get a pile of them. But never a peep from Apple. They like to reject our email for inscrutable reasons. We don't send bulk email. We only send email (almost entirely transactional) to about 5,000 paying customers who have accounts on our website. Two days ago I was however delighted to receive a reply to an email I had sent to them on May 1 asking for assistance and clarification on why they block us. They told me that the website of one of my wife's employers (not our website or domain), featured in her email signature, was hosting malicious software. It isn't on any blocklists and I can't find any evidence of this currently being the case. Maybe it did at one point and they cleaned it up? I suspect Apple blocks and never revisits if the reason is still valid.