10 ms·

What's an SPF Record?

156 points by albertgoeswoof 6y ago

XCSme 6y ago
> But SPF alone doesn’t completely solve spam, we also need DKIM, and to a lesser extent DMARC. More on those later! I recently wrote a short article on those, a tldr for setting up basic email authentication and anti-spam: https://www.usertrack.net/blog/stop-others-use-your-domain-emails https://www.usertrack.net/blog/stop-others-use-your-domain-e...
- deleted 6y ago
  [deleted]
- superkuh 6y ago
  What you describe, adding the DNS TXT record for DKIM, is just the icing on the cake. It's way more work and complex at postfix (+rspamd to sign or whatever) level. Suggesting that you can just, >Hostgator -> cPanel -> Email Authentication ...is ignoring 99% of the complexity and long term maintainance that DKIM implies. And why bother? It's not like the mega-corp walled gardens care if you have the perfect mailserver implementation. They'll mark you as spam or blacklist you anyway, because they can, because it's easier, because it's a job someone's doing for profit and you don't effect that. No. An SPF record is enough for determining if mail is legitimately from it's real mailserver. DKIM is cargo cult nonsense for most cases. A sacrifice to the google/microsoft/apple gods to hopefully earn their favor, but one that is almost always ignored anway.
  - XCSme 6y ago
    I agree, mail deliverability is really hard. In my case, the mailserver is only used to use a custom email address and forward the email, so deliverability was not an issue but other people using my domain name to send phishing emails. By making the fixes mentioned in the article the phishing attacks were stopped.
- superasn 6y ago
  That's a good writeup! Aren't DKIM and SPF supposedly doing the same thing of authenticating the email domain? If not, what's the difference?
  - Volundr 6y ago
    It's simplest terms, SPF allows you to prove (and equally importantly disprove) that a piece of mail comes from your domain based on IP address (there are a variety of mechanisms, but in the end it all boils down to IP). Basically SPF spells out specifically which hosts are allowed to send mail from your domain. DKIM allows you to prove a message originates from your domain via a cryptographic signature. So any mailserver with the private key can send messages from your domain, it's not limited to a predetermined subset. In different situations one can be more useful than the other. It's also recommended to implement both, as one or the other might fail for various reasons. It's also worth noting you really should implement DMARC as well, which lets you spell out a policy for what recipients should do if SPF and DMARC fail, including if they should notify you and how.
    - Moru 6y ago
      Just please set up DMARC so it allows forwarding a mail... Have had this problem a number of times lately, companies sending mail to us that just vanish in thin air without any explanation. Turns out an alias adress is forwarding their mail to whoever is supposed to handle their mail at the moment and this is not allowed due to how they set up DMARC. The mail server just drops the mail without warning.
      - beermonster 6y ago
        O2's billing emails, and a well known bank have broken DKIM signatures as they are improperly forwarded. No chance of getting through to their tech support, especially since these types of email are usually from a no-reply@ address and the people manning the phones are not the mail server admins anyway.
        6y ago
deleted 6y ago
[deleted]
_nickwhite 6y ago
I run a few moderately sized corporate e-mail systems. SPF, DKIM, DMARC are all necessities, but unfortunately they don't guarantee delivery. We have much better success sending e-mail out via Mailgun or Sendgrid. Since Verizon swallowed up Yahoo Mail and @aol.com, it's very easy to get on their shadow ban list, and nearly impossible to get off. Good luck telling all the employees they can't email @yahoo.com and @aol.com for at least 2 weeks. I see e-mail as another Internet service that is slowly centralizing, and if you operate outside of the big tech companies- for example, if you don't use Google/Yahoo/M365 e-mail, you're probably going to be in for a headache, even if you run perfect e-mail infrastructure.
- sebmellen 6y ago
  Sadly I've found this to be the case both with perfectly configured personal email servers and when using privacy-preserving email providers like Tutanota. Do you think there's anything that can be done to counter this centralizing force?
  - da_chicken 6y ago
    Now? Almost certainly not. On the side of centralization is a market worth unimagined billions, and perhaps seven providers that already routinely lobby to get what they want. On the side of decentralization? Little guys who are already being bullied and don't have the resources. Politicians who have other extremely dangerous problems currently, including and especially those in their own industry. Don't expect their help for a long time. Smaller service providers who only want the chance to switch sides. And large independent organizations (businesses, schools, governments) who are really only interested in a cheaper bill for the same service quality. Look, we have struggled to fix the MITM/plain text problem and spam/origin problem for well over 20 years, and the Internet Age is maybe 30? It's extremely likely that the only way to fix the core design problems with email is to have the service consolidate to a small consortium of services, have them improve the base design, and then fracture the monopolies. That's going to take decades. And mail is still such a pain in the butt for a user document database. And the user experience is a joke! Asynchronous delivery, best effort failure? I mean, what? The average web page is orders of magnitude larger now! People complain about FAX and FTP.
dillutedfixer 6y ago
One of the most important things I learned about SPF records was that the entire record had to be under 256 bytes. We were sending out emails from multiple servers on one domain, and were having so many deliverability issues because the SPF record had too many entries in it and would be seen as invalid by some spam firewalls. Once I cleaned up the record and split some email out to subdomains our delivery rates went way up. But of course as others mentioned SPF is just the beginning. DKIM, DMARC, reverse records, etc are all very important as well.
- brightball 6y ago
  You can fix this by putting a single SPF entry for each envelope/returnpath domain. SPF scales smoothly that way without any need to overload a single top level record with entries. This will pass DMARC as well. Very common issue.
- annoyingnoob 6y ago
  Sorta depends on how many lookups your record requires. You can just include a second record in the first and gain another 256 bytes.
  - marcbradshaw 6y ago
    * upto a limit of 10 lookups, which is surprisingly easy to hit these days
- wahern 6y ago
  > One of the most important things I learned about SPF records was that the entire record had to be under 256 bytes. It sounds like this is a case of buggy implementations. You can have TXT and even SPF records longer than 255 bytes. The fundamental issue is that TXT records are composed of segments of up to 255 bytes each, where each segment has an 8-bit length header. As Resource Records (RR) individually, as well as DNS packets as a whole, have a maximum length of 65535 (because of 16-bit length headers), DNS supports TXT RRs of approx. 256 segments, 255 bytes each. Both RFC 4408 (https://tools.ietf.org/html/rfc4408#section-3.1.3 https://tools.ietf.org/html/rfc4408#section-3.1.3) and it's successor, RFC 7208 https://tools.ietf.org/html/rfc7208#section-3.3 https://tools.ietf.org/html/rfc7208#section-3.3, require SPF implementations to concatenate the segments and process them as a single string. This means you should be able to publish a single SPF record of approx. 65025 bytes (256 x 255). It's not too surprising implementations might be buggy in this respect. It's a rather esoteric aspect of the TXT RR specification. (If only because few people, including developers of DNS-related tools, bother actually reading the specification!) And unfortunately neither the popular rfc4408-tests.yml nor rfc7208-tests.yml test suites seem to have a test case for long TXT records. Even if a test had been included it likely wouldn't have caught many violators as it's actually a matter of the low-level DNS library providing the proper interface(s), many DNS libraries are deficient in how well they expose DNS record parsing or packet injection interfaces, and most SPF libraries would have likely integrated the test suite at a higher level that didn't involve actually making queries using a low-level DNS library, which can be difficult to do. That said, RFC 4408 says that, > SPF implementations MUST limit the number of mechanisms and modifiers that do DNS lookups to at most 10 per SPF check, including any lookups caused by the use of the "include" mechanism or the "redirect" modifier. IME experience this limit is rather low and many, many SPF policies require more than 10 queries. (And if you need SPF policies larger than 255 bytes, you're probably skirting this or other similar limits.) When I was writing and deploying my own implementation I very quickly realized I needed to increase the minimum maximums, but choosing that value is a total crap shoot.
codazoda 6y ago
Apparently, having multiple A Records can cause problems with spf records. The founder of hanami.run, an email forwarding service, showed me how to fix mine.
- qu1mby 6y ago
  What was the fix?
marcbradshaw 6y ago
"We set a specific header in the email called Return-Path with all emails sent through our email service. If you take a peek at a raw email sent over OhMySMTP you’ll find that the return path ends in @mailer.ohmysmtp.com, and if you look this up in the DNS system, you’ll find an SPF record that points to our server IP address." While this will mean that and mail sent will pass SPF, it doesn't play well with DMARC. The SPF pass domain won't align with the From header domain, so the SPF authentication will be ignored. "But SPF alone doesn’t completely solve spam, we also need DKIM, and to a lesser extent DMARC. More on those later!" Realistically, SPF doesn't do anything to solve spam, nor does DKIM or DMARC. All of these technologies address spoofing and give a greater confidence of the sender responsible for the email which can then be used in reputation engines. Passing SPF/DKIM/DMARC say nothing about the message contents.
smiley1437 6y ago
I've always wondered - what could have been done to the original specification for email that would have avoided this hellscape of spam and antispam measures? Or does spam appear in any popular messaging system anyways because it is such a strong selective pressure?
- DanBC 6y ago
  People who send spam never think it is spam. They often think their UBE is somehow different from all the other stuff. They claim that the recipients want this bit of email, even if those same recipients complain about getting spam.
m-p-3 6y ago
This made me realize I had no SPF record set on my personal domain name, which I don't use for sending emails. https://serverfault.com/a/293123 https://serverfault.com/a/293123
citrin_ru 6y ago
SPF, DKIM, DMARC are must have for today mail and yet many domain owners fail to make even the first and easiest step - publish a valid SPF record. Some mistakes I commonly see (don't do this): * no space between directives: "v=spf1 ip4:192.0.2.0/24 include:example.org-all" * space inside a directive: "v=spf1 ip4:192. 0.2.0/24 -all" * bad mechanism: "v=spf1 ipv4:192.0.2.1 -all" * no mechanism: "v=spf1 192.0.2.1 example.com" * = instead of : "v=spf1 include=example.com" * Unicode, mostly dashes (but I've seen zero-width spaces too): "v=spf1 —all" * Two SPF records for the same domain: "v=spf1 mx:example.com -all", "v=spd1 include:example.net ?all" Don't know where it coming from, but IMHO we see a generation of developers/admins raised by Google search: you can type a search query with mistakes and still get what you want. Or may be web browsers to blame - you can screw up HTML/CSS in dozens ways and they'll render content anyway. And they start to think that all IT systems forgive small mistakes and typos. They just don't get that sometimes you have to read RFC (or at least simplified how-to) and follow it exactly.
deleted 6y ago
[deleted]

> But SPF alone doesn’t completely solve spam, we also need DKIM, and to a lesser extent DMARC. More on those later! I recently wrote a short article on those, a tldr for setting up basic email authentication and anti-spam: https://www.usertrack.net/blog/stop-others-use-your-domain-emails https://www.usertrack.net/blog/stop-others-use-your-domain-e...

deleted 6y ago
[deleted]
superkuh 6y ago
What you describe, adding the DNS TXT record for DKIM, is just the icing on the cake. It's way more work and complex at postfix (+rspamd to sign or whatever) level. Suggesting that you can just, >Hostgator -> cPanel -> Email Authentication ...is ignoring 99% of the complexity and long term maintainance that DKIM implies. And why bother? It's not like the mega-corp walled gardens care if you have the perfect mailserver implementation. They'll mark you as spam or blacklist you anyway, because they can, because it's easier, because it's a job someone's doing for profit and you don't effect that. No. An SPF record is enough for determining if mail is legitimately from it's real mailserver. DKIM is cargo cult nonsense for most cases. A sacrifice to the google/microsoft/apple gods to hopefully earn their favor, but one that is almost always ignored anway.
- XCSme 6y ago
  I agree, mail deliverability is really hard. In my case, the mailserver is only used to use a custom email address and forward the email, so deliverability was not an issue but other people using my domain name to send phishing emails. By making the fixes mentioned in the article the phishing attacks were stopped.
superasn 6y ago
That's a good writeup! Aren't DKIM and SPF supposedly doing the same thing of authenticating the email domain? If not, what's the difference?
- Volundr 6y ago
  It's simplest terms, SPF allows you to prove (and equally importantly disprove) that a piece of mail comes from your domain based on IP address (there are a variety of mechanisms, but in the end it all boils down to IP). Basically SPF spells out specifically which hosts are allowed to send mail from your domain. DKIM allows you to prove a message originates from your domain via a cryptographic signature. So any mailserver with the private key can send messages from your domain, it's not limited to a predetermined subset. In different situations one can be more useful than the other. It's also recommended to implement both, as one or the other might fail for various reasons. It's also worth noting you really should implement DMARC as well, which lets you spell out a policy for what recipients should do if SPF and DMARC fail, including if they should notify you and how.
  - Moru 6y ago
    Just please set up DMARC so it allows forwarding a mail... Have had this problem a number of times lately, companies sending mail to us that just vanish in thin air without any explanation. Turns out an alias adress is forwarding their mail to whoever is supposed to handle their mail at the moment and this is not allowed due to how they set up DMARC. The mail server just drops the mail without warning.
    - beermonster 6y ago
      O2's billing emails, and a well known bank have broken DKIM signatures as they are improperly forwarded. No chance of getting through to their tech support, especially since these types of email are usually from a no-reply@ address and the people manning the phones are not the mail server admins anyway.
      - 6y ago

deleted 6y ago

[deleted]

_nickwhite 6y ago

I run a few moderately sized corporate e-mail systems. SPF, DKIM, DMARC are all necessities, but unfortunately they don't guarantee delivery. We have much better success sending e-mail out via Mailgun or Sendgrid. Since Verizon swallowed up Yahoo Mail and @aol.com, it's very easy to get on their shadow ban list, and nearly impossible to get off. Good luck telling all the employees they can't email @yahoo.com and @aol.com for at least 2 weeks. I see e-mail as another Internet service that is slowly centralizing, and if you operate outside of the big tech companies- for example, if you don't use Google/Yahoo/M365 e-mail, you're probably going to be in for a headache, even if you run perfect e-mail infrastructure.

sebmellen 6y ago
Sadly I've found this to be the case both with perfectly configured personal email servers and when using privacy-preserving email providers like Tutanota. Do you think there's anything that can be done to counter this centralizing force?
- da_chicken 6y ago
  Now? Almost certainly not. On the side of centralization is a market worth unimagined billions, and perhaps seven providers that already routinely lobby to get what they want. On the side of decentralization? Little guys who are already being bullied and don't have the resources. Politicians who have other extremely dangerous problems currently, including and especially those in their own industry. Don't expect their help for a long time. Smaller service providers who only want the chance to switch sides. And large independent organizations (businesses, schools, governments) who are really only interested in a cheaper bill for the same service quality. Look, we have struggled to fix the MITM/plain text problem and spam/origin problem for well over 20 years, and the Internet Age is maybe 30? It's extremely likely that the only way to fix the core design problems with email is to have the service consolidate to a small consortium of services, have them improve the base design, and then fracture the monopolies. That's going to take decades. And mail is still such a pain in the butt for a user document database. And the user experience is a joke! Asynchronous delivery, best effort failure? I mean, what? The average web page is orders of magnitude larger now! People complain about FAX and FTP.

dillutedfixer 6y ago

One of the most important things I learned about SPF records was that the entire record had to be under 256 bytes. We were sending out emails from multiple servers on one domain, and were having so many deliverability issues because the SPF record had too many entries in it and would be seen as invalid by some spam firewalls. Once I cleaned up the record and split some email out to subdomains our delivery rates went way up. But of course as others mentioned SPF is just the beginning. DKIM, DMARC, reverse records, etc are all very important as well.

brightball 6y ago
You can fix this by putting a single SPF entry for each envelope/returnpath domain. SPF scales smoothly that way without any need to overload a single top level record with entries. This will pass DMARC as well. Very common issue.
annoyingnoob 6y ago
Sorta depends on how many lookups your record requires. You can just include a second record in the first and gain another 256 bytes.
- marcbradshaw 6y ago
  * upto a limit of 10 lookups, which is surprisingly easy to hit these days
wahern 6y ago
> One of the most important things I learned about SPF records was that the entire record had to be under 256 bytes. It sounds like this is a case of buggy implementations. You can have TXT and even SPF records longer than 255 bytes. The fundamental issue is that TXT records are composed of segments of up to 255 bytes each, where each segment has an 8-bit length header. As Resource Records (RR) individually, as well as DNS packets as a whole, have a maximum length of 65535 (because of 16-bit length headers), DNS supports TXT RRs of approx. 256 segments, 255 bytes each. Both RFC 4408 (https://tools.ietf.org/html/rfc4408#section-3.1.3 https://tools.ietf.org/html/rfc4408#section-3.1.3) and it's successor, RFC 7208 https://tools.ietf.org/html/rfc7208#section-3.3 https://tools.ietf.org/html/rfc7208#section-3.3, require SPF implementations to concatenate the segments and process them as a single string. This means you should be able to publish a single SPF record of approx. 65025 bytes (256 x 255). It's not too surprising implementations might be buggy in this respect. It's a rather esoteric aspect of the TXT RR specification. (If only because few people, including developers of DNS-related tools, bother actually reading the specification!) And unfortunately neither the popular rfc4408-tests.yml nor rfc7208-tests.yml test suites seem to have a test case for long TXT records. Even if a test had been included it likely wouldn't have caught many violators as it's actually a matter of the low-level DNS library providing the proper interface(s), many DNS libraries are deficient in how well they expose DNS record parsing or packet injection interfaces, and most SPF libraries would have likely integrated the test suite at a higher level that didn't involve actually making queries using a low-level DNS library, which can be difficult to do. That said, RFC 4408 says that, > SPF implementations MUST limit the number of mechanisms and modifiers that do DNS lookups to at most 10 per SPF check, including any lookups caused by the use of the "include" mechanism or the "redirect" modifier. IME experience this limit is rather low and many, many SPF policies require more than 10 queries. (And if you need SPF policies larger than 255 bytes, you're probably skirting this or other similar limits.) When I was writing and deploying my own implementation I very quickly realized I needed to increase the minimum maximums, but choosing that value is a total crap shoot.

codazoda 6y ago

Apparently, having multiple A Records can cause problems with spf records. The founder of hanami.run, an email forwarding service, showed me how to fix mine.

qu1mby 6y ago
What was the fix?

marcbradshaw 6y ago

"We set a specific header in the email called Return-Path with all emails sent through our email service. If you take a peek at a raw email sent over OhMySMTP you’ll find that the return path ends in @mailer.ohmysmtp.com, and if you look this up in the DNS system, you’ll find an SPF record that points to our server IP address." While this will mean that and mail sent will pass SPF, it doesn't play well with DMARC. The SPF pass domain won't align with the From header domain, so the SPF authentication will be ignored. "But SPF alone doesn’t completely solve spam, we also need DKIM, and to a lesser extent DMARC. More on those later!" Realistically, SPF doesn't do anything to solve spam, nor does DKIM or DMARC. All of these technologies address spoofing and give a greater confidence of the sender responsible for the email which can then be used in reputation engines. Passing SPF/DKIM/DMARC say nothing about the message contents.

smiley1437 6y ago

I've always wondered - what could have been done to the original specification for email that would have avoided this hellscape of spam and antispam measures? Or does spam appear in any popular messaging system anyways because it is such a strong selective pressure?

DanBC 6y ago
People who send spam never think it is spam. They often think their UBE is somehow different from all the other stuff. They claim that the recipients want this bit of email, even if those same recipients complain about getting spam.

m-p-3 6y ago

This made me realize I had no SPF record set on my personal domain name, which I don't use for sending emails. https://serverfault.com/a/293123 https://serverfault.com/a/293123

citrin_ru 6y ago

SPF, DKIM, DMARC are must have for today mail and yet many domain owners fail to make even the first and easiest step - publish a valid SPF record. Some mistakes I commonly see (don't do this): * no space between directives: "v=spf1 ip4:192.0.2.0/24 include:example.org-all" * space inside a directive: "v=spf1 ip4:192. 0.2.0/24 -all" * bad mechanism: "v=spf1 ipv4:192.0.2.1 -all" * no mechanism: "v=spf1 192.0.2.1 example.com" * = instead of : "v=spf1 include=example.com" * Unicode, mostly dashes (but I've seen zero-width spaces too): "v=spf1 —all" * Two SPF records for the same domain: "v=spf1 mx:example.com -all", "v=spd1 include:example.net ?all" Don't know where it coming from, but IMHO we see a generation of developers/admins raised by Google search: you can type a search query with mistakes and still get what you want. Or may be web browsers to blame - you can screw up HTML/CSS in dozens ways and they'll render content anyway. And they start to think that all IT systems forgive small mistakes and typos. They just don't get that sometimes you have to read RFC (or at least simplified how-to) and follow it exactly.

deleted 6y ago

[deleted]