5 ms·

Use SPF softfail, not (hard) fail

28 points by LeonM 4y ago

LinuxBender 4y ago
I will personally always use hard fail. If I need to relay mail then I configure the receiving site to make an exception for the relay. If that isn't possible then I do not relay. In a corporation this requires some level of discipline and knowing what is sending emails and why. Security teams should have a weekly test to see if they can spoof senders in their corporate email both from inside their company and from untrusted external IP's. I have found it too easy to spoof people in legal, security and leadership in most companies. Adding to this, hard fail should be used with parked domains without exception. If a domain is not used for sending emails then it should have a null MX and hard fail SPF stating not to trust any IP's or domains. @ 1d in mx 0 . @ 1din txt "v=spf1 -all" @ 1d in caa 0 issue ";" @ 1d in caa 0 issuewild ";" dmarc 1d in txt "v=DMARC1;p=reject;sp=reject;pct=100;aspf=s;" _domainkey 1d in txt "v=DKIM1; p=" _bimi 1d in txt "v=BIMI1; l=; a=;" * 1d in mx 0 . * 1d in txt "v=spf1 -all" *._domainkey 1d in txt "v=DKIM1; p=" * 1d in caa 0 issue ";" * 1d in caa 0 issuewild ";"
- LeonM 4y ago
  > Adding to this, hard fail should be used with parked domains without exception. If a domain is not used for sending emails then it should have a null MX and hard fail SPF stating not to trust any IP's or domains. This is correct, but this is also mentioned in the article multiple times. To add to that, a DMARC record with a "reject" policy is also recommended for domains that are not supposed to send any email, see also: https://www.mailhardener.com/kb/hardening-unused-domains https://www.mailhardener.com/kb/hardening-unused-domains
  - LinuxBender 4y ago
    Also null BIMI records as well, especially if one is going to use soft-fail. Or maybe a malware symbol.
    - LeonM 4y ago
      > Also null BIMI records as well Could be useful, but a successful attack on BIMI would require: - Getting access to the DNS of the domain (in which case the null record will simply be replaced) - Obtaining a VMC for the domain, which should not be possible due to the extended validation requirement for a VMC (such as: phone validation) - Spending ~$800 on a VMC > especially if one is going to use soft-fail. As explained in the article, if DMARC is deployed then softfail is equal to (hard) fail. With DMARC anything that is not an SPF pass, is not aligned, thus a fail. The only exception here would be a very old legacy email system that doesn't support DMARC. But those will also not support BIMI anyway. > Or maybe a malware symbol. BIMI should always require a VMC. It should be impossible to obtain a VMC with a 'malware symbol'. Unless of course you are willing to pay a lot to get it registered as your company logo at your trademark office.
- throwaway67743 4y ago
  I have the same aside from CA related (I don't see the point, if I lose control of a domain TLS is not a priority, and I can't stop rogue CA issuing a certificate as CAA is basically unsupported) - also no bimi as it's a solution looking for a problem and not even in the right place.
- pjkundert 4y ago
  As a result, emails from your domain will never reach anyone who has a "forwarding" email address: for example, someone who uses Cloudflare to manage their incoming email for their domains, and forwards it to their real email account. This is a lot of people. Why are you OK with that?
snowwrestler 4y ago
Hard fail is juggling with running chainsaws. If you do not absolutely have your shit together, you might lose something you really wanted to keep. Most orgs do not have their shit together. So if you are offering a service in the email space, telling your customers to set hard fail is just going to drive up your complaints and support costs. It makes total sense that this company is advising soft fail. Most people can’t juggle chainsaws safely.
- johnklos 4y ago
  If configuring SPF is juggling chainsaws, then there's no hope for humankind.
  - snowwrestler 4y ago
    Configuring SPF is easy. Keeping perfect track of all the various departments, services, vendors, IP addresses, etc sending on behalf of your domain is very hard in many organizations.
    - throwaway67743 4y ago
      It's almost like a working tech department and hierarchy might make sense (also includes mean you can delegate some, but I wouldn't)
    - bradknowles 4y ago
      Even if you can keep all that shit running fine for your organization, you can't make any guarantees with regards to these issues for all the other organizations on the planet. In fact, I would argue that you MUST assume that at least some non-trivial percentage of those other organizations are actually fucked up with regards to one or more of these areas. So, therefore, you must be even more conservative than you might otherwise have been. Even if everyone on the planet is a CMM-5 level organization, you're still going to lose the odd space shuttle or two. And maybe also some rockets. The forwarding problem has been with us since Internet e-mail was first created, and I don't see that problem going away. Ever. All you can do is decide how you're going to handle that for your own accounts where you set up forwarding, which may very well break e-mail coming from domains that don't fully understand the problem.
technion 4y ago
Look I get it on a technical level. But we've just had years of being yelled for anything but a hard fail. It's become a meme that anyone with a bug bounty program and softfail can expect to have it come in as a "vulnerability" no matter how out of scope you make it. It's valid to question and change strategies, but I don't think it's a good approach that people who've been chastised for years for softfails are suddenly being chasisted for hard fails. Edit: Note that users signing up to Office 365 are told to use hardfail: https://learn.microsoft.com/en-us/microsoft-365/security/office-365-security/email-authentication-spf-configure?view=o365-worldwide https://learn.microsoft.com/en-us/microsoft-365/security/off...
- LeonM 4y ago
  MS can't expect all their users to use DMARC, and blatantly advising their customers to do so can be dangerous. Not every domain is ready for DMARC and enabling it would cause undeliverable email. It wouldn't surprise me if MS was being harassed by angry 'experts' for not recommending 'hard' fail, because clearly that would be so much more secure (just look al all the comments in this topic). So I get it, advising to use a strict failure mode, even at the expense of some undeliverable email is acceptable for MS. But for those who have already adopted DMARC, the SPF 'hard' fail only increases the chance of undeliverable mail, while offering zero security benefits.
johnklos 4y ago
This is just flatly wrong: "One of them is that a relaying SMTP service (sometimes called a 'forwarder') will break SPF. This is known as the relaying problem." If relaying is happening, and you haven't added the relay to your SPF, then you're wrong. There is no scenario where this is problem with SPF. "If the email is bounced on the SMTP level due to an SPF 'hard' fail, the receiver may not perform further evaluation of the email, ignoring DKIM and DMARC." Well, of course. If SPF is configured to disallow email, then further evaluation isn't necessary. Either fix SPF, or don't blame SPF. These people are basically advocating for being OK with not configuring SPF properly. No. Just configure SPF properly.
- 0x073 4y ago
  If i receive an mail from e.g. paypal@paypal.com to myself@xyz.com that has no real mailbox but forwards the mail to myself2815@gmail.com. 1. paypal SMTP sends mail to myself@xyz.com 2. xyz.com receive mail SPF is valid 3. xyz.com sends the paypal mail to myself2815@gmail.com 4. gmail.com receives mail SPF is invalid I can't add XYZ.com to Paypal SPF and hardfail would mean email loss in this case. SPF is broken by design for many cases and should just used with softfail and dkim/dmarc. There is arc but it has its own problems.
  - throwaway67743 4y ago
    No, forwarding is broken by design, it always was. Configure whatever forwarding you use to do it properly, it's certainly doable (or just don't use gmail but a provider capable of handling this very common, if wrong, scenario)
    - pjkundert 4y ago
      SMTP "forwarding" (sending an email message with a header "To: <originaladdress>", to a different "RCPT TO: <differentaddress>") is "broken by design"? Sure, it's probably silly. But, SMTP is what it is, and you're not going to change it by complaining SPF's design just ... forgot about this, which happens all the time for email. Every MTA -- literally every forwarding MTA -- does this all day long, and every receiving MTA (which looks at the "Return-Path: <originaladdress>" to lookup the SPF TXT record), gets your fancy "v=spf1 ... nothing w/ forwarding MTA in it ... -all" record, and summarily executes the email w/ SPF: FAIL? It's just wrong. SPF can only serve as one (strong) hint that an email might be correct, if it passes. If it fails, all your MTA can assume is: "well, the email was very likely forwarded"; what else does it have going for it? If its "DKIM-Signature: ..." header is valid, that it's almost certainly A-OK! So, why do you want all your domain's nicely DKIM-signed emails dropped, just because some recipient decided to use a "forwarding" email address? That seems silly.
throwaway67743 4y ago
I use hard fail, because all of the problems described in the article are implementation (checking if mail from and from: are on the same domain/subdomain) and idiocy (I've seen people forwarding mail to Google workspace hosted stuff by their own Mx without stripping or rewriting headers) problems. The second one is really dumb, and results in 99.999% of my dmarc reports rather than actual forged emails. All of these are solvable by anyone with a basic understanding of mail server operation. Soft fail is also pointless, if you can't manage SPF records then you have no business emitting mail, pay a competent professional. (As a side point, there are a lot of inaccuracies in the article but I expect the target audience is not me)
- LeonM 4y ago
  > (As a side point, there are a lot of inaccuracies in the article but I expect the target audience is not me) Could you elaborate on those?
- pjkundert 4y ago
  You cannot "strip off and rewrite" headers without destroying the email message's DKIM signature -- which is the primary assurance of authenticity of incoming email! Also, you are ensuring that emails from your domain can never reach anyone who has a "forwarding" email address (which is anyone managing any significant number of domains). These people use Cloudflare, etc. to manage the MX for these domains, and forwarding incoming email to their central email address (which remains hidden). You can't get away from email forwarding; its sort of baked into the underlying assumptions of SMTP, which you are unlikely to single-handedly defeat and replace with something better. In the mean time, email from your domain will appear "broken" to a vast multitude of the world's email-receiving population. Your domain, not theirs.
  - throwaway67743 4y ago
    Yes, yes you can, that is exactly what you need to do to avoid DKIM failure, you've already broken the ability to verify anything by rewriting the destination. Also see: forwarding is broken (purely forwarding mail via SMTP is not the same as what the comment was about)
ehPReth 4y ago
isn't ARC supposed to fix the forwarding problem? https://en.wikipedia.org/wiki/Authenticated_Received_Chain https://en.wikipedia.org/wiki/Authenticated_Received_Chain
- LeonM 4y ago
  It does, but you can't rely on ARC being implemented by the receiver. ARC is one of the latest additions to email standards (introduced in 2019), there are a lot of legacy email systems that don't support it. If an SMTP service supports ARC, then it will probably also support DMARC (introduced in 2015) and DKIM (introduced in 2011). So you'll still be better of with using DKIM, as DKIM adds cryptographic proof of authentication and integrity, whereas SPF is just a simple allow list. The big challenge with email is finding a happy medium between security, and backwards compatibility for legacy systems.
citrin_ru 4y ago
In practice difference between SPF softfail and SPF hardfail is less significant than in theory (where everyone follows RFC7208 to the letter). Many mail system don't reject a message in case of SPF fail - SPF result is used one of many inputs for an anti-spam system. If we have SPF pass we can assume that a message was sent from an infrastructure approved by the domain owner and if domain has good reputation we can take it into account. If we don't have SPF pass then the sender address in the message can be spoofed. Be it SPF softfail or hardfail or even a missing SPF record we cannot trust the sender address. But if otherwise message looks good it still can be accepted. DMARC makes things a bit more complicated but it is similar - either you have DMARC pass (which is good) or you don't have it and why exactly you don't have DMARC pass is less important.

28 points by LeonM 4y ago

I will personally always use hard fail. If I need to relay mail then I configure the receiving site to make an exception for the relay. If that isn't possible then I do not relay. In a corporation this requires some level of discipline and knowing what is sending emails and why. Security teams should have a weekly test to see if they can spoof senders in their corporate email both from inside their company and from untrusted external IP's. I have found it too easy to spoof people in legal, security and leadership in most companies. Adding to this, hard fail should be used with parked domains without exception. If a domain is not used for sending emails then it should have a null MX and hard fail SPF stating not to trust any IP's or domains. @ 1d in mx 0 . @ 1din txt "v=spf1 -all" @ 1d in caa 0 issue ";" @ 1d in caa 0 issuewild ";" dmarc 1d in txt "v=DMARC1;p=reject;sp=reject;pct=100;aspf=s;" _domainkey 1d in txt "v=DKIM1; p=" _bimi 1d in txt "v=BIMI1; l=; a=;" * 1d in mx 0 . * 1d in txt "v=spf1 -all" *._domainkey 1d in txt "v=DKIM1; p=" * 1d in caa 0 issue ";" * 1d in caa 0 issuewild ";"

LeonM 4y ago
> Adding to this, hard fail should be used with parked domains without exception. If a domain is not used for sending emails then it should have a null MX and hard fail SPF stating not to trust any IP's or domains. This is correct, but this is also mentioned in the article multiple times. To add to that, a DMARC record with a "reject" policy is also recommended for domains that are not supposed to send any email, see also: https://www.mailhardener.com/kb/hardening-unused-domains https://www.mailhardener.com/kb/hardening-unused-domains
- LinuxBender 4y ago
  Also null BIMI records as well, especially if one is going to use soft-fail. Or maybe a malware symbol.
  - LeonM 4y ago
    > Also null BIMI records as well Could be useful, but a successful attack on BIMI would require: - Getting access to the DNS of the domain (in which case the null record will simply be replaced) - Obtaining a VMC for the domain, which should not be possible due to the extended validation requirement for a VMC (such as: phone validation) - Spending ~$800 on a VMC > especially if one is going to use soft-fail. As explained in the article, if DMARC is deployed then softfail is equal to (hard) fail. With DMARC anything that is not an SPF pass, is not aligned, thus a fail. The only exception here would be a very old legacy email system that doesn't support DMARC. But those will also not support BIMI anyway. > Or maybe a malware symbol. BIMI should always require a VMC. It should be impossible to obtain a VMC with a 'malware symbol'. Unless of course you are willing to pay a lot to get it registered as your company logo at your trademark office.
throwaway67743 4y ago
I have the same aside from CA related (I don't see the point, if I lose control of a domain TLS is not a priority, and I can't stop rogue CA issuing a certificate as CAA is basically unsupported) - also no bimi as it's a solution looking for a problem and not even in the right place.
pjkundert 4y ago
As a result, emails from your domain will never reach anyone who has a "forwarding" email address: for example, someone who uses Cloudflare to manage their incoming email for their domains, and forwards it to their real email account. This is a lot of people. Why are you OK with that?

snowwrestler 4y ago

Hard fail is juggling with running chainsaws. If you do not absolutely have your shit together, you might lose something you really wanted to keep. Most orgs do not have their shit together. So if you are offering a service in the email space, telling your customers to set hard fail is just going to drive up your complaints and support costs. It makes total sense that this company is advising soft fail. Most people can’t juggle chainsaws safely.

johnklos 4y ago
If configuring SPF is juggling chainsaws, then there's no hope for humankind.
- snowwrestler 4y ago
  Configuring SPF is easy. Keeping perfect track of all the various departments, services, vendors, IP addresses, etc sending on behalf of your domain is very hard in many organizations.
  - throwaway67743 4y ago
    It's almost like a working tech department and hierarchy might make sense (also includes mean you can delegate some, but I wouldn't)
  - bradknowles 4y ago
    Even if you can keep all that shit running fine for your organization, you can't make any guarantees with regards to these issues for all the other organizations on the planet. In fact, I would argue that you MUST assume that at least some non-trivial percentage of those other organizations are actually fucked up with regards to one or more of these areas. So, therefore, you must be even more conservative than you might otherwise have been. Even if everyone on the planet is a CMM-5 level organization, you're still going to lose the odd space shuttle or two. And maybe also some rockets. The forwarding problem has been with us since Internet e-mail was first created, and I don't see that problem going away. Ever. All you can do is decide how you're going to handle that for your own accounts where you set up forwarding, which may very well break e-mail coming from domains that don't fully understand the problem.

technion 4y ago

Look I get it on a technical level. But we've just had years of being yelled for anything but a hard fail. It's become a meme that anyone with a bug bounty program and softfail can expect to have it come in as a "vulnerability" no matter how out of scope you make it. It's valid to question and change strategies, but I don't think it's a good approach that people who've been chastised for years for softfails are suddenly being chasisted for hard fails. Edit: Note that users signing up to Office 365 are told to use hardfail: https://learn.microsoft.com/en-us/microsoft-365/security/office-365-security/email-authentication-spf-configure?view=o365-worldwide https://learn.microsoft.com/en-us/microsoft-365/security/off...

LeonM 4y ago
MS can't expect all their users to use DMARC, and blatantly advising their customers to do so can be dangerous. Not every domain is ready for DMARC and enabling it would cause undeliverable email. It wouldn't surprise me if MS was being harassed by angry 'experts' for not recommending 'hard' fail, because clearly that would be so much more secure (just look al all the comments in this topic). So I get it, advising to use a strict failure mode, even at the expense of some undeliverable email is acceptable for MS. But for those who have already adopted DMARC, the SPF 'hard' fail only increases the chance of undeliverable mail, while offering zero security benefits.

johnklos 4y ago

This is just flatly wrong: "One of them is that a relaying SMTP service (sometimes called a 'forwarder') will break SPF. This is known as the relaying problem." If relaying is happening, and you haven't added the relay to your SPF, then you're wrong. There is no scenario where this is problem with SPF. "If the email is bounced on the SMTP level due to an SPF 'hard' fail, the receiver may not perform further evaluation of the email, ignoring DKIM and DMARC." Well, of course. If SPF is configured to disallow email, then further evaluation isn't necessary. Either fix SPF, or don't blame SPF. These people are basically advocating for being OK with not configuring SPF properly. No. Just configure SPF properly.

0x073 4y ago
If i receive an mail from e.g. paypal@paypal.com to myself@xyz.com that has no real mailbox but forwards the mail to myself2815@gmail.com. 1. paypal SMTP sends mail to myself@xyz.com 2. xyz.com receive mail SPF is valid 3. xyz.com sends the paypal mail to myself2815@gmail.com 4. gmail.com receives mail SPF is invalid I can't add XYZ.com to Paypal SPF and hardfail would mean email loss in this case. SPF is broken by design for many cases and should just used with softfail and dkim/dmarc. There is arc but it has its own problems.
- throwaway67743 4y ago
  No, forwarding is broken by design, it always was. Configure whatever forwarding you use to do it properly, it's certainly doable (or just don't use gmail but a provider capable of handling this very common, if wrong, scenario)
  - pjkundert 4y ago
    SMTP "forwarding" (sending an email message with a header "To: <originaladdress>", to a different "RCPT TO: <differentaddress>") is "broken by design"? Sure, it's probably silly. But, SMTP is what it is, and you're not going to change it by complaining SPF's design just ... forgot about this, which happens all the time for email. Every MTA -- literally every forwarding MTA -- does this all day long, and every receiving MTA (which looks at the "Return-Path: <originaladdress>" to lookup the SPF TXT record), gets your fancy "v=spf1 ... nothing w/ forwarding MTA in it ... -all" record, and summarily executes the email w/ SPF: FAIL? It's just wrong. SPF can only serve as one (strong) hint that an email might be correct, if it passes. If it fails, all your MTA can assume is: "well, the email was very likely forwarded"; what else does it have going for it? If its "DKIM-Signature: ..." header is valid, that it's almost certainly A-OK! So, why do you want all your domain's nicely DKIM-signed emails dropped, just because some recipient decided to use a "forwarding" email address? That seems silly.

throwaway67743 4y ago

I use hard fail, because all of the problems described in the article are implementation (checking if mail from and from: are on the same domain/subdomain) and idiocy (I've seen people forwarding mail to Google workspace hosted stuff by their own Mx without stripping or rewriting headers) problems. The second one is really dumb, and results in 99.999% of my dmarc reports rather than actual forged emails. All of these are solvable by anyone with a basic understanding of mail server operation. Soft fail is also pointless, if you can't manage SPF records then you have no business emitting mail, pay a competent professional. (As a side point, there are a lot of inaccuracies in the article but I expect the target audience is not me)

LeonM 4y ago
> (As a side point, there are a lot of inaccuracies in the article but I expect the target audience is not me) Could you elaborate on those?
pjkundert 4y ago
You cannot "strip off and rewrite" headers without destroying the email message's DKIM signature -- which is the primary assurance of authenticity of incoming email! Also, you are ensuring that emails from your domain can never reach anyone who has a "forwarding" email address (which is anyone managing any significant number of domains). These people use Cloudflare, etc. to manage the MX for these domains, and forwarding incoming email to their central email address (which remains hidden). You can't get away from email forwarding; its sort of baked into the underlying assumptions of SMTP, which you are unlikely to single-handedly defeat and replace with something better. In the mean time, email from your domain will appear "broken" to a vast multitude of the world's email-receiving population. Your domain, not theirs.
- throwaway67743 4y ago
  Yes, yes you can, that is exactly what you need to do to avoid DKIM failure, you've already broken the ability to verify anything by rewriting the destination. Also see: forwarding is broken (purely forwarding mail via SMTP is not the same as what the comment was about)

ehPReth 4y ago

isn't ARC supposed to fix the forwarding problem? https://en.wikipedia.org/wiki/Authenticated_Received_Chain https://en.wikipedia.org/wiki/Authenticated_Received_Chain

LeonM 4y ago
It does, but you can't rely on ARC being implemented by the receiver. ARC is one of the latest additions to email standards (introduced in 2019), there are a lot of legacy email systems that don't support it. If an SMTP service supports ARC, then it will probably also support DMARC (introduced in 2015) and DKIM (introduced in 2011). So you'll still be better of with using DKIM, as DKIM adds cryptographic proof of authentication and integrity, whereas SPF is just a simple allow list. The big challenge with email is finding a happy medium between security, and backwards compatibility for legacy systems.

citrin_ru 4y ago

In practice difference between SPF softfail and SPF hardfail is less significant than in theory (where everyone follows RFC7208 to the letter). Many mail system don't reject a message in case of SPF fail - SPF result is used one of many inputs for an anti-spam system. If we have SPF pass we can assume that a message was sent from an infrastructure approved by the domain owner and if domain has good reputation we can take it into account. If we don't have SPF pass then the sender address in the message can be spoofed. Be it SPF softfail or hardfail or even a missing SPF record we cannot trust the sender address. But if otherwise message looks good it still can be accepted. DMARC makes things a bit more complicated but it is similar - either you have DMARC pass (which is good) or you don't have it and why exactly you don't have DMARC pass is less important.