4 ms·

I manage IT at a mid-size business. At least once a month, I get asked to release some incoming email from quarantine that got sent there because the sender's S

I manage IT at a mid-size business. At least once a month, I get asked to release some incoming email from quarantine that got sent there because the sender's SPF record is wrong or outdated and doesn't include all the email services they actually use. (What this really tells me is how many small businesses are out there running with no in-house IT expertise or support of any kind.) I don't do whitelisting. Instead, I always reach out and offer to help the other party correct their SPF record. It happens often enough that I wrote a script in Racket that will generate the email for me and paste it into the clipboard [1]. The email tells them exactly what they need to change, and links to docs from their current email provider (so they don't have to trust me about edits to their DNS). [1]: https://gist.github.com/otherjoel/6b8bf02f6db6e0c47ba6bca72ecf1c3e https://gist.github.com/otherjoel/6b8bf02f6db6e0c47ba6bca72e...

luckman212 2y ago
Neat! I must try this.
ziddoap 2y ago
What would you say the normal reception you receive from this email template is? I like the idea, but I would think sending a technical email (with industry-specific acronyms that you don't spell out!) to a business that has no in-house IT would just be ignored in most cases.
- velcrovan 2y ago
  Well if you read the template, you'll see I start out with a non-technical explanation, advise that they forward the email to an IT type person, and offer to help in any way I can. Then I put in a "More info" heading further down with all the details and instructions. Overall I'm pleased with how well this approach works. When people realize that their email is getting stuck in spam filters because of a problem on their end, they're usually motivated to get it fixed. Sometimes it gets sent to an owner who had barely enough tech mojo to stand up a gmail account at a custom domain, and even then the instructions are usually simple enough for them to follow.
  - ziddoap 2y ago
    >Well if you read the template I read the template, that's how I spotted the acronyms that weren't spelled out. Like DNS on the second line, before you recommend forwarding. >Overall I'm pleased with how well this approach works. Interesting, I definitely would have thought it'd be ignored more often than not, but I might have to look into rolling out something similar. Thanks for the idea.
    - ralferoo 2y ago
      I think you're being a bit unfair here. > If you do not have access to your company’s DNS records, please forward this email to someone in an IT role. If you don't even know what DNS records are, I'd imagine you'd assume you don't have access to them and so forward them to the IT person as suggested. But sure, maybe he could also add ", or don't know what they are" to this line.
      - ziddoap 2y ago
        I'm not trying to be unfair or critical or anything. My first question was genuine. I intended my note about acronyms to be just that: a side note. The response I got was "If you read the template [...]", which it should have been pretty obvious I did. Then I got an explanation of the template as if I hadn't read it, which was a bit patronizing. I think it's a good idea (and said so twice!). I was curious what the reception was like. Sorry if my comment about acronyms was too much. It is a pet peeve of mine to see acronyms not spelled out, especially technical ones in a document intended for non-technical people. I didn't intend it to derail the conversation. Obviously it was taken in a way more critical way than I had intended -- my fault.
        velcrovan 2y ago
        Sorry I didn’t pick up that you had read the template. I was just trying to give context for my answer without assuming or requiring anyone who might read it to have scrolled through all the code.
- egorfine 2y ago
  > ignored in most cases I see no problem here.
victorbjorklund 2y ago
That is really awesome. It can be easy to miss setting up SPF on every new tool.
rnewme 2y ago
Cool stuff! BTW, what's up with the license?
- velcrovan 2y ago
  Including the whole license in the gist was probably overkill, but I particularly like Blue Oak for permissive licensing and would like to see it more widely used. https://writing.kemitchell.com/2019/03/09/Deprecation-Notice.html https://writing.kemitchell.com/2019/03/09/Deprecation-Notice...
  - LelouBil 2y ago
    Hey, English is not my first language so I'm surely missing something here but: > Copyright > Each contributor licenses you to do everything with this software that would otherwise infringe that contributor's copyright in it. This sounds like the license specifically allows you to infringe on the contributor's copyright.
    - velcrovan 2y ago
      It's a permissive license. The licensor is saying “I own copyright in this work, which gives me the right to give you permission to do whatever you want with it.”
  - bb88 2y ago
    > If anyone notifies you in writing that you have not complied with [Notices](#notices), you can keep your license by taking all practical steps to comply within 30 days after the notice. If you do not do so, your license ends immediately. I'm not sure I like it. Like, what if the notification of notice was incorrect? You lose your license anyway?
    - velcrovan 2y ago
      By default under most open source licenses, particularly permissive ones, violating the terms even by accident ends your license instantly, with no notice whatsoever. Including a fair, common sense path to forgiveness severely limits legal risk for users, and is one of the things I like about the Blue Oak license.
hug 2y ago
Does the script handle macros in SPF? I've had a couple of other-company-IT-admins tell me that my MX is jacked because I use hosted SPF via proofpoint, and when they look up my SPF it looks like this: "v=spf1 include:%{ir}.%{v}.%{d}.spf.has.pphosted.com ~all" A surprising number of mail admins don't understand SPF macros.
- velcrovan 2y ago
  If I’m reaching for the script, it's because I’m already in a scenario where Proofpoint has quarantined legitimate email for failing SPF checks (we use Proofpoint too). So the script itself doesn’t do any analysis of the existing SPF record. It just shows them the existing record and tells them how to fix it based on the sender's IP for the email in question.
- TheNewsIsHere 2y ago
  In defense of those who haven’t read the RFCs personally — I can count on one hand the number of times I’ve seen SPF macros in the wild, which holds true if I included yours. Interestingly all Proofpoint customers too. I’ve seen it more common to isolate services to subdomains and specify subdomain SPF records rather than use macros. This is my preferred approach. I’m not hating on the macros. They’re just seemingly very rarely used. I know they’re on the table but I haven’t found a compelling use case in my own deployments.
EnigmaFlare 2y ago
I was on the receiving end of an automated version of this. However, when I looked into it, it seemed the problem (SPF record required more than 10 DNS lookups) was fairly common. I don't seem to have any other deliverability problems and my email and DNS is managed by some big hosting company so I assume it's not a real problem and didn't fix it.
- brightball 2y ago
  It’s a real problem. The solution is just to isolate each service on its own subdomain. The only thing that should be listed in your top level SPF record is the corporate email for the domain.
  - EnigmaFlare 2y ago
    The trouble is, it's full of stuff I don't understand because I'm not managing it, such as: include:_spf.qualtrics.com include:_spf.salesforce.com include:sparkpostmail.com include:spf.mailjet.com include:spf.protection.outlook.com include:_spf.myorderbox.com include:eig.spf .a.cloudfilter.net include:spf.websitewelcome.com
    - brightball 2y ago
      Yep, each one of those should be isolated to their own subdomain rather than being in the top level SPF record. Should have 8 total SPF records including the top level. Easiest thing to do is to start by moving just 1 include to its own subdomain.
rkagerer 2y ago
Awesome, mind if I send you an email solely to test if it gets through or if I get to be the recipient of your awesome script? ;-)
- velcrovan 2y ago
  Sure, I think?
deng 2y ago
> At least once a month, I get asked to release some incoming email from quarantine that got sent there because the sender's SPF record is wrong or outdated And at the same time, I regularly get Spam/Phishing with perfect SPF, DKIM, DMARC, etc. The domains and IPs they use might get blocked within a day, but of course, these people have no problem getting others. And although I have set up my MTA perfectly, my mail gets refused by MS/t-online/etc., because I don't have enough "sender reputation". In e-mail, we have an oligopoly of a few big mail providers, and in the end, they decide which mail gets delivered and which isn't, and to me it looks like they give a rat's ass about SPF and DKIM, and probably rightfully so, because most spammers are probably better at configuring MTAs than your average mail admin.
- taskforcegemini 2y ago
  t-online uses a global whitelist, which is pretty stupid for e-mail. sometimes it helps contacting them, other times they refuse to resolve it for arbitrary reasons (not because of actual spamming)
  - Leonelf 2y ago
    t-online told me I needed an imprint on the website that's reachable under my domain. Seems to be some misunderstanding of German law (German commercial websites need an imprint, legally, but t-online also apply this requirement to private domains).
    - persnickety 2y ago
      Last month they unblocked me even though the website is blank.
    - zelphirkalt 2y ago
      It is more complicated than that. There are more criteria for when you need an imprint: (1) any kind of journalistic content on your site (2) any kind of financial gain from showing ads or making ads (3) organizing any kind of group of people active on German territory (4) running a business website There might be more, but those are the ones I remember from reading the paragraphs a while ago. And these are, of course, vague, which means that even something like "my favorite restaurants in Berlin" could be considered an ad, or any kind of comment on politics might be considered a form of journalism. I dislike these rules, because they basically kill German blogging scene. Not so many people want to run a blog and have every idiot on the Internet know their personal address. And few bloggers want to rent a digital office or actual office, that will send mail to them (an indirection). The German law in this respect is terrible and working against a free Internet and against freedom of voicing your opinion. It works greatly in favor for tech giants, because people resort to putting their blogging on Facebook, Instagram and other disservices. It is very anti-decentralization.
jonathantf2 2y ago
I work at an IT provider - we see this daily. Have to whitelist to keep the customer happy, usually the other end is a 1 or 2 person business with an old hosted Yahoo system or similar
- egorfine 2y ago
  > happy or misguided? (I know the pain.)
pembrook 2y ago
If we care about keeping open protocols like HTTP and SMTP alive, we need to overhaul DNS. Or at least create a simplified common abstraction layer. It’s the most inherently user-hostile thing I’ve ever encountered - and I’m only just now starting to understand it, even though I’m almost 20 years into dealing with it.
AndrewDavis 2y ago
My favourite are queries "why are you rejecting my email?" Ehh... Because your dmarc policy told us to?
zelphirkalt 2y ago
How do I make use of the gist? I do (load "spf-fail.rkt") But afterwards none of the definitions are available in the Racket REPL. Maybe I need to (require ...) something? I do not see a module definition, that I would need to import and (require spf-fail) fails.
- velcrovan 2y ago
  If you’ve just run "racket" from the command line to get a REPL, you would use (require "spf-fail.rkt") to import all the bindings from this file (assuming it's in the same folder you ran racket from). The “Module Basics” chapter of the Racket is a good quick explainer of how this stuff works in Racket: https://docs.racket-lang.org/guide/module-basics.html https://docs.racket-lang.org/guide/module-basics.html Another method would be to open the file in DrRacket (or VSCode or Emacs or whatever editor you have set up with a Racket plugin/lang server) and just "run" it in the REPL. There are comments at the top that explain how to use it once it's loaded: ;; Generate a form email to let someone know their SPF records are misconfigured for their current email provider. ;; ;; Run (fill-report "domain.com" "1.2.3.4") where the 2nd arg is the sending email server's IP address. ;; It will copy the completed report to the clipboard for you. ;; ;; Only works on Windows for now. Personally, I have this file incorporated into a larger package (not published anywhere) for producing canned responses. With that package installed I can do this at a command prompt: raco canned spf domain.com 1.2.3.4
sylware 2y ago
Yeah, the only thing you need for simple SMTP services is: - with DNS, the simplest as possible SPF record. - Without DNS, (aka pure IPv[46] SMTP), your have implicit SPF: instead of querying the DNS and parsing the SPF record, you parse the mail header to check "reply-to/from/etc" fields (the appropriate fields) for the sending SMTP IPv[46] address, that to perform spam scoring.

luckman212 2y ago

Neat! I must try this.

ziddoap 2y ago

What would you say the normal reception you receive from this email template is? I like the idea, but I would think sending a technical email (with industry-specific acronyms that you don't spell out!) to a business that has no in-house IT would just be ignored in most cases.

velcrovan 2y ago
Well if you read the template, you'll see I start out with a non-technical explanation, advise that they forward the email to an IT type person, and offer to help in any way I can. Then I put in a "More info" heading further down with all the details and instructions. Overall I'm pleased with how well this approach works. When people realize that their email is getting stuck in spam filters because of a problem on their end, they're usually motivated to get it fixed. Sometimes it gets sent to an owner who had barely enough tech mojo to stand up a gmail account at a custom domain, and even then the instructions are usually simple enough for them to follow.
- ziddoap 2y ago
  >Well if you read the template I read the template, that's how I spotted the acronyms that weren't spelled out. Like DNS on the second line, before you recommend forwarding. >Overall I'm pleased with how well this approach works. Interesting, I definitely would have thought it'd be ignored more often than not, but I might have to look into rolling out something similar. Thanks for the idea.
  - ralferoo 2y ago
    I think you're being a bit unfair here. > If you do not have access to your company’s DNS records, please forward this email to someone in an IT role. If you don't even know what DNS records are, I'd imagine you'd assume you don't have access to them and so forward them to the IT person as suggested. But sure, maybe he could also add ", or don't know what they are" to this line.
    - ziddoap 2y ago
      I'm not trying to be unfair or critical or anything. My first question was genuine. I intended my note about acronyms to be just that: a side note. The response I got was "If you read the template [...]", which it should have been pretty obvious I did. Then I got an explanation of the template as if I hadn't read it, which was a bit patronizing. I think it's a good idea (and said so twice!). I was curious what the reception was like. Sorry if my comment about acronyms was too much. It is a pet peeve of mine to see acronyms not spelled out, especially technical ones in a document intended for non-technical people. I didn't intend it to derail the conversation. Obviously it was taken in a way more critical way than I had intended -- my fault.
      - velcrovan 2y ago
        Sorry I didn’t pick up that you had read the template. I was just trying to give context for my answer without assuming or requiring anyone who might read it to have scrolled through all the code.
egorfine 2y ago
> ignored in most cases I see no problem here.

victorbjorklund 2y ago

That is really awesome. It can be easy to miss setting up SPF on every new tool.

rnewme 2y ago

Cool stuff! BTW, what's up with the license?

velcrovan 2y ago
Including the whole license in the gist was probably overkill, but I particularly like Blue Oak for permissive licensing and would like to see it more widely used. https://writing.kemitchell.com/2019/03/09/Deprecation-Notice.html https://writing.kemitchell.com/2019/03/09/Deprecation-Notice...
- LelouBil 2y ago
  Hey, English is not my first language so I'm surely missing something here but: > Copyright > Each contributor licenses you to do everything with this software that would otherwise infringe that contributor's copyright in it. This sounds like the license specifically allows you to infringe on the contributor's copyright.
  - velcrovan 2y ago
    It's a permissive license. The licensor is saying “I own copyright in this work, which gives me the right to give you permission to do whatever you want with it.”
- bb88 2y ago
  > If anyone notifies you in writing that you have not complied with [Notices](#notices), you can keep your license by taking all practical steps to comply within 30 days after the notice. If you do not do so, your license ends immediately. I'm not sure I like it. Like, what if the notification of notice was incorrect? You lose your license anyway?
  - velcrovan 2y ago
    By default under most open source licenses, particularly permissive ones, violating the terms even by accident ends your license instantly, with no notice whatsoever. Including a fair, common sense path to forgiveness severely limits legal risk for users, and is one of the things I like about the Blue Oak license.

hug 2y ago

Does the script handle macros in SPF? I've had a couple of other-company-IT-admins tell me that my MX is jacked because I use hosted SPF via proofpoint, and when they look up my SPF it looks like this: "v=spf1 include:%{ir}.%{v}.%{d}.spf.has.pphosted.com ~all" A surprising number of mail admins don't understand SPF macros.

velcrovan 2y ago
If I’m reaching for the script, it's because I’m already in a scenario where Proofpoint has quarantined legitimate email for failing SPF checks (we use Proofpoint too). So the script itself doesn’t do any analysis of the existing SPF record. It just shows them the existing record and tells them how to fix it based on the sender's IP for the email in question.
TheNewsIsHere 2y ago
In defense of those who haven’t read the RFCs personally — I can count on one hand the number of times I’ve seen SPF macros in the wild, which holds true if I included yours. Interestingly all Proofpoint customers too. I’ve seen it more common to isolate services to subdomains and specify subdomain SPF records rather than use macros. This is my preferred approach. I’m not hating on the macros. They’re just seemingly very rarely used. I know they’re on the table but I haven’t found a compelling use case in my own deployments.

EnigmaFlare 2y ago

I was on the receiving end of an automated version of this. However, when I looked into it, it seemed the problem (SPF record required more than 10 DNS lookups) was fairly common. I don't seem to have any other deliverability problems and my email and DNS is managed by some big hosting company so I assume it's not a real problem and didn't fix it.

brightball 2y ago
It’s a real problem. The solution is just to isolate each service on its own subdomain. The only thing that should be listed in your top level SPF record is the corporate email for the domain.
- EnigmaFlare 2y ago
  The trouble is, it's full of stuff I don't understand because I'm not managing it, such as: include:_spf.qualtrics.com include:_spf.salesforce.com include:sparkpostmail.com include:spf.mailjet.com include:spf.protection.outlook.com include:_spf.myorderbox.com include:eig.spf .a.cloudfilter.net include:spf.websitewelcome.com
  - brightball 2y ago
    Yep, each one of those should be isolated to their own subdomain rather than being in the top level SPF record. Should have 8 total SPF records including the top level. Easiest thing to do is to start by moving just 1 include to its own subdomain.

rkagerer 2y ago

Awesome, mind if I send you an email solely to test if it gets through or if I get to be the recipient of your awesome script? ;-)

velcrovan 2y ago
Sure, I think?

deng 2y ago

> At least once a month, I get asked to release some incoming email from quarantine that got sent there because the sender's SPF record is wrong or outdated And at the same time, I regularly get Spam/Phishing with perfect SPF, DKIM, DMARC, etc. The domains and IPs they use might get blocked within a day, but of course, these people have no problem getting others. And although I have set up my MTA perfectly, my mail gets refused by MS/t-online/etc., because I don't have enough "sender reputation". In e-mail, we have an oligopoly of a few big mail providers, and in the end, they decide which mail gets delivered and which isn't, and to me it looks like they give a rat's ass about SPF and DKIM, and probably rightfully so, because most spammers are probably better at configuring MTAs than your average mail admin.

taskforcegemini 2y ago
t-online uses a global whitelist, which is pretty stupid for e-mail. sometimes it helps contacting them, other times they refuse to resolve it for arbitrary reasons (not because of actual spamming)
- Leonelf 2y ago
  t-online told me I needed an imprint on the website that's reachable under my domain. Seems to be some misunderstanding of German law (German commercial websites need an imprint, legally, but t-online also apply this requirement to private domains).
  - persnickety 2y ago
    Last month they unblocked me even though the website is blank.
  - zelphirkalt 2y ago
    It is more complicated than that. There are more criteria for when you need an imprint: (1) any kind of journalistic content on your site (2) any kind of financial gain from showing ads or making ads (3) organizing any kind of group of people active on German territory (4) running a business website There might be more, but those are the ones I remember from reading the paragraphs a while ago. And these are, of course, vague, which means that even something like "my favorite restaurants in Berlin" could be considered an ad, or any kind of comment on politics might be considered a form of journalism. I dislike these rules, because they basically kill German blogging scene. Not so many people want to run a blog and have every idiot on the Internet know their personal address. And few bloggers want to rent a digital office or actual office, that will send mail to them (an indirection). The German law in this respect is terrible and working against a free Internet and against freedom of voicing your opinion. It works greatly in favor for tech giants, because people resort to putting their blogging on Facebook, Instagram and other disservices. It is very anti-decentralization.

jonathantf2 2y ago

I work at an IT provider - we see this daily. Have to whitelist to keep the customer happy, usually the other end is a 1 or 2 person business with an old hosted Yahoo system or similar

egorfine 2y ago
> happy or misguided? (I know the pain.)

pembrook 2y ago

If we care about keeping open protocols like HTTP and SMTP alive, we need to overhaul DNS. Or at least create a simplified common abstraction layer. It’s the most inherently user-hostile thing I’ve ever encountered - and I’m only just now starting to understand it, even though I’m almost 20 years into dealing with it.

AndrewDavis 2y ago

My favourite are queries "why are you rejecting my email?" Ehh... Because your dmarc policy told us to?

zelphirkalt 2y ago

How do I make use of the gist? I do (load "spf-fail.rkt") But afterwards none of the definitions are available in the Racket REPL. Maybe I need to (require ...) something? I do not see a module definition, that I would need to import and (require spf-fail) fails.

velcrovan 2y ago
If you’ve just run "racket" from the command line to get a REPL, you would use (require "spf-fail.rkt") to import all the bindings from this file (assuming it's in the same folder you ran racket from). The “Module Basics” chapter of the Racket is a good quick explainer of how this stuff works in Racket: https://docs.racket-lang.org/guide/module-basics.html https://docs.racket-lang.org/guide/module-basics.html Another method would be to open the file in DrRacket (or VSCode or Emacs or whatever editor you have set up with a Racket plugin/lang server) and just "run" it in the REPL. There are comments at the top that explain how to use it once it's loaded: ;; Generate a form email to let someone know their SPF records are misconfigured for their current email provider. ;; ;; Run (fill-report "domain.com" "1.2.3.4") where the 2nd arg is the sending email server's IP address. ;; It will copy the completed report to the clipboard for you. ;; ;; Only works on Windows for now. Personally, I have this file incorporated into a larger package (not published anywhere) for producing canned responses. With that package installed I can do this at a command prompt: raco canned spf domain.com 1.2.3.4

sylware 2y ago

Yeah, the only thing you need for simple SMTP services is: - with DNS, the simplest as possible SPF record. - Without DNS, (aka pure IPv[46] SMTP), your have implicit SPF: instead of querying the DNS and parsing the SPF record, you parse the mail header to check "reply-to/from/etc" fields (the appropriate fields) for the sending SMTP IPv[46] address, that to perform spam scoring.