9 ms·

I run an email security/monitoring business [0], and we hear this issue with Gmail a lot, in fact: we have been bitten by it as well during the early days. The

by LeonM 7y ago

I run an email security/monitoring business [0], and we hear this issue with Gmail a lot, in fact: we have been bitten by it as well during the early days. The problem at hand is that it is really hard to debug situations like this. Google won't tell you why a particular email is blocked, because that information will be immediately exploited by spammers. We build our software suite because we found that mistakes in email configuration are easy to make and really hard to identify. I did a quick scan of rafa.eu.org and found the following: - SPF is setup [1] with a neutral 'all' mechanism, which basically disables SPF for this domain. Hence, DMARC will set SPF as 'pass' even though SPF did nothing here to help the spam algorithm to assess the sender. - A DMARC record exists and has a valid syntax [2], but the proposition (p value) is set to 'none', which basically disables DMARC altogether. It will enable reporting, but nothing more than that. - I don't know you DKIM selector, so I can't assess that, but make sure the DKIM signature address is aligned (using the same domain name). - The TLS configuration of your email server uses a self-signed certificate and is very much outdated (it offers SSLv3 and RC4 ciphers). I used testlssh.sh [3] to check this. So yes, you technically did setup TLS, SPF and DMARC, but in all 3 you have configuration errors. Running a mail server in 2019 requires a bit more work and maintenance than it used to be 10 years ago, but the reason is spammers, not Google. Remember: it is in Google's best interest to have as few false positives, as it ultimately benefits the users. It's just really frustrating that it is hard to figure out why Google is marking your email as spam. [0] https://www.mailhardener.com https://www.mailhardener.com [1] https://www.mailhardener.com/tools/spf-validator?domain=rafa.eu.org https://www.mailhardener.com/tools/spf-validator?domain=rafa... [2] https://www.mailhardener.com/tools/dmarc-validator?domain=rafa.eu.org https://www.mailhardener.com/tools/dmarc-validator?domain=ra... [3] https://testssl.sh/ https://testssl.sh/

throwaway2048 7y ago
Its inaccurate to say that SPF, DMARC and DKIM setup this way is a "misconfiguration", It still allows anybody you are mailing to verify that email is from your mail server, and your domain, which is good enough for anti-spam purposes. Setting SPF to neutral (or softfail) and DMARC to none just ensures stuff like mailing lists forwarding mail from your domain aren't automatically marked as spam. In my experience it does not matter if you have very strict SPF and DMARC policies, with a good reputation IP address, and an old domain, both of which have never sent spam. GMail will absolutely still spambox you.
- LeonM 7y ago
  > Setting SPF to neutral (or softfail) and DMARC to none just ensures stuff like mailing lists forwarding mail from your domain aren't automatically marked as spam. Not exactly. DMARC inspection passes if either SPF or DKIM validation passes. A mailing list forwarding mail will break SPF, but retain the DKIM signature. So you can definitely use strict SPF and DMARC with mailing lists. In fact, retaining mailing list compatibility had priority when the DMARC RFC was drafted. See the RFC.
  - throwaway2048 7y ago
    Lots of mailing lists add footers to the body of a mail, or otherwise mess with the message which is going to cause DKIM to fail.
    - LeonM 7y ago
      That's horrifying [0] An email service that alters the content of an email message in any way should definitely be marked as malicious. This allows for phishing (replacing a URL in the email), censorship, fake news, etc etc. [0] https://xkcd.com/1172/ https://xkcd.com/1172/
baobabKoodaa 7y ago
No, it's incorrect to say that running an email server requires just "a bit more work". Google basically no longer allows low volume servers to send mail. I wrote more about this here: https://www.attejuvonen.fi/dont-send-email-from-your-own-server/ https://www.attejuvonen.fi/dont-send-email-from-your-own-ser...
- PopeDotNinja 7y ago
  Sounds like there's a niche business and/or OSS project there. Create gmail accounts. Install email send bot on email server. Send messages to Gmail accounts. Figure out how to avoid spam.
  - dest 7y ago
    Won't this be exploited by spammers?
  - baobabKoodaa 7y ago
    A ton of services like this exist. For example, Postmark is specialized in sending transactional mail past Gmail's spam filters.
- Avamander 7y ago
  My low volume server can send mail very nicely. You can't however do your job half-assed, lack of strong SPF/TLS/DKIM isn't okay from both Google's and anti-spam perspective.
  - throwaway2048 7y ago
    How are you verifying this? I encourage you to try sending to a gmail address you have never sent to before, I virtually guarantee you are going to end up spamboxed. I know lots of people that run low volume personal type servers, many have claimed that gmail doesn't spambox them, but they were always testing on a (handful of) personal account(s), when I got them to check with an account they had never mailed before, they were spam boxed literally 100% of the time. Properly setup, and strict FCrDNS, DMARC, DKIM, SPF, TLS and a clean IP and an old domain that have both never sent spam are absolutely not enough to avoid gmail's spambox.
teejmya 7y ago
https://testssl.sh/ https://testssl.sh/ is super useful, thank you for sharing
cnst 7y ago
> The problem at hand is that it is really hard to debug situations like this. Google won't tell you why a particular email is blocked, because that information will be immediately exploited by spammers. This is a pretty big fallacy, because exactly this approach affects normal users much more than it affects spammers. The solutions themselves would be much different if you have to pass 10 emails per day with 100% certainty, versus a million where a 5% pass rate is good enough. A better explanation here is that these denials appear to be somewhat random in nature, driven by multiple layers of AI and proprietary systems that may be difficult to explain to the uninitiated, without revealing the whole code of the algorithm into OSS, and it's not quite possible to explain exactly why the AI does the things that it does. The only solution would be to actually make it possible to troubleshoot this to an actual resolution. Why it's an issue for a domain owner with a dedicated IP address to send a couple of emails per day from their domain name without ending up in Spam is a bigger question here. > I did a quick scan of rafa.eu.org and found the following: All the issues you list are non-issues, and merely the perpetuation of the security theatre. SPF is very useful for a domain like gmail.com, where there are hundreds of individual IP addresses that may be doing the sending, with little ways of finding out if they're legit without SPF. But the domain of the OP has a single server with a single dedicated IPv4 address. It is trivial to verify whether or not the domain owner has authorised a given email sent from such server, and whether it's authentic, even if it doesn't have DMARC, SPF or even MX records, not to mention DKIM, provided that it gets sent from the A or AAAA address of the domain. Yet it's these very emails that get denied, and even putting these useless and unnecessary for the situation DKIM, SPF and DMARC entries doesn't help, either. Further, your requirement of configuring SPF and DMARC to reject actual forged mail stems from a belief that someone else is abusing the domain reputation to send forged mail and spam, but, (1), that's never been asserted or implied by the sender, and, (2), why isn't Google smart enough to reject those messages then, instead of rejecting these ones that clearly do come from the owner of the domain, due to the match between reverse DNS, forward DNS, MX, EHLO, MAIL FROM et al. Because that's really what's going on here. We're not even talking about mailing lists or forwarding at this stage; the most basic mail setup results in his messages being rejected. Here and in your another comment in this thread, you're portraying all this advice as expert advice without even having anecdotal evidence to back it up that it actually works; this is sadly a common practice around the internet, to make suggestions for the sake of making suggestions, wasting people's time on implementing solutions that won't make any difference, where the proposed solution has little correlation to the issue at stake. And this is exactly why you get downvoted on the more technical forums like Hacker News for offering this sort of technical consulting for free — because folks here know better, and because these sort of blanket suggestions don't actually do anything for the situation the user experiences, don't solve the problem, and are technically incorrect and superficial to start with.

by LeonM 7y ago

throwaway2048 7y ago

Its inaccurate to say that SPF, DMARC and DKIM setup this way is a "misconfiguration", It still allows anybody you are mailing to verify that email is from your mail server, and your domain, which is good enough for anti-spam purposes. Setting SPF to neutral (or softfail) and DMARC to none just ensures stuff like mailing lists forwarding mail from your domain aren't automatically marked as spam. In my experience it does not matter if you have very strict SPF and DMARC policies, with a good reputation IP address, and an old domain, both of which have never sent spam. GMail will absolutely still spambox you.

LeonM 7y ago
> Setting SPF to neutral (or softfail) and DMARC to none just ensures stuff like mailing lists forwarding mail from your domain aren't automatically marked as spam. Not exactly. DMARC inspection passes if either SPF or DKIM validation passes. A mailing list forwarding mail will break SPF, but retain the DKIM signature. So you can definitely use strict SPF and DMARC with mailing lists. In fact, retaining mailing list compatibility had priority when the DMARC RFC was drafted. See the RFC.
- throwaway2048 7y ago
  Lots of mailing lists add footers to the body of a mail, or otherwise mess with the message which is going to cause DKIM to fail.
  - LeonM 7y ago
    That's horrifying [0] An email service that alters the content of an email message in any way should definitely be marked as malicious. This allows for phishing (replacing a URL in the email), censorship, fake news, etc etc. [0] https://xkcd.com/1172/ https://xkcd.com/1172/

baobabKoodaa 7y ago

No, it's incorrect to say that running an email server requires just "a bit more work". Google basically no longer allows low volume servers to send mail. I wrote more about this here: https://www.attejuvonen.fi/dont-send-email-from-your-own-server/ https://www.attejuvonen.fi/dont-send-email-from-your-own-ser...

PopeDotNinja 7y ago
Sounds like there's a niche business and/or OSS project there. Create gmail accounts. Install email send bot on email server. Send messages to Gmail accounts. Figure out how to avoid spam.
- dest 7y ago
  Won't this be exploited by spammers?
- baobabKoodaa 7y ago
  A ton of services like this exist. For example, Postmark is specialized in sending transactional mail past Gmail's spam filters.
Avamander 7y ago
My low volume server can send mail very nicely. You can't however do your job half-assed, lack of strong SPF/TLS/DKIM isn't okay from both Google's and anti-spam perspective.
- throwaway2048 7y ago
  How are you verifying this? I encourage you to try sending to a gmail address you have never sent to before, I virtually guarantee you are going to end up spamboxed. I know lots of people that run low volume personal type servers, many have claimed that gmail doesn't spambox them, but they were always testing on a (handful of) personal account(s), when I got them to check with an account they had never mailed before, they were spam boxed literally 100% of the time. Properly setup, and strict FCrDNS, DMARC, DKIM, SPF, TLS and a clean IP and an old domain that have both never sent spam are absolutely not enough to avoid gmail's spambox.

teejmya 7y ago

https://testssl.sh/ https://testssl.sh/ is super useful, thank you for sharing

cnst 7y ago

> The problem at hand is that it is really hard to debug situations like this. Google won't tell you why a particular email is blocked, because that information will be immediately exploited by spammers. This is a pretty big fallacy, because exactly this approach affects normal users much more than it affects spammers. The solutions themselves would be much different if you have to pass 10 emails per day with 100% certainty, versus a million where a 5% pass rate is good enough. A better explanation here is that these denials appear to be somewhat random in nature, driven by multiple layers of AI and proprietary systems that may be difficult to explain to the uninitiated, without revealing the whole code of the algorithm into OSS, and it's not quite possible to explain exactly why the AI does the things that it does. The only solution would be to actually make it possible to troubleshoot this to an actual resolution. Why it's an issue for a domain owner with a dedicated IP address to send a couple of emails per day from their domain name without ending up in Spam is a bigger question here. > I did a quick scan of rafa.eu.org and found the following: All the issues you list are non-issues, and merely the perpetuation of the security theatre. SPF is very useful for a domain like gmail.com, where there are hundreds of individual IP addresses that may be doing the sending, with little ways of finding out if they're legit without SPF. But the domain of the OP has a single server with a single dedicated IPv4 address. It is trivial to verify whether or not the domain owner has authorised a given email sent from such server, and whether it's authentic, even if it doesn't have DMARC, SPF or even MX records, not to mention DKIM, provided that it gets sent from the A or AAAA address of the domain. Yet it's these very emails that get denied, and even putting these useless and unnecessary for the situation DKIM, SPF and DMARC entries doesn't help, either. Further, your requirement of configuring SPF and DMARC to reject actual forged mail stems from a belief that someone else is abusing the domain reputation to send forged mail and spam, but, (1), that's never been asserted or implied by the sender, and, (2), why isn't Google smart enough to reject those messages then, instead of rejecting these ones that clearly do come from the owner of the domain, due to the match between reverse DNS, forward DNS, MX, EHLO, MAIL FROM et al. Because that's really what's going on here. We're not even talking about mailing lists or forwarding at this stage; the most basic mail setup results in his messages being rejected. Here and in your another comment in this thread, you're portraying all this advice as expert advice without even having anecdotal evidence to back it up that it actually works; this is sadly a common practice around the internet, to make suggestions for the sake of making suggestions, wasting people's time on implementing solutions that won't make any difference, where the proposed solution has little correlation to the issue at stake. And this is exactly why you get downvoted on the more technical forums like Hacker News for offering this sort of technical consulting for free — because folks here know better, and because these sort of blanket suggestions don't actually do anything for the situation the user experiences, don't solve the problem, and are technically incorrect and superficial to start with.