12 ms·

The annoying thing about the major email services (Gmail, Outlook, et al) using reputation as a key input in their spam prevention algorithms is that they don't

by bhauer 4y ago

The annoying thing about the major email services (Gmail, Outlook, et al) using reputation as a key input in their spam prevention algorithms is that they don't handle reputation in a way sympathetic to small and hobbyist mail server operators at all. Google and Microsoft have the computation and data storage resources to forever record how much spam they've received from the mail servers operated by any given small company or hobbyist. If a domain's authoritative mail servers have a history of never sending spam, when that company or hobbyist (for any reason) has to switch hosting providers and gets a new IP address, Google/Microsoft should recognize that the domain, and its referenced mail servers, has never been malicious. But instead, the majors treat a new IP address for a long-established domain as entirely new to the internet and assign it zero, or even negative, reputation. And they don't care to address this because, well, why care about hobbyists and the rare small company insolent enough to try to self-host email?

hamburglar 4y ago
Indeed, although I think there is a little more rationale than simply not caring. The hobbyist/small mail server is historically more likely to go unpatched and pwnage is more likely to go unnoticed, so it’s unfortunately a good defensive move to penalize them reputationally by default. PS yahoo is the worst for assuming poor reputation from a sender. In my experience they just introduce massive delivery delays at the drop of a hat even when the sender has a stellar reputation.
- bhauer 4y ago
  > The hobbyist/small mail server is historically more likely to go unpatched and pwnage is more likely to go unnoticed, so it’s unfortunately a good defensive move to penalize them reputationally by default. But to reiterate my earlier point: Google and Microsoft have the computational and storage capacity to have a detailed history of all domains' authoritative mail servers behavior. They will know whether a domain has a history of patch negligence. No, I think the far simpler explanation is that they just don't bother tracking reputation by domain. Or if they do, it's largely overshadowed by the weight given to IP-based reputation.
  - hamburglar 4y ago
    Oh, I’m not saying you’re wrong. They absolutely could put the effort in to solve this problem.
- alxlaz 4y ago
  > I think there is a little more rationale than simply not caring. You're right in one more way: both Google and Microsoft offer email hosting solutions for small businesses. Their main (and, if you've got more than 8-10 accounts or so, the only) selling point is that it makes managing email hassle-free. Making it as painful as possible for small businesses to host their email server helps these services tremendously. If they had real interoperability (either of their own accord or because it were forced upon them through regulatory measures), the biggest cash cows of these services -- companies that are well into "enough accounts that your own server would be much cheaper" territory but not large enough to afford or risk large infrastructure changes -- would evaporate pretty quickly.
gnarbarian 4y ago
This seems like a major legal liability for Google. It could be shown that Google and other major email hosting providers act like a cartel by unfairly discriminating against companies who don't use a major email provider.
- amenod 4y ago
  I hope so! I am tired of explaining people that it's actually Google's fault they didn't receive my mail. I'll be happy when they pay dearly for the disservice they do to e-mail.
  - gnarbarian 4y ago
    Are you using S/MIME certificates?
    - smartbit 4y ago
      Why are you asking? Is there a relation between spam handling and usage of S/MIME?
    - Kadin 4y ago
      Last time I looked into it (I run a mailserver and mailman list for one of my hobby groups), S/MIME wouldn't change your "spamminess" reputation score. DKIM, DMARC, and SPF do, though, and basically are table stakes if you want your mail (especially mailinglist messages) to go through to people at major providers.
    - amenod 4y ago
      No, would that help? I am however using SPF and one of DKIM / DMARC (I forget which). But anyway, I can live with a mail missing here and there. It is just annoying and isn't right.
  - Avamander 4y ago
    It's not really Google's fault to be honest. The need to warm up new IP's has existed for a while and a lot of providers do it. Any postmaster with experience knows how and why it's done.
bombcar 4y ago
Back when domains were easy to spoof (I could setup a server and send mail as ycombinator.com easily enough) it made sense to track the IPs, but now that you have DKIM and SPF links to cross-check, you should be able to use the domain reliability as a strong indicator. Sure you would have to catch people buying a "good" domain that expired, but that shouldn't be an insane hurdle. The real story is nobody cares.
- bhauer 4y ago
  > The real story is nobody cares. Exactly. Why should the major players care about those too small to matter? > Sure you would have to catch people buying a "good" domain that expired Adding to my earlier point, Google and Microsoft are well-enough connected to the domain registrars to know when that scenario has happened as well. If they put any effort into it, they could reliably determine whether a new IP address for an established domain is legitimate or a fraud. But as we've said, why put any effort into it when the only people complaining are not important?
  - bombcar 4y ago
    And it's even worse - if we theorize a email competitor appearing out of nowhere to rival Gmail, and people complaining, all that would happen is Google and Microsoft would special-case that provider, and the underlying issue wouldn't be solved.
    - gnarbarian 4y ago
      a class action lawsuit could make them care. as far as how to solve this problem technically, I think a reputation system based not on domains or ips but on email certificates is the real answer here.
      - toast0 4y ago
        DKIM is certificates, so I'm guessing you're talking about sender certificates? How would that help? Spammers can get certificates too. Maybe it cuts down on some of the misconfigured http email senders, maybe, but not enough to matter. Scam sites run https these days. You can't use like age of activity of the cert to help because a) things get compromised, b) you need to rotate your certs frequently anyway.
YetAnotherNick 4y ago
I could bet that most of mails from new IPs are spam because there are much fewer people willing to set up a mailserver than there are spammer operated mailservers. It is cheap to buy a new IP(comes free with many $2 hosting plans). The problem of identifying if a new IP is used for spam is not easy to solve.
- bhauer 4y ago
  > The problem of identifying if a new IP is used for spam is not easy to solve. How is it not easy to check a reputation database for domains when evaluating a mail server? Using DKIM records, Google could cross-reference the reputation for the sending domain and, where applicable, recognize that the domain in question has never been malicious or negligent. And in that case, extend a probationary reputation sufficient to allow the IP to establish its own new reputation.
  - YetAnotherNick 4y ago
    New IP means it is new and not part of any database.
    - bhauer 4y ago
      I'll quote myself: > How is it not easy to check a reputation database for domains when evaluating a mail server? The point I'm making in this thread is that they have the ability to maintain a database of domains that have been proven to be trustworthy. I am not talking about a database of IP addresses, and I am not sure why that is being raised here.
- megous 4y ago
  > The problem of identifying if a new IP is used for spam is not easy to solve. Huh? With dataset of ~100k classified hams/spams I get like >99% precisison in identifying spam/ham with just bogofilter and 0 heuristics whatsoever (my mail server accepts everything, and I just use bogofilter client side). I guratanee you MS has a dataset with > 100k emails, lol. It's not like this is unsolved problem.
w-j-w 4y ago
> small and hobbyist mail server operators From the perspective of an email receiver, I WANT things to be difficult for this group. Truly free email available to all would be unusable owing to spam. Oligopoly, I believe, results in the best experience for everyone.
ajross 4y ago
> If a domain's authoritative mail servers have a history of never sending spam, when that company or hobbyist (for any reason) has to switch hosting providers and gets a new IP address, Google/Microsoft should recognize that the domain, and its referenced mail servers, has never been malicious. That algorithm doesn't work. The internet is filled with parked domains that have "never been malicious". This just creates a new market for clean domains that you can use to evade protections. It makes spam a little more expensive, but it still gets through. None of these tricks work. There are no tricks. All rules can be gamed. The only thing that can't be easily faked is reality: if Microsoft knows you're a big org with a well-managed IT group running your output email setup, then they know they can (probably) trust you not to spam their customers. If you are too small to prove that to MS in a scalable way, no amount of heuristic trickery is going to help you.
- bhauer 4y ago
  I'm sorry, but I don't buy that argument. Google--a technology giant with algorithms and heuristics running most operations--is not up to the task of improving algorithms for email server reputation? No, they are definitely capable of improvement. It's simply not of interest to them. I can hardly blame them because the cost-benefit analysis clearly says, "why bother?" Of course anything can be gamed, but magnitude matters. I've had the same domain for 24 years, with many hundreds or thousands of email conversations between users of my server and those of the major email players over the years. I had to switch my mail server's IP address two years ago. Immediate zero reputation from many big players.
  - Avamander 4y ago
    > is not up to the task of improving algorithms for email server reputation? No, they are definitely capable of improvement. It's simply not of interest to them. Incorrect, if they stopped spending all that money and effort to keep up, their users would get flooded. > Of course anything can be gamed Sending spam/marketing e-mails is a multi-million industry. Both on illegal and legal markets. It's a constant race.
- BenjiWiebe 4y ago
kevincox 4y ago
FWIW I run a service that sends a decent amount of email to GMail users. While it was a bit slow to get started (messages being marked as spam) once I had sent for a month or two and had a few users that marked the messages as not-spam I don't appear to have any problems. I say this sending from a Digital Ocean IP address that occasionally changes. It appears that Google highly values domain reputation and that once I have got onto their good list I am doing OK. Disclaimers: I make sure to do everything else right. I have SPF and DKIM and my DMARC policy is to reject 100%. I also don't use IPv6 (DO Kube doesn't really support IPv6 well). I have found other major providers to be much worse. Microsoft seems to rely almost entirely on IP reputation and marks everything as spam, even accounts that have marked messages as "not spam" many, many times. Apple outright blocks the IP range.
- AshamedCaptain 4y ago
  My experience is that all of SPF, DKIM and DMARC are almost completely ignored by Gmail -- one day I simply stopped DKIM and DMARC and Google keep happily accepting emails (and to this day I still send emails without). In fact they will happily accept emails even when the SPF check didn't pass and the policy clearly says strict reject aka -all . While on the other hand I fully agree with TFA: I have _never_ been able to send an email to Gmail from a IPv6 address and have it not end up as spam, not even to accounts where I already whitelisted previous attempts. I don't think it's a reputation issue, since my IPv4 addresses likely have much worse reputation. It's as if they just handicap all IPv6 addresses.
  - kevincox 4y ago
    GMail definitely respects my DMARC reject policy. But IDK how missing one of SPF or DKIM affect its spam decision. But with DMARC reject and both missing or invalid it will bounce the message every time I have seen.
    - tomatocracy 4y ago
      Automatic mail forwarding without altering the From address will cause DMARC alignment for SPF to break. This is a common enough legitimate setup that most providers seem to effectively downgrade the DMARC policy applied when they see this (usually reject becomes quarantine, quarantine becomes ignore).
ipaddr 4y ago
The really annoying thing these days is google blocking zip files as spam requiring you to use google drive.
- rstupek 4y ago
  Zip files are being used to infect computers with malware which would explain why they're blocking them
  - DarylZero 4y ago
    I guess they're not allowed on Google Drive either.
walrus01 4y ago
they are basing their filters on the reputation of the netblock that your individual ipv4 /32 (or ipv6 equivalent) is contained in. And very often when you are hosting with a low cost VM or VPS provider the historical record of other IPs operated by the same hosting company in the same /24 or something is very poor. while I concur with all of the points you make, there is a logical and statistically accurate reason for some of these spam filters. even if your no-open-relay, rdns, spf, dkim, dmarc, SSL/TLS and other configuration is absolutely impeccable on your smtpd, your only recourse in a situation like this is to change to a new ISP that does not have a poor IP space reputation in all the adjacent IPs.
- Neil44 4y ago
  The netblock thing has happened to me. My Linode that I’ve had sending mail for 6 years gets blacklisted now purely for being on the netblock that it’s on.
  - apocalyptic0n3 4y ago
    Yep. I've encountered the exact same issues on multiple Linode servers in multiple data centers. I've also experienced it with an AWS server. It's often almost impossible to get it unblocked because the support teams at the email providers (if you can even contact them) don't realize that's what's happening. I didn't understand it myself until I got a call with an engineer at Microsoft who realized it in the middle of our call. He unlisted us on the call and we were fine for a while, but were eventually blocked again and haven't been able to get unblocked. It's really frustrating and I'm slowly ending all of my email services (both professional and hobby) as a result.
    - Neil44 4y ago
      It’s frustrating. I had outbound smtp throttled right down and a script running that shut smtp down completely if the queue got too big. That triggers an alert and everything gets sorted. But no, we don’t like where you’re from so jog on. Monopolistic.

by bhauer 4y ago

hamburglar 4y ago

Indeed, although I think there is a little more rationale than simply not caring. The hobbyist/small mail server is historically more likely to go unpatched and pwnage is more likely to go unnoticed, so it’s unfortunately a good defensive move to penalize them reputationally by default. PS yahoo is the worst for assuming poor reputation from a sender. In my experience they just introduce massive delivery delays at the drop of a hat even when the sender has a stellar reputation.

bhauer 4y ago
> The hobbyist/small mail server is historically more likely to go unpatched and pwnage is more likely to go unnoticed, so it’s unfortunately a good defensive move to penalize them reputationally by default. But to reiterate my earlier point: Google and Microsoft have the computational and storage capacity to have a detailed history of all domains' authoritative mail servers behavior. They will know whether a domain has a history of patch negligence. No, I think the far simpler explanation is that they just don't bother tracking reputation by domain. Or if they do, it's largely overshadowed by the weight given to IP-based reputation.
- hamburglar 4y ago
  Oh, I’m not saying you’re wrong. They absolutely could put the effort in to solve this problem.
alxlaz 4y ago
> I think there is a little more rationale than simply not caring. You're right in one more way: both Google and Microsoft offer email hosting solutions for small businesses. Their main (and, if you've got more than 8-10 accounts or so, the only) selling point is that it makes managing email hassle-free. Making it as painful as possible for small businesses to host their email server helps these services tremendously. If they had real interoperability (either of their own accord or because it were forced upon them through regulatory measures), the biggest cash cows of these services -- companies that are well into "enough accounts that your own server would be much cheaper" territory but not large enough to afford or risk large infrastructure changes -- would evaporate pretty quickly.

gnarbarian 4y ago

This seems like a major legal liability for Google. It could be shown that Google and other major email hosting providers act like a cartel by unfairly discriminating against companies who don't use a major email provider.

amenod 4y ago
I hope so! I am tired of explaining people that it's actually Google's fault they didn't receive my mail. I'll be happy when they pay dearly for the disservice they do to e-mail.
- gnarbarian 4y ago
  Are you using S/MIME certificates?
  - smartbit 4y ago
    Why are you asking? Is there a relation between spam handling and usage of S/MIME?
  - Kadin 4y ago
    Last time I looked into it (I run a mailserver and mailman list for one of my hobby groups), S/MIME wouldn't change your "spamminess" reputation score. DKIM, DMARC, and SPF do, though, and basically are table stakes if you want your mail (especially mailinglist messages) to go through to people at major providers.
  - amenod 4y ago
    No, would that help? I am however using SPF and one of DKIM / DMARC (I forget which). But anyway, I can live with a mail missing here and there. It is just annoying and isn't right.
- Avamander 4y ago
  It's not really Google's fault to be honest. The need to warm up new IP's has existed for a while and a lot of providers do it. Any postmaster with experience knows how and why it's done.

bombcar 4y ago

Back when domains were easy to spoof (I could setup a server and send mail as ycombinator.com easily enough) it made sense to track the IPs, but now that you have DKIM and SPF links to cross-check, you should be able to use the domain reliability as a strong indicator. Sure you would have to catch people buying a "good" domain that expired, but that shouldn't be an insane hurdle. The real story is nobody cares.

bhauer 4y ago
> The real story is nobody cares. Exactly. Why should the major players care about those too small to matter? > Sure you would have to catch people buying a "good" domain that expired Adding to my earlier point, Google and Microsoft are well-enough connected to the domain registrars to know when that scenario has happened as well. If they put any effort into it, they could reliably determine whether a new IP address for an established domain is legitimate or a fraud. But as we've said, why put any effort into it when the only people complaining are not important?
- bombcar 4y ago
  And it's even worse - if we theorize a email competitor appearing out of nowhere to rival Gmail, and people complaining, all that would happen is Google and Microsoft would special-case that provider, and the underlying issue wouldn't be solved.
  - gnarbarian 4y ago
    a class action lawsuit could make them care. as far as how to solve this problem technically, I think a reputation system based not on domains or ips but on email certificates is the real answer here.
    - toast0 4y ago
      DKIM is certificates, so I'm guessing you're talking about sender certificates? How would that help? Spammers can get certificates too. Maybe it cuts down on some of the misconfigured http email senders, maybe, but not enough to matter. Scam sites run https these days. You can't use like age of activity of the cert to help because a) things get compromised, b) you need to rotate your certs frequently anyway.

YetAnotherNick 4y ago

I could bet that most of mails from new IPs are spam because there are much fewer people willing to set up a mailserver than there are spammer operated mailservers. It is cheap to buy a new IP(comes free with many $2 hosting plans). The problem of identifying if a new IP is used for spam is not easy to solve.

bhauer 4y ago
> The problem of identifying if a new IP is used for spam is not easy to solve. How is it not easy to check a reputation database for domains when evaluating a mail server? Using DKIM records, Google could cross-reference the reputation for the sending domain and, where applicable, recognize that the domain in question has never been malicious or negligent. And in that case, extend a probationary reputation sufficient to allow the IP to establish its own new reputation.
- YetAnotherNick 4y ago
  New IP means it is new and not part of any database.
  - bhauer 4y ago
    I'll quote myself: > How is it not easy to check a reputation database for domains when evaluating a mail server? The point I'm making in this thread is that they have the ability to maintain a database of domains that have been proven to be trustworthy. I am not talking about a database of IP addresses, and I am not sure why that is being raised here.
megous 4y ago
> The problem of identifying if a new IP is used for spam is not easy to solve. Huh? With dataset of ~100k classified hams/spams I get like >99% precisison in identifying spam/ham with just bogofilter and 0 heuristics whatsoever (my mail server accepts everything, and I just use bogofilter client side). I guratanee you MS has a dataset with > 100k emails, lol. It's not like this is unsolved problem.

w-j-w 4y ago

> small and hobbyist mail server operators From the perspective of an email receiver, I WANT things to be difficult for this group. Truly free email available to all would be unusable owing to spam. Oligopoly, I believe, results in the best experience for everyone.

ajross 4y ago

> If a domain's authoritative mail servers have a history of never sending spam, when that company or hobbyist (for any reason) has to switch hosting providers and gets a new IP address, Google/Microsoft should recognize that the domain, and its referenced mail servers, has never been malicious. That algorithm doesn't work. The internet is filled with parked domains that have "never been malicious". This just creates a new market for clean domains that you can use to evade protections. It makes spam a little more expensive, but it still gets through. None of these tricks work. There are no tricks. All rules can be gamed. The only thing that can't be easily faked is reality: if Microsoft knows you're a big org with a well-managed IT group running your output email setup, then they know they can (probably) trust you not to spam their customers. If you are too small to prove that to MS in a scalable way, no amount of heuristic trickery is going to help you.

bhauer 4y ago
I'm sorry, but I don't buy that argument. Google--a technology giant with algorithms and heuristics running most operations--is not up to the task of improving algorithms for email server reputation? No, they are definitely capable of improvement. It's simply not of interest to them. I can hardly blame them because the cost-benefit analysis clearly says, "why bother?" Of course anything can be gamed, but magnitude matters. I've had the same domain for 24 years, with many hundreds or thousands of email conversations between users of my server and those of the major email players over the years. I had to switch my mail server's IP address two years ago. Immediate zero reputation from many big players.
- Avamander 4y ago
  > is not up to the task of improving algorithms for email server reputation? No, they are definitely capable of improvement. It's simply not of interest to them. Incorrect, if they stopped spending all that money and effort to keep up, their users would get flooded. > Of course anything can be gamed Sending spam/marketing e-mails is a multi-million industry. Both on illegal and legal markets. It's a constant race.
BenjiWiebe 4y ago

kevincox 4y ago

FWIW I run a service that sends a decent amount of email to GMail users. While it was a bit slow to get started (messages being marked as spam) once I had sent for a month or two and had a few users that marked the messages as not-spam I don't appear to have any problems. I say this sending from a Digital Ocean IP address that occasionally changes. It appears that Google highly values domain reputation and that once I have got onto their good list I am doing OK. Disclaimers: I make sure to do everything else right. I have SPF and DKIM and my DMARC policy is to reject 100%. I also don't use IPv6 (DO Kube doesn't really support IPv6 well). I have found other major providers to be much worse. Microsoft seems to rely almost entirely on IP reputation and marks everything as spam, even accounts that have marked messages as "not spam" many, many times. Apple outright blocks the IP range.

AshamedCaptain 4y ago
My experience is that all of SPF, DKIM and DMARC are almost completely ignored by Gmail -- one day I simply stopped DKIM and DMARC and Google keep happily accepting emails (and to this day I still send emails without). In fact they will happily accept emails even when the SPF check didn't pass and the policy clearly says strict reject aka -all . While on the other hand I fully agree with TFA: I have _never_ been able to send an email to Gmail from a IPv6 address and have it not end up as spam, not even to accounts where I already whitelisted previous attempts. I don't think it's a reputation issue, since my IPv4 addresses likely have much worse reputation. It's as if they just handicap all IPv6 addresses.
- kevincox 4y ago
  GMail definitely respects my DMARC reject policy. But IDK how missing one of SPF or DKIM affect its spam decision. But with DMARC reject and both missing or invalid it will bounce the message every time I have seen.
  - tomatocracy 4y ago
    Automatic mail forwarding without altering the From address will cause DMARC alignment for SPF to break. This is a common enough legitimate setup that most providers seem to effectively downgrade the DMARC policy applied when they see this (usually reject becomes quarantine, quarantine becomes ignore).

ipaddr 4y ago

The really annoying thing these days is google blocking zip files as spam requiring you to use google drive.

rstupek 4y ago
Zip files are being used to infect computers with malware which would explain why they're blocking them
- DarylZero 4y ago
  I guess they're not allowed on Google Drive either.

walrus01 4y ago

they are basing their filters on the reputation of the netblock that your individual ipv4 /32 (or ipv6 equivalent) is contained in. And very often when you are hosting with a low cost VM or VPS provider the historical record of other IPs operated by the same hosting company in the same /24 or something is very poor. while I concur with all of the points you make, there is a logical and statistically accurate reason for some of these spam filters. even if your no-open-relay, rdns, spf, dkim, dmarc, SSL/TLS and other configuration is absolutely impeccable on your smtpd, your only recourse in a situation like this is to change to a new ISP that does not have a poor IP space reputation in all the adjacent IPs.

Neil44 4y ago
The netblock thing has happened to me. My Linode that I’ve had sending mail for 6 years gets blacklisted now purely for being on the netblock that it’s on.
- apocalyptic0n3 4y ago
  Yep. I've encountered the exact same issues on multiple Linode servers in multiple data centers. I've also experienced it with an AWS server. It's often almost impossible to get it unblocked because the support teams at the email providers (if you can even contact them) don't realize that's what's happening. I didn't understand it myself until I got a call with an engineer at Microsoft who realized it in the middle of our call. He unlisted us on the call and we were fine for a while, but were eventually blocked again and haven't been able to get unblocked. It's really frustrating and I'm slowly ending all of my email services (both professional and hobby) as a result.
  - Neil44 4y ago
    It’s frustrating. I had outbound smtp throttled right down and a script running that shut smtp down completely if the queue got too big. That triggers an alert and everything gets sorted. But no, we don’t like where you’re from so jog on. Monopolistic.