6 ms·

I remember reading pg's post, it was a classic. Anyone here working on anti-spam? Where are things at today?

amaccuish 8y ago
Not "working" on anti-spam, but for my family's email server, we run rspamd and it works really well.
Felz 8y ago
I'm not directly working on spam, but it seems like there's a heavy reliance on ipv4 address reputation these days, which is probably one reason mailservers won't use ipv6 any time soon.
- thatsaguy 8y ago
  Address reputation was always big, even back then. In the last years though, I cannot really recommend to use any of the DNSBL anymore. I've encountered more cases where legitimate servers were blocked due to netblock vicinity or indeed previous ownership than actual spam issues. Greylisting will still catch dynamic allocations almost as effectively, while you won't reject legitimate mail due to server and/or DNSBL issues.
  - jorangreef 8y ago
    Have you tried using a quorum of DNSBL (e.g. barracudacentral.org, cbl.abuseat.org, truncate.gbudb.net) to reduce false positives? In other words, if at least two DNSBL queries agree, then reject, or feed this information to the rest of the spam pipeline?
    - thatsaguy 8y ago
      This is pretty common, and systems like SA do this for you by batching responses and calculating a score. I found this to be pretty much worthless if you already have greylisting, even for high-quality curated lists such as spamhaus SBL/XBL.
thatsaguy 8y ago
From my perspective (~500 employee mail server), greylisting had a much larger impact at the time, thanks to the spambots/viruses attempting direct connection to mail servers. Extremely effective, zero false positives, much lighter on resources. I did use both, of course, so that I could keep a record of how effective the systems were. Today the situation has flipped. Most of the spam we get is coming from authoritative servers (ie: gmail, yahoo, etc), making stuff like SPF/DKIM/etc next to worthless from a spam perspective (it's still marginally useful for forgeries), while bayes (or in general, trainable) filters are essentially the only thing that can differentiate it reliably. With a modern setup, you can basically next to zero spam and no false positives. In fact, honest email marketing (ie: mailing lists you've actually subscribed to) are from my experience the only thing that throws these filters off.
- jorangreef 8y ago
  Thanks, one thing we also found is that spammers tend to be poor at RFC standards, in a way that Gmail etc. will have no problem with, but which are obviously broken. For example, we use our own https://github.com/ronomon/mime https://github.com/ronomon/mime to detect and reject email which has missing multi-parts (no terminating boundary delimiter). All of this has been spam so far, and we are yet to see a false positive. I don't think SpamAssassin has a rule for this (yet)? Another example is illegal header characters, which are almost always spam, with a handful of false positives (usually machine-generated).
  - readingnews 8y ago
    That is an interesting approach. Care to let us know how you go from https://github.com/ronomon/mime https://github.com/ronomon/mime to some kind of SMTP server plugin (like for postfix for example)?
    - jorangreef 8y ago
      Thanks, you might find Haraka to be easiest since it's already Javascript. Postfix may require a process callout, you might need to write a milter.
pjc50 8y ago
I was running my own email server round about this time, and gave it up mid-2000s, in great part because of spam. Both inbound filtering and outbound deliverability into other people's filters. It seems that centralisation has been highly effective in spam-fighting. Google must have a huge corpus of spam and ham, and they also have the benefit of being able to spot patterns of incoming mail "live" across a very large number of accounts.

Not "working" on anti-spam, but for my family's email server, we run rspamd and it works really well.

Felz 8y ago

I'm not directly working on spam, but it seems like there's a heavy reliance on ipv4 address reputation these days, which is probably one reason mailservers won't use ipv6 any time soon.

thatsaguy 8y ago
Address reputation was always big, even back then. In the last years though, I cannot really recommend to use any of the DNSBL anymore. I've encountered more cases where legitimate servers were blocked due to netblock vicinity or indeed previous ownership than actual spam issues. Greylisting will still catch dynamic allocations almost as effectively, while you won't reject legitimate mail due to server and/or DNSBL issues.
- jorangreef 8y ago
  Have you tried using a quorum of DNSBL (e.g. barracudacentral.org, cbl.abuseat.org, truncate.gbudb.net) to reduce false positives? In other words, if at least two DNSBL queries agree, then reject, or feed this information to the rest of the spam pipeline?
  - thatsaguy 8y ago
    This is pretty common, and systems like SA do this for you by batching responses and calculating a score. I found this to be pretty much worthless if you already have greylisting, even for high-quality curated lists such as spamhaus SBL/XBL.

thatsaguy 8y ago

From my perspective (~500 employee mail server), greylisting had a much larger impact at the time, thanks to the spambots/viruses attempting direct connection to mail servers. Extremely effective, zero false positives, much lighter on resources. I did use both, of course, so that I could keep a record of how effective the systems were. Today the situation has flipped. Most of the spam we get is coming from authoritative servers (ie: gmail, yahoo, etc), making stuff like SPF/DKIM/etc next to worthless from a spam perspective (it's still marginally useful for forgeries), while bayes (or in general, trainable) filters are essentially the only thing that can differentiate it reliably. With a modern setup, you can basically next to zero spam and no false positives. In fact, honest email marketing (ie: mailing lists you've actually subscribed to) are from my experience the only thing that throws these filters off.

jorangreef 8y ago
Thanks, one thing we also found is that spammers tend to be poor at RFC standards, in a way that Gmail etc. will have no problem with, but which are obviously broken. For example, we use our own https://github.com/ronomon/mime https://github.com/ronomon/mime to detect and reject email which has missing multi-parts (no terminating boundary delimiter). All of this has been spam so far, and we are yet to see a false positive. I don't think SpamAssassin has a rule for this (yet)? Another example is illegal header characters, which are almost always spam, with a handful of false positives (usually machine-generated).
- readingnews 8y ago
  That is an interesting approach. Care to let us know how you go from https://github.com/ronomon/mime https://github.com/ronomon/mime to some kind of SMTP server plugin (like for postfix for example)?
  - jorangreef 8y ago
    Thanks, you might find Haraka to be easiest since it's already Javascript. Postfix may require a process callout, you might need to write a milter.

pjc50 8y ago

I was running my own email server round about this time, and gave it up mid-2000s, in great part because of spam. Both inbound filtering and outbound deliverability into other people's filters. It seems that centralisation has been highly effective in spam-fighting. Google must have a huge corpus of spam and ham, and they also have the benefit of being able to spot patterns of incoming mail "live" across a very large number of accounts.