4 ms·

Well this is pretty neat. I've been working on custom software to improve the spam filtering on my mail server for the last year (side project). It currently w

Well this is pretty neat. I've been working on custom software to improve the spam filtering on my mail server for the last year (side project). It currently works by letting hosted users forward spam messages to a flytrap account, and then the daemon runs, reads the forwarded message, tracks down the original in the user's mail directory, does a whois on the origin in the mail headers, consults its logs, and then adds a temporary network-wide blackhole to iptables. Originally it was intended to work alongside SpamAssassin and SQLGrey and all that, but last night I started considering replacing SpamAssassin altogether. I love SA, but the spammers are beating it regularly now. My TODO notes in the code actually say, "reputation tracking for embedded URLs, domains, ccTLDs and gTLDs, sender addresses, and content keywords." I wrote the first bits of code for reputation tracking this morning. It's not much of a step for the software really, because it already uses embedded URLs in a message as part of the profile "fingerprint" for finding the original message from a forwarded version. But I'm a bit chuffed to hear that I'm on the right track, considering how effective Gmail's tactics have been. :-) Small service providers have it really tough right now. Users don't tolerate any spam at all. A few years ago, the state of the art for small independent services was SpamAssassin + SQLGrey (or other greylisting) plus a few other tricks; that's not sufficient anymore, and most of us smallfry lack the resources to come up with something much better. After just 6 weeks in production, the software already has 20+million IPs blocked at any given time.

oalders 12y ago
Do you have any plans to release this software? I've got a small hosting business and SpamAssassin seems to be getting beaten on a very regular basis.
- thaumaturgy 12y ago
  Yeah, I've been thinking about that recently. There are some challenges though. Probably the biggest one is that it's designed for a specific mail server setup: postfix + dovecot (configured for maildir) + fail2ban + php (the code is in php, because it was convenient) + mysql. I don't know yet how portable it will be. If you'd like to try it anyway, let me know and I'll post what I've got to GitHub in the next few days. edit: alternatively, I've been more seriously considering making the current network ban list available as an RBL. Since I already have DNS servers, it would be pretty trivial to do.
  - baudehlo 12y ago
    Make sure you read the DNSBL rfc if you implement that :)
  - bgaluszka 12y ago
    I run personal email server with postfix + dovecot and was thinking about replacing SA with something custom written in php. I would love to see what you have written.
  - oalders 12y ago
    Your setup sounds pretty much like mine, so it would be portable in my case. :) Having said that, if this were available as an RBL that would also be really great.
baudehlo 12y ago
This sounds a lot like DCC. Have you considered adding it into your filtering layer?
baudehlo 12y ago
I think SA has suffered from some of the original core developers (myself included) moving on to other projects in a completely different tech area. The good news is that other projects have taken up some of the mantle, like Haraka, check out the karma plugin. It does some amazing blocking of spam and penalizing clients. Beyond that also one of the things SA doesn't do well is actually rejecting hard on sensible blacklists like the CBL. We worked hard to make everything heuristic based but it wasn't always the right choice. Some things need to be black and white. There's some code on SA for short circuiting now but it's not really the best solution. In my own spam filtering I have a bunch of hard rejects and they work really well. Anyway, check out Haraka or Qpsmtpd for solid anti spam mail serving solutions. They work really well.
- patio11 12y ago
  Hey wait, I did vaguely recognize your username from somewhere. Thanks for SpamAssassin! I spent more hours spelunking through your guys' source code and community rulesets than you'd want to know, about a decade ago, while working for the anti-spam group in our R&D department at a tech incubator.
- xorcist 12y ago
  Thank you for SA. I do not share the experience that it is outsmarted, I still achieve > 98% accuracy just as I did ten years ago. That is really a testament to all the hard work of its developers.

oalders 12y ago

Do you have any plans to release this software? I've got a small hosting business and SpamAssassin seems to be getting beaten on a very regular basis.

thaumaturgy 12y ago
Yeah, I've been thinking about that recently. There are some challenges though. Probably the biggest one is that it's designed for a specific mail server setup: postfix + dovecot (configured for maildir) + fail2ban + php (the code is in php, because it was convenient) + mysql. I don't know yet how portable it will be. If you'd like to try it anyway, let me know and I'll post what I've got to GitHub in the next few days. edit: alternatively, I've been more seriously considering making the current network ban list available as an RBL. Since I already have DNS servers, it would be pretty trivial to do.
- baudehlo 12y ago
  Make sure you read the DNSBL rfc if you implement that :)
- bgaluszka 12y ago
  I run personal email server with postfix + dovecot and was thinking about replacing SA with something custom written in php. I would love to see what you have written.
- oalders 12y ago
  Your setup sounds pretty much like mine, so it would be portable in my case. :) Having said that, if this were available as an RBL that would also be really great.

baudehlo 12y ago

This sounds a lot like DCC. Have you considered adding it into your filtering layer?

baudehlo 12y ago

I think SA has suffered from some of the original core developers (myself included) moving on to other projects in a completely different tech area. The good news is that other projects have taken up some of the mantle, like Haraka, check out the karma plugin. It does some amazing blocking of spam and penalizing clients. Beyond that also one of the things SA doesn't do well is actually rejecting hard on sensible blacklists like the CBL. We worked hard to make everything heuristic based but it wasn't always the right choice. Some things need to be black and white. There's some code on SA for short circuiting now but it's not really the best solution. In my own spam filtering I have a bunch of hard rejects and they work really well. Anyway, check out Haraka or Qpsmtpd for solid anti spam mail serving solutions. They work really well.

patio11 12y ago
Hey wait, I did vaguely recognize your username from somewhere. Thanks for SpamAssassin! I spent more hours spelunking through your guys' source code and community rulesets than you'd want to know, about a decade ago, while working for the anti-spam group in our R&D department at a tech incubator.
xorcist 12y ago
Thank you for SA. I do not share the experience that it is outsmarted, I still achieve > 98% accuracy just as I did ten years ago. That is really a testament to all the hard work of its developers.