5 ms·

The Cat-And-Mouse Story of Implementing Anti-Spam for Mail.ru

63 points by yarapavan 10y ago

yarapavan 10y ago
Tarantool, mentioned in here, is an in-memory database and application server with 100% compatible drop-in replacement for Lua 5.1 Github page: https://github.com/tarantool/tarantool/ https://github.com/tarantool/tarantool/
kazinator 10y ago
One powerful rule that I use in my Exim setup is to drop any SMTP connection that doesn't have forward and backward DNS that match. This catches a big majority of all spam, with hardly any false positive downside. No credible in SMTP forwarding host has broken DNS (by definition, practically). Lots of spamming machines do. Of course, not all, but this is a game of percentages and multiple stages.
- treve 10y ago
  You'll definitely lose email from Amazon SES then. What about checking SPF records instead?
  - kazinator 10y ago
    My darned residential cable line has forward and reverse DNS that match. If the Amazon SES can't figure this out, they have no business sending SMTP e-mail. I do apply the SPF check and reject the SMTP if it comes up as a confirmed negative: the sending domain has an SPF record, with strict semantics, and the IP doesn't match. SPF speaks to a different problem (not the problem of "is this host a spammy host?"), and generally has no teeth. Passing SPF only informs me that the IP address sending to me is allowed to use the given envelope sender domain. A spammer can register a bunch of domains and set up SPF records for them, and then use the domains as envelope addresses. (Of course, these envelope sender addresses don't necessarily have anything to do with what is in the From: header). You can't expect some sort of free pass just because you made a SPF record. What the SPF record does is protect your domain from being used by other people the bases of a sender envelope address. This is of low value, because most recipients don't even see the envelope address. SPF can be applied against From: too but that's fraught with problems, because of forwarding via mailing lists, or people using alternative sending identities which are legitimate.
  - r00fus 10y ago
    Is this a bad thing, if you're trying to avoid spam?
- Nux 10y ago
  So, do you require the PTR and A records of the remote host to match?
  - kazinator 10y ago
    I don't work with DNS at the protocol level; it's done via the Exim "verify = reverse_host_lookup". The documentation says: This condition ensures that a verified host name has been looked up from the IP address of the client host. (This may have happened already if the host name was needed for checking a host list, or if the host matched host_lookup.) Verification ensures that the host name obtained from a reverse DNS lookup, or one of its aliases, does, when it is itself looked up in the DNS, yield the original IP address. The phrase "or one of its aliases" seems to imply that the match does not strictly require the A record.
    - jlgaddis 10y ago
      The general term for this is "Forward Confirmed reverse DNS" (FCrDNS). I do these checks (plus others) on my Postfix mail systems. I would assume most of the other (sane) MTAs can perform these checks as well.
- marcosdumay 10y ago
  Never thought about it (and I've been thinking about email a lot recently), but now that people mentioned it... Every mail sever defaults should be to trust an encrypted connection with a valid MX host name. Failing that, it may choose to check either reverse DNS or SPF. I see no reason for preferring any of those. Not checking anything by default isn't a good idea nowadays.
kazinator 10y ago
I use a home-grown web app called Tamarind for managing throwaway mail aliases, in conjunction with Exim: http://www.kylheku.com/cgit/tamarind/tree/README http://www.kylheku.com/cgit/tamarind/tree/README
dmm 10y ago
You suck! I ran an email server from my home for several years. I could get a static ip but my isp wouldn't set my rdns for some reason. I had both spf and dkim correctly configured but people like you eventually forced me to move to commercial hosting.
- kazinator 10y ago
  Resolving MX records from your home and contacting hosts directly on their port 25 is a nonstarter. I wouldn't even think of doing it. Instead, I forward to my ISP's SMTP forwarding hosts (via an authenticated SMTP connection). It's not necessarily the "working DNS" rule that will reject you, but other methods, like anti-spam databases that list residential line IP addresses. For example, my residential line, in fact, has forward and backward DNS that match. If I directly contact someone's mail server which has this rule, I can get past that rule. That's not what will "get" me.
  - dmm 10y ago
    > Resolving MX records from your home and contacting hosts directly on their port 25 is a nonstarter. Google, Yahoo, and Microsoft accepted mail from my server without any issues. My server never sent spam and was never on spam blacklists. What's your problem? Maybe you suck.
    - deleted 10y ago
      [deleted]
    - deleted 10y ago
      [deleted]
  - AstralStorm 10y ago
    Funny thing, my own old mail servers allowed that for AUTHed SMTP w/ correct TLS, but then disallowed invalid domains or wrong account in From field. There are really no legitimate reasons to allow spoofed outgoing mail.
- dang 10y ago
Robin_Message 10y ago
The spam bit sounds cool but the nosql db sounds odd. E.G. That tree implementation with batched rebalancing sounds downright scary, and I thought avl was considered the worst kind of balanced tree due to excessive overhead (an int per node to red-black's bit) .
- bigbes 10y ago
  > Our investigation revealed that in case of frequent insertions and deletions Tarantool initiated a complex process of tree rebalancing (all our indexes were of TREE type). It's all about 1.5. New version (1.6) uses brand new bps-tree, not sg/avl-tree. It behaves better on all workloads. AVLTree was "temporary" hack. Our implementation works better, for their needs. BTW - AVL is not bad, but it's hard to implement a good one (believe me :) ).
  - Robin_Message 10y ago
    Nice. Yeah, AVL is definitely okay but it makes me happy to hear something better is in there :) What sort of tree is this new bps-tree?
    - bigbes 10y ago
      bps-tree means B+-tree: unique combination of B+ and B tree. You can read wiki or "The Ubiquitous B-Tree" whitepaper https://wwwold.cs.umd.edu/class/fall2002/cmsc818s/Readings/b-tree.pdf https://wwwold.cs.umd.edu/class/fall2002/cmsc818s/Readings/b... . You may read code of it here: https://github.com/tarantool/tarantool/blob/1.6/src/lib/salad/bps_tree.h https://github.com/tarantool/tarantool/blob/1.6/src/lib/sala... . It's very thoroughly commented.

63 points by yarapavan 10y ago

Tarantool, mentioned in here, is an in-memory database and application server with 100% compatible drop-in replacement for Lua 5.1 Github page: https://github.com/tarantool/tarantool/ https://github.com/tarantool/tarantool/

kazinator 10y ago

One powerful rule that I use in my Exim setup is to drop any SMTP connection that doesn't have forward and backward DNS that match. This catches a big majority of all spam, with hardly any false positive downside. No credible in SMTP forwarding host has broken DNS (by definition, practically). Lots of spamming machines do. Of course, not all, but this is a game of percentages and multiple stages.

treve 10y ago
You'll definitely lose email from Amazon SES then. What about checking SPF records instead?
- kazinator 10y ago
  My darned residential cable line has forward and reverse DNS that match. If the Amazon SES can't figure this out, they have no business sending SMTP e-mail. I do apply the SPF check and reject the SMTP if it comes up as a confirmed negative: the sending domain has an SPF record, with strict semantics, and the IP doesn't match. SPF speaks to a different problem (not the problem of "is this host a spammy host?"), and generally has no teeth. Passing SPF only informs me that the IP address sending to me is allowed to use the given envelope sender domain. A spammer can register a bunch of domains and set up SPF records for them, and then use the domains as envelope addresses. (Of course, these envelope sender addresses don't necessarily have anything to do with what is in the From: header). You can't expect some sort of free pass just because you made a SPF record. What the SPF record does is protect your domain from being used by other people the bases of a sender envelope address. This is of low value, because most recipients don't even see the envelope address. SPF can be applied against From: too but that's fraught with problems, because of forwarding via mailing lists, or people using alternative sending identities which are legitimate.
- r00fus 10y ago
  Is this a bad thing, if you're trying to avoid spam?
Nux 10y ago
So, do you require the PTR and A records of the remote host to match?
- kazinator 10y ago
  I don't work with DNS at the protocol level; it's done via the Exim "verify = reverse_host_lookup". The documentation says: This condition ensures that a verified host name has been looked up from the IP address of the client host. (This may have happened already if the host name was needed for checking a host list, or if the host matched host_lookup.) Verification ensures that the host name obtained from a reverse DNS lookup, or one of its aliases, does, when it is itself looked up in the DNS, yield the original IP address. The phrase "or one of its aliases" seems to imply that the match does not strictly require the A record.
  - jlgaddis 10y ago
    The general term for this is "Forward Confirmed reverse DNS" (FCrDNS). I do these checks (plus others) on my Postfix mail systems. I would assume most of the other (sane) MTAs can perform these checks as well.
marcosdumay 10y ago
Never thought about it (and I've been thinking about email a lot recently), but now that people mentioned it... Every mail sever defaults should be to trust an encrypted connection with a valid MX host name. Failing that, it may choose to check either reverse DNS or SPF. I see no reason for preferring any of those. Not checking anything by default isn't a good idea nowadays.

kazinator 10y ago

I use a home-grown web app called Tamarind for managing throwaway mail aliases, in conjunction with Exim: http://www.kylheku.com/cgit/tamarind/tree/README http://www.kylheku.com/cgit/tamarind/tree/README

dmm 10y ago

You suck! I ran an email server from my home for several years. I could get a static ip but my isp wouldn't set my rdns for some reason. I had both spf and dkim correctly configured but people like you eventually forced me to move to commercial hosting.

kazinator 10y ago
Resolving MX records from your home and contacting hosts directly on their port 25 is a nonstarter. I wouldn't even think of doing it. Instead, I forward to my ISP's SMTP forwarding hosts (via an authenticated SMTP connection). It's not necessarily the "working DNS" rule that will reject you, but other methods, like anti-spam databases that list residential line IP addresses. For example, my residential line, in fact, has forward and backward DNS that match. If I directly contact someone's mail server which has this rule, I can get past that rule. That's not what will "get" me.
- dmm 10y ago
  > Resolving MX records from your home and contacting hosts directly on their port 25 is a nonstarter. Google, Yahoo, and Microsoft accepted mail from my server without any issues. My server never sent spam and was never on spam blacklists. What's your problem? Maybe you suck.
  - deleted 10y ago
    [deleted]
  - deleted 10y ago
    [deleted]
- AstralStorm 10y ago
  Funny thing, my own old mail servers allowed that for AUTHed SMTP w/ correct TLS, but then disallowed invalid domains or wrong account in From field. There are really no legitimate reasons to allow spoofed outgoing mail.
dang 10y ago

Robin_Message 10y ago

The spam bit sounds cool but the nosql db sounds odd. E.G. That tree implementation with batched rebalancing sounds downright scary, and I thought avl was considered the worst kind of balanced tree due to excessive overhead (an int per node to red-black's bit) .

bigbes 10y ago
> Our investigation revealed that in case of frequent insertions and deletions Tarantool initiated a complex process of tree rebalancing (all our indexes were of TREE type). It's all about 1.5. New version (1.6) uses brand new bps-tree, not sg/avl-tree. It behaves better on all workloads. AVLTree was "temporary" hack. Our implementation works better, for their needs. BTW - AVL is not bad, but it's hard to implement a good one (believe me :) ).
- Robin_Message 10y ago
  Nice. Yeah, AVL is definitely okay but it makes me happy to hear something better is in there :) What sort of tree is this new bps-tree?
  - bigbes 10y ago
    bps-tree means B+-tree: unique combination of B+ and B tree. You can read wiki or "The Ubiquitous B-Tree" whitepaper https://wwwold.cs.umd.edu/class/fall2002/cmsc818s/Readings/b-tree.pdf https://wwwold.cs.umd.edu/class/fall2002/cmsc818s/Readings/b... . You may read code of it here: https://github.com/tarantool/tarantool/blob/1.6/src/lib/salad/bps_tree.h https://github.com/tarantool/tarantool/blob/1.6/src/lib/sala... . It's very thoroughly commented.