8 ms·

Lamson's Bounce Detection Algorithm

23 points by twampss 17y ago

snprbob86 17y ago
As I discussed in an earlier thread, our custom mailer uses Gmail on a custom domain for storage; fetchmail connects via IMAP and then pipes to procmail which pipes to a Python script. Lamson looks like it can offer a lot of great features, but I'll have to run my own SMTP server and I'm worried about downtime. If I let Gmail store the messages, I don't need to worry (much) about messages going missing. If my server crashes, fetchmail will just pick up the missed messages in the next fetch, only a few minutes late. For others who run mail servers: how do you handle downtime?
- sho 17y ago
  With redundancy, basically. But there's not much "downtime" with mail servers, it's not like web servers where they're always being updated with the latest code and the app server needs restarting. Any admin worth his salt should be able to guarantee very good uptime (barring HW/power/network failure) for mail under normal/medium load conditions; high load has its own challenges but that's a separate issue. Of course, real 100% uptime means multiple machines at multiple sites, cue standard cost/benefit analysis. And it's also important to bear in mind that it's an SMTP server; unless it's actually your users trying to connect directly it will be other mail servers talking to it. If you're down for a while, they will queue at their end and retry periodically. You'd have to be down for days to lose mail, although your users might get warning notices, which might look unprofessional. One thing you could do to get the benefit of having your own server, while keeping the gmail redundancy, is simply set gmail to forward all those messages to you. If you went down, they'd just build up until you were back up, and that way you could avoid users ever getting those embarrassing failure notices. I think Lamson looks awesome, btw. I started an SMTP server in Ruby before but gave up in horror when I started having to actually read those hideous RFCs. Shaw certainly has a knack for getting the dirty work done. A real mail server is a horrible job to write. He should be given a govt grant or something to do this shit. (edited to sound less like an asshole, unfortunately not before OP had replied .. sorry >_<)
  - snprbob86 17y ago
    It's not much of a trick and it certainly isn't very complex. fetchmailrc: poll pop.gmail.com with proto pop3 port 995 user "user@mydomain.com" password "pa$$w0rd" no rewrite ssl mda "procmail -f %F" procmailrc: LOGFILE=$HOME/log/procmail.log # Process the mail and let the application do its own logging. :0 | python2.5 $HOME/procmail.py # Alert the admins of any failures and keep a copy of the failed message. :0 c | python2.5 $HOME/alertfailure.py Then start fetchmail to run every 10 seconds or so: fetchmail -d 10
    - sho 17y ago
      Oops, sorry, I actually removed that bit since I thought I sounded like a douchebag. Serves me right for using "edit" to change my "drafts". Thanks for the explanation, that's simpler than I imagined. But you had said IMAP, which is way more fiddly than pop3, since you'd be holding open connections and all that stuff, so I was envisaging having to run long-running processes, etc, which can be a headache in itself. Is gmail really happy with you polling a POP server every 10 seconds all day every day?
      - snprbob86 17y ago
        I meant pop. Oops myself :-) Honestly, I haven't the slightest clue if Gmail minds the polling. I assumed that by the time we had enough mail traffic for it to be a problem, we'd be prepared to migrate to something like Lamson. Release early/often and all that jazz :-)
        sho 17y ago
        Hm, interesting. I hadn't even thought to use pop like that because I just assumed the admins or their automated scripts would get annoyed and block me. Yeah, good approach. Just do the simplest thing that could possibly work; upgrade if and when you need to. I should perhaps tattoo that somewhere since it never seems to sink in : /
- wmf 17y ago
  Besides what sho said about SMTP uptime not being that important, Lamson can read messages from a Maildir queue. You could use Gmail -> fetchmail -> Lamson. http://lamsonproject.org/docs/deferred_processing_to_queues.html http://lamsonproject.org/docs/deferred_processing_to_queues....
  - snprbob86 17y ago
    Aaahh cool. I didn't think about putting fetchmail in between. I was assuming I'd have to do Gmail forwarding to Lamson. We'll probably migrate to Lamson eventually to make good use of things like this neat bounce handling. Thanks!
sielskr 17y ago
From the page that says what the Lamson project is about (emphasis mine): "I’m always looking for people doing interesting things with e-mail. Even if you’re a spammer, or someone trying to destroy the spammers, I want to hear from you" (Zed Shaw, maintainer of the Lamson project).
- sp332 17y ago
  He's made the point before that any advance in email technology will be primarily used for spam. http://www.theregister.co.uk/2009/07/03/lamson/print.html http://www.theregister.co.uk/2009/07/03/lamson/print.html "I totally think Lamson will be used more for spam than for anything really useful. My hope with Lamson is that it acts like a modern sewage system for the internet. Everyone's gotta poop. You can't escape it, so why not try to do it in a clean fashion that improves the general health of everyone involved?"
  - icey 17y ago
    He's also said that he hopes it will mean spammers end up better at targeting so that there will be less shit flowing overall.
    - eli 17y ago
      Makes sense. The huge amount of backscatter ain't helping anyone
mkyc 17y ago
I don't check my spam folder. If it's in there, it's not delivered. Email providers should (optionally) bounce spam, rather than sending it to a spam folder. This would eliminate the problem of false-positives, since senders would be notified. It would reduce traffic, since spammers would know the message didn't get through to any inbox, and so would focus elsewhere. Finally, it would eliminate the temptation of a spam folder for people who don't want spam, but peek 'just to make sure', and thereafter go on to support spammers. Perhaps someone here is running an email provider, or working on such software?
- eli 17y ago
  Bounce it where? The return address is always fake and sometimes it's that of an innocent bystander.
  - tome 17y ago
    Not if it's a false positive!
  - nailer 17y ago
    I'd 550 the spamming server sending it to me, they can take care of the bounce themselves.
- pwmanagerdied 17y ago
  The problem is that the overwhelming majority of spam has a false return address, and the technologies to verify that mail came from the server it claims aren't yet widely enough used to be reliable. The bounces create an extremely large amount of traffic for servers unrelated to the spammers. Besides, many servers do behave in the way you specify. Before I had the sense to disable catch-all email addresses on a few domains I owned, they were all being flooded with bounced spam, much of which my spam filters didn't catch. Blah. Everything's a mess.
  - mkyc 17y ago
    The important part of what I said above is that the spam would not arrive in any inbox. The temptation (and ability) to look through spam is the current problem, and would be eliminated. If some servers are already doing that, then they are doing the right thing - except, the message body of a spam should not be bounced. The extra traffic on the bounce is worth expending, since it is necessary to keep spam out of sight. If they can't get their message through, they lose the incentive to send spam. That's all that matters.

23 points by twampss 17y ago

As I discussed in an earlier thread, our custom mailer uses Gmail on a custom domain for storage; fetchmail connects via IMAP and then pipes to procmail which pipes to a Python script. Lamson looks like it can offer a lot of great features, but I'll have to run my own SMTP server and I'm worried about downtime. If I let Gmail store the messages, I don't need to worry (much) about messages going missing. If my server crashes, fetchmail will just pick up the missed messages in the next fetch, only a few minutes late. For others who run mail servers: how do you handle downtime?

sho 17y ago
With redundancy, basically. But there's not much "downtime" with mail servers, it's not like web servers where they're always being updated with the latest code and the app server needs restarting. Any admin worth his salt should be able to guarantee very good uptime (barring HW/power/network failure) for mail under normal/medium load conditions; high load has its own challenges but that's a separate issue. Of course, real 100% uptime means multiple machines at multiple sites, cue standard cost/benefit analysis. And it's also important to bear in mind that it's an SMTP server; unless it's actually your users trying to connect directly it will be other mail servers talking to it. If you're down for a while, they will queue at their end and retry periodically. You'd have to be down for days to lose mail, although your users might get warning notices, which might look unprofessional. One thing you could do to get the benefit of having your own server, while keeping the gmail redundancy, is simply set gmail to forward all those messages to you. If you went down, they'd just build up until you were back up, and that way you could avoid users ever getting those embarrassing failure notices. I think Lamson looks awesome, btw. I started an SMTP server in Ruby before but gave up in horror when I started having to actually read those hideous RFCs. Shaw certainly has a knack for getting the dirty work done. A real mail server is a horrible job to write. He should be given a govt grant or something to do this shit. (edited to sound less like an asshole, unfortunately not before OP had replied .. sorry >_<)
- snprbob86 17y ago
  It's not much of a trick and it certainly isn't very complex. fetchmailrc: poll pop.gmail.com with proto pop3 port 995 user "user@mydomain.com" password "pa$$w0rd" no rewrite ssl mda "procmail -f %F" procmailrc: LOGFILE=$HOME/log/procmail.log # Process the mail and let the application do its own logging. :0 | python2.5 $HOME/procmail.py # Alert the admins of any failures and keep a copy of the failed message. :0 c | python2.5 $HOME/alertfailure.py Then start fetchmail to run every 10 seconds or so: fetchmail -d 10
  - sho 17y ago
    Oops, sorry, I actually removed that bit since I thought I sounded like a douchebag. Serves me right for using "edit" to change my "drafts". Thanks for the explanation, that's simpler than I imagined. But you had said IMAP, which is way more fiddly than pop3, since you'd be holding open connections and all that stuff, so I was envisaging having to run long-running processes, etc, which can be a headache in itself. Is gmail really happy with you polling a POP server every 10 seconds all day every day?
    - snprbob86 17y ago
      I meant pop. Oops myself :-) Honestly, I haven't the slightest clue if Gmail minds the polling. I assumed that by the time we had enough mail traffic for it to be a problem, we'd be prepared to migrate to something like Lamson. Release early/often and all that jazz :-)
      - sho 17y ago
        Hm, interesting. I hadn't even thought to use pop like that because I just assumed the admins or their automated scripts would get annoyed and block me. Yeah, good approach. Just do the simplest thing that could possibly work; upgrade if and when you need to. I should perhaps tattoo that somewhere since it never seems to sink in : /
wmf 17y ago
Besides what sho said about SMTP uptime not being that important, Lamson can read messages from a Maildir queue. You could use Gmail -> fetchmail -> Lamson. http://lamsonproject.org/docs/deferred_processing_to_queues.html http://lamsonproject.org/docs/deferred_processing_to_queues....
- snprbob86 17y ago
  Aaahh cool. I didn't think about putting fetchmail in between. I was assuming I'd have to do Gmail forwarding to Lamson. We'll probably migrate to Lamson eventually to make good use of things like this neat bounce handling. Thanks!

sielskr 17y ago

From the page that says what the Lamson project is about (emphasis mine): "I’m always looking for people doing interesting things with e-mail. Even if you’re a spammer, or someone trying to destroy the spammers, I want to hear from you" (Zed Shaw, maintainer of the Lamson project).

sp332 17y ago
He's made the point before that any advance in email technology will be primarily used for spam. http://www.theregister.co.uk/2009/07/03/lamson/print.html http://www.theregister.co.uk/2009/07/03/lamson/print.html "I totally think Lamson will be used more for spam than for anything really useful. My hope with Lamson is that it acts like a modern sewage system for the internet. Everyone's gotta poop. You can't escape it, so why not try to do it in a clean fashion that improves the general health of everyone involved?"
- icey 17y ago
  He's also said that he hopes it will mean spammers end up better at targeting so that there will be less shit flowing overall.
  - eli 17y ago
    Makes sense. The huge amount of backscatter ain't helping anyone

mkyc 17y ago

I don't check my spam folder. If it's in there, it's not delivered. Email providers should (optionally) bounce spam, rather than sending it to a spam folder. This would eliminate the problem of false-positives, since senders would be notified. It would reduce traffic, since spammers would know the message didn't get through to any inbox, and so would focus elsewhere. Finally, it would eliminate the temptation of a spam folder for people who don't want spam, but peek 'just to make sure', and thereafter go on to support spammers. Perhaps someone here is running an email provider, or working on such software?

eli 17y ago
Bounce it where? The return address is always fake and sometimes it's that of an innocent bystander.
- tome 17y ago
  Not if it's a false positive!
- nailer 17y ago
  I'd 550 the spamming server sending it to me, they can take care of the bounce themselves.
pwmanagerdied 17y ago
The problem is that the overwhelming majority of spam has a false return address, and the technologies to verify that mail came from the server it claims aren't yet widely enough used to be reliable. The bounces create an extremely large amount of traffic for servers unrelated to the spammers. Besides, many servers do behave in the way you specify. Before I had the sense to disable catch-all email addresses on a few domains I owned, they were all being flooded with bounced spam, much of which my spam filters didn't catch. Blah. Everything's a mess.
- mkyc 17y ago
  The important part of what I said above is that the spam would not arrive in any inbox. The temptation (and ability) to look through spam is the current problem, and would be eliminated. If some servers are already doing that, then they are doing the right thing - except, the message body of a spam should not be bounced. The extra traffic on the bounce is worth expending, since it is necessary to keep spam out of sight. If they can't get their message through, they lose the incentive to send spam. That's all that matters.