4 ms·

Ask HN: Email handler

18 points by snprbob86 18y ago

I am currently building a Django web application which involves listening for email. I looked high and low, but couldn't find an existing solution for implementing automated email handlers in Python. So I decided to build one. I'm still hoping there is a great library out there that I missed. Does anyone know of one? Getting a simple handler up and running was easy, but I have run into some trouble with graceful handling of error conditions. For example, I don't always get SMTP error codes when sending mail to bad addresses or full mailboxes. Instead, I often get non-delivery receipts as emails a while later. Unfortunately, these NDRs appear to differ wildly by server and do not include structured or standardized meta-data such as the message ID that was not delivered. I noticed some striking similarities in requirements between the system we are building and Anyvite. Anyvite automatically expanded a gmail address to a full name and parsed a non-delivery receipt (try sending an invite to a known bad email address). I am curious how they accomplished those two tasks. Since Anyvite was a Y Combinator company, I was hoping one of those guys could comment too :-) Thanks!

apgwoz 18y ago
I would use procmail (http://www.procmail.org/ http://www.procmail.org/). You might also check out the comments at this older post: http://news.ycombinator.com/item?id=186875 http://news.ycombinator.com/item?id=186875
- snprbob86 18y ago
  I looked briefly at procmail, but it seems to only have a primitive mechanism for identify bounce messages. There does not appear to be a way to extract meta data from them. """ If the regular expression contains ‘^FROM_DAEMON’ it will be substituted by ‘(^(Mailing-List : |Precedence :.(junk |bulk |list) |To : Multiple recipients of |(((Resent-)?(From |Sender) |X-Envelope-From) : |>?From )([^>][^(.%@a-z0-9])?(Post(ma?(st(e?r)? |n) |office) |(send)?Mail(er)? |daemon |m(mdf |ajordomo) |n?uucp |LIST(SERV |proc) |NETSERV |o(wner |ps) |r(e(quest |sponse) |oot) |b(ounce |bs\.smtp) |echo |mirror |s(erv(ices? |er) |mtp(error)? |ystem) |A(dmin(istrator)? |MMGR |utoanswer))(([^).! :a-z0-9][-_a-z0-9])?[%@>\t ][^<)]($.$.)?)?$([^>] |$)))’, which should catch mails coming from most daemons """ I still need a way to get meaningful information from the bounce messages. For example, my system sometimes emails a file on behalf of a user. If I receive a "mailbox does not exist" bounce, all I can do is notify the user that they may have mistyped an email address. However, if I receive a "mailbox full" bounce, I want to try again without the attachment and only with a web download link.
jasonkester 18y ago
You're looking for this: http://www.smtp2web.com/ http://www.smtp2web.com/ They listen for mail and hit your site with HTTP POSTs whenever they show up.
- snprbob86 18y ago
  smtp2web solves the problem of getting the messages. As far as I can tell, there are 3 ways to get access to the messages: 1) Control your mail server, use something like procmon to forward mail to a script 2) Rely on smtp2web or some other mail server to forward mail to a script 3) Poll for messages via SMTP or IMAP I chose #3 with IMAP over Gmail (configured for my domain). The real problem I am having is identifying and categorizing non-delivery responses / bounce mails.
  - aaronsw 18y ago
    I assume you mean procmail, not procmon. Decent MTAs (like qmail) make it pretty easy to have mail delivered to a script.
    - snprbob86 18y ago
      You assume correctly. My day job involves a lot of Windows programming and a healthy dose of procmon use :-)
aaronsw 18y ago
Do what ezmlm does with bounce messages: use a different envelope sender for each outgoing mail and encode the message ID in that address. If you get a mail to the envelope sender you can typically assume it's a non-delivery report of some kind. For finer-grained information, you can search the resulting message for HCMSSCs: http://cr.yp.to/proto/hcmssc.txt http://cr.yp.to/proto/hcmssc.txt Further reading: http://cr.yp.to/proto/verp.txt http://cr.yp.to/proto/verp.txt http://support.kavi.com/khelp/kmlm/user_help/html/bounces.html#automated_bounce_handling http://support.kavi.com/khelp/kmlm/user_help/html/bounces.ht...
- snprbob86 18y ago
  This is certainly interesting. In the grand problem of making sense of bounce messages, there seems to be three sub-problems: 1) Determining if the message is in fact a bounce. VERP doesn't seem to solve this. 2) Correctly linking a bounce message to the original message. VERP seems to solve that via encoding the message ID into the return path. I could implement VERP with "plus addressing". Thanks! 3) Identifying the error that the bounce message represents. Unfortunately, the presence of HCMSSCs in the hash format does not seem common. Even Gmail's bounce messages do not provide them.
brianm 18y ago
Not python, but simple and works great: http://subethasmtp.tigris.org/ http://subethasmtp.tigris.org/
drm237 18y ago
Hey, sorry we didn't respond earlier, just saw this post and the Anyvite reference. Our system is very basic right now and needs to be expanded on in the future. We start by setting a return path on all emails we send. This directs any bounce messages to a single mailbox that we can then process. Parsing the bounce messages for us is fairly easy. We took some ideas from David Rusenko's post at: http://david.weebly.com/1/post/2008/06/startup-email-monitor-your-bounce-messages.html http://david.weebly.com/1/post/2008/06/startup-email-monitor... You seem to be trying to tie a bounce back to a specific outgoing message, which may or may not be the right thing to do depending on your application. For us, we only care about the email address it was sent to so we know not to send them anything else (if it's a hard bounce). Best of luck.
- snprbob86 18y ago
  Thanks! I sent a longer note via email :-)

18 points by snprbob86 18y ago

apgwoz 18y ago

I would use procmail (http://www.procmail.org/ http://www.procmail.org/). You might also check out the comments at this older post: http://news.ycombinator.com/item?id=186875 http://news.ycombinator.com/item?id=186875

snprbob86 18y ago
I looked briefly at procmail, but it seems to only have a primitive mechanism for identify bounce messages. There does not appear to be a way to extract meta data from them. """ If the regular expression contains ‘^FROM_DAEMON’ it will be substituted by ‘(^(Mailing-List : |Precedence :.(junk |bulk |list) |To : Multiple recipients of |(((Resent-)?(From |Sender) |X-Envelope-From) : |>?From )([^>][^(.%@a-z0-9])?(Post(ma?(st(e?r)? |n) |office) |(send)?Mail(er)? |daemon |m(mdf |ajordomo) |n?uucp |LIST(SERV |proc) |NETSERV |o(wner |ps) |r(e(quest |sponse) |oot) |b(ounce |bs\.smtp) |echo |mirror |s(erv(ices? |er) |mtp(error)? |ystem) |A(dmin(istrator)? |MMGR |utoanswer))(([^).! :a-z0-9][-_a-z0-9])?[%@>\t ][^<)]($.$.)?)?$([^>] |$)))’, which should catch mails coming from most daemons """ I still need a way to get meaningful information from the bounce messages. For example, my system sometimes emails a file on behalf of a user. If I receive a "mailbox does not exist" bounce, all I can do is notify the user that they may have mistyped an email address. However, if I receive a "mailbox full" bounce, I want to try again without the attachment and only with a web download link.

jasonkester 18y ago

You're looking for this: http://www.smtp2web.com/ http://www.smtp2web.com/ They listen for mail and hit your site with HTTP POSTs whenever they show up.

snprbob86 18y ago
smtp2web solves the problem of getting the messages. As far as I can tell, there are 3 ways to get access to the messages: 1) Control your mail server, use something like procmon to forward mail to a script 2) Rely on smtp2web or some other mail server to forward mail to a script 3) Poll for messages via SMTP or IMAP I chose #3 with IMAP over Gmail (configured for my domain). The real problem I am having is identifying and categorizing non-delivery responses / bounce mails.
- aaronsw 18y ago
  I assume you mean procmail, not procmon. Decent MTAs (like qmail) make it pretty easy to have mail delivered to a script.
  - snprbob86 18y ago
    You assume correctly. My day job involves a lot of Windows programming and a healthy dose of procmon use :-)

aaronsw 18y ago

Do what ezmlm does with bounce messages: use a different envelope sender for each outgoing mail and encode the message ID in that address. If you get a mail to the envelope sender you can typically assume it's a non-delivery report of some kind. For finer-grained information, you can search the resulting message for HCMSSCs: http://cr.yp.to/proto/hcmssc.txt http://cr.yp.to/proto/hcmssc.txt Further reading: http://cr.yp.to/proto/verp.txt http://cr.yp.to/proto/verp.txt http://support.kavi.com/khelp/kmlm/user_help/html/bounces.html#automated_bounce_handling http://support.kavi.com/khelp/kmlm/user_help/html/bounces.ht...

snprbob86 18y ago
This is certainly interesting. In the grand problem of making sense of bounce messages, there seems to be three sub-problems: 1) Determining if the message is in fact a bounce. VERP doesn't seem to solve this. 2) Correctly linking a bounce message to the original message. VERP seems to solve that via encoding the message ID into the return path. I could implement VERP with "plus addressing". Thanks! 3) Identifying the error that the bounce message represents. Unfortunately, the presence of HCMSSCs in the hash format does not seem common. Even Gmail's bounce messages do not provide them.

brianm 18y ago

Not python, but simple and works great: http://subethasmtp.tigris.org/ http://subethasmtp.tigris.org/

drm237 18y ago

Hey, sorry we didn't respond earlier, just saw this post and the Anyvite reference. Our system is very basic right now and needs to be expanded on in the future. We start by setting a return path on all emails we send. This directs any bounce messages to a single mailbox that we can then process. Parsing the bounce messages for us is fairly easy. We took some ideas from David Rusenko's post at: http://david.weebly.com/1/post/2008/06/startup-email-monitor-your-bounce-messages.html http://david.weebly.com/1/post/2008/06/startup-email-monitor... You seem to be trying to tie a bounce back to a specific outgoing message, which may or may not be the right thing to do depending on your application. For us, we only care about the email address it was sent to so we know not to send them anything else (if it's a hard bounce). Best of luck.

snprbob86 18y ago
Thanks! I sent a longer note via email :-)