6 ms·

Show HN: Switchboard – open-source email processing in Erlang

92 points by mickhagen 12y ago

christophe971 12y ago
Looks nice so far, worked with IMAP for a while, not fun... So is it on top of IMAP or ... ?
- jtmoulia 12y ago
  Yup, IMAP is powerful, but it's gathered some baggage over the years. Switchboard uses a simplified interface based on JMAP[1] for client's and workers, which it does proxy into IMAP commands. One of the conveniences of Switchboard is it manages the IMAP connections: restarting them as they fail and allowing concurrent access. [1] http://jmap.io/ http://jmap.io/
- oelmekki 12y ago
  It seems to be, as far as I understand. If that's actually the case, how do you handle the case where someone read a new mail (and thus, mark it as read) before app processed it ?
  - jtmoulia 12y ago
    The application uses IMAP UID/UIDVALIDITY values, typically along with IDLE, to identify new emails in a mailbox. These don't change when an email is read, so marking as read doesn't introduce any race conditions in this case.
sherm8n 12y ago
I worked on an email application for a year that was on top of imap. It was slow as hell. How does switchboard get around that? On another note, I think context.io provides a decent paid solution. How is this project different?
- jtmoulia 12y ago
  For Switchboard's intended task of as-they-arrive email processing, speed is not a huge issue. That being said, there are plans for using more than one IMAP connection to parallelize some requests. Switchboard is fairly different from context.io. Switchboard is open-source -- being able to keep in control of services accessing emails on your users' behalf is great. Also, Switchboard is intended for processing emails as they arrive using worker processes, as opposed to context.io which appears to be primarily a proxy.
buf 12y ago
I did a project like this a few years ago and here are my takeaways: 1. Building on top of IMAP is very difficult. It's not a technology that is really designed for the mobile world of today (e.g. streaming, sockets for chat) 2. Threading is a bitch. The old folder tree layout is formidable. ---------------- That said, it's so cool to see something like this in the wild. As an engineer, I would've used this as a boilerplate for my project years ago, and it would've saved me literally six months of efforts. Kudos to this. It's gods work. Free startup idea: Use this to help larger businesses collaborate more efficiently (much like if Yammer and GTasks had a baby).
- austenallred 12y ago
  It seems to me that this is the startup idea, built on top of this tech.
  - buf 12y ago
    Seemed to me that it wasn't particularly meant for internal business comms. Just random comms.
    - askura 12y ago
      Would be a good idea to market it for those two verticals.
- jtmoulia 12y ago
  1. IMAP is powerful and ubiquitous, but it does put a lot of work on the client: implementing a parser, caching values, correctly handling unsolicited messages. Switchboard tries to carry some of this boilerplate weight. 2. Yes! We still have to work this into Switchboard, but conversations in the JMAP (http://jmap.io http://jmap.io) spec give an idea of how it will be implemented.
- angersock 12y ago
  Interesting article about message threading: http://www.jwz.org/doc/threading.html http://www.jwz.org/doc/threading.html
jtmoulia 12y ago
I worked with Spatch to make Switchboard, and will be around to answer any questions. Our motivation in building and continuing to build Switchboard is to make it easier for developers to process emails. Looking forward to more ideas!
- fit2rule 12y ago
  Thanks for the nice product, and well done on a polished release - I've been playing around with it for a few minutes and I'm finding that its not only a great tool, but I'm learning erlang to boot. :) Since I sent my first email in the 80's, I've wanted to build business-logic systems with mail-based interfaces - things like, have an address such as "calendar@mydomain.com" which can be cc'ed on emails which might contain future scheduling events, and have the calendar@ client automatically work out the details of reminding everyone of the date-target that was set .. Seems to me that Switchboard would be the ideal platform upon which to develop such services .. among other things too. See any gotcha's? Its always amazed me that automated email is not used more and more for basic business processes - sure, we have mailing lists and autoresponders and all sorts of things like that, but the fact that most language/tech stacks don't really treat email as the human-maintained message queue, with first class priorities, seems a little lacking .. I guess Switchboard is an attempt to make up for that. Anyway, thanks again - I'll certainly be devoting some research time to this tool, and I hope I can build something useful on top of it ..
  - makeshifthoop 12y ago
    It should work very well for specifying the logic around the calendar example: Switchboard would accept the credentials of the accounts to monitor, e.g. calendar@mydomain.com, and then notify a "schedule notification" worker of new messages. The burden of scheduling reminders for participants would be on the worker, but there are a lot of tools for that. I like the idea! One note: Switchboard is especially useful relative to a bare IMAP connection when you're monitoring many accounts. This is because it provides a lot of the boilerplate around restarting failed connections and issuing IMAP commands while an IMAP connection has an active IDLE continuation. I agree that there's tons of room for more automation around emails. I think the lack of automation is due to not having high-level tools, which is where Switchboard comes in, and that emails are typically natural language. I'm personally excited to see Switchboard get used for email classification -- imagine if your business-logic could include the email "class", e.g. "sent by a human", or "marketing", ... etc. A bit like Gmail's categories, but open and accessible to developers.
nwalter08 12y ago
Can't wait to see what's built on this
ing33k 12y ago
will evaluate this thoroughly , needed something like this .
- mickhagen 12y ago
  Great, I hope it's a good fit! If not, please let us know about any sticking points either through the mailing list, switchboarddev _at_ librelist.com, or the GitHub issues page.
dmix 12y ago
Could this be used for automatic GPG encryption before storage on disk?
- mickhagen 12y ago
  I'm guessing that you want the emails to be encrypted before being saved to any disk? In that case, Switchboard isn't sufficient: it runs in front of email providers, and can't prevent them from saving an email to disk. Switchboard could be used to fetch the email, encrypt it, and then replace the unencrypted email with the encrypted email on the server. However, replacing emails is invasive if the user doesn’t expect it, and other than the email provider's word you have no guarantee that the unencrypted email was actually deleted.
wmf 12y ago
Why is this based on IMAP? When I see "server-side email processing" I think SMTP.
- bismark 12y ago
  The wording is a bit ambiguous — “server-side email processing” refers to processing emails which are stored with an email provider, e.g. Gmail. IMAP is used to interact with the email providers because it is a mail retrieval protocol, and is widely used. If we were doing “server-side email processing” of emails as they were sent, SMTP would have fit in.
soupdiver 12y ago
I think I haven't clearly understood where to use Switchboard. In my normal mail stack (postfix, dovecot) where would Switchboard be placed?
- bismark 12y ago
  It would sit along side your Dovecot server. The Switchboard core subscribes to events in a folder (using IMAP IDLE) and issues change events to connected workers.
  - X4 12y ago
    so is this like zeromq for mails?
    - makeshifthoop 12y ago
      I can't give an unqualified yes, but Switchboard is similar to zeromq in that it fans out events [new emails] to subscribed processes [workers]. We actually thought about using zeromq, but went with a simpler solution for now, i.e. using websockets.
sandGorgon 12y ago
Does anyone know how this compares to Lamson (which has been around for quite some time) ? [1] http://lamsonproject.org/ http://lamsonproject.org/
- askura 12y ago
  I've not checked this out properly just yet but I'm having a look. I can give you a heads up later.
- mickhagen 12y ago
  Lamson is an SMTP server which acts as the gateway for emails from the outside world to be put in your mail store. Switchboard doesn't have any direct connection to SMTP. It monitors changes in your mail store using IMAP IDLE and issues events to workers which can then do basically whatever your heart desires. (btw, we host our switchboard mailing lists on librelist.com which is a showcase for Lamson. cool project)
  - sandGorgon 12y ago
    Correct me if I'm wrong, but Lamson enables those hooks/workers as well - but integrates with an FSM and a queuing mechanism (Celery) to get this done. Additionally, there is Apache James (Java) which does something similar.
    - jtmoulia 12y ago
      You're correct that Lamson provides tools for processing emails, but it doesn't include a mechanism for feeding it emails from IMAP servers. Switchboard actually complements Lamson -- Switchboard could manage IMAP connections, collect incoming emails, and feed those emails to Lamson. You'd be able to use Lamson in front of Gmail/Yahoo/etc I'm not super familiar with Apache James, but it appears to be a mail server with pluggable components. Switchboard fits in as a layer between any mail server that supports IMAP, and its version of pluggable components [processes connection over the interface]. Thanks, this really helps us see where the documentation needs to be improved :)
fit2rule 12y ago
This is a really neat tool, and I look forward to using it. I can think of a number of uses, already - analyses of mailbox activity, statistics and metrics collection, business-processes being interfaced with email, and so on. Heck, I guess I could use this to replace my recent interest in PlanLeaf (email-based todo manager) with my own custom version. On a side note - its funny how things flow on HN sometimes - yesterday I played with PlanLeaf, and had the thought 'nice, but it'd be even nicer to just have the code to process emails properly', and now .. along comes a stack that gives me a way to do just that ..
joshdance 12y ago
Looks great. Any future plans for implementations or API in other languages?
- makeshifthoop 12y ago
  Currently, there are worker/client implementations in Python, Ruby, and JavaScript. New languages can be added in response to demand -- is there a language you'd be interested in using on the client-side in particular? Also, because it uses JSON over websockets, it's fairly easy to implement a new worker/client that follows the Switchboard interface -- here's the Python implementation in 132 LOC: https://github.com/jtmoulia/switchboard-python/blob/master/switchboard/__init__.py https://github.com/jtmoulia/switchboard-python/blob/master/s...
erlangod 12y ago
I fuckin love it. I have a huge hard on for Erlang lately.

92 points by mickhagen 12y ago

christophe971 12y ago

Looks nice so far, worked with IMAP for a while, not fun... So is it on top of IMAP or ... ?

jtmoulia 12y ago
Yup, IMAP is powerful, but it's gathered some baggage over the years. Switchboard uses a simplified interface based on JMAP[1] for client's and workers, which it does proxy into IMAP commands. One of the conveniences of Switchboard is it manages the IMAP connections: restarting them as they fail and allowing concurrent access. [1] http://jmap.io/ http://jmap.io/
oelmekki 12y ago
It seems to be, as far as I understand. If that's actually the case, how do you handle the case where someone read a new mail (and thus, mark it as read) before app processed it ?
- jtmoulia 12y ago
  The application uses IMAP UID/UIDVALIDITY values, typically along with IDLE, to identify new emails in a mailbox. These don't change when an email is read, so marking as read doesn't introduce any race conditions in this case.

sherm8n 12y ago

I worked on an email application for a year that was on top of imap. It was slow as hell. How does switchboard get around that? On another note, I think context.io provides a decent paid solution. How is this project different?

jtmoulia 12y ago
For Switchboard's intended task of as-they-arrive email processing, speed is not a huge issue. That being said, there are plans for using more than one IMAP connection to parallelize some requests. Switchboard is fairly different from context.io. Switchboard is open-source -- being able to keep in control of services accessing emails on your users' behalf is great. Also, Switchboard is intended for processing emails as they arrive using worker processes, as opposed to context.io which appears to be primarily a proxy.

buf 12y ago

I did a project like this a few years ago and here are my takeaways: 1. Building on top of IMAP is very difficult. It's not a technology that is really designed for the mobile world of today (e.g. streaming, sockets for chat) 2. Threading is a bitch. The old folder tree layout is formidable. ---------------- That said, it's so cool to see something like this in the wild. As an engineer, I would've used this as a boilerplate for my project years ago, and it would've saved me literally six months of efforts. Kudos to this. It's gods work. Free startup idea: Use this to help larger businesses collaborate more efficiently (much like if Yammer and GTasks had a baby).

austenallred 12y ago
It seems to me that this is the startup idea, built on top of this tech.
- buf 12y ago
  Seemed to me that it wasn't particularly meant for internal business comms. Just random comms.
  - askura 12y ago
    Would be a good idea to market it for those two verticals.
jtmoulia 12y ago
1. IMAP is powerful and ubiquitous, but it does put a lot of work on the client: implementing a parser, caching values, correctly handling unsolicited messages. Switchboard tries to carry some of this boilerplate weight. 2. Yes! We still have to work this into Switchboard, but conversations in the JMAP (http://jmap.io http://jmap.io) spec give an idea of how it will be implemented.
angersock 12y ago
Interesting article about message threading: http://www.jwz.org/doc/threading.html http://www.jwz.org/doc/threading.html

jtmoulia 12y ago

I worked with Spatch to make Switchboard, and will be around to answer any questions. Our motivation in building and continuing to build Switchboard is to make it easier for developers to process emails. Looking forward to more ideas!

fit2rule 12y ago
Thanks for the nice product, and well done on a polished release - I've been playing around with it for a few minutes and I'm finding that its not only a great tool, but I'm learning erlang to boot. :) Since I sent my first email in the 80's, I've wanted to build business-logic systems with mail-based interfaces - things like, have an address such as "calendar@mydomain.com" which can be cc'ed on emails which might contain future scheduling events, and have the calendar@ client automatically work out the details of reminding everyone of the date-target that was set .. Seems to me that Switchboard would be the ideal platform upon which to develop such services .. among other things too. See any gotcha's? Its always amazed me that automated email is not used more and more for basic business processes - sure, we have mailing lists and autoresponders and all sorts of things like that, but the fact that most language/tech stacks don't really treat email as the human-maintained message queue, with first class priorities, seems a little lacking .. I guess Switchboard is an attempt to make up for that. Anyway, thanks again - I'll certainly be devoting some research time to this tool, and I hope I can build something useful on top of it ..
- makeshifthoop 12y ago
  It should work very well for specifying the logic around the calendar example: Switchboard would accept the credentials of the accounts to monitor, e.g. calendar@mydomain.com, and then notify a "schedule notification" worker of new messages. The burden of scheduling reminders for participants would be on the worker, but there are a lot of tools for that. I like the idea! One note: Switchboard is especially useful relative to a bare IMAP connection when you're monitoring many accounts. This is because it provides a lot of the boilerplate around restarting failed connections and issuing IMAP commands while an IMAP connection has an active IDLE continuation. I agree that there's tons of room for more automation around emails. I think the lack of automation is due to not having high-level tools, which is where Switchboard comes in, and that emails are typically natural language. I'm personally excited to see Switchboard get used for email classification -- imagine if your business-logic could include the email "class", e.g. "sent by a human", or "marketing", ... etc. A bit like Gmail's categories, but open and accessible to developers.

nwalter08 12y ago

Can't wait to see what's built on this

ing33k 12y ago

will evaluate this thoroughly , needed something like this .

mickhagen 12y ago
Great, I hope it's a good fit! If not, please let us know about any sticking points either through the mailing list, switchboarddev _at_ librelist.com, or the GitHub issues page.

dmix 12y ago

Could this be used for automatic GPG encryption before storage on disk?

mickhagen 12y ago
I'm guessing that you want the emails to be encrypted before being saved to any disk? In that case, Switchboard isn't sufficient: it runs in front of email providers, and can't prevent them from saving an email to disk. Switchboard could be used to fetch the email, encrypt it, and then replace the unencrypted email with the encrypted email on the server. However, replacing emails is invasive if the user doesn’t expect it, and other than the email provider's word you have no guarantee that the unencrypted email was actually deleted.

wmf 12y ago

Why is this based on IMAP? When I see "server-side email processing" I think SMTP.

bismark 12y ago
The wording is a bit ambiguous — “server-side email processing” refers to processing emails which are stored with an email provider, e.g. Gmail. IMAP is used to interact with the email providers because it is a mail retrieval protocol, and is widely used. If we were doing “server-side email processing” of emails as they were sent, SMTP would have fit in.

soupdiver 12y ago

I think I haven't clearly understood where to use Switchboard. In my normal mail stack (postfix, dovecot) where would Switchboard be placed?

bismark 12y ago
It would sit along side your Dovecot server. The Switchboard core subscribes to events in a folder (using IMAP IDLE) and issues change events to connected workers.
- X4 12y ago
  so is this like zeromq for mails?
  - makeshifthoop 12y ago
    I can't give an unqualified yes, but Switchboard is similar to zeromq in that it fans out events [new emails] to subscribed processes [workers]. We actually thought about using zeromq, but went with a simpler solution for now, i.e. using websockets.

sandGorgon 12y ago

Does anyone know how this compares to Lamson (which has been around for quite some time) ? [1] http://lamsonproject.org/ http://lamsonproject.org/

askura 12y ago
I've not checked this out properly just yet but I'm having a look. I can give you a heads up later.
mickhagen 12y ago
Lamson is an SMTP server which acts as the gateway for emails from the outside world to be put in your mail store. Switchboard doesn't have any direct connection to SMTP. It monitors changes in your mail store using IMAP IDLE and issues events to workers which can then do basically whatever your heart desires. (btw, we host our switchboard mailing lists on librelist.com which is a showcase for Lamson. cool project)
- sandGorgon 12y ago
  Correct me if I'm wrong, but Lamson enables those hooks/workers as well - but integrates with an FSM and a queuing mechanism (Celery) to get this done. Additionally, there is Apache James (Java) which does something similar.
  - jtmoulia 12y ago
    You're correct that Lamson provides tools for processing emails, but it doesn't include a mechanism for feeding it emails from IMAP servers. Switchboard actually complements Lamson -- Switchboard could manage IMAP connections, collect incoming emails, and feed those emails to Lamson. You'd be able to use Lamson in front of Gmail/Yahoo/etc I'm not super familiar with Apache James, but it appears to be a mail server with pluggable components. Switchboard fits in as a layer between any mail server that supports IMAP, and its version of pluggable components [processes connection over the interface]. Thanks, this really helps us see where the documentation needs to be improved :)

fit2rule 12y ago

This is a really neat tool, and I look forward to using it. I can think of a number of uses, already - analyses of mailbox activity, statistics and metrics collection, business-processes being interfaced with email, and so on. Heck, I guess I could use this to replace my recent interest in PlanLeaf (email-based todo manager) with my own custom version. On a side note - its funny how things flow on HN sometimes - yesterday I played with PlanLeaf, and had the thought 'nice, but it'd be even nicer to just have the code to process emails properly', and now .. along comes a stack that gives me a way to do just that ..

joshdance 12y ago

Looks great. Any future plans for implementations or API in other languages?

makeshifthoop 12y ago
Currently, there are worker/client implementations in Python, Ruby, and JavaScript. New languages can be added in response to demand -- is there a language you'd be interested in using on the client-side in particular? Also, because it uses JSON over websockets, it's fairly easy to implement a new worker/client that follows the Switchboard interface -- here's the Python implementation in 132 LOC: https://github.com/jtmoulia/switchboard-python/blob/master/switchboard/__init__.py https://github.com/jtmoulia/switchboard-python/blob/master/s...

erlangod 12y ago

I fuckin love it. I have a huge hard on for Erlang lately.