3 ms·

I’m really impressed by the email parsing success. How did you tackle the hellscape that is parsing replies out of emails?

Alex3917 8y ago
Thanks! I didn't really understand all the email specs when I started working on the parser, so it was mostly written using guess and check. For plaintext emails without forwards, we use a slightly modified version of Email Reply Parser to strip replies. We made a couple changes so that messages with inline replies aren't stripped. The next step is running everything through through BeautifulSoup. We use this to strip out most of the garbage from the HTML and rewrite it to make it beautiful. For example, paragraph text within divs gets unwrapped and placed within paragraphs. And text within divs that's styled to look like blockquotes gets placed within actual blockquote elements. This is not only good for SEO, but makes it easy for us to normalize the typography and style across threads to create a consistent reading experience. There's also some logic here for formatting emails from things like Zendesk and Mixmax correctly. And this is also where the highlighting and redaction get applied. Right now we just unwrap text within tables and place it in paragraph elements, I need to change this algorithm a little though to allow actual tabular data. The next step is running the text through Bleach, first to linkify any plaintext URLs (and sanitize URLs), and next to remove all the remaining markup and attributes we don't allow. Then you enter the death field of regex. The algorithm we use to unwrap text within divs and tables creates some extra whitespace before and after the newly created elements, so when you have nested divs and tables some of this needs to be stripped. We also strip out semantically invalid markup and empty elements. E.g. there is a lot of stuff that looks vaguely like this: body = body.replace('<li></ol>', '</ol>') The last step is that we run everything through BeautifulSoup again to make sure the whole thing is idempotent. I hear this isn't the Computer Science approved method of guaranteeing idempotence, but in this case it works. (Basically there are just a couple cases where the regex can lead to having unbalanced tags, so we need to make sure it's valid markup again.) Overall the whole process is pretty easy to read and understand, and it generally produces better results than Gmail so I'm pretty happy with it. If we were processing millions of emails per hour I'd probably put more work into making it more efficient, but for now it's fast enough to feel instantaneous even for threads with dozens of messages.
- natdempk 8y ago
  Cool, much thanks for the detailed reply!

Alex3917 8y ago

Thanks! I didn't really understand all the email specs when I started working on the parser, so it was mostly written using guess and check. For plaintext emails without forwards, we use a slightly modified version of Email Reply Parser to strip replies. We made a couple changes so that messages with inline replies aren't stripped. The next step is running everything through through BeautifulSoup. We use this to strip out most of the garbage from the HTML and rewrite it to make it beautiful. For example, paragraph text within divs gets unwrapped and placed within paragraphs. And text within divs that's styled to look like blockquotes gets placed within actual blockquote elements. This is not only good for SEO, but makes it easy for us to normalize the typography and style across threads to create a consistent reading experience. There's also some logic here for formatting emails from things like Zendesk and Mixmax correctly. And this is also where the highlighting and redaction get applied. Right now we just unwrap text within tables and place it in paragraph elements, I need to change this algorithm a little though to allow actual tabular data. The next step is running the text through Bleach, first to linkify any plaintext URLs (and sanitize URLs), and next to remove all the remaining markup and attributes we don't allow. Then you enter the death field of regex. The algorithm we use to unwrap text within divs and tables creates some extra whitespace before and after the newly created elements, so when you have nested divs and tables some of this needs to be stripped. We also strip out semantically invalid markup and empty elements. E.g. there is a lot of stuff that looks vaguely like this: body = body.replace('<li></ol>', '</ol>') The last step is that we run everything through BeautifulSoup again to make sure the whole thing is idempotent. I hear this isn't the Computer Science approved method of guaranteeing idempotence, but in this case it works. (Basically there are just a couple cases where the regex can lead to having unbalanced tags, so we need to make sure it's valid markup again.) Overall the whole process is pretty easy to read and understand, and it generally produces better results than Gmail so I'm pretty happy with it. If we were processing millions of emails per hour I'd probably put more work into making it more efficient, but for now it's fast enough to feel instantaneous even for threads with dozens of messages.

natdempk 8y ago
Cool, much thanks for the detailed reply!