4 ms·

Wait until you get to IMAP and Exchange -- you'll learn that you're only 5% done. :(

by dmbaggett 11y ago

Estragon 11y ago
What are some of the most surprising complexities?
- dmbaggett 11y ago
  There are at least 75 nontrivial problems, each of which is worth of its own startup-amount-of-effort. A few random examples: - sanitizing HTML email: you must eliminate XSS while preserving the appearance; this requires real HTML and CSS parsing; we do this on the end user's phone at MB/sec rates, which is not easy - conversation threading: any email in a conversation that's been sent with Outlook will have stripped the standard threading headers and added the proprietary Thread-Index header, so in practice you must rely on heuristics - properly mapping the IMAP/Exchange/POP "database" onto UI views is immensely complex in a real client, and requires exactly the right set of abstractions (which are very subtle) - it is a vast understatement to stay that the IMAP protocol is a mess; it is also plagued by some really bad design choices (such as relying on message sequence numbers which vary by connection rather than exclusively using UIDs, which don't). - instant full text search over a million message inbox is hard, but this is a real-world use case - converting arbitrary MIME emails to HTML for rendering purposes is a huge rathole - making offline mode really work (supporting search and modification of cached mails) is complicated, and most clients punt on it (because they are implemented server-side and therefore require a network connection for anything to work) - implementing everything server side is (relatively) easier, but requires you to scale bandwidth per user and mailbox, which is expensive; on the other hand, doing everything client-side like Inky does is much harder because you've got tremendous variation among hardware platforms and operating systems, and it's a really heavyweight app In general, an email client subsumes many other highly nontrivial subproblems, somewhat like a browser does. Just doing the security-related stuff alone properly -- TLS, certs, revocation, encrypted email -- is really labor-intensive.
  - hammerandtongs 11y ago
    Supporting POP3 seems like an anti-feature in 2015. It seems to me, that muddling its lack of features into the middle of an application that properly supports IMAP would make your internal api much more complex?
    - dmbaggett 11y ago
      Actually POP is easy except for efficiently determining what's changed in the mailbox since the last time you checked. And some providers still only offer POP access -- for example, Verizon.
      - narrowrail 11y ago
        From the outside, it looks like you have great engineering talent, but on the product management side you seem to lack some vision other than to cover backwards compatibility. This is frustrating to witness, even at a distance.
        narrowrail 11y ago
        Perhaps I should clarify that I worked for a founder where we had this problem, and after a year almost of the engineers I started with had left. It was maddening, and I finally had to leave as well. The company was eventually sold for pennies on the dollar.
  - dugmartin 11y ago
    I think sanitizing email is a solved problem with this: https://github.com/cure53/DOMPurify https://github.com/cure53/DOMPurify I'm not the author but I've used it in the past. According to the readme it is what Fastmail uses.
    - dmbaggett 11y ago
      Probably not fast enough for us, but it is a very nice library. (We use libxml2.) The reference we test against, by the way, is Mike Cardwell's excellent Email Privacy Tester: https://emailprivacytester.com/ Try it on your own mail client!

Wait until you get to IMAP and Exchange -- you'll learn that you're only 5% done. :(

Estragon 11y ago

What are some of the most surprising complexities?

dmbaggett 11y ago
There are at least 75 nontrivial problems, each of which is worth of its own startup-amount-of-effort. A few random examples: - sanitizing HTML email: you must eliminate XSS while preserving the appearance; this requires real HTML and CSS parsing; we do this on the end user's phone at MB/sec rates, which is not easy - conversation threading: any email in a conversation that's been sent with Outlook will have stripped the standard threading headers and added the proprietary Thread-Index header, so in practice you must rely on heuristics - properly mapping the IMAP/Exchange/POP "database" onto UI views is immensely complex in a real client, and requires exactly the right set of abstractions (which are very subtle) - it is a vast understatement to stay that the IMAP protocol is a mess; it is also plagued by some really bad design choices (such as relying on message sequence numbers which vary by connection rather than exclusively using UIDs, which don't). - instant full text search over a million message inbox is hard, but this is a real-world use case - converting arbitrary MIME emails to HTML for rendering purposes is a huge rathole - making offline mode really work (supporting search and modification of cached mails) is complicated, and most clients punt on it (because they are implemented server-side and therefore require a network connection for anything to work) - implementing everything server side is (relatively) easier, but requires you to scale bandwidth per user and mailbox, which is expensive; on the other hand, doing everything client-side like Inky does is much harder because you've got tremendous variation among hardware platforms and operating systems, and it's a really heavyweight app In general, an email client subsumes many other highly nontrivial subproblems, somewhat like a browser does. Just doing the security-related stuff alone properly -- TLS, certs, revocation, encrypted email -- is really labor-intensive.
- hammerandtongs 11y ago
  Supporting POP3 seems like an anti-feature in 2015. It seems to me, that muddling its lack of features into the middle of an application that properly supports IMAP would make your internal api much more complex?
  - dmbaggett 11y ago
    Actually POP is easy except for efficiently determining what's changed in the mailbox since the last time you checked. And some providers still only offer POP access -- for example, Verizon.
    - narrowrail 11y ago
      From the outside, it looks like you have great engineering talent, but on the product management side you seem to lack some vision other than to cover backwards compatibility. This is frustrating to witness, even at a distance.
      - narrowrail 11y ago
        Perhaps I should clarify that I worked for a founder where we had this problem, and after a year almost of the engineers I started with had left. It was maddening, and I finally had to leave as well. The company was eventually sold for pennies on the dollar.
- dugmartin 11y ago
  I think sanitizing email is a solved problem with this: https://github.com/cure53/DOMPurify https://github.com/cure53/DOMPurify I'm not the author but I've used it in the past. According to the readme it is what Fastmail uses.
  - dmbaggett 11y ago
    Probably not fast enough for us, but it is a very nice library. (We use libxml2.) The reference we test against, by the way, is Mike Cardwell's excellent Email Privacy Tester: https://emailprivacytester.com/ Try it on your own mail client!