7 ms·

To be fair to him, though, pretty much every programmer I know since I've been working on (http://inky.com http://inky.com) has asked me why it's hard. Email is

To be fair to him, though, pretty much every programmer I know since I've been working on (http://inky.com http://inky.com) has asked me why it's hard. Email is partly hard because of the historical cruft, but there are genuinely difficult design/abstraction challenges with email as well, and even experienced developers tend to mis-predict how hard it is to make a "real" mail client. The one exception to this was Carl de Marcken, one of my co-founders at ITA Software, and chief scientist there. He predicted -- correctly -- that it would take us 4+ years to get to an MVP.

plehoux 11y ago
I agree, at Missive (https://missiveapp.com https://missiveapp.com) it took way longer than we originally thought to get to MVP, ~around 1.75 year (and we only support Gmail/Google app for the moment)
- dmbaggett 11y ago
  Wait until you get to IMAP and Exchange -- you'll learn that you're only 5% done. :(
  - Estragon 11y ago
    What are some of the most surprising complexities?
    - dmbaggett 11y ago
      There are at least 75 nontrivial problems, each of which is worth of its own startup-amount-of-effort. A few random examples: - sanitizing HTML email: you must eliminate XSS while preserving the appearance; this requires real HTML and CSS parsing; we do this on the end user's phone at MB/sec rates, which is not easy - conversation threading: any email in a conversation that's been sent with Outlook will have stripped the standard threading headers and added the proprietary Thread-Index header, so in practice you must rely on heuristics - properly mapping the IMAP/Exchange/POP "database" onto UI views is immensely complex in a real client, and requires exactly the right set of abstractions (which are very subtle) - it is a vast understatement to stay that the IMAP protocol is a mess; it is also plagued by some really bad design choices (such as relying on message sequence numbers which vary by connection rather than exclusively using UIDs, which don't). - instant full text search over a million message inbox is hard, but this is a real-world use case - converting arbitrary MIME emails to HTML for rendering purposes is a huge rathole - making offline mode really work (supporting search and modification of cached mails) is complicated, and most clients punt on it (because they are implemented server-side and therefore require a network connection for anything to work) - implementing everything server side is (relatively) easier, but requires you to scale bandwidth per user and mailbox, which is expensive; on the other hand, doing everything client-side like Inky does is much harder because you've got tremendous variation among hardware platforms and operating systems, and it's a really heavyweight app In general, an email client subsumes many other highly nontrivial subproblems, somewhat like a browser does. Just doing the security-related stuff alone properly -- TLS, certs, revocation, encrypted email -- is really labor-intensive.
      - hammerandtongs 11y ago
        Supporting POP3 seems like an anti-feature in 2015. It seems to me, that muddling its lack of features into the middle of an application that properly supports IMAP would make your internal api much more complex?
        dmbaggett 11y ago
        Actually POP is easy except for efficiently determining what's changed in the mailbox since the last time you checked. And some providers still only offer POP access -- for example, Verizon.
        narrowrail 11y ago
        From the outside, it looks like you have great engineering talent, but on the product management side you seem to lack some vision other than to cover backwards compatibility. This is frustrating to witness, even at a distance.
        narrowrail 11y ago
        Perhaps I should clarify that I worked for a founder where we had this problem, and after a year almost of the engineers I started with had left. It was maddening, and I finally had to leave as well. The company was eventually sold for pennies on the dollar.
      - dugmartin 11y ago
        I think sanitizing email is a solved problem with this: https://github.com/cure53/DOMPurify https://github.com/cure53/DOMPurify I'm not the author but I've used it in the past. According to the readme it is what Fastmail uses.
        dmbaggett 11y ago
        Probably not fast enough for us, but it is a very nice library. (We use libxml2.) The reference we test against, by the way, is Mike Cardwell's excellent Email Privacy Tester: https://emailprivacytester.com/ Try it on your own mail client!
- deleted 11y ago
  [deleted]
- deleted 11y ago
  [deleted]

plehoux 11y ago

I agree, at Missive (https://missiveapp.com https://missiveapp.com) it took way longer than we originally thought to get to MVP, ~around 1.75 year (and we only support Gmail/Google app for the moment)

dmbaggett 11y ago
Wait until you get to IMAP and Exchange -- you'll learn that you're only 5% done. :(
- Estragon 11y ago
  What are some of the most surprising complexities?
  - dmbaggett 11y ago
    There are at least 75 nontrivial problems, each of which is worth of its own startup-amount-of-effort. A few random examples: - sanitizing HTML email: you must eliminate XSS while preserving the appearance; this requires real HTML and CSS parsing; we do this on the end user's phone at MB/sec rates, which is not easy - conversation threading: any email in a conversation that's been sent with Outlook will have stripped the standard threading headers and added the proprietary Thread-Index header, so in practice you must rely on heuristics - properly mapping the IMAP/Exchange/POP "database" onto UI views is immensely complex in a real client, and requires exactly the right set of abstractions (which are very subtle) - it is a vast understatement to stay that the IMAP protocol is a mess; it is also plagued by some really bad design choices (such as relying on message sequence numbers which vary by connection rather than exclusively using UIDs, which don't). - instant full text search over a million message inbox is hard, but this is a real-world use case - converting arbitrary MIME emails to HTML for rendering purposes is a huge rathole - making offline mode really work (supporting search and modification of cached mails) is complicated, and most clients punt on it (because they are implemented server-side and therefore require a network connection for anything to work) - implementing everything server side is (relatively) easier, but requires you to scale bandwidth per user and mailbox, which is expensive; on the other hand, doing everything client-side like Inky does is much harder because you've got tremendous variation among hardware platforms and operating systems, and it's a really heavyweight app In general, an email client subsumes many other highly nontrivial subproblems, somewhat like a browser does. Just doing the security-related stuff alone properly -- TLS, certs, revocation, encrypted email -- is really labor-intensive.
    - hammerandtongs 11y ago
      Supporting POP3 seems like an anti-feature in 2015. It seems to me, that muddling its lack of features into the middle of an application that properly supports IMAP would make your internal api much more complex?
      - dmbaggett 11y ago
        Actually POP is easy except for efficiently determining what's changed in the mailbox since the last time you checked. And some providers still only offer POP access -- for example, Verizon.
        narrowrail 11y ago
        From the outside, it looks like you have great engineering talent, but on the product management side you seem to lack some vision other than to cover backwards compatibility. This is frustrating to witness, even at a distance.
        narrowrail 11y ago
        Perhaps I should clarify that I worked for a founder where we had this problem, and after a year almost of the engineers I started with had left. It was maddening, and I finally had to leave as well. The company was eventually sold for pennies on the dollar.
    - dugmartin 11y ago
      I think sanitizing email is a solved problem with this: https://github.com/cure53/DOMPurify https://github.com/cure53/DOMPurify I'm not the author but I've used it in the past. According to the readme it is what Fastmail uses.
      - dmbaggett 11y ago
        Probably not fast enough for us, but it is a very nice library. (We use libxml2.) The reference we test against, by the way, is Mike Cardwell's excellent Email Privacy Tester: https://emailprivacytester.com/ Try it on your own mail client!
deleted 11y ago
[deleted]
deleted 11y ago
[deleted]