5 ms·

He dwells too much upon the age and cruft of email, but this article reads more like a layman learning for the first time about Hofstadter's law, general progra

He dwells too much upon the age and cruft of email, but this article reads more like a layman learning for the first time about Hofstadter's law, general programming project mis-scoping, and dealing with technical debt he himself created... which is kinda cool to hear about from "the outside".

netghost 11y ago
What's great though is that at the end of it, he has a tool that works for him. That's a huge accomplishment. All too often I obsess over tiny details in my personal projects and never get to anything useful at all. Meanwhile this guy built something that's probably not elegant, but totally does what he needed.
dmbaggett 11y ago
To be fair to him, though, pretty much every programmer I know since I've been working on (http://inky.com http://inky.com) has asked me why it's hard. Email is partly hard because of the historical cruft, but there are genuinely difficult design/abstraction challenges with email as well, and even experienced developers tend to mis-predict how hard it is to make a "real" mail client. The one exception to this was Carl de Marcken, one of my co-founders at ITA Software, and chief scientist there. He predicted -- correctly -- that it would take us 4+ years to get to an MVP.
- plehoux 11y ago
  I agree, at Missive (https://missiveapp.com https://missiveapp.com) it took way longer than we originally thought to get to MVP, ~around 1.75 year (and we only support Gmail/Google app for the moment)
  - dmbaggett 11y ago
    Wait until you get to IMAP and Exchange -- you'll learn that you're only 5% done. :(
    - Estragon 11y ago
      What are some of the most surprising complexities?
      - dmbaggett 11y ago
        There are at least 75 nontrivial problems, each of which is worth of its own startup-amount-of-effort. A few random examples: - sanitizing HTML email: you must eliminate XSS while preserving the appearance; this requires real HTML and CSS parsing; we do this on the end user's phone at MB/sec rates, which is not easy - conversation threading: any email in a conversation that's been sent with Outlook will have stripped the standard threading headers and added the proprietary Thread-Index header, so in practice you must rely on heuristics - properly mapping the IMAP/Exchange/POP "database" onto UI views is immensely complex in a real client, and requires exactly the right set of abstractions (which are very subtle) - it is a vast understatement to stay that the IMAP protocol is a mess; it is also plagued by some really bad design choices (such as relying on message sequence numbers which vary by connection rather than exclusively using UIDs, which don't). - instant full text search over a million message inbox is hard, but this is a real-world use case - converting arbitrary MIME emails to HTML for rendering purposes is a huge rathole - making offline mode really work (supporting search and modification of cached mails) is complicated, and most clients punt on it (because they are implemented server-side and therefore require a network connection for anything to work) - implementing everything server side is (relatively) easier, but requires you to scale bandwidth per user and mailbox, which is expensive; on the other hand, doing everything client-side like Inky does is much harder because you've got tremendous variation among hardware platforms and operating systems, and it's a really heavyweight app In general, an email client subsumes many other highly nontrivial subproblems, somewhat like a browser does. Just doing the security-related stuff alone properly -- TLS, certs, revocation, encrypted email -- is really labor-intensive.
        hammerandtongs 11y ago
        Supporting POP3 seems like an anti-feature in 2015. It seems to me, that muddling its lack of features into the middle of an application that properly supports IMAP would make your internal api much more complex?
  - deleted 11y ago
    [deleted]
  - deleted 11y ago
    [deleted]

netghost 11y ago

What's great though is that at the end of it, he has a tool that works for him. That's a huge accomplishment. All too often I obsess over tiny details in my personal projects and never get to anything useful at all. Meanwhile this guy built something that's probably not elegant, but totally does what he needed.

dmbaggett 11y ago

To be fair to him, though, pretty much every programmer I know since I've been working on (http://inky.com http://inky.com) has asked me why it's hard. Email is partly hard because of the historical cruft, but there are genuinely difficult design/abstraction challenges with email as well, and even experienced developers tend to mis-predict how hard it is to make a "real" mail client. The one exception to this was Carl de Marcken, one of my co-founders at ITA Software, and chief scientist there. He predicted -- correctly -- that it would take us 4+ years to get to an MVP.

plehoux 11y ago
I agree, at Missive (https://missiveapp.com https://missiveapp.com) it took way longer than we originally thought to get to MVP, ~around 1.75 year (and we only support Gmail/Google app for the moment)
- dmbaggett 11y ago
  Wait until you get to IMAP and Exchange -- you'll learn that you're only 5% done. :(
  - Estragon 11y ago
    What are some of the most surprising complexities?
    - dmbaggett 11y ago
      There are at least 75 nontrivial problems, each of which is worth of its own startup-amount-of-effort. A few random examples: - sanitizing HTML email: you must eliminate XSS while preserving the appearance; this requires real HTML and CSS parsing; we do this on the end user's phone at MB/sec rates, which is not easy - conversation threading: any email in a conversation that's been sent with Outlook will have stripped the standard threading headers and added the proprietary Thread-Index header, so in practice you must rely on heuristics - properly mapping the IMAP/Exchange/POP "database" onto UI views is immensely complex in a real client, and requires exactly the right set of abstractions (which are very subtle) - it is a vast understatement to stay that the IMAP protocol is a mess; it is also plagued by some really bad design choices (such as relying on message sequence numbers which vary by connection rather than exclusively using UIDs, which don't). - instant full text search over a million message inbox is hard, but this is a real-world use case - converting arbitrary MIME emails to HTML for rendering purposes is a huge rathole - making offline mode really work (supporting search and modification of cached mails) is complicated, and most clients punt on it (because they are implemented server-side and therefore require a network connection for anything to work) - implementing everything server side is (relatively) easier, but requires you to scale bandwidth per user and mailbox, which is expensive; on the other hand, doing everything client-side like Inky does is much harder because you've got tremendous variation among hardware platforms and operating systems, and it's a really heavyweight app In general, an email client subsumes many other highly nontrivial subproblems, somewhat like a browser does. Just doing the security-related stuff alone properly -- TLS, certs, revocation, encrypted email -- is really labor-intensive.
      - hammerandtongs 11y ago
        Supporting POP3 seems like an anti-feature in 2015. It seems to me, that muddling its lack of features into the middle of an application that properly supports IMAP would make your internal api much more complex?
- deleted 11y ago
  [deleted]
- deleted 11y ago
  [deleted]