5 ms·

What are some of the most surprising complexities?

dmbaggett 11y ago
There are at least 75 nontrivial problems, each of which is worth of its own startup-amount-of-effort. A few random examples: - sanitizing HTML email: you must eliminate XSS while preserving the appearance; this requires real HTML and CSS parsing; we do this on the end user's phone at MB/sec rates, which is not easy - conversation threading: any email in a conversation that's been sent with Outlook will have stripped the standard threading headers and added the proprietary Thread-Index header, so in practice you must rely on heuristics - properly mapping the IMAP/Exchange/POP "database" onto UI views is immensely complex in a real client, and requires exactly the right set of abstractions (which are very subtle) - it is a vast understatement to stay that the IMAP protocol is a mess; it is also plagued by some really bad design choices (such as relying on message sequence numbers which vary by connection rather than exclusively using UIDs, which don't). - instant full text search over a million message inbox is hard, but this is a real-world use case - converting arbitrary MIME emails to HTML for rendering purposes is a huge rathole - making offline mode really work (supporting search and modification of cached mails) is complicated, and most clients punt on it (because they are implemented server-side and therefore require a network connection for anything to work) - implementing everything server side is (relatively) easier, but requires you to scale bandwidth per user and mailbox, which is expensive; on the other hand, doing everything client-side like Inky does is much harder because you've got tremendous variation among hardware platforms and operating systems, and it's a really heavyweight app In general, an email client subsumes many other highly nontrivial subproblems, somewhat like a browser does. Just doing the security-related stuff alone properly -- TLS, certs, revocation, encrypted email -- is really labor-intensive.
- hammerandtongs 11y ago
  Supporting POP3 seems like an anti-feature in 2015. It seems to me, that muddling its lack of features into the middle of an application that properly supports IMAP would make your internal api much more complex?
  - dmbaggett 11y ago
    Actually POP is easy except for efficiently determining what's changed in the mailbox since the last time you checked. And some providers still only offer POP access -- for example, Verizon.
    - narrowrail 11y ago
      From the outside, it looks like you have great engineering talent, but on the product management side you seem to lack some vision other than to cover backwards compatibility. This is frustrating to witness, even at a distance.
      - narrowrail 11y ago
        Perhaps I should clarify that I worked for a founder where we had this problem, and after a year almost of the engineers I started with had left. It was maddening, and I finally had to leave as well. The company was eventually sold for pennies on the dollar.
- dugmartin 11y ago
  I think sanitizing email is a solved problem with this: https://github.com/cure53/DOMPurify https://github.com/cure53/DOMPurify I'm not the author but I've used it in the past. According to the readme it is what Fastmail uses.
  - dmbaggett 11y ago
    Probably not fast enough for us, but it is a very nice library. (We use libxml2.) The reference we test against, by the way, is Mike Cardwell's excellent Email Privacy Tester: https://emailprivacytester.com/ Try it on your own mail client!
    - nmjenkins 11y ago
      We have no speed problems using this in production at FastMail (we're kind of known for our speed in fact…). We use it at render time, both on desktop and mobile.
  - Alex3917 11y ago
    Also, Google has a tool called Caja that is open source: https://github.com/google/caja https://github.com/google/caja
- mwcampbell 11y ago
  You mention that the client is a really heavyweight app. A quick look at the packages for Windows, iOS, and Android suggests that you're using Python, specifically the CPython interpreter. Would you choose something else if you were starting today? In particular, I assume that stuffing Python into the mobile apps was challenging. These days I'd use .NET/Xamarin or the Elements compiler (www.elementscompiler.com) when starting on a new multi-platform client-heavy application like this.
  - dmbaggett 11y ago
    If starting today, I'd probably try to use Rust. However, using Python actually makes it much lighter weight than it would be running all native code, because Python bytecode is quite compact, and we use small C extensions for things that need to be fast. Being able to link with C code is hugely important for us, and I don't know if .NET/Xamarin would allow that..? Getting Python to run on a POSIX environment is actually pretty trivial. Windows RT is harder, which is one reason we've never bothered with a Windows RT port. And on iOS you are limited to using a single core under Python, because you aren't allowed by the O/S to fork. The biggest pain I've had with Python is memory management: it's almost impossible not to leak memory in a Python program because the garbage collector is quite primitive and reference counting is just so error prone. For an email app you need to be able to run for days/weeks without crashing or blowing up memory usage; you can't just restart the process every 3am like you can server-side. That's one reason Rust is appealing: memory management is helped by the type system.
    - ghuntley 11y ago
      Re: invoking C/C++ from .NET https://msdn.microsoft.com/en-us/library/aa288468(v=vs.71).aspx https://msdn.microsoft.com/en-us/library/aa288468(v=vs.71).a...
    - mwcampbell 11y ago
      I seriously doubt that using Python bytecode rather than native code makes the application more lightweight. Here are the problems I see: 1. Code size: Bytecode instructions may be more compact than their native code counterparts, since the bytecode instructions are dealing with higher-level abstractions. But the dynamism of Python prevents some major optimizations. First, the bytecode has to include identifiers, since everything is late-bound. More importantly, a packaged Python application tends to include a lot of dead code, because the packaging tool can't accurately tell what modules, let alone individual functions, are actually used. And that's assuming you're using something like py2exe which at least tries to prune the set of distributed modules. 2. Startup time: Of course, even native code can vary here. If you have an executable that's statically linked except for OS libraries, the OS can just read that executable into memory and jump to the entry pooint; that's fast even for a cold start. But if your application consists of an executable plus several DLLs or shared libraries, the OS has to do a lot more work to get it loaded. Python modules tend to be more fine-grained than typical DLLs, and for each module, the interpreter has to find it on the search path, then open the file and read it. 3. Memory footprint: Native code is mapped into memory as shared read-only pages. The OS doesn't actually have to add any of those pages to the working set until they're executed, and it can page them out at any time without having to write to a swap file, which mobile platforms don't have. A few bytecode interpreters, most notably Dalvik, have managed to exploit this, but not Python. So all the Python bytecode has to be in memory all of the time. This is exacerbated by the limited ability to trim the code size as described above. More generally, my thinking on performance, especially perceived performance, in end-user applications has been heavily influenced by this blog post: http://chadaustin.me/2009/02/logic-vs-array-processing/ http://chadaustin.me/2009/02/logic-vs-array-processing/ Here's the money quote: > With disk and memory speeds improving so much more slowly than CPU speeds, the difference between a snappy desktop application and a sluggish application is a handful of page faults. When choosing a technology platform for a project, it’s worth considering the impact to overall responsiveness down the road. And I’m pretty sure I just recommended writing your entire application in C++, which sounds insane, even to me. I’ll leave it at that. That post was several years old. These days I'd expect him to recommend Rust over C++. Of course, you've been developing software for a long time, and you've actually used Python in a mobile app; I haven't. So maybe the issues I've raised just don't matter. I'm particularly curious if startup time has been an issue for Inky at all, especially on the mobile platforms.
- jcranmer 11y ago
  You've managed to miss a few of the biggies I'd count: - the charset parameter is nowhere near as correct as you'd like it to be - many RFCs are not followed in practice. In particular, RFC 2047 is literally more often incorrectly implemented than correctly implemented. I also once recall trying an admittedly weird message (a message whose body is of type message/rfc822) on 4 different IMAP server implementations and getting 4 different answers for part numbering. I'd say building an email client is harder than building a web browser because at least the people who write the specifications for the latter have been trying to thoroughly document all the idiosyncrasies and how to handle them. Also, screw CFWS.
  - dmbaggett 11y ago
    Oh God, IMAP part numbering. I'd blocked that out until now. And regarding RFC 2047, I still remember when I first learned about RFC 2231. It really reads like an April Fools joke. Actual text from this RFC: Character set and language information may be combined with the parameter continuation mechanism. For example: Content-Type: application/x-stuff title*0*=us-ascii'en'This%20is%20even%20more%20 title*1*=%2A%2A%2Afun%2A%2A%2A%20 title*2="isn't it!"
    - jcranmer 11y ago
      You (well, the spec writer) forgot the semicolons (it's errata'd). :-) And now I'm stuck trying to remember off the top of my head whether or not the first few lines are permitted to use quoted-strings in the first place. With regards to RFC 2047, the worst header I've seen to date is this: Re: [Kitchen Nightmares] Meow! Gordon Ramsay Is =?ISO-8859-1?B?UEgR lqZ VuIEhlYWQgVH rbGeOIFNob BJc RP2JzZXNzZW?= With My =?ISO-8859-1?B?SHVzYmFuZ JzX0JhbGxzL JfU2F5c19BbXiScw==?= Baking Company Owner I've been unable to find any possible decoding of those encoded-words that make sense. ... I can't wait until everyone decides to just use UTF-8 for all the headers. Except then I get to yell at everyone who uses Windows-1252 headers right now. >:(

There are at least 75 nontrivial problems, each of which is worth of its own startup-amount-of-effort. A few random examples: - sanitizing HTML email: you must eliminate XSS while preserving the appearance; this requires real HTML and CSS parsing; we do this on the end user's phone at MB/sec rates, which is not easy - conversation threading: any email in a conversation that's been sent with Outlook will have stripped the standard threading headers and added the proprietary Thread-Index header, so in practice you must rely on heuristics - properly mapping the IMAP/Exchange/POP "database" onto UI views is immensely complex in a real client, and requires exactly the right set of abstractions (which are very subtle) - it is a vast understatement to stay that the IMAP protocol is a mess; it is also plagued by some really bad design choices (such as relying on message sequence numbers which vary by connection rather than exclusively using UIDs, which don't). - instant full text search over a million message inbox is hard, but this is a real-world use case - converting arbitrary MIME emails to HTML for rendering purposes is a huge rathole - making offline mode really work (supporting search and modification of cached mails) is complicated, and most clients punt on it (because they are implemented server-side and therefore require a network connection for anything to work) - implementing everything server side is (relatively) easier, but requires you to scale bandwidth per user and mailbox, which is expensive; on the other hand, doing everything client-side like Inky does is much harder because you've got tremendous variation among hardware platforms and operating systems, and it's a really heavyweight app In general, an email client subsumes many other highly nontrivial subproblems, somewhat like a browser does. Just doing the security-related stuff alone properly -- TLS, certs, revocation, encrypted email -- is really labor-intensive.

hammerandtongs 11y ago
Supporting POP3 seems like an anti-feature in 2015. It seems to me, that muddling its lack of features into the middle of an application that properly supports IMAP would make your internal api much more complex?
- dmbaggett 11y ago
  Actually POP is easy except for efficiently determining what's changed in the mailbox since the last time you checked. And some providers still only offer POP access -- for example, Verizon.
  - narrowrail 11y ago
    From the outside, it looks like you have great engineering talent, but on the product management side you seem to lack some vision other than to cover backwards compatibility. This is frustrating to witness, even at a distance.
    - narrowrail 11y ago
      Perhaps I should clarify that I worked for a founder where we had this problem, and after a year almost of the engineers I started with had left. It was maddening, and I finally had to leave as well. The company was eventually sold for pennies on the dollar.
dugmartin 11y ago
I think sanitizing email is a solved problem with this: https://github.com/cure53/DOMPurify https://github.com/cure53/DOMPurify I'm not the author but I've used it in the past. According to the readme it is what Fastmail uses.
- dmbaggett 11y ago
  Probably not fast enough for us, but it is a very nice library. (We use libxml2.) The reference we test against, by the way, is Mike Cardwell's excellent Email Privacy Tester: https://emailprivacytester.com/ Try it on your own mail client!
  - nmjenkins 11y ago
    We have no speed problems using this in production at FastMail (we're kind of known for our speed in fact…). We use it at render time, both on desktop and mobile.
- Alex3917 11y ago
  Also, Google has a tool called Caja that is open source: https://github.com/google/caja https://github.com/google/caja
mwcampbell 11y ago
You mention that the client is a really heavyweight app. A quick look at the packages for Windows, iOS, and Android suggests that you're using Python, specifically the CPython interpreter. Would you choose something else if you were starting today? In particular, I assume that stuffing Python into the mobile apps was challenging. These days I'd use .NET/Xamarin or the Elements compiler (www.elementscompiler.com) when starting on a new multi-platform client-heavy application like this.
- dmbaggett 11y ago
  If starting today, I'd probably try to use Rust. However, using Python actually makes it much lighter weight than it would be running all native code, because Python bytecode is quite compact, and we use small C extensions for things that need to be fast. Being able to link with C code is hugely important for us, and I don't know if .NET/Xamarin would allow that..? Getting Python to run on a POSIX environment is actually pretty trivial. Windows RT is harder, which is one reason we've never bothered with a Windows RT port. And on iOS you are limited to using a single core under Python, because you aren't allowed by the O/S to fork. The biggest pain I've had with Python is memory management: it's almost impossible not to leak memory in a Python program because the garbage collector is quite primitive and reference counting is just so error prone. For an email app you need to be able to run for days/weeks without crashing or blowing up memory usage; you can't just restart the process every 3am like you can server-side. That's one reason Rust is appealing: memory management is helped by the type system.
  - ghuntley 11y ago
    Re: invoking C/C++ from .NET https://msdn.microsoft.com/en-us/library/aa288468(v=vs.71).aspx https://msdn.microsoft.com/en-us/library/aa288468(v=vs.71).a...
  - mwcampbell 11y ago
    I seriously doubt that using Python bytecode rather than native code makes the application more lightweight. Here are the problems I see: 1. Code size: Bytecode instructions may be more compact than their native code counterparts, since the bytecode instructions are dealing with higher-level abstractions. But the dynamism of Python prevents some major optimizations. First, the bytecode has to include identifiers, since everything is late-bound. More importantly, a packaged Python application tends to include a lot of dead code, because the packaging tool can't accurately tell what modules, let alone individual functions, are actually used. And that's assuming you're using something like py2exe which at least tries to prune the set of distributed modules. 2. Startup time: Of course, even native code can vary here. If you have an executable that's statically linked except for OS libraries, the OS can just read that executable into memory and jump to the entry pooint; that's fast even for a cold start. But if your application consists of an executable plus several DLLs or shared libraries, the OS has to do a lot more work to get it loaded. Python modules tend to be more fine-grained than typical DLLs, and for each module, the interpreter has to find it on the search path, then open the file and read it. 3. Memory footprint: Native code is mapped into memory as shared read-only pages. The OS doesn't actually have to add any of those pages to the working set until they're executed, and it can page them out at any time without having to write to a swap file, which mobile platforms don't have. A few bytecode interpreters, most notably Dalvik, have managed to exploit this, but not Python. So all the Python bytecode has to be in memory all of the time. This is exacerbated by the limited ability to trim the code size as described above. More generally, my thinking on performance, especially perceived performance, in end-user applications has been heavily influenced by this blog post: http://chadaustin.me/2009/02/logic-vs-array-processing/ http://chadaustin.me/2009/02/logic-vs-array-processing/ Here's the money quote: > With disk and memory speeds improving so much more slowly than CPU speeds, the difference between a snappy desktop application and a sluggish application is a handful of page faults. When choosing a technology platform for a project, it’s worth considering the impact to overall responsiveness down the road. And I’m pretty sure I just recommended writing your entire application in C++, which sounds insane, even to me. I’ll leave it at that. That post was several years old. These days I'd expect him to recommend Rust over C++. Of course, you've been developing software for a long time, and you've actually used Python in a mobile app; I haven't. So maybe the issues I've raised just don't matter. I'm particularly curious if startup time has been an issue for Inky at all, especially on the mobile platforms.
jcranmer 11y ago
You've managed to miss a few of the biggies I'd count: - the charset parameter is nowhere near as correct as you'd like it to be - many RFCs are not followed in practice. In particular, RFC 2047 is literally more often incorrectly implemented than correctly implemented. I also once recall trying an admittedly weird message (a message whose body is of type message/rfc822) on 4 different IMAP server implementations and getting 4 different answers for part numbering. I'd say building an email client is harder than building a web browser because at least the people who write the specifications for the latter have been trying to thoroughly document all the idiosyncrasies and how to handle them. Also, screw CFWS.
- dmbaggett 11y ago
  Oh God, IMAP part numbering. I'd blocked that out until now. And regarding RFC 2047, I still remember when I first learned about RFC 2231. It really reads like an April Fools joke. Actual text from this RFC: Character set and language information may be combined with the parameter continuation mechanism. For example: Content-Type: application/x-stuff title*0*=us-ascii'en'This%20is%20even%20more%20 title*1*=%2A%2A%2Afun%2A%2A%2A%20 title*2="isn't it!"
  - jcranmer 11y ago
    You (well, the spec writer) forgot the semicolons (it's errata'd). :-) And now I'm stuck trying to remember off the top of my head whether or not the first few lines are permitted to use quoted-strings in the first place. With regards to RFC 2047, the worst header I've seen to date is this: Re: [Kitchen Nightmares] Meow! Gordon Ramsay Is =?ISO-8859-1?B?UEgR lqZ VuIEhlYWQgVH rbGeOIFNob BJc RP2JzZXNzZW?= With My =?ISO-8859-1?B?SHVzYmFuZ JzX0JhbGxzL JfU2F5c19BbXiScw==?= Baking Company Owner I've been unable to find any possible decoding of those encoded-words that make sense. ... I can't wait until everyone decides to just use UTF-8 for all the headers. Except then I get to yell at everyone who uses Windows-1252 headers right now. >:(