15 ms·
Email made me miserable, so I decided to build my own email app from scratch
- dmbaggett 11y agoChris Kirk, the author of this piece, interviewed me briefly. I have to admit that when he told me he was writing his own email client for the story, he became my tech journalism hero.
- stevekemp 11y agoIt does sound pretty impressive; I've written a (console-based) mail-client, and between fighting with different approaches to UI, scripting, and MIME issues, I've a lot of respect for somebody tackling the problem.
- egometry 11y agoHe dwells too much upon the age and cruft of email, but this article reads more like a layman learning for the first time about Hofstadter's law, general programming project mis-scoping, and dealing with technical debt he himself created... which is kinda cool to hear about from "the outside".
- netghost 11y agoWhat's great though is that at the end of it, he has a tool that works for him. That's a huge accomplishment. All too often I obsess over tiny details in my personal projects and never get to anything useful at all. Meanwhile this guy built something that's probably not elegant, but totally does what he needed.
- dmbaggett 11y agoTo be fair to him, though, pretty much every programmer I know since I've been working on (http://inky.com http://inky.com) has asked me why it's hard. Email is partly hard because of the historical cruft, but there are genuinely difficult design/abstraction challenges with email as well, and even experienced developers tend to mis-predict how hard it is to make a "real" mail client. The one exception to this was Carl de Marcken, one of my co-founders at ITA Software, and chief scientist there. He predicted -- correctly -- that it would take us 4+ years to get to an MVP.
- plehoux 11y agoI agree, at Missive (https://missiveapp.com https://missiveapp.com) it took way longer than we originally thought to get to MVP, ~around 1.75 year (and we only support Gmail/Google app for the moment)
- dmbaggett 11y agoWait until you get to IMAP and Exchange -- you'll learn that you're only 5% done. :(
- Estragon 11y agoWhat are some of the most surprising complexities?
- dmbaggett 11y agoThere are at least 75 nontrivial problems, each of which is worth of its own startup-amount-of-effort. A few random examples: - sanitizing HTML email: you must eliminate XSS while preserving the appearance; this requires real HTML and CSS parsing; we do this on the end user's phone at MB/sec rates, which is not easy - conversation threading: any email in a conversation that's been sent with Outlook will have stripped the standard threading headers and added the proprietary Thread-Index header, so in practice you must rely on heuristics - properly mapping the IMAP/Exchange/POP "database" onto UI views is immensely complex in a real client, and requires exactly the right set of abstractions (which are very subtle) - it is a vast understatement to stay that the IMAP protocol is a mess; it is also plagued by some really bad design choices (such as relying on message sequence numbers which vary by connection rather than exclusively using UIDs, which don't). - instant full text search over a million message inbox is hard, but this is a real-world use case - converting arbitrary MIME emails to HTML for rendering purposes is a huge rathole - making offline mode really work (supporting search and modification of cached mails) is complicated, and most clients punt on it (because they are implemented server-side and therefore require a network connection for anything to work) - implementing everything server side is (relatively) easier, but requires you to scale bandwidth per user and mailbox, which is expensive; on the other hand, doing everything client-side like Inky does is much harder because you've got tremendous variation among hardware platforms and operating systems, and it's a really heavyweight app In general, an email client subsumes many other highly nontrivial subproblems, somewhat like a browser does. Just doing the security-related stuff alone properly -- TLS, certs, revocation, encrypted email -- is really labor-intensive.
- CaptSpify 11y agokeeping up with email became a snap for me once I discovered mutt and procmail. This post definitely falls under "reinventing the wheel", but I can't say that's a bad thing. Unfortunately, most people's workflow is so different, that having one email workflow/interface just won't cut it, and sometimes it takes a custom solution.
- pmlnr 11y agoWhy procmail, why not Sieve on the server side?
- pjc50 11y agoPerhaps they went over to mutt+procmail before Sieve was written? Perhaps they don't use Cyrus IMAP, or lack the relevant permissions on their host? Both of those apply to me; I have a mutt+procmail account and a Gmail account for different purposes.
- SwellJoe 11y agoDovecot also supports Sieve. Nonetheless, I still use procmail, even though I could turn on Sieve on my mail servers. Habits die hard, and the tools I use to manage my procmail rules (Virtualmin and Usermin) don't yet support Sieve directly (though its on the roadmap for the Usermin update coming in a few months).
- rakoo 11y agoWhich mail providers provide Sieve support ? I see Fastmail does, but Gmail doesn't.
- pdonis 11y ago> Which mail providers provide Sieve support ? Nobody really does, which I find extremely frustrating since sieve is supposed to be the Internet standard for mail filtering. Fastmail sort of does, in that they use sieve rules; but last time I looked, they didn't have a managesieve interface, which means I can't just upload the sieve rules I have spent a lot of time creating on my home email server. I have to create them from scratch using Fastmail's rather clunky (IMO) interface. As far as I can tell, no other email provider uses sieve at all.
- oever 11y agoThe repo is here https://github.com/cperryk/slatemail https://github.com/cperryk/slatemail I've no idea how to run the app. It seems to be a graphical node.js application.
- satai 11y agoNo README.md in the root of a git repo is definitely a bad sign.
- oopsies49 11y agoIt's a personal project that the author didn't expect anyone would ever look at. He's also a journalist, not a software developer. It's an accomplishment that he had the determination to complete his project, even if it is rough around the edges.
- ohitsdom 11y agoHe's a journalist, so I'm sure he expected people would be looking at it. Reasons were probably more due to inexperience and lack of time.
- nkrisc 11y agoI bet if you asked him nicely or kindly pointed out that he's missing that, he might add one.
- rokhayakebe 11y agoPerhaps Email serves its purpose perfectly: send/receive messages. For everything else, we could either extended it via plugins or create different tools SMS/Chat apps/etc...
- pmlnr 11y agoThis. Email is pluggable, on client and on server side. The basic idea of email is done and finished; everything that extends it should be a plugin.
- afarrell 11y agoIsn't Nylas (https://www.nylas.com https://www.nylas.com) supposed to fix this problem?
- Silhouette 11y agoIt's hard to say. That web site is so buzzword-driven and content-free that after reading a couple of the pages I still have literally no idea what it does or why I might need it.
- david-given 11y agoWow. That's site's awful. I got a bit further than you --- I actually found the API docs, before giving up when a manual Javascript page-load hook broke the back button. AFAICT it's a web service which talks the email providers for you, thus making writing email clients easier. I'm not quite sure how they do this without the end user having to give them their login credentials...
- tlrobinson 11y agoThey have an open source email sync engine/REST API https://github.com/nylas/sync-engine https://github.com/nylas/sync-engine and client https://github.com/nylas/N1 https://github.com/nylas/N1
- grinich 11y agoHey-- I work at Nylas. Would love any feedback you have for the site! There's a bit more info on our "Features" page: https://nylas.com/features/ https://nylas.com/features/ In particular: > Nylas brings email into the 21st century with modern tools, RESTful developer APIs, and open source code. Integrating with Gmail or Exchange now takes minutes instead of months. Nylas is the easiest way to build an email app. Pretty much this means we talk IMAP/SMTP/ActiveSync/MIME so your app doesn't have to. Plus there are some nice new features, like a transactional "delta" endpoint and webhooks, which makes writing mail processing code a breeze. If you're a developer, the easiest place to dig in might be our docs: https://nylas.com/docs/platform https://nylas.com/docs/platform or the open source sync engine: https://github.com/nylas/sync-engine https://github.com/nylas/sync-engine We also shipped a new desktop email app this fall, which started as a fork of Atom. You can see that code here: https://github.com/nylas/n1 https://github.com/nylas/n1 Our website is sorely in need of a refresh, but right now our team is busy scaling and fixing bugs like crazy. (Relatedly, if any designers are looking for a job... feel free to email mg@nylas.com !)
- plehoux 11y ago"Maybe the dream of an email-free future isn't dead; maybe it just means a future in which email is merely a sliver of our communication rather than the whole pie." Email is victim of its own success, like cars & highways in America. Highways are not bad per se, they were and are an incredible revolution. But too much of them means the value we collectively subtract constantly diminish. We need alternatives: walkable neighborhoods, bike lanes, tramways, trains, bus, etc. It's the same with email, we need alternatives. Chat services like Slack & HipChat are good examples for internal communication. Email's decentralized nature and ubiquity mean it's the perfect tool for external communication. I don't see why this would change.
- plehoux 11y agoLast week I did a visualization of all email clients in time from '79 to now (http://email-apps-timeline.missiveapp.com http://email-apps-timeline.missiveapp.com). It shows that there is a lot happening in the email space right now, even if the technical barrier to entry, as the author implies, is so high.
- whatever_dude 11y agoGreat chart! Some of those new email clients look great. I wonder if any of them will become solid and popular enough.
- baldfat 11y agoI am still waiting for the application that kills email and the need for clients..
- simoncion 11y agoGiven that the set of folks that are interested in and capable of making widely used end-user communcations software, and the set of people who are not interested in making walled gardens appears to have no overlap, you're going to be waiting for approximately forever. :(
- siscia 11y agoWe are working on this same problem providing unlimited, disposable, email for all. We are getting close to something similar to an alpha version, feel free to subscribe: mailroad.co/
- abluecloud 11y agoThe CSS negative margin you've applied to `.row` is causing your page to add vertical scrollbars. You should remove the padding from `.container-fluid` and apply margins to the inner elements to stop it. It looks naff.
- ForHackernews 11y agoAm I the only person in the world who thinks email works pretty well? Maybe I just get less mail than many people, but I just delete (or mark "read" and ignore) most of the marketing ham mails I get. I read and respond to the actually-important messages I get for work or from personal contacts. Actual spam is pretty well filtered. I vastly prefer email to Facebook messages, or Slack notifications, or any of the other purported "solutions" to communication. Why do people hate email so much?
- wpietri 11y agoAs the author explains in the first few paragraphs, he was basically fine with email until he was thrust into a job with a much higher email volume. That's been the story I've heard from most people who hate email.
- creeble 11y agoExactly, the problem isn't email, it's the level of communications you believe you are expected to maintain. Maybe you are expected to maintain this level, in which case you have a problem that no level of email-client fixes will solve.
- pjc50 11y ago> For example: I often find myself completely losing track of something that urgently requires my response, or struggling for several minutes to find a single email from a few weeks ago. That’s because email was not designed with the assumption that anyone would get hundreds of messages a day. To a computer, a direct email from your boss looks the same as your water bill. It's on you to put it where it needs to be. IMAP provides only two tools for this: mailboxes and flags. You're expected to sort your messages into mailboxes as if you're sorting paperwork into folders, and/or flag them with a limited set of colors. Do this long enough, however, and you'll soon end up like I did, rifling through an elaborate hierarchy to find the proper folder. The more complex your system of organization, the longer it takes you to organize or find each individual email, and the less useful the system becomes. Eventually, the time you spend maintaining the system outweighs the time it saves you. It's not so much email-the-communications-system, but "email" as a collective noun for all the different communications demands from other people and system he faces.
- aaronem 11y agoWhy use email as a to-do list, when it's obviously so poorly suited for the role? Why not use email for things email's good at, and use one of the wide variety of to-do list tools for task tracking? When you need to refer to an email from a to-do item, there's ways to do that; some clients support direct hyperlinks, and others can be dealt with by just including the date/time and sender or subject of the email in the to-do item's notes, so that you have the information you need to quickly find the message via search. You can also include excerpts from email, or even whole messages, directly via copy-and-paste; after all, it's not as though they are going to change once received, so there's no sync problem to manage. Granted, this might seem like being more effort than just mapping task list semantics onto email. But the article under discussion here chronicles its author's essentially having reinvented every task abstraction all the way up to the project, solely in order to overcome the inherent drawbacks of trying to use email as a task list. It seems impossible that this could be easier than just finding a task list tool that has the features you need, and learning to use it alongside your preferred email client.
- pjc50 11y agoQuite a lot of people use their inbox as a task list where the task is "respond to / deal with this email". The problem here is that such a task list is world-writable and chronologically sorted.
- aaronem 11y agoRight, I get that, but the point I'm making is that the tool isn't suited to the job it's being used for. "Deal with this email" could mean anything from an immediate off-the-cuff answer to a trivial information request, to a project that takes days to complete; email clients generally provide no good method (or just no method, period) of capturing and tracking such tasks, because that's not the job they are designed to do. Worse, as you rightly note, if your email account is also your to-do list, anyone can add to it as frequently as they like. This may or may not make an actual difference in the amount of work you have to do, as for example in a case where you get ten different emails all pertaining to the same task, but it might well have a psychological effect, and it certainly makes it harder to keep track of what you actually need to do. I mean, I don't know. I'm not here to tell anyone they're wrong to do what they've found works for them. But every time the subject comes up, I hear a lot of people saying that what they're currently doing really doesn't work for them, and they've been as yet unable to come up with a better way. From the entrepreneur/innovator side, these pleas tend to be met with yet another attempt to solve problems with email which only exist as a result of making email do things it's neither designed nor well suited to do. I think people would be better served by instead suggesting the use of a tool which actually is suited for the job they're trying to make email do. There are lots of task list managers out there, and almost any of them will do the job more ably than yet another attempt to cram email into a paradigm to which it so plainly isn't suited.
- golergka 11y agoMay be the author is a bad programmer — but undeniably, he's a great tech journalist.
- randyrand 11y agoHah, very true. Though, we all start off as bad programmers =P
- T3RMINATED 11y agoI once made an email client and no one used it, not even me.
- Murk 11y agoAddspam;DR
- aplorbust 11y agoHow is Slack different from IRC or using netcat on a LAN?
- eclipxe 11y agoMore unicorn dust and SV hype-machine
- detaro 11y agoIt comes ready to run with tons of stuff you would have to add to an IRC setup. Each time the topic comes up there are tons of people insisting that they could do the same with an IRC software + X, and they are probably right, but that doesn't come nicely packaged and ready-to-use for non-techies. + it is a hosted service, which some people like.
- codezero 11y agomanaged permissions, built in history (doesn't rely on you being connected to get a log of a conversation that happened when you were gone), great full text search, integrated rich content (posts, videos, images) which are also searchable in the same way, adding context to conversations, built-in system for notifying people, channels, and groups, with centralized options to tailor those notifications to yourself, a fully featured mobile application that synchronizes with the desktop app seamlessly. Slack basically removes the requirement for you to have an IRC server, client, and a suite of IRC scripts and bots. It unifies the experience across the team, which, yes, does remove some levels of customization you would get from your personal IRC client, but in practice, it's been a highly productive work-chat tool. I've basically replaced Dropbox as my source for important files, instead, putting them into a Slack channel. That way what ever made that file relevant is surrounded by the chat that necessitates the file. It's not for everyone, and I don't think anyone would claim it should be, but if you work on a team, especially a distributed one, Slack brings a lot to the table "batteries included."
- rocky1138 11y agoYes, but what happens when Slack goes away?
- larksimian 11y agoLiked the article but it produced a weird dissonance in me. On the one hand I totally empathized with his frustration/excitement at building a cool bit of software for himself. On the other hand I've never had problems managing my email inbox, mainly because I don't get that much relevant email. In particular, since the Gmail tab redesign I've not had an issue keeping my Inbox unread count at 0(though with my Promotions/Updates/Forums tab unread count in the who-knows-how-manies). I guess this confirms his point about email protocols being dinosaurs, since the way Gmail implements its tabs is tied into their client implementations and doesn't work in other email clients. The overall impression I ended up with was that Slate is simply using email for workflows it was never meant to support and is not very good at supporting. Email chains are not a good way to do project management/planning. It's good that they've started using Slack for chat-like communication. They should adopt some sort of project management app to manage projects. If you've got a user that's used to Gmail, they can learn Trello, for instance, in about an hour. I tend to think about 3 levels of communication: chatting(text, audio, video); email; project related. Each level requiring additional structure and being a bit more cumbersome to use than the last when writing, but easier to use when reading/searching for information. In that sense it makes sense to use 3 (sets of) tools for 3 different types of communication. The main downside is that it's somewhat complicated to transfer a conversation from one channel to another. However doing all 3 in one application leads to the risk of making a simple use case too complicated or a complicated use case impossible. The ideal integrated communication app would need to provide IRC-like chat power, Hangouts/Skype like chat simplicity(+audio/video support), Gmail like email usability and auto filtering, and more than Trello-like project management/collaboration tools(for instance, having Google Doc-style shared documents properly integrated into it). All usable in interfaces designed for the particular level of communication you are engaging it at the time, but allowing you to move 'threads' of conversation up or down the hierarchy easily. I've not encountered anything that does this right. Heck just being able to properly integrate Gmail, hangouts chats and google docs/google doc comments between each other isn't possible and all those are made by the same company.
- tmail21 11y agoI like the way you imagine 3 levels of communication, viz. chatting, email and project related. However, I think the jump from email to project-related is too large. The amount of ceremony required for project tools is too high relative to email. In particular, these tools are based on static 'groups' or 'teams' and hence are nothing like email's per-message variation in users. Furthermore they are based on asymmetric permissions (owner vs. non-owner) which leads to an entirely different set of dynamics. I think a level between email and projects can bring a lot of benefits the author is looking for with only a modest increase in cognitive overhead. In this approach, there would be a first-class entity at the thread-level. So, the entire Discussion would be a first class entity. This solves a lot of the author's problems. You don't have random stuff showing up and disappearing from threads (based on Subject and based on whether you happened to be copied or not). Threads could easily have tracking numbers which would open up email to the 'outside world'. Also, once one has a first-class thread, one could associate append-only updateable content sections. This would be like attachments but without the versioning problems. In fact it would be easy to build a lightweight process framework, with tasks etc. on such a foundation. Disclaimer: Our company (https://tmail21.com https://tmail21.com) has built just such a system. It adds a first class entity at the thread level (which we call a TMail). It allows users to start with Simple Topic-Oriented Discussions and from there progress to Goal-oriented Discussions and from there to Lean Processes. Take a look and let us know what you think.
- tunesmith 11y agoI've always thought an effective if brutal strategy might be the one that won't even deliver your email to you if you're not at Inbox Zero, and then it delivers a day at a time so you might end up days or weeks behind. And maybe it would give you clues, like, "you have 47 additional emails waiting from this sender", so if they're spammy, you could just unsubscribe right there and it would delete them upstream without you even having to look at them.
- patates 11y agodo people really have this problem? I never remember being bombarded by a single source (be it a person or a service), email becomes a burden because there are many sources and the level of urgency is alternating even for the same source through time
- LoSboccacc 11y agoYeah I manage that by treating all mail as low priority and every chat as high priority. People chatting for spam or to bring attention to a low priority inbox gets told to send an email. This way I'm sure not to miss critical messages without relying on pepople self tagging
- r00fus 11y agoI would think this could be a real situation if you have a problem customer (read: demands more than their SLA) and you're not ruthless/effective in your communication. Sometimes not responding in those situations can lead to better outcomes than replying in haste or frustration.
- vpeters25 11y agoI find very interesting how the author describes some of the shortcomings of IMAP. The article seems like a nice starting point for requirements to create a new email protocol. Like the author, for years i've been bothered by email, but as someone who has been maintaining mail servers for 20+ years my biggest issue is the server side of things. SMTP, POP and IMAP have served their purpose way beyond their original scope. I think going to the drawing board and coming up with a replacement is long overdue.
- dmm 11y agoI think fastmail has proposed jmap as a replacement for imap: http://jmap.io/ http://jmap.io/
- vpeters25 11y agoFrom JMAP description on github: > It aims to be compatible with the IMAP data model, Unlike this project, I think user requirements should drive the protocol's design, not a need to be compatible with SMTP/IMAP.
- rgbrgb 11y agoCompatibility with the rest of the network is a user requirement.
- brongondwana 11y agoYes, this. You need to remain compatible with the rest of the world. IMAP plus every extension in the world isn't even that awful, if everyone supported all of them. The problem is that they don't, so you have massive complexity of servers to support.
- vpeters25 11y agoIt should not be, that's the main premise of my idea: create a new email protocol that completely replaces both SMTP and IMAP based on actual use cases instead of being constrained by 40+ years of legacy protocols and extensions. Client and servers using this protocol would not be required to implement SMTP/POP/IMAP at all, bridge services could be built for that.
- jakejake 11y agoHaving built several in multiple languages, I'd recommend for anyone to build their own email client. It's quite easy to get the basics up and running but quickly becomes tricky getting all of the messages to display consistently. Then you can add embedded images and attachments to the mix. The list goes on and on. But, you really will learn a lot about protocols, parsing text & binary data, working with legacy formats and just all kinds of interesting stuff. Even if you don't wind up with an amazing client, it's still a lot of fun!
- stevekemp 11y agoI'd second that. The basics are easy, but there will be constant niggles relating to RFC2047 header-parsing, MIME-issues, and issues relating to handling some crazy mails that people will send. I've written my mail-clients in C++, but I suspect Perl, Python, or Ruby, would be fast enough with the massive caveat that you need to have decent libraries/interfaces for the low-level parts.
- noddingham 11y agoThe most salient point he makes is when he says there's a difference in what email was designed and how we use it today. And it sounds like Slate needs better tools. Do you even Confluence / JIRA (or insert your preference here)? If you'd stop trying to use email as the one tool for everything, you'd stop (or significantly reduce) the times you run into it's supposed shortcomings.
- jethro_tell 11y agothe problem with a lot of those tools is that they use email as a notification system. When they should run something more like the notification system for iOS or android. (you have seventeen new notices from jira in 3 issues). You look at it once, then it is gone and you go to the site to check it out.
- nodivbyzero 11y agoDiscover Emacs with powerful Gnus.
- jrapdx3 11y agoA few years ago I wrote a webmail client to access my mail on the server host. I had developed a web server (in Scheme no less) that was/is running (now on a VPS) and at the time it seemed only natural to extend it with webmail functionality. The effort was successful (for some definition of success), though it proved a to be a daunting task. It taught me a lot about email, that it's riddled with arcane protocols and dark corners, these "features" pretty much without consistency or order. On top of that there was constructing the UI needed to actually use the thing. Would I do it again if starting from scratch? I don't think so, it sure seems the hard way to go about it. No doubt I could recommend it as a great learning exercise, but for creating a product I don't think it's the pragmatic way to go. I've considered publicly sharing my project, but not before refactoring some prodigiously opaque, obscure and convoluted parts, and who knows when I'll have the time (or courage) to tackle it. Email remains at once a great wonder and horror, yet for all that email continues to be widely used, and remains essential to most everyone. I still use my email client most every day, despite its warts. I guess it's proven its worth after all.
- hackuser 11y agoAnother, much longer and more technical view of the complexities of email clients: Why email is hard by Joshua Cranmer of Mozilla: http://quetzalcoatal.blogspot.com/2013/09/why-email-is-hard-part-1-architecture.html http://quetzalcoatal.blogspot.com/2013/09/why-email-is-hard-...
- listic 11y agoAren't there already email solutions out there that treat email as a to-do list, divide it into discrete projects and so on? I believe there should be some, based on Allen's GTD method, if nothing else. (https://en.wikipedia.org/wiki/Getting_Things_Done https://en.wikipedia.org/wiki/Getting_Things_Done)
- camgunz 11y agoThe most remarkable thing about this article to me was that a journalist with moderate programming ability was able to build an email client. I think as coding becomes more accessible we'll start to see truly novel approaches and applications. People say C succeeded because of UNIX, and JavaScript succeeded because of the web. I'd guess the next language will succeed because it makes true software development fully accessible to people who haven't spent years learning how to do it.
- tim333 11y agoReading his background (http://cperryk.com/ http://cperryk.com/) it says "I built my first website as an 11-year-old." "I earned my first dollars building websites for family friends and fellow students' parents." So he may be a bit more experienced than the impression he gives in the article.
- z3t4 11y agoI still believe that multi-part e-mail conversations can be cleaned up with a few "diffs" and presented in a chat-like view.
- barbs 11y ago> I found the process frustrating, humbling, and strangely fun. I imagined I was on CSI—but instead of a person, it was my app dead on the floor, and I had to find out who killed it and how What a great way to think about bug-fixing!
- slowmovintarget 11y agoI couldn't help thinking of Google Wave while reading this. There was one missing feature that made a handful of my friends and I stop using it: regular GMail in the same client. If we didn't have to switch back and forth, Wave would have slowly grown to dominate most electronic conversations I had with friends and colleagues. Sigh.
- 0x445424 11y agoThat's exactly why I actually liked Buzz... It was integrated with Gmail.
- deleted 11y ago[deleted]
- deleted 11y ago[deleted]
- clscott 11y agoThis is an interesting read, when I saw a few of the comments here ask "how hard can it be?" I was reminded of this presentation by Rik Signes at YAPC::NA (2008) Email Hates the Living https://www.youtube.com/watch?v=4s9IjkMAmns https://www.youtube.com/watch?v=4s9IjkMAmns
- collinmanderson 11y ago"When Gmail launched as a public beta in 2007" - Wasn't it April 1, 2004?