9 ms·
Why kernel development still uses email (2016)
- IncRnd 9y agoIt doesn't make sense for large projects to move to other systems. Control is lost of the history, which automatically disqualifies any platform. Using services others than e-mail will also entail a revamp and move to a different service every few years. Email is a far lower cost and more reliable medium most other methods.
- u801e 9y agoI've always wondered why newsgroups aren't used instead of email. They pretty much have the same advantages of email and they're far better in terms of searching archives compared to email. Web interfaces for searching email list archives are painful to use. If it wasn't for services like gmane (and later public-inbox), it would be very difficult to view archives of those lists.
- IncRnd 9y agoNewsgroups could be an exception to what I wrote. That's a good point. But, newsgroups don't use the special protocols that I meant, which may not be long-lived enough.
- TazeTSchnitzel 9y agoOne of the best things about email is that, with the right client, you can view discussions as hierarchical trees. That means you can have multiple simultaneous discussions about a topic asynchronously and not lose track of anything. GitHub and the like only work that well when there's maybe one or two things being discussed at once, because there's just a single linear stream of comments.
- carussell 9y agoOn the other hand, the unthreaded approach can help keep the discussion focused and tight. There are few things worse than setting out to get some work done and having to wade through a bunch of comments to find the meat. In the age of Bugzilla, if you wanted to discuss something related, you'd open up a new bug. You'd string things together with a tracking (meta) bug if you liked. This Bugzilla-style workflow, however, doesn't accommodate the stuff that you wouldn't feel you should open a bug for. That's a feature. Like so many other things, though, this only works if you enforce it. It's pretty horrifying what the GitHub generation tends to think passes for acceptable on bugtrackers, for example.
- u801e 9y ago> On the other hand, the unthreaded approach can help keep the discussion focused and tight. Or that earlier posts get lost because no one actually sees them. With threaded discussion, if one subthread has gone off-topic, you can always collapse or kill that subthread. And your point doesn't always hold true. For example https://github.com/torvalds/linux/pull/17 https://github.com/torvalds/linux/pull/17
- carussell 9y ago> With threaded discussion, if one subthread has gone off-topic, you can always collapse or kill that subthread. This is a social problem. I deliberately added a disclaimer about the tendency for things to turn to crap if you tolerate it. I don't know what this is supposed to mean: > And your point doesn't always hold true. For example https://github.com/torvalds/linux/pull/17 https://github.com/torvalds/linux/pull/17
- u801e 9y ago>> And your point doesn't always hold true. For example https://github.com/torvalds/linux/pull/17 https://github.com/torvalds/linux/pull/17 > I don't know what this is supposed to mean What I linked to is an example of a discussion that didn't stay focused or tight despite the fact that it's not threaded.
- pmoriarty 9y agoThis is one of the things that web-based forums have yet to get right. Email (and NNTP news) clients from 20 or 30 years ago are far superior in this respect, because they can intelligently deal with threading and folding. These features alone makes large conversations much easier to deal with than on web-forums. To add to that, email (and NNTP news) clients even from 20 or 30 years ago have other powerful features that web forums have yet to catch up on: - kill files[1] (which you can use to filter out unwanted articles/mails based on content or metadata such as subject, user, etc) - scoring - user-configurable anti-spam filtering or other "intelligent" filtering (such as bayesian filtering not just for spam/ham, but for interesting/unintersting content) - tagging not just on a site-wide level but at the client level so each user can tag messages/articles the way they make sense to them - other advanced filtering and scripting based on any of the above Web-based forums are just incredibly primitive compared to this many-decade-old technology. [1] - https://wikivisually.com/wiki/Kill_file https://wikivisually.com/wiki/Kill_file
- SubiculumCode 9y agoCan we do that with a chat system? Because Slack threading is just inadequate, imo.
- nextos 9y agoExactly. Gnus is a marvelous client because it has really good NNTP and threading support (silencing and automatic scoring) that allows you to cope with gigantic volumes of email. It's a bit sad the code is messy and it's a tad slow compared to other options. Notmuch for Emacs has so much potential to replace Gnus. The interface is incredibly fast and friendly (a classic tree view and a threaded view that IMHO surpasses Gmail). Plus super quick search. I think it could become a killer application for Emacs (the other two now are org-mode and Magit IMHO). However, there's no good tooling to keep Notmuch in two-way sync with maildirs right now. Everything works based on tags, which is incredibly flexible, but there should be some tooling to implement logic that allows moving mails around based on their tags, so that the remote IMAP store gets tidy. People usually rely on scripts for simple use cases, which is fine albeit a bit hacky. Some DSL that resembles Sieve, but implemented in ELisp would be cool.
- leeoniya 9y agoi'm implementing a webmail app as we speak (from the bottom up). my immediate thought was "wtf are 'conversations', gmail? of course i want threads!". then i started actually handling the raw email messages. i got threads working pretty well but noticed frequent rendering issues and digging into the cause. my conclusion is, email threading is mostly a pipe dream; it can only be done properly if you have a view of the entire conversation (as is the case with centralized forums/message stores like HN). the 'References:' header only represents the message ids of any particular branch up to the root. if you try turning a number of messages with the same root into proper threads, you quickly discover missing branches, partial branches/holes, missing leaf nodes, etc. If your knowledge of a thread starts with a message forwarded to you, no thread is possible (even though one clearly exists as indicated by the References chain in the header!). side-conversations that happen (selective replies, cc/bcc), additional fwd/reply chains are all invisible to you. sadly, an incomplete thread view is an extremely common thing (at least in my inbox/sent of ~15k messages). the client can only thread the actual raw messages it has; those simply quoted in any replies or forwards (which established the Reference chains) don't count. the only commonality you have is knowing the root message id and the timestamp for ORDER BY. everything in between, you may or may not have to reconstruct a partial view of the thread...and each user will have a different view of that thread based on which parts they've been party to. the threading that i did get working was nice, but not as useful as a bulletproof solution would have been. unfortunately, i don't believe that's possible with email. i now understand why gmail does flat convos.
- ori_b 9y ago> my conclusion is, email threading is mostly a pipe dream Them why has it been working so well for me since I started using email? Maybe take a look at this for inspiration: https://fkref.com/?RbC_-NEE https://fkref.com/?RbC_-NEE
- leeoniya 9y agothats basically what i did. the problem is with this step: 4. Prune empty containers. you can prune/flatten these containers but then the deeper messages that you do have are not direct replies to their ancestors in the flattened/pruned version of the tree. i didnt say that threading did not work. it does. but it is imperfect at best and can be misleading at worst. in my case i was constructing an MPTT structure (Nested Set) for optimal thread fetching and simply leaving in the dummy message placeholders that i didnt have a copy of. i plan to experiment further, but there arent really any earth-shaking revelations in that post. it's all fairly obvious.
- lloeki 9y agoGitLab has discussions, which thread topics within an issue/MR/commit. In the context of a MR, individual messages or whole threads can be marked as resolved. This is an invaluable tool to get out of linear quote hell.
- thg 9y agoDiscussion from a year ago: https://news.ycombinator.com/item?id=12620468 https://news.ycombinator.com/item?id=12620468
- blauditore 9y ago(Unrelated to the e-mail topic) > A poor craftsman famously complains about his tools, Greg said, but a good craftsman knows how to choose excellent tools. Unless their company forces tools on them. :(
- Kenji 9y agoThe best craftsmen create their own tools.
- ivan_gammel 9y agoActually good craftsman knows how to use the tools available to him to do the job in the most effective way. Choice is a luxury that is not always present.
- 482794793792894 9y agoJust because you say that doesn't make it true, though. Sometimes a tool just is worse, even if you become skilled at it. I can try to work around Microsoft Word's UI by learning the keyboard shortcuts, but I can't do anything against Internet Explorer taking its sweet time loading webpages. No skill in the world will make that faster.
- ivan_gammel 9y agoWell, that's the point. If you can get your job done with IE (I can only guess what kind of job is that), you just do it without pointless clicking, pressing F1 button or anything else that does not belong to the path of success. Talks about "it's worse", "omg it's so slow", "why not firefox/chrome/whatever" are meaningless when you are in strict corporate environment with only IE allowed or even possible. True professionals do not pay much attention to the tools. They pick the best when they have choice or find the way to use what's available, because craftsmanship is not about the tools.
- na85 9y agoFor 99% of folks outside the HN echo chamber, email works fine. Spam is a problem, but I've never heard or read of anyone who thinks email needs to go the way of the dodo except on this site.
- AndyMcConachie 9y agoI have no idea why you're downvoted. Email is great especially because it's an open standard. I don't understand why people expect Linux kernel development to adopt some kind of proprietary platform for communications. I think it's a terrible idea.
- na85 9y agoHN doesn't like to be reminded that this is an echo chamber. Dissent is not permitted. Interrupting the circle jerk is not permitted.
- abiox 9y agobtw "echo chamber" is generally perceived as disparaging, so it's not weird people wouldn't be impressed by that. and feigning victimhood after antagonizing people is pretty gauche.
- na85 9y agoIt was intended to be disparaging: one of my major beefs with this community is the massive culture of groupthink and aggressive self reinforcement/adulation.
- abiox 9y ago> It was intended to be disparaging and thus your thesis of 'HN doesn't like to be reminded' falls to the simpler explanation of 'trashtalk usually gets downvoted'.
- 9y ago
- tzahola 9y agoReally? I thought they switched to Google Wave long ago...
- Sir_Cmpwn 9y agoI've been working on a side project to bring the best of both worlds. It aims to provide git repository hosting, issue tracking, continuous integration, and mailing lists to organize efforts. But I'll also be providing web tools for submitting and reviewing patches that provides a workflow familiar to GitHub users, but backed by mailing lists. If anyone is interested in getting involved in the closed alpha for this software, my email's in my profile. Hit me up!
- hdhzy 9y agoI wonder if they considered git appraise [0] as it allows offline work and synchronization due to underlying transport (git). [0]: https://github.com/google/git-appraise https://github.com/google/git-appraise
- CoreXtreme 9y agoHow about using Discourse? I think it scales well. Would love to hear your experience with this.
- marksomnian 9y agoAFAIK it doesn't really have proper tools for reviewing patches.
- usernam 9y agoI find Discourse absolutely horrendous. Discourse mailing-list mode is a tack-on at best, and the formatting you get in mail notifications basically require you to be online to visit the link. And the gamification is also a huge put-off. Web forums made us forgot how far we came with NNTP years ago. Even the shittiest NNTP could do subthread splitting, hiding and scoring. Handling hundreds of messages per day was easy. The convenience of doing-it-all under one powerful client was massive. You can still do that with mailing lists, as long as it's not as badly managed as forums-with-a-wanna-be-mailing-list-mode addon as Discourse is.
- jasode 9y ago>, kernel developers still use email because it is faster than any of the alternatives. Over the course of the last year, the project accepted about eight changes per hour — every hour — from over 4,000 developers sponsored by over 400 companies Compare with Google's ~1875-per-hour from 2015[1]: >, and Google’s 25,000 engineers make about 45,000 commits (changes) to the repository each day. Obviously, we can't extrapolate that Google should switch to an internal SMTP project management system layered on top of Perforce because "email is faster than any of the alternatives". Same for other companies with high commit rates to their source repos like Microsoft and Facebook. Since Linus & Linux community never did an A-vs-B experiment of email vs open-source-JIRA-clone as a project workflow, I think we can only safely say that email works best for a disparate group of volunteers with different employers. Email is the lowest common denominator that everyone could agree to use. You couldn't get 4000 opinionated developers to agree to use an open-source JIRA-clone or MS-Team-Foundation-clone or some other ALM[2]-in-the-cloud. Therefore, listing advantages of email clients such as "threaded tree views" etc is actually hiding the underlying reason email-as-workflow works for Linux. Email isn't about "throughput"; it's "acceptance". [1] https://www.wired.com/2015/09/google-2-billion-lines-codeand-one-place/ https://www.wired.com/2015/09/google-2-billion-lines-codeand... [2] https://en.wikipedia.org/wiki/Application_lifecycle_management https://en.wikipedia.org/wiki/Application_lifecycle_manageme...
- walkingolof 9y agoIts not comparing the same thing I think, commits to a repo and diffs sent over email, and even if it was, its not really the same thing anyway... Kernel development seems to progress fine, I'm pretty sure if there was a problem, Linus would have made some effort to change it, just like the whole source control thing that ended up becoming Git.
- jasode 9y ago>Kernel development seems to progress fine, I'm not claiming that there's a bottleneck in kernel commits. (I'd guess human code reviews checking quality is the ultimate bottleneck.) I'm saying that using emails as "building blocks to reconstruct an pseudo ALM" instead of using an actual special-purpose ALM -- is the only realistic workflow that would be accepted by the disparate Linux community who don't work for a single employer. (If you're a single company employing all 4000 of Linux kernel programmers, you could impose an ALM tool be mandatory for code reviews, assigning work, tagging bug reports, etc.) Discussing positive bullet points of email ... even though each bullet point is true actually obscures the underlying reason the devs won't use an ALM.
- uiri 9y agoIronically enough, Google actually does use Gerrit for their own kernel development.
- jwilk 9y agoWhat's ironic about it?
- kashyapc 9y agoI split my time between Gerrit-based and email-based projects. While I appreciate Gerrit for its benefits (like the excellent CI integration), I instinctively tend to gravitate towards my email-based workflow that is blazing fast, allows me to work completely offline (once OfflineIMAP fetches email in the morning), and lets me process tons of email / patches. The said offline workflow (setup with: Mutt, OfflineIMAP, Postfix for local mail queueing, and Notmuch for fast indexing / searching / tagging) gives me unalloyed joy compared to the clickety-clicky, RSI-inducing pain that is Gerrit. The problems listed in the below email, by Dan Berrangé, about Gerrit are still a bloody pain in the neck for me: https://lists.nongnu.org/archive/html/qemu-devel/2014-02/msg00232.html https://lists.nongnu.org/archive/html/qemu-devel/2014-02/msg... [I learnt 2 days ago (can't find the URL to it; typing from my phone) that Gerrit now allows you to respond via email (strangely the announcement only mentioned the dreaded HTML email). I'm a bit cautious to say "sounds promising" without trying it yet.]
- unlmtd1 9y agoEasy: because using a webapp to track issues sucks donkey balls, and matrix still has no threading and good terminal client.
- m-j-fox 9y agoWhat I want to know is: which of Tower, GitKraken, GitHub Desktop or Stash do the kernel developers prefer?
- pritambaral 9y agoDoubt they'd prefer any of those. None of those are open source, every single one is a GUI app (which has a history of limiting access to the plumbing that can be provided), and some of those don't even run on Linux.
- slrz 9y agoYes, and do they use Outlook or the Yahoo webmail interface to compose their patches?
- shmerl 9y agoI can understand it for the kernel. What about Debian bugs?
- zozo123 9y agoBecause you can read email from the command line.
- Iv 9y agoI think a crucial point is missed: email does not have a single point of failure. If you go through a website, a server downtime halts the development. If the mailing list crashes, patches can still be sent and some discussions kind of continue until it is resolved. Individual email servers may fail, but it is easy to work around it. Many people have several email and other channels through which to be reached. Also, a very important point that Linus explained was at the core of some design decisions for git: it works offline. You have your email archive offline so if you take a plan, are at a conference in a place with no internet (does still exist, sadly), are on the road, you can still go through your archives and the patches you have to examine are still available.