14 ms·

Why kernel development still uses email

294 points by dankohn1 10y ago

deleted 10y ago
[deleted]
ncouture 10y ago
I like to think that Linux kernel developers master most of the tools they use, especially maintainers who have power to accept patches. From my own perspective and in different contexts, I can tell that email can be as efficient as it can be inefficient and that depends mostly on its users.
drewg123 10y ago
When I was at Google, gerrit was indeed used internally. Things may have changed in the last 18months, but moving off the old tool to Gerrit was such a huge project that I'd be surprised if they'd moved on to something else.
- xjia 10y ago
  +1. AFAIK Gerrit is still being used for kernel-related projects.
na85 10y ago
I find the idea that a technology should be replaced simply because it is old to be very irksome. Those developers out there blindly contributing to the trend of replacing "tried and true" with "new and shiny" really need to take a step back and gain some perspective. Email works great for the vast, vast majority of users.
- Klathmon 10y ago
  While I agree, the opposite is also an issue. Developers which either look down upon anything new as "pointless" or in some cases actively try to stop improvements. Looking at the kernel dev system, it works great, but it has its share of problems. I've never even thought about contributing to the kernel because of the work involved to just submit a patch, that process can be improved. Making it easier for newcomers, and updating some of the workflow to be more aligned with current tools could really help.
  - taeric 10y ago
    I didn't realize the kernel had a large process to submit. Any particular part scare you off?
  - userbinator 10y ago
    I think it's a deliberate barrier to entry: they're basically saying "if you can't be bothered to learn how to do it, we don't want your contributions." Linus occasionally rants about some of the crap that gets in, and I have a feeling he certainly doesn't want more, which would happen if the barrier was lowered. Here's a somewhat recent example: http://lkml.iu.edu/hypermail/linux/kernel/1404.0/01331.html http://lkml.iu.edu/hypermail/linux/kernel/1404.0/01331.html
    - digi_owl 10y ago
      I really do wonder what it is with Sievers. The handing of udev went into a nosedive after GregKH handed him the maintainership. Yet GregKH seems to trust him completely and Torvalds for some reason trust GregKH. It really makes me worry about the future of the kernel once Torvalds decides that he has had enough.
    - qb45 10y ago
      Probably you are right, though in the case you linked the barrier was clearly insufficient. And OTOH, I have seen many cases where actually sane one-time contributors had to spend unreasonable time changing trivial patches to match the guidelines. [RFC PATCH] cmdline: Hide "debug" from /proc/cmdline It has come to our attention that a system running a specific user space init program will not boot if you add "debug" to the kernel command line. Lol. I sometimes get disgusted by the kind of crap people merge, but it's nice that at least this one got the ridicule it deserved.
  - Manishearth 10y ago
    I will note that contributing to GitHub is pretty much just as complicated, and requires just as many steps and git commands. The difference is that folks are used to contributing to GitHub. And that's an important difference. I feel that projects should allow newcomers to contribute through multiple channels. For tiny patches from newcomers, it doesn't really hurt to let them make patches on github, go through review (without annoying other folks with nitfix review emails), and mail the final patch to the list for them. Of course, as they start to contribute more, nudge them towards the "real" patch process. This moves them towards contributing the regular way without the process being one more barrier in the way of entry. I've mentioned this in http://manishearth.github.io/blog/2016/01/03/making-your-open-source-project-newcomer-friendly/ http://manishearth.github.io/blog/2016/01/03/making-your-ope... -- you can tailor your process for newcomers. Your current process is probably great for power users but might be complicated for newcomers, but at the same time a lot of what makes the process great really isn't necessary for the patches that most newcomers contribute. Of course, all this works when the project actually cares about helping newcomers. As mentioned in the other comments, it seems like Linux doesn't care about this; it prefers to restrict itself to contributors who can (and want to) figure it out on their own. This could end up filtering out smart people who could have figured it out (but found better uses of their time), but that's their choice. Not to say that projects which don't do this don't care about this. GDB has a pretty complicated process, but they're also very helpful to newcomers. I might work on providing an alternate workflow or a mentoring process at some point. Firefox is similar; the bugzilla-based workflow can be new and tricky, but there's mentorship. (Another thing I may work on improving at some point)
- deckard1 10y ago
  Web development is the absolute worst at this. None of the technology is new, whatsoever (mostly stealing from Lisp, rebranding it, and doing it wrong or calling MVC by yet-another name). It's impossible to quantitatively state that the new tools we use today are any better than the tools we just tossed out yesterday. Because no one became a master at the tools of yesterday. There is no basis for comparison. No expertise. Just the eternal trend following.
  - madshiva 10y ago
    So damn true!
taeric 10y ago
The number of systems we have created to try and move collaboration off of email is hilariously large. Between email and lisp, is there any other system that you can find in any sufficiently large system?
- willismichael 10y ago
  Also hilarious is the fact that every single one of them will provide email notification when you're offline.
- brandmeyer 10y ago
  > Between email and lisp, is there any other system that you can find in any sufficiently large system? Make. How many systems try really really hard to just update the things that need to be changed in an output store from a different input store using a serial number? Nah, just use the date. That's almost like a serial number, isn't it? Except for those handful of things for which we couldn't be bothered to set up the dependencies correctly - those get updated every time. It won't be that expensive to refresh just a handful every run...
DSMan195276 10y ago
I think it's worth adding, in the age of git sending patches by email is easier then ever - it definitely shows that git was developed by Linus with the kernel's development in mind. There is literally a `git format-patch` command which spits out a full formatted email (or emails) of your various commits, which can then be sent to a mail client to be edited and sent out. Or even easier, just use git to send the patches as email direct via `git send-email`. The harder part is touched on in the article - email clients and email servers. All the patches sent to the mailing-list are plain text, and lots of clients or servers either mess with the plain text before sending or after receiving. And with that they also take some configuring to avoid sending attachments as base64 - which will be rejected by the kernel. Gmail thankfully doesn't mess anything up on the sever side, but the web client wraps your email to 80 characters per line making it unusable. Because of the above issues, using a separate mail client is essentially required, and setting them up can be annoying and there aren't really tons of them to choose from. I personally use `mutt`, but I can definitely understand why some wouldn't want to bother.
- bluedino 10y ago
  Outlook loves messing with messages. "Detecting and removing" extra line breaks is one of its features
  - digi_owl 10y ago
    Ah, MS And their hatred of line breaks...
  - DSMan195276 10y ago
    I remember reading that when Microsoft started contributing some code to the Linux kernel (to support Hyper-V mostly) they had to run their own Linux server in house so they could send properly formatted emails for their patches, since they had no way to send email that wouldn't be messed-up by their Outlook servers.
    - qb45 10y ago
      Not sure. I have successfully submitted patches to Linux using Outlook and Exchange. All it took was going through configuration and disabling some features which intuitively looked like they may mangle my emails.
blaisio 10y ago
So I get and agree with what he's saying about how GitHub duplicated email, but as a millennial who uses Vim, I find GitHub much more intuitive and useful than sending patches via email. In general, I disagree with the issues he has with GitHub - most of them are wishy washy disagreements that could easily be solved by adding things to GitHub or using a chrome extension. IMO the only real, logical reason they need to keep using email is "Email works just as well as GitHub, and we've already been using email, so we might as well continue using it." Which is a good reason on it's own - there's no reason to try and come up with problems with GitHub. (Again, not saying GitHub is perfect, but I don't think the reasons he gave were particularly valid.)
- viraptor 10y ago
  > that could easily be solved by adding things to GitHub or using a chrome extension I don't believe "adding things to GitHub" is easy unless you're the GitHub product manager. Also requiring some extensions would mean auto-verifying all the requirements on submitted PRs and half of them ending up with "redo it our way and submit again". Not impossible, but likely annoying to contributors and spammy for anyone receiving PRs.
- vasilia 10y ago
  Just try to use GitHub Enterprise for 400 or more users. You get a bare metal image of the virtual machine with no replication, load balancing, etc. Nothing more for a huge price. I don't know how they scaling public version.
  - cm3 10y ago
    FWIW, Gitlab now has DC/OS (Mesosphere) support https://about.gitlab.com/2016/09/16/announcing-gitlab-and-mesosphere/ https://about.gitlab.com/2016/09/16/announcing-gitlab-and-me...
- michaelmrose 10y ago
  I don't really feel that you being a millennial or using vim is relevant. I have no idea how you would go about getting github to implement whatever functionality is desired and even so would come with the following disadvantages -- Requiring a chrome extension limiting people to using chrome + a singular website vs any email client under the sun -- wouldn't work offline -- Presumably it would be more challenging for sight impaired individuals to use a js heavy website vs their email. They stated they had existing contributors who were so impaired. -- Since people still presumably do a lot of discussion via email it would mean using at minimum github AND email some of which would refer back and forth to other discussions. -- Relying on github would require tying free software infrastructure to a single non free software tool. Recalling bitkeeper this didn't go well last time. -- Hosted github costs additional money. -- The primary and perhaps only benefit would be making it easier for people to come in off the street and drop patches with minimal effort but this would actually cost effort to manage and the kind of individual who can't or likely wont figure out how to use existing infrastructure may not be an optimal contributor in any case. Basically your solution involves not only effort to create tools that don't exist, the cooperation of a third party, but the end result would almost certainly be objectively worse in a number of ways. I feel like you didn't really address any of the things the actual expert on the matter said and just hand waved his entire argument away.
notacoward 10y ago
As always, it's worth pointing out that this is just about the Linux kernel. Different scale, different developer community, different code structure, different testing/CI methodology (i.e. practically none) than just about every other project out there. Email works for them - great. Other things work for other people - also great. Theirs is one interesting data point, but that's all it is.
- catern 10y ago
  >As always, it's worth pointing out that this is just about the Linux kernel. Different scale, different developer community, different code structure, different testing/CI methodology (i.e. practically none) than just about every other project out there. Eh? Patches on mailing lists is the "traditional" way for open source projects to operate. Many projects "still" do things this way - indeed I wish more projects did, for the reasons outlined in this article!
  - notacoward 10y ago
    You're trying to refute a much stronger claim than I actually made. I was pointing out these differences to explain why what's best for the Linux kernel might not be best for another project. I wasn't saying email is bad, that it couldn't be the best for any other project, etc. The space of projects for which email is a good choice might be larger than just the Linux kernel, but it's much much smaller than the whole space of open-source projects as well. Given this community's tendency to cargo-cult whatever the highest-profile projects do without thinking about how different circumstances might affect applicability, I thought it was worth highlighting.
deleted 10y ago
[deleted]
ludamad 10y ago
"Some systems, like Outlook, will uniformly corrupt patches; as a result, companies doing kernel development tend to keep a Linux machine that they can use to send patches in a corner somewhere." Isn't a Linux desktop also convenient for the development itself?
- technion 10y ago
  I developed a series of patches for the RPi kernel fork. I use the Pi headless, and generally access it from a Windows machine. I never tried to send those patches upstream - and the Pi fork uses Github for such things - but I'm entirely confident that if I tried to paste those patches into an email client somewhere along the line, I'd get an upstream maintainer pointing out that magical line breaks blew something up somewhere.
  - aisofteng 10y ago
    What I read is "I am entirely confident about what would happen if I tried to do something I've never tried to do", which is useless speculation.
    - ludamad 10y ago
      What I read is that he develops kernel patches entirely on a Windows desktop, which is a relevant anecdote.
nxc18 10y ago
Email is an amazing tool. I don't understand the rush to get rid of it. It is a free standard with many implementations on both the client and server. It is exceptionally easy to manage email. Many servers have great systems for categorizing emails and storing them long-term. Email is fast. Everyone has one. There is this weird myth that Email magically kills productivity while Slack and GitHub don't. I never understood this - slack and GitHub have a lot more distraction with their social media components. I see the day coming soon when in order to work with other people I'll need to find and install one of the x tens of IM clients. The day when I can't find anything because it is in a million places and its real hard to search 10 years of conversation when messages don't have basic metadata like a subject. :(
- digi_owl 10y ago
  The pups always goes for the new shiny (get off my lawn)...
- dredmorbius 10y ago
  Email is tremendously useful, and whatever replaces it should borrow from those useful bits. It also has numerous problems. Some are listed in the article: 1. Clients (and some servers) mangle email. You can't be assured that what you send is what someone else will receive. I blame the clients and servers here, but ... 2. Privacy. This has become among the biggest downsides. Whether it's data-in-flight or data-at-rest, email remains an exceptionally poor choice for private communications, and there's been little real progress on addressing this, in large part because of vendor reluctance at both client and server / system level. E.g., Microsoft, Google, Yahoo, and AOL. 3. Users. Systems which work well with email rely strongly on compliance by users, which relies strongly on enforcement by communications managers or moderators. For LKML, that would be the kernel mailing list etiquette. Battles over header formats, quoting styles, and quote before vs. after exist because these strongly impact workflow and efficiency of groups as a whole. Failure to acculturise hampers the group, and acculturisation rates limit group size and growth. Much of Linus's noted "personality problems" come from the task of having to corral a group of people over which he has little if any formal control, a decidedly limited-bandwidth social communications channel (e.g., no tone, expression, voice, facial gestures, etc. signals), and the necessity TO BE UNAMBIGUOUSLY CLEAR when someone is transgressing norms. (Yes, I'm aware of HN's norms on ALL CAPS.) 4. File formats. Both a blessing and a curse: you can send anything. This means any given group can specify its preferences, but also raises the problem that there are few global preferences given. Kernel development works well with email by having specific tools and formats, from coding style to patch and git tools which create workable code fragments, to conventions for list discussion. This is strongly linked to #3 above. 5. The directory. In thinking about how email compares with other proposed alternatives, one key is that email lacks the concept of a central directory. Or at least, not a formal one (spammers may trade in same). On Twitter, or Facebook, or HN, every individual user is uniquely identified within a global directory. In Email, by contrast, what exist are addresses, which is to say, <user>@<domain>, where the domain has its own sub-directory. But there's no guarantee that domain1 would know that user1 is some valid directory entry at domain2. At best, domain1 knows how to reach domain2, and attempt delivery, via DNS, generally though not necessarily through MX records. This becomes more of a problem with ... 6. Spam and spoofing. Because there's no global directory, the validity of an email from <user2>@<domain2> cannot be assessed. In fact, there's no requirement that that even be a valid address, though as a convention it's generally considerate and useful. There've been numerous hacks to add increasing levels of authentication to email (SPF, DKIM), but the protocols remain weak. There's a much broader problem of metadata leakage. Deriving from this: 7. Reputation. Much of the spam problem derives from poor tools and support for establishing, assessing, sharing, and acting on the reputation of specific senders -- whether individual originators, email hosts, domains, or IP ranges. Changes to how the Internet is being used, including very large aggregated email providers (e.g., Google, Microsoft, Yahoo, AOL), and point-of-origin masking systems such as Tor make formerly useful concepts of IP and domain-based reptuations of limited reliability. And a consequence is that the assurance of getting messages through to any one person are becoming quite low. The more users are added to a system, the higher the noise level. Email was designed for a friendly, unencrypted, lightly used system of mostly known and fairly trusted users numbering in the thousands. It's scaled remarkably well, by roughly six orders of magnitude. But it's become exceptionally creaky.
wildchild 10y ago
What to use instead? Blockchain?
rpedela 10y ago
For Github organizations, can't you forward notifications to a mailing list? Then everyone on the mailing list can see everyone's comments and respond through email but you still get to keep Github and all of its advantages.
- Manishearth 10y ago
  Or people can just subscribe to a repo. Github provides pretty decent tools for filtering these via mail filters, too (sadly the on-site github notifications aren't as fine grained). GKHs assertion that you can't follow threads via email is false.
begriffs 10y ago
Are there any good references online about how to switch from Github-based collaboration to one based on email and patches? I'd like to give it a try and see how it feels.
- gkya 10y ago
  Put mailman/majordomo and cgit/gitweb on a server, and a little front page for the project if you like. Use spamassassin with mailman, restrict allowed mail mime types to plain text and patches. Maybe use one of those tools that pick patches up from your mbox. Quilt is a good patch manager. I'd use emacs org mode for issues and todos.
icc97 10y ago
> "On the other hand, GitHub does not scale to larger projects. He pointed at the Kubernetes project" It seems according to this, that Github is fine for a project until it scales to such a size as you need an alternative. So even this is agreeing that small projects should start with Github. It would be useful to know the lower bound of users/developers at which point you need to switch to a more flexible system like email.
- stephenr 10y ago
  s/should/could/
sambe 10y ago
Email is great and I can see why they continue to use it, but there is a danger of being blinkered. Saying that GitHub reinvented email because they have comment threads is... almost trolling. Especially when also saying email integrates well with patchwork (presumably written due to the inability of email to handle issue/PR tracking). Both have strengths and weaknesses.
csl 10y ago
Is Gerrit really that terrible? I do agree that clicking through each file is a waste, but I love the ability to work on several different reviews and keep track of their different build statuses. Also, where I use it we have different levels of requirements for different branches (more people usually need to approve reviews on soon-to-be-released point branches). We use it for a large project in MLOCs but relatively few developers (less than a hundred). I can't see it would work for 4000 devs at all, though.
- Too 10y ago
  Could you elaborate why you think gerrit wouldn't work for 4000 devs? At that scale the project would be divided into smaller responsibility areas essentially operating on their own anyway.
  - csl 10y ago
    Right, you could do that and probably do it well. I was mostly thinking about the simple matter of scaling the resources and maintenance crew to tackle 4000 devs. But my biggest concern about it is that it introduces centralization into the workflow. I do truly love Gerrit, but I get a bit worried about keeping a lot of important stuff in a centralized and not so transparent database.
- adrianratnapala 10y ago
  Perhaps I don't understand the Gerrit workflow, but last time I read about Gerrit it looked like it had a philosophy of "one commit, one review". That sounds like an anethema for the kernel, where they want to review a series of tidied up commits, one per email.
  - kiallmacinnes 10y ago
    Gerrit does have "one commit, one review", but - that's not a bad thing. If a change is well written and structured, merging the first in the chain without the second+ is both harmless and is progress towards the end goal. There's also patch chains in Gerrit, so you can see the X commits that make up the full change, and the second+ cannot be merged before the first.
Daviey 10y ago
The article totally missed the mark on Gerrit. I think the heavy user is now OpenStack and not Android... The article claims that Gerrit is just a small part, but if used fully it really isn't. It can be used to track the entire patch lifecycle. Which LKML doens't do via email.. because it isn't very good at it... it uses 'patchwork'. OpenStack uses Gerrit to manage 537 git repositories. In the last year, 86,027 changesets were merged in the last year out of 111,292 commits. [0] The real values are the smart searching capability (with email subscription to that search),integration with CI, inline code reviews.. simple rebasing, acls and easy digestion who reviewed. Once setup, the workflow is super simple: $ git commit $ git review $ # celebrate success Further, why all the hate on project managers? [0] http://activity.openstack.org/dash/browser/ http://activity.openstack.org/dash/browser/
known 10y ago
Email solves https://en.wikipedia.org/wiki/Producer%E2%80%93consumer_problem https://en.wikipedia.org/wiki/Producer%E2%80%93consumer_prob... AND http://www.beej.us/guide/bgipc/output/html/singlepage/bgipc.html#fifopc http://www.beej.us/guide/bgipc/output/html/singlepage/bgipc....
exDM69 10y ago
tl;dr: my wish is a code review convention for git-notes and a friendly UI on top. If only Git would allow storing reviews and issues in the repo proper. Something like Git notes with conventions to refer to files/lines in a file/different revisions etc. Something similar to Fossil. This would solve half of the problem. The other half would be the UI layer. Now the reviews live in a database, outside of the repo, and can't be downloaded locally for offline work. This would allow doing code reviews and managing issues e.g. while commuting in a train. And everything would be pretty fast compared to waiting for a web page to load for every file you want to review and connecting to a server for every comment you write.
- erikb 10y ago
  Why does email not solve your problem? It allows offline work, you can store the reviews on your computer, you can even commit it to git if you want. Also check out emacs' org-mode for an alternative approach. And last but not least, does it just sound like you value the review higher than the code? Why should it be more than a text file like any other? Meaning is something we add to the text. A tool with a specific goal like git shouldn't enhance a meaning to an object type.
  - exDM69 10y ago
    The review comments still live outside of the repo, in an unformatted ad-hoc format that's not easy to write a nice UI for. Ideally, I should be able to add review comments in a graphical diff viewer or web front-end and every dev should be able to choose the tools they prefer. Email workflow would solve the connectivity issues, but the reviews, issues etc are not stored for posterity along with the code. Reviews don't need to be much more than text, but they need some metadata to refer to files/revisions/lines. Basically this could be solved with some conventions on top of text files that machines could understand. What I'm after is something like Fossil, where all the project management can be done offline and in a structured way.
    - erikb 10y ago
      Fossil looks quite interesting. Thanks for the reference. https://www.fossil-scm.org/index.html/doc/trunk/www/concepts.wiki https://www.fossil-scm.org/index.html/doc/trunk/www/concepts...
cm3 10y ago
For continued decentralized operations something like https://github.com/google/git-appraise https://github.com/google/git-appraise looks good. It's like Fossil-SCM's distributed tickets but for code review. It can also mirror to Gihtub and Phabricator and has a web interface too. Most importantly it doesn't break the decentralized aspect of git.
dschiptsov 10y ago
Because it ain't broken.)
crististm 10y ago
I don't have anything against e-mail but I don't get what's wrong with Gerrit either [1]. We used it on several projects and it was pretty transparent to the work flow. [1]except for some Java gc issues that required the process to be restarted every now and then - pretty annoying though
piotrkaminski 10y ago
Amusingly, the article references Kubernetes and its growing pains a couple times, but fails to mention that it adopted Reviewable (on top of GitHub) for doing more complicated code reviews. I don't know if Reviewable would scale to kernel development -- and it certainly doesn't support offline work, etc. -- but at least Kubernetes seems to be doing all right with it. (Disclosure: I'm the founder of Reviewable.)
lucaspottersky 10y ago
Simple: humans don't like change. We say we do like it, but the truth is that we don't.
u801e 10y ago
One thing I've always wondered is why email based patch submission and review was never done via a newsgroup over NNTP. One advantage a newsgroup has over an email list is that you don't have to set up filters in your email client, and that every message posted doesn't have to be specifically CC'd to a certain group of people. Also, it's much easier to browse past posts. Currently this can be done via the gmane NNTP gateway for the kernel and git mailing lists, but I can't see a good reason why a newsgroup wouldn't be preferred over a mailing list.

294 points by dankohn1 10y ago

deleted 10y ago

[deleted]

I like to think that Linux kernel developers master most of the tools they use, especially maintainers who have power to accept patches. From my own perspective and in different contexts, I can tell that email can be as efficient as it can be inefficient and that depends mostly on its users.

drewg123 10y ago

When I was at Google, gerrit was indeed used internally. Things may have changed in the last 18months, but moving off the old tool to Gerrit was such a huge project that I'd be surprised if they'd moved on to something else.

xjia 10y ago
+1. AFAIK Gerrit is still being used for kernel-related projects.

na85 10y ago

I find the idea that a technology should be replaced simply because it is old to be very irksome. Those developers out there blindly contributing to the trend of replacing "tried and true" with "new and shiny" really need to take a step back and gain some perspective. Email works great for the vast, vast majority of users.

Klathmon 10y ago
While I agree, the opposite is also an issue. Developers which either look down upon anything new as "pointless" or in some cases actively try to stop improvements. Looking at the kernel dev system, it works great, but it has its share of problems. I've never even thought about contributing to the kernel because of the work involved to just submit a patch, that process can be improved. Making it easier for newcomers, and updating some of the workflow to be more aligned with current tools could really help.
- taeric 10y ago
  I didn't realize the kernel had a large process to submit. Any particular part scare you off?
- userbinator 10y ago
  I think it's a deliberate barrier to entry: they're basically saying "if you can't be bothered to learn how to do it, we don't want your contributions." Linus occasionally rants about some of the crap that gets in, and I have a feeling he certainly doesn't want more, which would happen if the barrier was lowered. Here's a somewhat recent example: http://lkml.iu.edu/hypermail/linux/kernel/1404.0/01331.html http://lkml.iu.edu/hypermail/linux/kernel/1404.0/01331.html
  - digi_owl 10y ago
    I really do wonder what it is with Sievers. The handing of udev went into a nosedive after GregKH handed him the maintainership. Yet GregKH seems to trust him completely and Torvalds for some reason trust GregKH. It really makes me worry about the future of the kernel once Torvalds decides that he has had enough.
  - qb45 10y ago
    Probably you are right, though in the case you linked the barrier was clearly insufficient. And OTOH, I have seen many cases where actually sane one-time contributors had to spend unreasonable time changing trivial patches to match the guidelines. [RFC PATCH] cmdline: Hide "debug" from /proc/cmdline It has come to our attention that a system running a specific user space init program will not boot if you add "debug" to the kernel command line. Lol. I sometimes get disgusted by the kind of crap people merge, but it's nice that at least this one got the ridicule it deserved.
- Manishearth 10y ago
  I will note that contributing to GitHub is pretty much just as complicated, and requires just as many steps and git commands. The difference is that folks are used to contributing to GitHub. And that's an important difference. I feel that projects should allow newcomers to contribute through multiple channels. For tiny patches from newcomers, it doesn't really hurt to let them make patches on github, go through review (without annoying other folks with nitfix review emails), and mail the final patch to the list for them. Of course, as they start to contribute more, nudge them towards the "real" patch process. This moves them towards contributing the regular way without the process being one more barrier in the way of entry. I've mentioned this in http://manishearth.github.io/blog/2016/01/03/making-your-open-source-project-newcomer-friendly/ http://manishearth.github.io/blog/2016/01/03/making-your-ope... -- you can tailor your process for newcomers. Your current process is probably great for power users but might be complicated for newcomers, but at the same time a lot of what makes the process great really isn't necessary for the patches that most newcomers contribute. Of course, all this works when the project actually cares about helping newcomers. As mentioned in the other comments, it seems like Linux doesn't care about this; it prefers to restrict itself to contributors who can (and want to) figure it out on their own. This could end up filtering out smart people who could have figured it out (but found better uses of their time), but that's their choice. Not to say that projects which don't do this don't care about this. GDB has a pretty complicated process, but they're also very helpful to newcomers. I might work on providing an alternate workflow or a mentoring process at some point. Firefox is similar; the bugzilla-based workflow can be new and tricky, but there's mentorship. (Another thing I may work on improving at some point)
deckard1 10y ago
Web development is the absolute worst at this. None of the technology is new, whatsoever (mostly stealing from Lisp, rebranding it, and doing it wrong or calling MVC by yet-another name). It's impossible to quantitatively state that the new tools we use today are any better than the tools we just tossed out yesterday. Because no one became a master at the tools of yesterday. There is no basis for comparison. No expertise. Just the eternal trend following.
- madshiva 10y ago
  So damn true!

taeric 10y ago

The number of systems we have created to try and move collaboration off of email is hilariously large. Between email and lisp, is there any other system that you can find in any sufficiently large system?

willismichael 10y ago
Also hilarious is the fact that every single one of them will provide email notification when you're offline.
brandmeyer 10y ago
> Between email and lisp, is there any other system that you can find in any sufficiently large system? Make. How many systems try really really hard to just update the things that need to be changed in an output store from a different input store using a serial number? Nah, just use the date. That's almost like a serial number, isn't it? Except for those handful of things for which we couldn't be bothered to set up the dependencies correctly - those get updated every time. It won't be that expensive to refresh just a handful every run...

DSMan195276 10y ago

I think it's worth adding, in the age of git sending patches by email is easier then ever - it definitely shows that git was developed by Linus with the kernel's development in mind. There is literally a `git format-patch` command which spits out a full formatted email (or emails) of your various commits, which can then be sent to a mail client to be edited and sent out. Or even easier, just use git to send the patches as email direct via `git send-email`. The harder part is touched on in the article - email clients and email servers. All the patches sent to the mailing-list are plain text, and lots of clients or servers either mess with the plain text before sending or after receiving. And with that they also take some configuring to avoid sending attachments as base64 - which will be rejected by the kernel. Gmail thankfully doesn't mess anything up on the sever side, but the web client wraps your email to 80 characters per line making it unusable. Because of the above issues, using a separate mail client is essentially required, and setting them up can be annoying and there aren't really tons of them to choose from. I personally use `mutt`, but I can definitely understand why some wouldn't want to bother.

bluedino 10y ago
Outlook loves messing with messages. "Detecting and removing" extra line breaks is one of its features
- digi_owl 10y ago
  Ah, MS And their hatred of line breaks...
- DSMan195276 10y ago
  I remember reading that when Microsoft started contributing some code to the Linux kernel (to support Hyper-V mostly) they had to run their own Linux server in house so they could send properly formatted emails for their patches, since they had no way to send email that wouldn't be messed-up by their Outlook servers.
  - qb45 10y ago
    Not sure. I have successfully submitted patches to Linux using Outlook and Exchange. All it took was going through configuration and disabling some features which intuitively looked like they may mangle my emails.

blaisio 10y ago

So I get and agree with what he's saying about how GitHub duplicated email, but as a millennial who uses Vim, I find GitHub much more intuitive and useful than sending patches via email. In general, I disagree with the issues he has with GitHub - most of them are wishy washy disagreements that could easily be solved by adding things to GitHub or using a chrome extension. IMO the only real, logical reason they need to keep using email is "Email works just as well as GitHub, and we've already been using email, so we might as well continue using it." Which is a good reason on it's own - there's no reason to try and come up with problems with GitHub. (Again, not saying GitHub is perfect, but I don't think the reasons he gave were particularly valid.)

viraptor 10y ago
> that could easily be solved by adding things to GitHub or using a chrome extension I don't believe "adding things to GitHub" is easy unless you're the GitHub product manager. Also requiring some extensions would mean auto-verifying all the requirements on submitted PRs and half of them ending up with "redo it our way and submit again". Not impossible, but likely annoying to contributors and spammy for anyone receiving PRs.
vasilia 10y ago
Just try to use GitHub Enterprise for 400 or more users. You get a bare metal image of the virtual machine with no replication, load balancing, etc. Nothing more for a huge price. I don't know how they scaling public version.
- cm3 10y ago
  FWIW, Gitlab now has DC/OS (Mesosphere) support https://about.gitlab.com/2016/09/16/announcing-gitlab-and-mesosphere/ https://about.gitlab.com/2016/09/16/announcing-gitlab-and-me...
michaelmrose 10y ago
I don't really feel that you being a millennial or using vim is relevant. I have no idea how you would go about getting github to implement whatever functionality is desired and even so would come with the following disadvantages -- Requiring a chrome extension limiting people to using chrome + a singular website vs any email client under the sun -- wouldn't work offline -- Presumably it would be more challenging for sight impaired individuals to use a js heavy website vs their email. They stated they had existing contributors who were so impaired. -- Since people still presumably do a lot of discussion via email it would mean using at minimum github AND email some of which would refer back and forth to other discussions. -- Relying on github would require tying free software infrastructure to a single non free software tool. Recalling bitkeeper this didn't go well last time. -- Hosted github costs additional money. -- The primary and perhaps only benefit would be making it easier for people to come in off the street and drop patches with minimal effort but this would actually cost effort to manage and the kind of individual who can't or likely wont figure out how to use existing infrastructure may not be an optimal contributor in any case. Basically your solution involves not only effort to create tools that don't exist, the cooperation of a third party, but the end result would almost certainly be objectively worse in a number of ways. I feel like you didn't really address any of the things the actual expert on the matter said and just hand waved his entire argument away.

notacoward 10y ago

As always, it's worth pointing out that this is just about the Linux kernel. Different scale, different developer community, different code structure, different testing/CI methodology (i.e. practically none) than just about every other project out there. Email works for them - great. Other things work for other people - also great. Theirs is one interesting data point, but that's all it is.

catern 10y ago
>As always, it's worth pointing out that this is just about the Linux kernel. Different scale, different developer community, different code structure, different testing/CI methodology (i.e. practically none) than just about every other project out there. Eh? Patches on mailing lists is the "traditional" way for open source projects to operate. Many projects "still" do things this way - indeed I wish more projects did, for the reasons outlined in this article!
- notacoward 10y ago
  You're trying to refute a much stronger claim than I actually made. I was pointing out these differences to explain why what's best for the Linux kernel might not be best for another project. I wasn't saying email is bad, that it couldn't be the best for any other project, etc. The space of projects for which email is a good choice might be larger than just the Linux kernel, but it's much much smaller than the whole space of open-source projects as well. Given this community's tendency to cargo-cult whatever the highest-profile projects do without thinking about how different circumstances might affect applicability, I thought it was worth highlighting.

deleted 10y ago

[deleted]

ludamad 10y ago

"Some systems, like Outlook, will uniformly corrupt patches; as a result, companies doing kernel development tend to keep a Linux machine that they can use to send patches in a corner somewhere." Isn't a Linux desktop also convenient for the development itself?

technion 10y ago
I developed a series of patches for the RPi kernel fork. I use the Pi headless, and generally access it from a Windows machine. I never tried to send those patches upstream - and the Pi fork uses Github for such things - but I'm entirely confident that if I tried to paste those patches into an email client somewhere along the line, I'd get an upstream maintainer pointing out that magical line breaks blew something up somewhere.
- aisofteng 10y ago
  What I read is "I am entirely confident about what would happen if I tried to do something I've never tried to do", which is useless speculation.
  - ludamad 10y ago
    What I read is that he develops kernel patches entirely on a Windows desktop, which is a relevant anecdote.

nxc18 10y ago

Email is an amazing tool. I don't understand the rush to get rid of it. It is a free standard with many implementations on both the client and server. It is exceptionally easy to manage email. Many servers have great systems for categorizing emails and storing them long-term. Email is fast. Everyone has one. There is this weird myth that Email magically kills productivity while Slack and GitHub don't. I never understood this - slack and GitHub have a lot more distraction with their social media components. I see the day coming soon when in order to work with other people I'll need to find and install one of the x tens of IM clients. The day when I can't find anything because it is in a million places and its real hard to search 10 years of conversation when messages don't have basic metadata like a subject. :(

digi_owl 10y ago
The pups always goes for the new shiny (get off my lawn)...
dredmorbius 10y ago
Email is tremendously useful, and whatever replaces it should borrow from those useful bits. It also has numerous problems. Some are listed in the article: 1. Clients (and some servers) mangle email. You can't be assured that what you send is what someone else will receive. I blame the clients and servers here, but ... 2. Privacy. This has become among the biggest downsides. Whether it's data-in-flight or data-at-rest, email remains an exceptionally poor choice for private communications, and there's been little real progress on addressing this, in large part because of vendor reluctance at both client and server / system level. E.g., Microsoft, Google, Yahoo, and AOL. 3. Users. Systems which work well with email rely strongly on compliance by users, which relies strongly on enforcement by communications managers or moderators. For LKML, that would be the kernel mailing list etiquette. Battles over header formats, quoting styles, and quote before vs. after exist because these strongly impact workflow and efficiency of groups as a whole. Failure to acculturise hampers the group, and acculturisation rates limit group size and growth. Much of Linus's noted "personality problems" come from the task of having to corral a group of people over which he has little if any formal control, a decidedly limited-bandwidth social communications channel (e.g., no tone, expression, voice, facial gestures, etc. signals), and the necessity TO BE UNAMBIGUOUSLY CLEAR when someone is transgressing norms. (Yes, I'm aware of HN's norms on ALL CAPS.) 4. File formats. Both a blessing and a curse: you can send anything. This means any given group can specify its preferences, but also raises the problem that there are few global preferences given. Kernel development works well with email by having specific tools and formats, from coding style to patch and git tools which create workable code fragments, to conventions for list discussion. This is strongly linked to #3 above. 5. The directory. In thinking about how email compares with other proposed alternatives, one key is that email lacks the concept of a central directory. Or at least, not a formal one (spammers may trade in same). On Twitter, or Facebook, or HN, every individual user is uniquely identified within a global directory. In Email, by contrast, what exist are addresses, which is to say, <user>@<domain>, where the domain has its own sub-directory. But there's no guarantee that domain1 would know that user1 is some valid directory entry at domain2. At best, domain1 knows how to reach domain2, and attempt delivery, via DNS, generally though not necessarily through MX records. This becomes more of a problem with ... 6. Spam and spoofing. Because there's no global directory, the validity of an email from <user2>@<domain2> cannot be assessed. In fact, there's no requirement that that even be a valid address, though as a convention it's generally considerate and useful. There've been numerous hacks to add increasing levels of authentication to email (SPF, DKIM), but the protocols remain weak. There's a much broader problem of metadata leakage. Deriving from this: 7. Reputation. Much of the spam problem derives from poor tools and support for establishing, assessing, sharing, and acting on the reputation of specific senders -- whether individual originators, email hosts, domains, or IP ranges. Changes to how the Internet is being used, including very large aggregated email providers (e.g., Google, Microsoft, Yahoo, AOL), and point-of-origin masking systems such as Tor make formerly useful concepts of IP and domain-based reptuations of limited reliability. And a consequence is that the assurance of getting messages through to any one person are becoming quite low. The more users are added to a system, the higher the noise level. Email was designed for a friendly, unencrypted, lightly used system of mostly known and fairly trusted users numbering in the thousands. It's scaled remarkably well, by roughly six orders of magnitude. But it's become exceptionally creaky.

wildchild 10y ago

What to use instead? Blockchain?

rpedela 10y ago

For Github organizations, can't you forward notifications to a mailing list? Then everyone on the mailing list can see everyone's comments and respond through email but you still get to keep Github and all of its advantages.

Manishearth 10y ago
Or people can just subscribe to a repo. Github provides pretty decent tools for filtering these via mail filters, too (sadly the on-site github notifications aren't as fine grained). GKHs assertion that you can't follow threads via email is false.

begriffs 10y ago

Are there any good references online about how to switch from Github-based collaboration to one based on email and patches? I'd like to give it a try and see how it feels.

gkya 10y ago
Put mailman/majordomo and cgit/gitweb on a server, and a little front page for the project if you like. Use spamassassin with mailman, restrict allowed mail mime types to plain text and patches. Maybe use one of those tools that pick patches up from your mbox. Quilt is a good patch manager. I'd use emacs org mode for issues and todos.

icc97 10y ago

> "On the other hand, GitHub does not scale to larger projects. He pointed at the Kubernetes project" It seems according to this, that Github is fine for a project until it scales to such a size as you need an alternative. So even this is agreeing that small projects should start with Github. It would be useful to know the lower bound of users/developers at which point you need to switch to a more flexible system like email.

stephenr 10y ago
s/should/could/

sambe 10y ago

Email is great and I can see why they continue to use it, but there is a danger of being blinkered. Saying that GitHub reinvented email because they have comment threads is... almost trolling. Especially when also saying email integrates well with patchwork (presumably written due to the inability of email to handle issue/PR tracking). Both have strengths and weaknesses.

csl 10y ago

Is Gerrit really that terrible? I do agree that clicking through each file is a waste, but I love the ability to work on several different reviews and keep track of their different build statuses. Also, where I use it we have different levels of requirements for different branches (more people usually need to approve reviews on soon-to-be-released point branches). We use it for a large project in MLOCs but relatively few developers (less than a hundred). I can't see it would work for 4000 devs at all, though.

Too 10y ago
Could you elaborate why you think gerrit wouldn't work for 4000 devs? At that scale the project would be divided into smaller responsibility areas essentially operating on their own anyway.
- csl 10y ago
  Right, you could do that and probably do it well. I was mostly thinking about the simple matter of scaling the resources and maintenance crew to tackle 4000 devs. But my biggest concern about it is that it introduces centralization into the workflow. I do truly love Gerrit, but I get a bit worried about keeping a lot of important stuff in a centralized and not so transparent database.
adrianratnapala 10y ago
Perhaps I don't understand the Gerrit workflow, but last time I read about Gerrit it looked like it had a philosophy of "one commit, one review". That sounds like an anethema for the kernel, where they want to review a series of tidied up commits, one per email.
- kiallmacinnes 10y ago
  Gerrit does have "one commit, one review", but - that's not a bad thing. If a change is well written and structured, merging the first in the chain without the second+ is both harmless and is progress towards the end goal. There's also patch chains in Gerrit, so you can see the X commits that make up the full change, and the second+ cannot be merged before the first.

Daviey 10y ago

The article totally missed the mark on Gerrit. I think the heavy user is now OpenStack and not Android... The article claims that Gerrit is just a small part, but if used fully it really isn't. It can be used to track the entire patch lifecycle. Which LKML doens't do via email.. because it isn't very good at it... it uses 'patchwork'. OpenStack uses Gerrit to manage 537 git repositories. In the last year, 86,027 changesets were merged in the last year out of 111,292 commits. [0] The real values are the smart searching capability (with email subscription to that search),integration with CI, inline code reviews.. simple rebasing, acls and easy digestion who reviewed. Once setup, the workflow is super simple: $ git commit $ git review $ # celebrate success Further, why all the hate on project managers? [0] http://activity.openstack.org/dash/browser/ http://activity.openstack.org/dash/browser/

known 10y ago

Email solves https://en.wikipedia.org/wiki/Producer%E2%80%93consumer_problem https://en.wikipedia.org/wiki/Producer%E2%80%93consumer_prob... AND http://www.beej.us/guide/bgipc/output/html/singlepage/bgipc.html#fifopc http://www.beej.us/guide/bgipc/output/html/singlepage/bgipc....

exDM69 10y ago

tl;dr: my wish is a code review convention for git-notes and a friendly UI on top. If only Git would allow storing reviews and issues in the repo proper. Something like Git notes with conventions to refer to files/lines in a file/different revisions etc. Something similar to Fossil. This would solve half of the problem. The other half would be the UI layer. Now the reviews live in a database, outside of the repo, and can't be downloaded locally for offline work. This would allow doing code reviews and managing issues e.g. while commuting in a train. And everything would be pretty fast compared to waiting for a web page to load for every file you want to review and connecting to a server for every comment you write.

erikb 10y ago
Why does email not solve your problem? It allows offline work, you can store the reviews on your computer, you can even commit it to git if you want. Also check out emacs' org-mode for an alternative approach. And last but not least, does it just sound like you value the review higher than the code? Why should it be more than a text file like any other? Meaning is something we add to the text. A tool with a specific goal like git shouldn't enhance a meaning to an object type.
- exDM69 10y ago
  The review comments still live outside of the repo, in an unformatted ad-hoc format that's not easy to write a nice UI for. Ideally, I should be able to add review comments in a graphical diff viewer or web front-end and every dev should be able to choose the tools they prefer. Email workflow would solve the connectivity issues, but the reviews, issues etc are not stored for posterity along with the code. Reviews don't need to be much more than text, but they need some metadata to refer to files/revisions/lines. Basically this could be solved with some conventions on top of text files that machines could understand. What I'm after is something like Fossil, where all the project management can be done offline and in a structured way.
  - erikb 10y ago
    Fossil looks quite interesting. Thanks for the reference. https://www.fossil-scm.org/index.html/doc/trunk/www/concepts.wiki https://www.fossil-scm.org/index.html/doc/trunk/www/concepts...

cm3 10y ago

For continued decentralized operations something like https://github.com/google/git-appraise https://github.com/google/git-appraise looks good. It's like Fossil-SCM's distributed tickets but for code review. It can also mirror to Gihtub and Phabricator and has a web interface too. Most importantly it doesn't break the decentralized aspect of git.

dschiptsov 10y ago

Because it ain't broken.)

crististm 10y ago

I don't have anything against e-mail but I don't get what's wrong with Gerrit either [1]. We used it on several projects and it was pretty transparent to the work flow. [1]except for some Java gc issues that required the process to be restarted every now and then - pretty annoying though

piotrkaminski 10y ago

Amusingly, the article references Kubernetes and its growing pains a couple times, but fails to mention that it adopted Reviewable (on top of GitHub) for doing more complicated code reviews. I don't know if Reviewable would scale to kernel development -- and it certainly doesn't support offline work, etc. -- but at least Kubernetes seems to be doing all right with it. (Disclosure: I'm the founder of Reviewable.)

lucaspottersky 10y ago

Simple: humans don't like change. We say we do like it, but the truth is that we don't.

u801e 10y ago

One thing I've always wondered is why email based patch submission and review was never done via a newsgroup over NNTP. One advantage a newsgroup has over an email list is that you don't have to set up filters in your email client, and that every message posted doesn't have to be specifically CC'd to a certain group of people. Also, it's much easier to browse past posts. Currently this can be done via the gmane NNTP gateway for the kernel and git mailing lists, but I can't see a good reason why a newsgroup wouldn't be preferred over a mailing list.