4 ms·

>, kernel developers still use email because it is faster than any of the alternatives. Over the course of the last year, the project accepted about eight chang

by jasode 9y ago

>, kernel developers still use email because it is faster than any of the alternatives. Over the course of the last year, the project accepted about eight changes per hour — every hour — from over 4,000 developers sponsored by over 400 companies Compare with Google's ~1875-per-hour from 2015[1]: >, and Google’s 25,000 engineers make about 45,000 commits (changes) to the repository each day. Obviously, we can't extrapolate that Google should switch to an internal SMTP project management system layered on top of Perforce because "email is faster than any of the alternatives". Same for other companies with high commit rates to their source repos like Microsoft and Facebook. Since Linus & Linux community never did an A-vs-B experiment of email vs open-source-JIRA-clone as a project workflow, I think we can only safely say that email works best for a disparate group of volunteers with different employers. Email is the lowest common denominator that everyone could agree to use. You couldn't get 4000 opinionated developers to agree to use an open-source JIRA-clone or MS-Team-Foundation-clone or some other ALM[2]-in-the-cloud. Therefore, listing advantages of email clients such as "threaded tree views" etc is actually hiding the underlying reason email-as-workflow works for Linux. Email isn't about "throughput"; it's "acceptance". [1] https://www.wired.com/2015/09/google-2-billion-lines-codeand-one-place/ https://www.wired.com/2015/09/google-2-billion-lines-codeand... [2] https://en.wikipedia.org/wiki/Application_lifecycle_management https://en.wikipedia.org/wiki/Application_lifecycle_manageme...

walkingolof 9y ago
Its not comparing the same thing I think, commits to a repo and diffs sent over email, and even if it was, its not really the same thing anyway... Kernel development seems to progress fine, I'm pretty sure if there was a problem, Linus would have made some effort to change it, just like the whole source control thing that ended up becoming Git.
- jasode 9y ago
  >Kernel development seems to progress fine, I'm not claiming that there's a bottleneck in kernel commits. (I'd guess human code reviews checking quality is the ultimate bottleneck.) I'm saying that using emails as "building blocks to reconstruct an pseudo ALM" instead of using an actual special-purpose ALM -- is the only realistic workflow that would be accepted by the disparate Linux community who don't work for a single employer. (If you're a single company employing all 4000 of Linux kernel programmers, you could impose an ALM tool be mandatory for code reviews, assigning work, tagging bug reports, etc.) Discussing positive bullet points of email ... even though each bullet point is true actually obscures the underlying reason the devs won't use an ALM.
  - wmf 9y ago
    It seems like that argument cuts both ways; what about developers who refuse to use the email-based process?
    - jasode 9y ago
      >; what about developers who refuse to use the email-based process? I guess I don't understand your question. In Linux, the email-as-the-workflow[1] has been agreed upon from Linux Torvald and his lieutenants all the way down. I suppose that if a one refused to use email-based-workflow, they'd have to find a Linux kernel maintainer to be a champion for their proposed changes and submit them on their behalf. (Is this a common scenario? I don't know.) If a rebel Linux kernel programmer insisted on staging changes for review in Gerritt/JIRA/github/etc, he's going to be isolated in an island because he's not part of the agreed-upon workflow: emails. [1] https://github.com/torvalds/linux/blob/master/Documentation/process/submitting-patches.rst#14-the-canonical-patch-format https://github.com/torvalds/linux/blob/master/Documentation/...
      - wmf 9y ago
        If Linus declared that everyone must use, say, Gitlab from now on, some Linux developers might leave. But developers who don't use email may never start working on Linux. That's why it cuts both ways: either option alienates some developers.
        walshemj 9y ago
        A developer that does not use mail :-) that is very unlikely
        dlp211 9y ago
        A developer that does not use mail for code submission.
codemac 9y ago
I got in this same discussion with gregkh on Reddit, and Torvalds in person. Linux development is much slower than many large software engineering firms, because the gatekeeper model has many more integrators and reviewers as a proportion of the members than the "pull request"/ team code review model Acceptance is certainly part of it, but 8 changes per hour is 8-10 email threads per hour, which is just not that hard to track as an individual, thus no need to change anything. Right now at work on a large (200ish) team, I get cc'd on about the same amount of patches.
abiox 9y ago
> Compare with Google isn't google's situation a monorepo composed of many distinct projects? there doesn't seem much utility in the comparison. > Email isn't about "throughput"; it's "acceptance". this seems to be the similar, in the other direction. using (or not) an alm in an organization isn't about "throughput", it's about "accepting" a company's policy assertions.
- jasode 9y ago
  >using (or not) an alm in an organization isn't about "throughput", it's about "accepting" a company's policy assertions. I don't think that equating it from the reverse direction works. Here's why: Using the article's count of ~4000 developers, let's consider all software companies that employ 4000+ developers as part of a "natural experiment"[1]. This includes companies and organizations like MS, Facebook, NASA, SAP, Oracle, etc. Each company "votes" in the marketplace of ideas to coordinate software development for a complex project. One company might use JIRA. Another uses MS Team Foundation. Another uses FogCreek. Maybe another employer might use SMTP emails similar to Linux kernel development. Since companies have every incentive to develop software with cost-efficiency and velocity, they will want to use the best workflow possible. Since no single software company I know of uses SMTP emails as their ALM workflow, it either means that (1) companies are wasting money & losing iteration velocity by avoiding email -- or -- (2) the Linux kernel community has special circumstances that makes using a true unified ALM unrealistic. I believe that special circumstance is: 4000 programmers work for different employers instead of a single employer. [1] https://en.wikipedia.org/wiki/Natural_experiment https://en.wikipedia.org/wiki/Natural_experiment
  - abiox 9y ago
    > Since companies have every incentive to develop software with cost-efficiency and velocity, they will want to use the best workflow possible. in the abstract these are generally viewed as or presumed to be positive, but the pressures and objectives of a business can be rather complex and other things will often compete for finite resources. having worked in a number of large organizations, the waste and inefficiencies one can find is absolutely staggering. outside the sales pitches, it's also not clear how an alm interacts with notions of 'cost-efficienty' and 'velocity'. introducing process and bureaucracy isn't always a win. perhaps the presumed predilection of large organizations for alms can be found in alms addressing (or seeming to address) other needs, pressures or requirements.
bassman9000 9y ago
4000 developers sponsored by 400 companies. Let than sink in. It's not only 4000 opinionated devs. It's, potentially, 400 completely different organizations; 400 management layers on top; 400 different priority sets; devs in every time zone; etc. Google is just 1 company (a dozen, if you want to break down Alphabet). It's amazing that they manage to get 8 changes an hour, and still be one of the most successful open source, and software project in general. Why changing the system?
- erikbye 9y ago
  14000 developers from 1300 companies have contributed, since 2005. From 3.19 to 4.7, an average of 4600 lines of code was added daily. Over 80% of all kernel development is done by developers paid for their work. A small number of companies are responsible for a large portion of the total changes. These being, from biggest to lowest contributor: Intel (12.9%), Red Hat (8,0%), Linaro (4%), Samsung, SUSE, IBM, Google (2%), AMD, Texas Instruments, ARM, Oracle. Contributions from unpaid developers have been in decline for many years. 14.6% in 2012, 7.7% in 2016.
  - lifeisstillgood 9y ago
    Can I ask how you got that info? Are committers listed as paid / unpaid ?
    - erikbye 9y ago
      http://go.linuxfoundation.org/linux-kernel-development-report-2016 http://go.linuxfoundation.org/linux-kernel-development-repor...
yahna 9y ago
Probably more like email works best for linus and the other gatekeepers.
emmelaich 9y ago
The great thing about email is that it is asynchronous. The problem with a ALM is it is a global lock. It is enormously disruptive if it is unavailable or even slow.

by jasode 9y ago

walkingolof 9y ago

Its not comparing the same thing I think, commits to a repo and diffs sent over email, and even if it was, its not really the same thing anyway... Kernel development seems to progress fine, I'm pretty sure if there was a problem, Linus would have made some effort to change it, just like the whole source control thing that ended up becoming Git.

jasode 9y ago
>Kernel development seems to progress fine, I'm not claiming that there's a bottleneck in kernel commits. (I'd guess human code reviews checking quality is the ultimate bottleneck.) I'm saying that using emails as "building blocks to reconstruct an pseudo ALM" instead of using an actual special-purpose ALM -- is the only realistic workflow that would be accepted by the disparate Linux community who don't work for a single employer. (If you're a single company employing all 4000 of Linux kernel programmers, you could impose an ALM tool be mandatory for code reviews, assigning work, tagging bug reports, etc.) Discussing positive bullet points of email ... even though each bullet point is true actually obscures the underlying reason the devs won't use an ALM.
- wmf 9y ago
  It seems like that argument cuts both ways; what about developers who refuse to use the email-based process?
  - jasode 9y ago
    >; what about developers who refuse to use the email-based process? I guess I don't understand your question. In Linux, the email-as-the-workflow[1] has been agreed upon from Linux Torvald and his lieutenants all the way down. I suppose that if a one refused to use email-based-workflow, they'd have to find a Linux kernel maintainer to be a champion for their proposed changes and submit them on their behalf. (Is this a common scenario? I don't know.) If a rebel Linux kernel programmer insisted on staging changes for review in Gerritt/JIRA/github/etc, he's going to be isolated in an island because he's not part of the agreed-upon workflow: emails. [1] https://github.com/torvalds/linux/blob/master/Documentation/process/submitting-patches.rst#14-the-canonical-patch-format https://github.com/torvalds/linux/blob/master/Documentation/...
    - wmf 9y ago
      If Linus declared that everyone must use, say, Gitlab from now on, some Linux developers might leave. But developers who don't use email may never start working on Linux. That's why it cuts both ways: either option alienates some developers.
      - walshemj 9y ago
        A developer that does not use mail :-) that is very unlikely
        dlp211 9y ago
        A developer that does not use mail for code submission.

codemac 9y ago

I got in this same discussion with gregkh on Reddit, and Torvalds in person. Linux development is much slower than many large software engineering firms, because the gatekeeper model has many more integrators and reviewers as a proportion of the members than the "pull request"/ team code review model Acceptance is certainly part of it, but 8 changes per hour is 8-10 email threads per hour, which is just not that hard to track as an individual, thus no need to change anything. Right now at work on a large (200ish) team, I get cc'd on about the same amount of patches.

abiox 9y ago

> Compare with Google isn't google's situation a monorepo composed of many distinct projects? there doesn't seem much utility in the comparison. > Email isn't about "throughput"; it's "acceptance". this seems to be the similar, in the other direction. using (or not) an alm in an organization isn't about "throughput", it's about "accepting" a company's policy assertions.

jasode 9y ago
>using (or not) an alm in an organization isn't about "throughput", it's about "accepting" a company's policy assertions. I don't think that equating it from the reverse direction works. Here's why: Using the article's count of ~4000 developers, let's consider all software companies that employ 4000+ developers as part of a "natural experiment"[1]. This includes companies and organizations like MS, Facebook, NASA, SAP, Oracle, etc. Each company "votes" in the marketplace of ideas to coordinate software development for a complex project. One company might use JIRA. Another uses MS Team Foundation. Another uses FogCreek. Maybe another employer might use SMTP emails similar to Linux kernel development. Since companies have every incentive to develop software with cost-efficiency and velocity, they will want to use the best workflow possible. Since no single software company I know of uses SMTP emails as their ALM workflow, it either means that (1) companies are wasting money & losing iteration velocity by avoiding email -- or -- (2) the Linux kernel community has special circumstances that makes using a true unified ALM unrealistic. I believe that special circumstance is: 4000 programmers work for different employers instead of a single employer. [1] https://en.wikipedia.org/wiki/Natural_experiment https://en.wikipedia.org/wiki/Natural_experiment
- abiox 9y ago
  > Since companies have every incentive to develop software with cost-efficiency and velocity, they will want to use the best workflow possible. in the abstract these are generally viewed as or presumed to be positive, but the pressures and objectives of a business can be rather complex and other things will often compete for finite resources. having worked in a number of large organizations, the waste and inefficiencies one can find is absolutely staggering. outside the sales pitches, it's also not clear how an alm interacts with notions of 'cost-efficienty' and 'velocity'. introducing process and bureaucracy isn't always a win. perhaps the presumed predilection of large organizations for alms can be found in alms addressing (or seeming to address) other needs, pressures or requirements.

bassman9000 9y ago

4000 developers sponsored by 400 companies. Let than sink in. It's not only 4000 opinionated devs. It's, potentially, 400 completely different organizations; 400 management layers on top; 400 different priority sets; devs in every time zone; etc. Google is just 1 company (a dozen, if you want to break down Alphabet). It's amazing that they manage to get 8 changes an hour, and still be one of the most successful open source, and software project in general. Why changing the system?

erikbye 9y ago
14000 developers from 1300 companies have contributed, since 2005. From 3.19 to 4.7, an average of 4600 lines of code was added daily. Over 80% of all kernel development is done by developers paid for their work. A small number of companies are responsible for a large portion of the total changes. These being, from biggest to lowest contributor: Intel (12.9%), Red Hat (8,0%), Linaro (4%), Samsung, SUSE, IBM, Google (2%), AMD, Texas Instruments, ARM, Oracle. Contributions from unpaid developers have been in decline for many years. 14.6% in 2012, 7.7% in 2016.
- lifeisstillgood 9y ago
  Can I ask how you got that info? Are committers listed as paid / unpaid ?
  - erikbye 9y ago
    http://go.linuxfoundation.org/linux-kernel-development-report-2016 http://go.linuxfoundation.org/linux-kernel-development-repor...

yahna 9y ago

Probably more like email works best for linus and the other gatekeepers.

emmelaich 9y ago

The great thing about email is that it is asynchronous. The problem with a ALM is it is a global lock. It is enormously disruptive if it is unavailable or even slow.