5 ms·

Only a few vendor-paid developers do almost all open-source work

151 points by tomxor 8y ago

RangerScience 8y ago
Really good insights on how this is tots fine and works well, but I think it also misses something key: As company, you can make money by giving something away for free.
- maxxxxx 8y ago
  It may also be a tool to to push out competition. Keep your own money making stuff proprietary but make everything else open source so others can't survive. this seems to be the Google and Facebook strategy.
  - detuur 8y ago
    If you make the stuff that powers your way of doing business free, you'll push out everyone who tries anything different. If your way is rule, you're king.
- rectang 8y ago
  Under this analysis, some portion of the Open Source efforts out there fall into a very old tradition that includes promotional giveaways, loss leaders, and so on. It is the application of old principles to a new arena.
- sunir 8y ago
  That isn’t key. Making money by giving away something is free is physically impossible. Therefore there is something critical you are missing about what is going on. It’s only key in the other sense that it will unlock a door to greater understanding if you seek out what underlies the contradiction.
mhsabbagh 8y ago
This is hilarious: >> It is fair to say that for almost all of the projects in the CNCF, specific vendors account for most of the development work being done. > Not just “many” open source projects—all of them. The author moves from an analysis written about CNCF projects to generalize a conclusion about all open source software. What a misleading title and content, it remains hanged without any evidence on those claims.
- sheetjs 8y ago
  The original "study" http://redmonk.com/fryan/2018/04/03/who-contributes-an-analysis-of-cncf-projects/ http://redmonk.com/fryan/2018/04/03/who-contributes-an-analy... has an appropriate title: > TL; DR: The vast of code contributions to each CNCF projects outside of Kubernetes come from a small subset of users and companies. InfoWorld sensationalized the title.
  - ghaff 8y ago
    Matt is generalizing outside of the CNCF though. He specifically mentions Linux. Personally, I wouldn't have used the term "all" absent having actually looked at the numbers myself. That said, it's probably at least mostly true, especially if you look at the subproject level in the case of projects like Linux and OpenStack.
- recentdarkness 8y ago
  > it remains hanged without any evidence on those claims. While I get where you are coming from, give me a counter example/evidence. My observation of the open source world is pretty much in sync with the generalization of this article. Even if you might find some good counter example it most likely will be the exception to the rule.
  - bscphil 8y ago
    I'm having trouble understanding this view. Of course most of the big projects like Linux or the browsers have corporate backing, but that ignores the huge amount of work done on much smaller projects by unpaid devs. I don't understand how anyone could say "I can’t think of a significant counterexample" as the author does; surely that just reflects his ignorance of anything beyond big corporate projects. Most of the programs I'm interacting with daily are built by volunteers; that's pretty typical of the Linux ecosystem. Sure, Firefox has corporate backing, but the half dozen or so extensions that make the browser useful for me are all built by volunteers. uBlock, uMatrix, TreeStyleTab, Stylus, etc etc. My backup software, borg? Built by volunteers. My music software, mpd? Built by volunteers. GIMP, useful for quick edits? Volunteers. My text editor, most of the games I play, terminal utilities, torrent clients? Volunteers too. Even the enormous amount of packaging work and bug triage done for my distribution (Arch Linux) is done by volunteers. Even KDE and Gnome, I suspect, don't have many devs hired by corps to work on those projects. (Though I might be mistaken.) Maybe by lines of open source code, rather than number of projects, you'd have an argument that corps do most of the work. But when it comes to the program ecosystem on Linux desktops, you're completely surrounded by a bunch of volunteer projects.
tspiteri 8y ago
The article uses "commercial" when it should really use "proprietary". The open source projects mentioned are all very much commercial software.
- ghaff 8y ago
  I'd rephrase that slightly. In many cases, there is commercial software that makes use of many of those open source projects.
kerng 8y ago
>>The irony is that what makes open source work—and differ from commercial software—is that only a few developers do the major work on any project. Comparing Open Source vs. Commercial software - that's the fail of the article right at the beginning... Apple vs Oranges... What has one to do with the other? Maybe they meant proprietary?
- ghaff 8y ago
  I'm a bit surprised that Matt used that terminology. It's pretty obvious from the context that he's using commercial as a synonym for proprietary and he certainly knows that's sloppy IMO.
- vram22 8y ago
  Yes, good point. I've seen a few well-known/famous tech writers make that mistake, either unknowingly or carelessly. Open source or free software (as in speech) vs. closed source (proprietary) and commercial or paid / non-commercial or free software (as in beer) are 2 X 2 (orthogonal) variations, giving 4 possibilities in total. Odd that not many people see this. Maybe sloppy as ghaff said. Correct me if I got it wrong, anyone, that's my understanding of "free as in speech" vs. "free as in beer".
  - franciscop 8y ago
    I prefer libre (as in speech) and gratis (as in no cost) but it might be because I'm Spanish. The thing is, they are conceptually orthogonal, but in practice there's strong correlation. It might be for many reasons, but I'd guesstimate 90+% of open source is also non-commercial.
    - heavenlyblue 8y ago
      Are you implying Linux is not commercial? What about ElasticSearch? GitLab? PostgreSQL?
      - franciscop 8y ago
peterwwillis 8y ago
Open source contributions are actually mostly not from the project in question. You'll almost always depend on a mountain of other code, in the form of the kernel, software libraries, user utilities, and so on. A lot of these are specialized projects and have their own core developers. Your project may be possible without them, on another platform for example, but in Linux there's still thousands of random devs contributing to a whole. Cloud software has suffered from NIH for a good long time now, and required proprietary APIs to function. You have to "integrate" your app with another app, it never just has implicit compatibility. Sad really.
- ghaff 8y ago
  That's a good point although it's not wholly unique to open source. In addition, container platforms for example may be made up of a fair number of different projects including container orchestration, the base container capabilities, monitoring, logging, CI/CD, etc.
- kanox 8y ago
  That claim that "vendor-paid developers do almost all work" also fits the Linux kernel. The kernel is very large and has many contributors but work done by random volunteers is dwarfed by that done by large companies. This is visible in contributor statistics: https://lwn.net/Articles/750054/ https://lwn.net/Articles/750054/ For the kernel the "community" is to a very large extent a community of companies.
  - mr_toad 8y ago
    I get the impression that many of the kernel developers in these companies would be active in the community no matter where they worked. For example see this interview with one of the names on the list: ‘I can't imagine doing anything else for a living after having looked into most kernel subsystems at some point over the last 15 years.’ https://www.linux.com/news/30-linux-kernel-developers-30-weeks-arnd-bergmann https://www.linux.com/news/30-linux-kernel-developers-30-wee...
starefossen 8y ago
Keep in mind that many of these corporate employed developers would likely contribute to the same OSS projects as a part of their job regardless of what company they happen to be employed at.
- pests 8y ago
  I don't think this is true. First of all some workplaces won't allow it. Some companies don't even run on OSS. Some companies will work on it but not contribute back. Some companies won't have a budget to fund random / semi-related OSS project development.
  - rossdavidh 8y ago
    There are examples. For instance, both Linux (Linus Torvalds) and Python (Guido Rossum) have been steered by original contributors who changed employers several times, always still retaining their role as primary developer/coordinator for that open source project. Not saying it's the majority, but Linux and Python are two pretty important examples.
    - yayana 8y ago
      Those aren't relevant examples for drawing a conclusion about the relationship between the average contributor to Linux or python and their employer.. Or for the lead of an average project and their employer.
      - pests 8y ago
        You made this point better than I could. If anything GP mentioning _important projects_ is by definition the exception.
- ghaff 8y ago
  There are doubtless some prominent developers on a number of major open source projects who could find employment at a variety of companies without changing much about what they work on. They're a minority. Most companies expect developers to focus on what's important to their specific business and not what was important to their previous employer's business. They can't just work on whatever they feel like, especially on company time.
xw917222 8y ago
Open source has turned into a net-negative for the profession. I work on an open source project (paid), and I dislike the constant interaction with random strangers who think they have a right to influence the project and make architectural decisions. Competition is fierce and drives down the salaries. I'm very cautious about Microsoft apparently also adapting the open source serf model. Open source eradicated too many solid desktop-apps and forced everyone into broken web-apps.
josmala 8y ago
Infoworld: "We care about your privacy" That is the headline, in an interface which allows me to scroll 32 times so that I can disable sharing my personal information to all the ~300+ different parties they share it by default.
Proven 8y ago
If most work is done by two people why pay more than $200K/year for any open source software
donbright 8y ago
you know what would be interesting... is if there was some kind of equivalent for dtrace or strace or gdb.. but instead of collecting statistics on performance or call trees of functions, it could instead tell you statistics on who wrote the code that your machine is running, line by line.
- haihaibye 8y ago
  Could modify debug symbols to contain git blame output then collect it in a profiler. We'd probably just learn who wrote the busy/wait kernel loop.
- dankohn1 8y ago
  This is by project instead of by machine, but CNCF develops DevStats to show a number of statistics on each of our projects, including the companies behind the development. https://k8s.devstats.cncf.io/d/9/companies-table?orgId=1 https://k8s.devstats.cncf.io/d/9/companies-table?orgId=1 Disclosure: I am executive director of CNCF.
maxander 8y ago
As others have noted, the article is based on a study that only looked at a small subject of OSS projects, so it's not very solid. But the results fit with many commenters' perceptions. It's surprising that so little is known about the dynamics of how OSS gets written, given how crucially important it is to the global economy these days. (Even) more broadly; I wonder if a similar system will wind up dominating in some fields of science. There's been a bunch of cases where prominent research papers in AI or biomedicine have been published by entirely for-profit companies; one supposes they benefit in the same fashion that more traditional software companies benefit from publishing open source work, namely by building a reputation and cultivating a larger ecosystem around their expertise.
- dnomad 8y ago
  Not much is known about the reality of open source because people do not want to know. This is the biggest story of the last decade: a few major technology corporations have banded together to give away billions of dollars of software completely for free. It is, I suspect, the greatest act of charity our planet has ever known. It's not difficult to see how virtually all of the major trends from the rise of China to the ever expanding proliferation of smartphones and IoT are being driven by this. The disconnect between the narrative around technology companies and and the reality is due to a severe cultural blindspot. It would simply never occur to most people that private companies would compete to give way most of their products. Even people who should know better don't want to admit that much of the software they consume (for absolutely zero dollars!) is provided by the same companies they love to whine about. We seem to be missing a whole new phase of capitalism happening right under our noses and there's really not enough research into it.
anfilt 8y ago
Well, it kinda takes more people working part time to do the same amount as someone working on it full time. Throw in the fact they also probably get paid. Yes, the will code more the most volunteers. Sadly, volunteers who would contribute more have to eat too. Although, I find the statement "all" is a bit misleading. I know of quite a few projects done by people on the side.
AkshatM 8y ago
While each individual project may have a core group of committers, are they the same people each time? This conclusion was not clear to me from the article, though the Hacker News title implies it. If that is the case, it goes against my experience with open-source. True, individual projects have core committers who drive the project forward - but the ecosystem that springs up around a project doesn't typically derive entirely from this core subset. NPM is likely the best example with 350,000 packages existing on the platform alone with the distribution of developers to packages being fairly uniform (reference: https://github.com/substack/npmtop/blob/master/README.markdown https://github.com/substack/npmtop/blob/master/README.markdo... - mind that the numbers are from 2011, but even back then it was highly uniform).

151 points by tomxor 8y ago

Really good insights on how this is tots fine and works well, but I think it also misses something key: As company, you can make money by giving something away for free.

maxxxxx 8y ago
It may also be a tool to to push out competition. Keep your own money making stuff proprietary but make everything else open source so others can't survive. this seems to be the Google and Facebook strategy.
- detuur 8y ago
  If you make the stuff that powers your way of doing business free, you'll push out everyone who tries anything different. If your way is rule, you're king.
rectang 8y ago
Under this analysis, some portion of the Open Source efforts out there fall into a very old tradition that includes promotional giveaways, loss leaders, and so on. It is the application of old principles to a new arena.
sunir 8y ago
That isn’t key. Making money by giving away something is free is physically impossible. Therefore there is something critical you are missing about what is going on. It’s only key in the other sense that it will unlock a door to greater understanding if you seek out what underlies the contradiction.

mhsabbagh 8y ago

This is hilarious: >> It is fair to say that for almost all of the projects in the CNCF, specific vendors account for most of the development work being done. > Not just “many” open source projects—all of them. The author moves from an analysis written about CNCF projects to generalize a conclusion about all open source software. What a misleading title and content, it remains hanged without any evidence on those claims.

sheetjs 8y ago
The original "study" http://redmonk.com/fryan/2018/04/03/who-contributes-an-analysis-of-cncf-projects/ http://redmonk.com/fryan/2018/04/03/who-contributes-an-analy... has an appropriate title: > TL; DR: The vast of code contributions to each CNCF projects outside of Kubernetes come from a small subset of users and companies. InfoWorld sensationalized the title.
- ghaff 8y ago
  Matt is generalizing outside of the CNCF though. He specifically mentions Linux. Personally, I wouldn't have used the term "all" absent having actually looked at the numbers myself. That said, it's probably at least mostly true, especially if you look at the subproject level in the case of projects like Linux and OpenStack.
recentdarkness 8y ago
> it remains hanged without any evidence on those claims. While I get where you are coming from, give me a counter example/evidence. My observation of the open source world is pretty much in sync with the generalization of this article. Even if you might find some good counter example it most likely will be the exception to the rule.
- bscphil 8y ago
  I'm having trouble understanding this view. Of course most of the big projects like Linux or the browsers have corporate backing, but that ignores the huge amount of work done on much smaller projects by unpaid devs. I don't understand how anyone could say "I can’t think of a significant counterexample" as the author does; surely that just reflects his ignorance of anything beyond big corporate projects. Most of the programs I'm interacting with daily are built by volunteers; that's pretty typical of the Linux ecosystem. Sure, Firefox has corporate backing, but the half dozen or so extensions that make the browser useful for me are all built by volunteers. uBlock, uMatrix, TreeStyleTab, Stylus, etc etc. My backup software, borg? Built by volunteers. My music software, mpd? Built by volunteers. GIMP, useful for quick edits? Volunteers. My text editor, most of the games I play, terminal utilities, torrent clients? Volunteers too. Even the enormous amount of packaging work and bug triage done for my distribution (Arch Linux) is done by volunteers. Even KDE and Gnome, I suspect, don't have many devs hired by corps to work on those projects. (Though I might be mistaken.) Maybe by lines of open source code, rather than number of projects, you'd have an argument that corps do most of the work. But when it comes to the program ecosystem on Linux desktops, you're completely surrounded by a bunch of volunteer projects.

tspiteri 8y ago

The article uses "commercial" when it should really use "proprietary". The open source projects mentioned are all very much commercial software.

ghaff 8y ago
I'd rephrase that slightly. In many cases, there is commercial software that makes use of many of those open source projects.

kerng 8y ago

>>The irony is that what makes open source work—and differ from commercial software—is that only a few developers do the major work on any project. Comparing Open Source vs. Commercial software - that's the fail of the article right at the beginning... Apple vs Oranges... What has one to do with the other? Maybe they meant proprietary?

ghaff 8y ago
I'm a bit surprised that Matt used that terminology. It's pretty obvious from the context that he's using commercial as a synonym for proprietary and he certainly knows that's sloppy IMO.
vram22 8y ago
Yes, good point. I've seen a few well-known/famous tech writers make that mistake, either unknowingly or carelessly. Open source or free software (as in speech) vs. closed source (proprietary) and commercial or paid / non-commercial or free software (as in beer) are 2 X 2 (orthogonal) variations, giving 4 possibilities in total. Odd that not many people see this. Maybe sloppy as ghaff said. Correct me if I got it wrong, anyone, that's my understanding of "free as in speech" vs. "free as in beer".
- franciscop 8y ago
  I prefer libre (as in speech) and gratis (as in no cost) but it might be because I'm Spanish. The thing is, they are conceptually orthogonal, but in practice there's strong correlation. It might be for many reasons, but I'd guesstimate 90+% of open source is also non-commercial.
  - heavenlyblue 8y ago
    Are you implying Linux is not commercial? What about ElasticSearch? GitLab? PostgreSQL?
    - franciscop 8y ago

peterwwillis 8y ago

Open source contributions are actually mostly not from the project in question. You'll almost always depend on a mountain of other code, in the form of the kernel, software libraries, user utilities, and so on. A lot of these are specialized projects and have their own core developers. Your project may be possible without them, on another platform for example, but in Linux there's still thousands of random devs contributing to a whole. Cloud software has suffered from NIH for a good long time now, and required proprietary APIs to function. You have to "integrate" your app with another app, it never just has implicit compatibility. Sad really.

ghaff 8y ago
That's a good point although it's not wholly unique to open source. In addition, container platforms for example may be made up of a fair number of different projects including container orchestration, the base container capabilities, monitoring, logging, CI/CD, etc.
kanox 8y ago
That claim that "vendor-paid developers do almost all work" also fits the Linux kernel. The kernel is very large and has many contributors but work done by random volunteers is dwarfed by that done by large companies. This is visible in contributor statistics: https://lwn.net/Articles/750054/ https://lwn.net/Articles/750054/ For the kernel the "community" is to a very large extent a community of companies.
- mr_toad 8y ago
  I get the impression that many of the kernel developers in these companies would be active in the community no matter where they worked. For example see this interview with one of the names on the list: ‘I can't imagine doing anything else for a living after having looked into most kernel subsystems at some point over the last 15 years.’ https://www.linux.com/news/30-linux-kernel-developers-30-weeks-arnd-bergmann https://www.linux.com/news/30-linux-kernel-developers-30-wee...

starefossen 8y ago

Keep in mind that many of these corporate employed developers would likely contribute to the same OSS projects as a part of their job regardless of what company they happen to be employed at.

pests 8y ago
I don't think this is true. First of all some workplaces won't allow it. Some companies don't even run on OSS. Some companies will work on it but not contribute back. Some companies won't have a budget to fund random / semi-related OSS project development.
- rossdavidh 8y ago
  There are examples. For instance, both Linux (Linus Torvalds) and Python (Guido Rossum) have been steered by original contributors who changed employers several times, always still retaining their role as primary developer/coordinator for that open source project. Not saying it's the majority, but Linux and Python are two pretty important examples.
  - yayana 8y ago
    Those aren't relevant examples for drawing a conclusion about the relationship between the average contributor to Linux or python and their employer.. Or for the lead of an average project and their employer.
    - pests 8y ago
      You made this point better than I could. If anything GP mentioning _important projects_ is by definition the exception.
ghaff 8y ago
There are doubtless some prominent developers on a number of major open source projects who could find employment at a variety of companies without changing much about what they work on. They're a minority. Most companies expect developers to focus on what's important to their specific business and not what was important to their previous employer's business. They can't just work on whatever they feel like, especially on company time.

xw917222 8y ago

Open source has turned into a net-negative for the profession. I work on an open source project (paid), and I dislike the constant interaction with random strangers who think they have a right to influence the project and make architectural decisions. Competition is fierce and drives down the salaries. I'm very cautious about Microsoft apparently also adapting the open source serf model. Open source eradicated too many solid desktop-apps and forced everyone into broken web-apps.

josmala 8y ago

Infoworld: "We care about your privacy" That is the headline, in an interface which allows me to scroll 32 times so that I can disable sharing my personal information to all the ~300+ different parties they share it by default.

Proven 8y ago

If most work is done by two people why pay more than $200K/year for any open source software

donbright 8y ago

you know what would be interesting... is if there was some kind of equivalent for dtrace or strace or gdb.. but instead of collecting statistics on performance or call trees of functions, it could instead tell you statistics on who wrote the code that your machine is running, line by line.

haihaibye 8y ago
Could modify debug symbols to contain git blame output then collect it in a profiler. We'd probably just learn who wrote the busy/wait kernel loop.
dankohn1 8y ago
This is by project instead of by machine, but CNCF develops DevStats to show a number of statistics on each of our projects, including the companies behind the development. https://k8s.devstats.cncf.io/d/9/companies-table?orgId=1 https://k8s.devstats.cncf.io/d/9/companies-table?orgId=1 Disclosure: I am executive director of CNCF.

maxander 8y ago

As others have noted, the article is based on a study that only looked at a small subject of OSS projects, so it's not very solid. But the results fit with many commenters' perceptions. It's surprising that so little is known about the dynamics of how OSS gets written, given how crucially important it is to the global economy these days. (Even) more broadly; I wonder if a similar system will wind up dominating in some fields of science. There's been a bunch of cases where prominent research papers in AI or biomedicine have been published by entirely for-profit companies; one supposes they benefit in the same fashion that more traditional software companies benefit from publishing open source work, namely by building a reputation and cultivating a larger ecosystem around their expertise.

dnomad 8y ago
Not much is known about the reality of open source because people do not want to know. This is the biggest story of the last decade: a few major technology corporations have banded together to give away billions of dollars of software completely for free. It is, I suspect, the greatest act of charity our planet has ever known. It's not difficult to see how virtually all of the major trends from the rise of China to the ever expanding proliferation of smartphones and IoT are being driven by this. The disconnect between the narrative around technology companies and and the reality is due to a severe cultural blindspot. It would simply never occur to most people that private companies would compete to give way most of their products. Even people who should know better don't want to admit that much of the software they consume (for absolutely zero dollars!) is provided by the same companies they love to whine about. We seem to be missing a whole new phase of capitalism happening right under our noses and there's really not enough research into it.

anfilt 8y ago

Well, it kinda takes more people working part time to do the same amount as someone working on it full time. Throw in the fact they also probably get paid. Yes, the will code more the most volunteers. Sadly, volunteers who would contribute more have to eat too. Although, I find the statement "all" is a bit misleading. I know of quite a few projects done by people on the side.

AkshatM 8y ago

While each individual project may have a core group of committers, are they the same people each time? This conclusion was not clear to me from the article, though the Hacker News title implies it. If that is the case, it goes against my experience with open-source. True, individual projects have core committers who drive the project forward - but the ecosystem that springs up around a project doesn't typically derive entirely from this core subset. NPM is likely the best example with 350,000 packages existing on the platform alone with the distribution of developers to packages being fairly uniform (reference: https://github.com/substack/npmtop/blob/master/README.markdown https://github.com/substack/npmtop/blob/master/README.markdo... - mind that the numbers are from 2011, but even back then it was highly uniform).