5 ms·

GitHub Language Trends

113 points by spatten 12y ago

err4nt 12y ago
Very interesting graphs, I was surprised that CSS has had a recent uptick but as somebody who has specialized in responsive layout in the past two years I guess that uptick represents my life too. I can't get enough language statistics on Github! I run 'gitinspector' on my web server to compute language stats in individual git repositories, but one thing I haven't been able to figure out it how to chart the language stats for one git repository over time in a branch. Does anybody know how you can chart language use over time in one repository?
- AYBABTME 12y ago
  I have projects (markdown repos) marked as CSS projects for which I haven't written more than 10 lines of CSS. Bootstrap makes it appear so, I guess.
gexla 12y ago
I don't know this is a good indication of trends. As the article (somewhat shallowly) shows, there are stories behind of these graphs. Ruby has probably just settled to a normal position post early adopter. Shows that Ruby is still strong. Javascript is probably building the sorts of libraries that other languages already have. These guys have had a lot of work to do.
- davidw 12y ago
  Yes, Ruby was very much "overrepresented" in the early days of github, so the decline in its growth relative to other languages is fairly normal sounding, rather than indicative of its actual decline. Also, Javascript is something that is going to get checked in to lots of web projects one way or another. I wonder if they weed out duplicate copies of, say, jQuery.
  - grey-area 12y ago
    I wonder if they weed out duplicate copies of, say, jQuery. These stats have not in my experience been very reliable at all, and tend to massively overcount javascript. I've had a few projects tagged 'javascript' when in fact they were web projects and just had some common js libraries like jquery included (ruby and golang projects).
  - coyotebush 12y ago
    > I wonder if they weed out duplicate copies of, say, jQuery. Linguist tries to filter out things like that: https://github.com/github/linguist/blob/master/lib/linguist/vendor.yml https://github.com/github/linguist/blob/master/lib/linguist/...
    - nox_ 12y ago
      "Tries" is the operative word here. There is a documented history of utter incompetence in Linguist.
      - jhcnt 12y ago
        Likewise for most of GitHub's other libraries. Sundown is some of the worst code I've ever seen.
- courtson5 12y ago
  Ah, it is only on internet forums where someone can argue that a sharply downward trending graph shows that a language is 'still strong'.
  - davorb 12y ago
    "GitHub is a specific community that’s grown very quickly since it launched [writeup]. It was not initially reflective of open source as a whole but rather centered around the Ruby on Rails community" Github has moved out of a niche over the past five years, and the graph demonstrates that.
  - andreasvc 12y ago
    Not really. It is universal that statistics can easily misrepresent if one doesn't take a critical look at choice of data and how it is presented. For example in this case the focus on new repos/issues/etc. is suspicious to me. It could very well be that we just see a rise in one-off Javascript uploads to Github here, instead of a comparison of sustained development on mature projects.
largehotcoffee 12y ago
>Language detection is based on lines of code
- Karunamon 12y ago
  This is important to note. As a huge ruby fanboy, one of my favorite things about the language are the shortcuts and syntactic sugar that are provided to make things easier. I saw the decline and got a bit worried, then came here and had those fears assuaged. I do wonder if the LOC difference has anything to do with people getting more familiar with the language and doing more things in less code.
rumcajz 12y ago
This looks like commercial ecosystem (Java, JavaScript) migrating to GitHub rather than hobbyist ecosystems (Ruby, Python, Perl) being on decline.
- camus2 12y ago
  say what? Ruby isnt for hobbyists, neither is Python(dont know Perl).But you just wanted to hear yourself say that didnt you?
  - threeseed 12y ago
    It isn't entirely untrue. Ruby/Python are more popular in the hobbyist space and less so in the enterprise. Java is the opposite. It shouldn't be seen as a reflection of the platform.
    - meepmorp 12y ago
      Does the scientific computing community count as hobbyist? It seems to be a significant niche for python these days, largely due to numpy. Also, as another user noted, hobbyist/enterprise is a false dichotomy. There's shedloads of non-hobbyist code that also non-enterprise.
    - destron 12y ago
      You have set up a false dichotomy here between enterprise and hobbyist.
      - deleted 12y ago
        [deleted]
- The_Double 12y ago
  I think a large part of the rise in Java code is because of a large amount of open source android projects. (applications and mods/forks of the operating system)
alexchamberlain 12y ago
What is the cause for the correlation between Java and JavaScript?
- pcmonk 12y ago
  Probably random noise. Or else because it kind of tracks the arc of github going mainstream, and Java and Javascript are very mainstream.
- romanovcode 12y ago
  Java: Rise of Android ~2009 Javascript: Random noise.
sheetjs 12y ago
FWIW many projects written in web languages like PHP seem to be treated as javascript by Github because they also have javascript code (which happens to be more significant / larger than the underlying code). It's unfortunate that there is no way to specify the primary language of a project. The Github language system is also somewhat unpredictable: https://github.com/SheetJS/test_files https://github.com/SheetJS/test_files seems to alternate between AppleScript and Shell with each commit (even if no .scpt or .sh file was changed or added)
- flohofwoe 12y ago
  Indeed, Github's language detection is absolutely useless, and a manual setting is very much needed. One of my projects is a header-only C++ lib, which is detected as 100% C, another one is a fairly typical C++11 codebase, which is detected as 50% C and 50% C++, third is a Python project but has some big example 3D asset files in three.js format which is then listed as 1.2% Python and 98.8% Javascript. So at least for my github projects, the language classification is completely off.
- zongitsrinzler 12y ago
  If a project mainly consists of JavaScript it should be a JavaScript project, no?
  - deleted 12y ago
    [deleted]
  - ubernostrum 12y ago
    Well, to take an example: I work on the Mozilla Developer Network (MDN). MDN is mostly a wiki, and the underlying codebase for that is all in Python. There's also the Demo Studio, which lets people upload web technology demos, and that's Python too. But the wiki also has an embedded script-ish language (letting people create macros/templates for specific purposes and re-use them across multiple articles). That's JavaScript-based, and so is implemented in node.js. And there's a WYSIWYG editor for the wiki pages, which is a couple of MDN-specific plugins we've developed, plus off-the-shelf components (the editor + jQuery, which we also use for a few other things on-site). So MDN is a Python project with a couple JS utilities attached. And if you set up a local copy, you can pretty easily see that Python is by far the heaviest-used language. But due to the way GitHub counts and reports statistics, it shows up as a JavaScript project. To pick on an easy example: the copy of jQuery we have in our repository weighs in at just over 9,000 lines. So just having jQuery means you need to write over 10k lines of code in order to get the "real" language of your project recognized.
jcbrand 12y ago
One of their reasons for the growth of Javascript: > the JavaScript development philosophy that encourages bundling of dependencies in the same repo as the primary codebase I wonder how much this is still an issue with the rise of npm, bower and other package managers.
- mateuszf 12y ago
  node_modules directory is usually added to .gitignore. I don't think it is usually commited by accident as it results in a huge file list in diffs and git status command.
  - iLoch 12y ago
    That's what he's saying. While the author's statement may not necessarily be incorrect, the trend these days is to manage your JS (both server and client side code) with package managers like NPM and Bower.
PeterisP 12y ago
It would be interesting to see which languages are rising. In those stats, everything except Javascript seems to be declining, and the total relative decline is much larger than JS growth, so something must be growing - but what it is?
kyberias 12y ago
It would be much more useful to show the absolute numbers. Or provide the raw data for others to do more meaningful graphs.
- arscan 12y ago
  It depends on what story you are trying to tell... Absolute numbers might not be all that meaningful when you are talking about relative popularity. Or it might be. Data visualization is hard because you typically have to trade information density for comprehensibility. I assume the data came from http://www.githubarchive.org/ http://www.githubarchive.org/
louthy 12y ago
Not sure if it's reasonable to make this call, but it seems the dynamically typed languages have significantly higher percentage of issues overall.
- kijin 12y ago
  It could be that dynamic languages have a lower barrier of entry, which encourages many more people to fork the code, find bugs, and make pull requests. It's easy for a PHP guy to find and report bugs in a JavaScript project. It's not so easy for a JavaScript guy to find bugs in a C++ project.
chx 12y ago
I find the complete lack of Go surprising.
- threeseed 12y ago
  Go is a new language so not that surprising. It is also over represented here on HN. You don't see Go jobs in the wild for example.
  - d0ugie 12y ago
    > You don't see Go jobs in the wild.. Do you fellas expect that to change over the next few years? I want to learn a new language, Go seems alluring, though I'd like to put myself on the right side of history.
    - frostmatthew 12y ago
      > I want to learn a new language, Go seems alluring, though I'd like to put myself on the right side of history. * if you want to learn a new language because you enjoy learning new languages learn Go since you find that interesting * if you want to learn a new language to make you a better overall developer learn a language different than what you're used to (i.e. if you've only worked with Python learning Ruby won't benefit you as much as learning a functional language like Clojure or a lower level language like C) * if you want to learn a new language solely (or primarily) for the job prospects (and it sounds like you do) learn one that presently has a large number of openings (like Java) instead of learning a new language and hoping it flourishes (you can always learn Go [or whatever] when/if it becomes widely used)
  - Matrixik 12y ago
    Search here: http://www.golangprojects.com/ http://www.golangprojects.com/
  - chx 12y ago
    I do not think Go as something that would go on itself as a job. Instead if you already have a site (like Drupal which is my primary skill) and you want to add a very performant REST interface then Go provides a pleasant way to write one.
pron 12y ago
GitHub is slowly starting to reflect the software world at large, although the true picture is Java and C leading by a huge, huge margin. I don't expect GitHub to ever fully reflect that, as most Java shops, and nearly all C shops, would never host their code on GitHub.
- gurkendoktor 12y ago
  I think it's safe to say that most projects on github are libraries and snippets, not finished products. Not even Ruby shops push all of their actual work to github. But at least there is a culture of sharing libraries, which drives the Ruby % up, and I don't see this happening a lot with C.
m_ram 12y ago
> Violating all expectations and trends, new Java users on GitHub even grew as a percentage of overall new users, while everything else went downhill. This further supports the assertion that GitHub is reaching the enterprise. This is more likely to be due to the rise of Android since 2009.
- gurkendoktor 12y ago
  I would also expect a few CS students to push their homework to github nowadays, and most homework is Java.
skrebbel 12y ago
The last 2 graphs look like they were drawn by someone who hates colour blind people.
platz 12y ago
2010-2011 appears to be the "year of the great inflection point"
jhcnt 12y ago
In my experience, GitHub's so-called "detection" gets it wrong far more often than right. It's worse than useless -- it's misleading.
lkrubner 12y ago
To my mind, the big trend is toward polygot programming, which perhaps reveals what a transitional and perhaps revolutionary time this is in the world of computer programming. This paragraph struck me as the most important: "Almost every language shows a long-term downhill trend... My initial guess is that users of languages below the top 12 are growing in share to counterbalance the decreases here. It’s also possible that GitHub may leave some users unclassified, which would tend to lower everything else’s proportion over time."
peteretep 12y ago
Presumably the big Perl spike is the migration of CPAN modules over to Github en-masse.
dham 12y ago
Just need to have a dropdown of the primary language like Bitbucket does. Probably have different results.
pecanpieyw 12y ago
Github should really just enable the developers to specify the language of their repo by themselves. Bitbucket get this right in the first place. Auto detection for language sounds cool, while doesn't work for most of the web project.
hcarvalhoalves 12y ago
Javascript is really hard to measure I believe, the numbers are always skewed because of 3 things: 1. A lot of repositories include 3rd party libraries. 2. A lot of software includes a web interface, even if the backend language is something else, but the LOC for Javascript can be equal or even higher because of 1. 3. JSON is counted as Javascript sometimes

113 points by spatten 12y ago

Very interesting graphs, I was surprised that CSS has had a recent uptick but as somebody who has specialized in responsive layout in the past two years I guess that uptick represents my life too. I can't get enough language statistics on Github! I run 'gitinspector' on my web server to compute language stats in individual git repositories, but one thing I haven't been able to figure out it how to chart the language stats for one git repository over time in a branch. Does anybody know how you can chart language use over time in one repository?

AYBABTME 12y ago
I have projects (markdown repos) marked as CSS projects for which I haven't written more than 10 lines of CSS. Bootstrap makes it appear so, I guess.

gexla 12y ago

I don't know this is a good indication of trends. As the article (somewhat shallowly) shows, there are stories behind of these graphs. Ruby has probably just settled to a normal position post early adopter. Shows that Ruby is still strong. Javascript is probably building the sorts of libraries that other languages already have. These guys have had a lot of work to do.

davidw 12y ago
Yes, Ruby was very much "overrepresented" in the early days of github, so the decline in its growth relative to other languages is fairly normal sounding, rather than indicative of its actual decline. Also, Javascript is something that is going to get checked in to lots of web projects one way or another. I wonder if they weed out duplicate copies of, say, jQuery.
- grey-area 12y ago
  I wonder if they weed out duplicate copies of, say, jQuery. These stats have not in my experience been very reliable at all, and tend to massively overcount javascript. I've had a few projects tagged 'javascript' when in fact they were web projects and just had some common js libraries like jquery included (ruby and golang projects).
- coyotebush 12y ago
  > I wonder if they weed out duplicate copies of, say, jQuery. Linguist tries to filter out things like that: https://github.com/github/linguist/blob/master/lib/linguist/vendor.yml https://github.com/github/linguist/blob/master/lib/linguist/...
  - nox_ 12y ago
    "Tries" is the operative word here. There is a documented history of utter incompetence in Linguist.
    - jhcnt 12y ago
      Likewise for most of GitHub's other libraries. Sundown is some of the worst code I've ever seen.
courtson5 12y ago
Ah, it is only on internet forums where someone can argue that a sharply downward trending graph shows that a language is 'still strong'.
- davorb 12y ago
  "GitHub is a specific community that’s grown very quickly since it launched [writeup]. It was not initially reflective of open source as a whole but rather centered around the Ruby on Rails community" Github has moved out of a niche over the past five years, and the graph demonstrates that.
- andreasvc 12y ago
  Not really. It is universal that statistics can easily misrepresent if one doesn't take a critical look at choice of data and how it is presented. For example in this case the focus on new repos/issues/etc. is suspicious to me. It could very well be that we just see a rise in one-off Javascript uploads to Github here, instead of a comparison of sustained development on mature projects.

largehotcoffee 12y ago

>Language detection is based on lines of code

Karunamon 12y ago
This is important to note. As a huge ruby fanboy, one of my favorite things about the language are the shortcuts and syntactic sugar that are provided to make things easier. I saw the decline and got a bit worried, then came here and had those fears assuaged. I do wonder if the LOC difference has anything to do with people getting more familiar with the language and doing more things in less code.

rumcajz 12y ago

This looks like commercial ecosystem (Java, JavaScript) migrating to GitHub rather than hobbyist ecosystems (Ruby, Python, Perl) being on decline.

camus2 12y ago
say what? Ruby isnt for hobbyists, neither is Python(dont know Perl).But you just wanted to hear yourself say that didnt you?
- threeseed 12y ago
  It isn't entirely untrue. Ruby/Python are more popular in the hobbyist space and less so in the enterprise. Java is the opposite. It shouldn't be seen as a reflection of the platform.
  - meepmorp 12y ago
    Does the scientific computing community count as hobbyist? It seems to be a significant niche for python these days, largely due to numpy. Also, as another user noted, hobbyist/enterprise is a false dichotomy. There's shedloads of non-hobbyist code that also non-enterprise.
  - destron 12y ago
    You have set up a false dichotomy here between enterprise and hobbyist.
    - deleted 12y ago
      [deleted]
The_Double 12y ago
I think a large part of the rise in Java code is because of a large amount of open source android projects. (applications and mods/forks of the operating system)

alexchamberlain 12y ago

What is the cause for the correlation between Java and JavaScript?

pcmonk 12y ago
Probably random noise. Or else because it kind of tracks the arc of github going mainstream, and Java and Javascript are very mainstream.
romanovcode 12y ago
Java: Rise of Android ~2009 Javascript: Random noise.

sheetjs 12y ago

FWIW many projects written in web languages like PHP seem to be treated as javascript by Github because they also have javascript code (which happens to be more significant / larger than the underlying code). It's unfortunate that there is no way to specify the primary language of a project. The Github language system is also somewhat unpredictable: https://github.com/SheetJS/test_files https://github.com/SheetJS/test_files seems to alternate between AppleScript and Shell with each commit (even if no .scpt or .sh file was changed or added)

flohofwoe 12y ago
Indeed, Github's language detection is absolutely useless, and a manual setting is very much needed. One of my projects is a header-only C++ lib, which is detected as 100% C, another one is a fairly typical C++11 codebase, which is detected as 50% C and 50% C++, third is a Python project but has some big example 3D asset files in three.js format which is then listed as 1.2% Python and 98.8% Javascript. So at least for my github projects, the language classification is completely off.
zongitsrinzler 12y ago
If a project mainly consists of JavaScript it should be a JavaScript project, no?
- deleted 12y ago
  [deleted]
- ubernostrum 12y ago
  Well, to take an example: I work on the Mozilla Developer Network (MDN). MDN is mostly a wiki, and the underlying codebase for that is all in Python. There's also the Demo Studio, which lets people upload web technology demos, and that's Python too. But the wiki also has an embedded script-ish language (letting people create macros/templates for specific purposes and re-use them across multiple articles). That's JavaScript-based, and so is implemented in node.js. And there's a WYSIWYG editor for the wiki pages, which is a couple of MDN-specific plugins we've developed, plus off-the-shelf components (the editor + jQuery, which we also use for a few other things on-site). So MDN is a Python project with a couple JS utilities attached. And if you set up a local copy, you can pretty easily see that Python is by far the heaviest-used language. But due to the way GitHub counts and reports statistics, it shows up as a JavaScript project. To pick on an easy example: the copy of jQuery we have in our repository weighs in at just over 9,000 lines. So just having jQuery means you need to write over 10k lines of code in order to get the "real" language of your project recognized.

jcbrand 12y ago

One of their reasons for the growth of Javascript: > the JavaScript development philosophy that encourages bundling of dependencies in the same repo as the primary codebase I wonder how much this is still an issue with the rise of npm, bower and other package managers.

mateuszf 12y ago
node_modules directory is usually added to .gitignore. I don't think it is usually commited by accident as it results in a huge file list in diffs and git status command.
- iLoch 12y ago
  That's what he's saying. While the author's statement may not necessarily be incorrect, the trend these days is to manage your JS (both server and client side code) with package managers like NPM and Bower.

PeterisP 12y ago

It would be interesting to see which languages are rising. In those stats, everything except Javascript seems to be declining, and the total relative decline is much larger than JS growth, so something must be growing - but what it is?

kyberias 12y ago

It would be much more useful to show the absolute numbers. Or provide the raw data for others to do more meaningful graphs.

arscan 12y ago
It depends on what story you are trying to tell... Absolute numbers might not be all that meaningful when you are talking about relative popularity. Or it might be. Data visualization is hard because you typically have to trade information density for comprehensibility. I assume the data came from http://www.githubarchive.org/ http://www.githubarchive.org/

louthy 12y ago

Not sure if it's reasonable to make this call, but it seems the dynamically typed languages have significantly higher percentage of issues overall.

kijin 12y ago
It could be that dynamic languages have a lower barrier of entry, which encourages many more people to fork the code, find bugs, and make pull requests. It's easy for a PHP guy to find and report bugs in a JavaScript project. It's not so easy for a JavaScript guy to find bugs in a C++ project.

chx 12y ago

I find the complete lack of Go surprising.

threeseed 12y ago
Go is a new language so not that surprising. It is also over represented here on HN. You don't see Go jobs in the wild for example.
- d0ugie 12y ago
  > You don't see Go jobs in the wild.. Do you fellas expect that to change over the next few years? I want to learn a new language, Go seems alluring, though I'd like to put myself on the right side of history.
  - frostmatthew 12y ago
    > I want to learn a new language, Go seems alluring, though I'd like to put myself on the right side of history. * if you want to learn a new language because you enjoy learning new languages learn Go since you find that interesting * if you want to learn a new language to make you a better overall developer learn a language different than what you're used to (i.e. if you've only worked with Python learning Ruby won't benefit you as much as learning a functional language like Clojure or a lower level language like C) * if you want to learn a new language solely (or primarily) for the job prospects (and it sounds like you do) learn one that presently has a large number of openings (like Java) instead of learning a new language and hoping it flourishes (you can always learn Go [or whatever] when/if it becomes widely used)
- Matrixik 12y ago
  Search here: http://www.golangprojects.com/ http://www.golangprojects.com/
- chx 12y ago
  I do not think Go as something that would go on itself as a job. Instead if you already have a site (like Drupal which is my primary skill) and you want to add a very performant REST interface then Go provides a pleasant way to write one.

pron 12y ago

GitHub is slowly starting to reflect the software world at large, although the true picture is Java and C leading by a huge, huge margin. I don't expect GitHub to ever fully reflect that, as most Java shops, and nearly all C shops, would never host their code on GitHub.

gurkendoktor 12y ago
I think it's safe to say that most projects on github are libraries and snippets, not finished products. Not even Ruby shops push all of their actual work to github. But at least there is a culture of sharing libraries, which drives the Ruby % up, and I don't see this happening a lot with C.

m_ram 12y ago

> Violating all expectations and trends, new Java users on GitHub even grew as a percentage of overall new users, while everything else went downhill. This further supports the assertion that GitHub is reaching the enterprise. This is more likely to be due to the rise of Android since 2009.

gurkendoktor 12y ago
I would also expect a few CS students to push their homework to github nowadays, and most homework is Java.

skrebbel 12y ago

The last 2 graphs look like they were drawn by someone who hates colour blind people.

platz 12y ago

2010-2011 appears to be the "year of the great inflection point"

jhcnt 12y ago

In my experience, GitHub's so-called "detection" gets it wrong far more often than right. It's worse than useless -- it's misleading.

lkrubner 12y ago

To my mind, the big trend is toward polygot programming, which perhaps reveals what a transitional and perhaps revolutionary time this is in the world of computer programming. This paragraph struck me as the most important: "Almost every language shows a long-term downhill trend... My initial guess is that users of languages below the top 12 are growing in share to counterbalance the decreases here. It’s also possible that GitHub may leave some users unclassified, which would tend to lower everything else’s proportion over time."

peteretep 12y ago

Presumably the big Perl spike is the migration of CPAN modules over to Github en-masse.

dham 12y ago

Just need to have a dropdown of the primary language like Bitbucket does. Probably have different results.

pecanpieyw 12y ago

Github should really just enable the developers to specify the language of their repo by themselves. Bitbucket get this right in the first place. Auto detection for language sounds cool, while doesn't work for most of the web project.

hcarvalhoalves 12y ago

Javascript is really hard to measure I believe, the numbers are always skewed because of 3 things: 1. A lot of repositories include 3rd party libraries. 2. A lot of software includes a web interface, even if the backend language is something else, but the LOC for Javascript can be equal or even higher because of 1. 3. JSON is counted as Javascript sometimes