7 ms·

Counting Stars on GitHub

70 points by Adrock 13y ago

Cthulhu_ 13y ago
Well, his interpretation of the numbers is rather off, I'm afraid. Mostly due to the language categoriziation; Bootstrap, Node, Zurb Foundation and Adobe Brackets aren't Javascript projects, for example.
- laymil 13y ago
  I thought that was well addressed by this comment in the post: "Looking at the top repo for each language also exposes a weakness in the methodology: GitHub’s language identification isn’t perfect and there are number of polyglot projects. The top Java repo is Storm, which uses enough Clojure (20.1% by GitHub’s measure) to make this identification questionable when you take into account Clojure’s conciseness over Java’s."
- Adrock 13y ago
  OP here. I agree that the language categorization is questionable. It's all based on what the GitHub Search API returns as the language. Looking at Bootstrap, the classification doesn't seem to be strictly based on percentage: - CSS 59.8% - JavaScript 39.7% - Python 0.5% ...but GitHub Search says JavaScript. I do see that if I do a search query with a language filter, I get different results.
  - misterdai 13y ago
    GitHub uses its Linguist project to detect languages and index them as far as I know. I've used the list of languages from this file: https://github.com/github/linguist/blob/master/lib/linguist/languages.yml https://github.com/github/linguist/blob/master/lib/linguist/... I did pick an older version as they don't always run the latest release. Handy as you can filter out items flagged as "markup". Plus, have to watch out for languages like Mirah that are oddly indexed (same repo count as Ruby).
kevinbowman 13y ago
I like that the visualisations are done with D3, the 5th-highest project on the list.
- Adrock 13y ago
  D3 is awesome. It was really nice when working on the post to just be able to re-run the queries to generate new CSV or JSON and not have to go through a whole separate visualization generation pipeline.
danso 13y ago
I'm really amazed that at how Github is such a vital service to developers everywhere, yet there hasn't been a huge amount of alternative ways to dig for hidden and popular repos across different metrics. Even the metric the OP uses, just pure number of stars, has a lot of interesting things in it, as the OP shows. Maybe it's a reflection of how well-designed Github's site-explore is that people haven't yet reason to augment it. When I say that the even pure star count has a lot of interesting things in it...I mean even in the simplest sense. I know or have used all of those repos with the exception of jQuery-File-Upload and adobe/brackets, but I can be a real dilettante. But in the top 100, there's plenty that I've never heard of. Even within plain Ruby-land, I'm always running into gems that have been around for a few years or are highly starred (Spring and Middleman among them, in the last couple of weeks). What I'd like to see is even further categorization of the gems...something similar to the Ruby Toolbox (https://www.ruby-toolbox.com/categories/Active_Record_DB_Adapters https://www.ruby-toolbox.com/categories/Active_Record_DB_Ada...)...where you can find repos based on use-case, not just language. I do like Toolbox's emphasis on release dates and latest updates, as well as pure total download numbers. I've found that number of contributors, rate of pull requests/issues dealt with, and rate of recent updates, is as useful a metric as star count.
hashtree 13y ago
As to the language rankings, some are at a large disadvantage when their standard libs are outstanding. A prime example being Julia (http://docs.julialang.org/en/latest/stdlib/ http://docs.julialang.org/en/latest/stdlib/). The same goes for smaller languages like Roy (https://github.com/puffnfresh/roy https://github.com/puffnfresh/roy) since GitHub doesn't detect them.
misterdai 13y ago
Interesting post, it's worth playing around with the GitHub API to see what you can uncover. Watch out with basing information off their search API, sometimes it seems to miss users or repositories from being indexed, screwing up your results. Caused me a few headaches recently as I've a little site built on daily statistics collected from GitHub. For example: Mojomdo vanished from their user results making Linus Torvalds most followed user, fixed on GitHub now but still evident on my site (boo). Here's my site for anyone curious: http://hubreports.yougeezer.co.uk http://hubreports.yougeezer.co.uk
chimeracoder 13y ago
This is an interesting idea, but it suffers from one maor problem: Github's language detection is rather inflexible, and sometimes misleading. For example: I have a number of small Go projects that are incorrectly categorized as CSS or Javascript. One such example is a project explicitly designed to be a Go webserver that you can just "go get" (or git clone) and run immediately, with no extra setup, sort of like Twitter Bootstrap, but for the backend: https://github.com/ChimeraCoder/go-server-bootstrap https://github.com/ChimeraCoder/go-server-bootstrap Because it self-hosts the Bootstrap files, and because the lines of Go code required are relatively small, less than 5% of this project is detected as being written in Go, even though I didn't write a single line of Javascript or CSS myself for the entire project. (Yes, I know I could pull in the Bootstrap, etc. from a CDN, but this is just to illustrate the limitations of an automatic language detection model that the developer has zero control over).
- vezzy-fnord 13y ago
  Linguist also has an odd tendency to miscategorize projects as being written in Prolog, thus inflating the repository rates of an already very niche language. Perhaps it has something to do with (at least in part) Prolog and Perl sharing the same extensions?
  - draegtun 13y ago
    Same problem happens with Rebol which often get categorised as R on Github (both share .r extension). some ref: https://github.com/search?l=R&q=rebol https://github.com/search?l=R&q=rebol | https://github.com/search?l=R&q=rebol&type=Code https://github.com/search?l=R&q=rebol&type=Code
- thedaniel 13y ago
  > an automatic language detection model that the developer has zero control over Pull requests welcome :) https://github.com/github/linguist https://github.com/github/linguist
  - chimeracoder 13y ago
deleted 13y ago
[deleted]
EGreg 13y ago
Said no more counting dollars, we'll be counting stars....
- Adrock 13y ago
  OP here. I was waiting for that :P
  - EGreg 13y ago
    Glad to oblige :) EDIT: I just realized that the entire song can be about switching to open source development on github. http://youtu.be/hT_nvWreIhg http://youtu.be/hT_nvWreIhg A young, but not that bold, startup founder is losing sleep, dreaming about the things that they can be. He doesnt think the world is sold, he's just doing what they're told by VCs. All his risky career decisions make him feel alive. And sure enough like many other startups his hits the deadpool. Take that money watch it burn, sink in the river, the lessons are learned. So he moves to release the tech as open source software and starts counting the stars on github instead of dollars... LOL
prezjordan 13y ago
I have a strange relationship with GitHub stars. On one hand, I love them - they give my projects exposure (I always mention cleaver [0] during my interviews) and let me know how many people are "using" my project. On the other hand, I feel it sort of pushes me in the wrong direction. Stars aren't a good indicator of use, or even importance. For instance, cleaver is not technically challenging, yet it shares the same number of stars as numpy [1]. So, not sure how I feel about them - not sure what they measure. [0]: https://github.com/jdan/cleaver https://github.com/jdan/cleaver [1]: https://github.com/numpy/numpy https://github.com/numpy/numpy
- sheetjs 13y ago
  To an extent, projects that lean more on github tend to receive more github stars. I don't know when numpy switched over to github, but their website still uses SF to distribute the official binaries. On the other hand, many projects initially started as github repos and naturally draw people that way. As an example, consider d3, where the main page rests on d3js.org but most of the links point to the github project wiki.
- beliu 13y ago
  prezjordan, you should check out Sourcegraph (https://sourcegraph.com https://sourcegraph.com). It shows you how many other projects/people actually use a library and how they use it (code snippets). [0]: https://sourcegraph.com/github.com/jdan/cleaver https://sourcegraph.com/github.com/jdan/cleaver [1]: https://sourcegraph.com/github.com/numpy/numpy https://sourcegraph.com/github.com/numpy/numpy disclaimer: I am one of the creators of Sourcegraph and would love to hear what you think about it!
pedalpete 13y ago
As a few have mentioned, the numbers are flawed, but I see them as flawed differently than others have mentioned. The problem with 'language popularity' is that we looking probably 90% at web-projects being based on Github, so within those projects, CSS and Javascript are pretty much a given, as they are the default client side-libraries. You can't build a website without CSS and rarely do without Javascript (on the front end).

70 points by Adrock 13y ago

Well, his interpretation of the numbers is rather off, I'm afraid. Mostly due to the language categoriziation; Bootstrap, Node, Zurb Foundation and Adobe Brackets aren't Javascript projects, for example.

laymil 13y ago
I thought that was well addressed by this comment in the post: "Looking at the top repo for each language also exposes a weakness in the methodology: GitHub’s language identification isn’t perfect and there are number of polyglot projects. The top Java repo is Storm, which uses enough Clojure (20.1% by GitHub’s measure) to make this identification questionable when you take into account Clojure’s conciseness over Java’s."
Adrock 13y ago
OP here. I agree that the language categorization is questionable. It's all based on what the GitHub Search API returns as the language. Looking at Bootstrap, the classification doesn't seem to be strictly based on percentage: - CSS 59.8% - JavaScript 39.7% - Python 0.5% ...but GitHub Search says JavaScript. I do see that if I do a search query with a language filter, I get different results.
- misterdai 13y ago
  GitHub uses its Linguist project to detect languages and index them as far as I know. I've used the list of languages from this file: https://github.com/github/linguist/blob/master/lib/linguist/languages.yml https://github.com/github/linguist/blob/master/lib/linguist/... I did pick an older version as they don't always run the latest release. Handy as you can filter out items flagged as "markup". Plus, have to watch out for languages like Mirah that are oddly indexed (same repo count as Ruby).

kevinbowman 13y ago

I like that the visualisations are done with D3, the 5th-highest project on the list.

Adrock 13y ago
D3 is awesome. It was really nice when working on the post to just be able to re-run the queries to generate new CSV or JSON and not have to go through a whole separate visualization generation pipeline.

danso 13y ago

I'm really amazed that at how Github is such a vital service to developers everywhere, yet there hasn't been a huge amount of alternative ways to dig for hidden and popular repos across different metrics. Even the metric the OP uses, just pure number of stars, has a lot of interesting things in it, as the OP shows. Maybe it's a reflection of how well-designed Github's site-explore is that people haven't yet reason to augment it. When I say that the even pure star count has a lot of interesting things in it...I mean even in the simplest sense. I know or have used all of those repos with the exception of jQuery-File-Upload and adobe/brackets, but I can be a real dilettante. But in the top 100, there's plenty that I've never heard of. Even within plain Ruby-land, I'm always running into gems that have been around for a few years or are highly starred (Spring and Middleman among them, in the last couple of weeks). What I'd like to see is even further categorization of the gems...something similar to the Ruby Toolbox (https://www.ruby-toolbox.com/categories/Active_Record_DB_Adapters https://www.ruby-toolbox.com/categories/Active_Record_DB_Ada...)...where you can find repos based on use-case, not just language. I do like Toolbox's emphasis on release dates and latest updates, as well as pure total download numbers. I've found that number of contributors, rate of pull requests/issues dealt with, and rate of recent updates, is as useful a metric as star count.

hashtree 13y ago

As to the language rankings, some are at a large disadvantage when their standard libs are outstanding. A prime example being Julia (http://docs.julialang.org/en/latest/stdlib/ http://docs.julialang.org/en/latest/stdlib/). The same goes for smaller languages like Roy (https://github.com/puffnfresh/roy https://github.com/puffnfresh/roy) since GitHub doesn't detect them.

misterdai 13y ago

Interesting post, it's worth playing around with the GitHub API to see what you can uncover. Watch out with basing information off their search API, sometimes it seems to miss users or repositories from being indexed, screwing up your results. Caused me a few headaches recently as I've a little site built on daily statistics collected from GitHub. For example: Mojomdo vanished from their user results making Linus Torvalds most followed user, fixed on GitHub now but still evident on my site (boo). Here's my site for anyone curious: http://hubreports.yougeezer.co.uk http://hubreports.yougeezer.co.uk

chimeracoder 13y ago

This is an interesting idea, but it suffers from one maor problem: Github's language detection is rather inflexible, and sometimes misleading. For example: I have a number of small Go projects that are incorrectly categorized as CSS or Javascript. One such example is a project explicitly designed to be a Go webserver that you can just "go get" (or git clone) and run immediately, with no extra setup, sort of like Twitter Bootstrap, but for the backend: https://github.com/ChimeraCoder/go-server-bootstrap https://github.com/ChimeraCoder/go-server-bootstrap Because it self-hosts the Bootstrap files, and because the lines of Go code required are relatively small, less than 5% of this project is detected as being written in Go, even though I didn't write a single line of Javascript or CSS myself for the entire project. (Yes, I know I could pull in the Bootstrap, etc. from a CDN, but this is just to illustrate the limitations of an automatic language detection model that the developer has zero control over).

vezzy-fnord 13y ago
Linguist also has an odd tendency to miscategorize projects as being written in Prolog, thus inflating the repository rates of an already very niche language. Perhaps it has something to do with (at least in part) Prolog and Perl sharing the same extensions?
- draegtun 13y ago
  Same problem happens with Rebol which often get categorised as R on Github (both share .r extension). some ref: https://github.com/search?l=R&q=rebol https://github.com/search?l=R&q=rebol | https://github.com/search?l=R&q=rebol&type=Code https://github.com/search?l=R&q=rebol&type=Code
thedaniel 13y ago
> an automatic language detection model that the developer has zero control over Pull requests welcome :) https://github.com/github/linguist https://github.com/github/linguist
- chimeracoder 13y ago

deleted 13y ago

[deleted]

EGreg 13y ago

Said no more counting dollars, we'll be counting stars....

Adrock 13y ago
OP here. I was waiting for that :P
- EGreg 13y ago
  Glad to oblige :) EDIT: I just realized that the entire song can be about switching to open source development on github. http://youtu.be/hT_nvWreIhg http://youtu.be/hT_nvWreIhg A young, but not that bold, startup founder is losing sleep, dreaming about the things that they can be. He doesnt think the world is sold, he's just doing what they're told by VCs. All his risky career decisions make him feel alive. And sure enough like many other startups his hits the deadpool. Take that money watch it burn, sink in the river, the lessons are learned. So he moves to release the tech as open source software and starts counting the stars on github instead of dollars... LOL

prezjordan 13y ago

I have a strange relationship with GitHub stars. On one hand, I love them - they give my projects exposure (I always mention cleaver [0] during my interviews) and let me know how many people are "using" my project. On the other hand, I feel it sort of pushes me in the wrong direction. Stars aren't a good indicator of use, or even importance. For instance, cleaver is not technically challenging, yet it shares the same number of stars as numpy [1]. So, not sure how I feel about them - not sure what they measure. [0]: https://github.com/jdan/cleaver https://github.com/jdan/cleaver [1]: https://github.com/numpy/numpy https://github.com/numpy/numpy

sheetjs 13y ago
To an extent, projects that lean more on github tend to receive more github stars. I don't know when numpy switched over to github, but their website still uses SF to distribute the official binaries. On the other hand, many projects initially started as github repos and naturally draw people that way. As an example, consider d3, where the main page rests on d3js.org but most of the links point to the github project wiki.
beliu 13y ago
prezjordan, you should check out Sourcegraph (https://sourcegraph.com https://sourcegraph.com). It shows you how many other projects/people actually use a library and how they use it (code snippets). [0]: https://sourcegraph.com/github.com/jdan/cleaver https://sourcegraph.com/github.com/jdan/cleaver [1]: https://sourcegraph.com/github.com/numpy/numpy https://sourcegraph.com/github.com/numpy/numpy disclaimer: I am one of the creators of Sourcegraph and would love to hear what you think about it!

pedalpete 13y ago

As a few have mentioned, the numbers are flawed, but I see them as flawed differently than others have mentioned. The problem with 'language popularity' is that we looking probably 90% at web-projects being based on Github, so within those projects, CSS and Javascript are pretty much a given, as they are the default client side-libraries. You can't build a website without CSS and rarely do without Javascript (on the front end).