5 ms·

I mainly look at the ratio of stars to open issues. I find this to be a very good heuristic: http://repoq.com/baidu/Familia/ http://repoq.com/baidu/Familia/ h

I mainly look at the ratio of stars to open issues. I find this to be a very good heuristic: http://repoq.com/baidu/Familia/ http://repoq.com/baidu/Familia/ http://repoq.com/nltk/nltk http://repoq.com/nltk/nltk http://repoq.com/RaRe-Technologies/gensim http://repoq.com/RaRe-Technologies/gensim http://repoq.com/lists/natural_language_processing http://repoq.com/lists/natural_language_processing

techie128 8y ago
China has a population of 1.3B. Isn't it possible that a lot of them have starred it because it is in Mandarin and targeted primarily towards them? You're not accounting for bias.
- JesseAldridge 8y ago
  Presumeably the number of open issues should be proportional, no? A high star/open-issue ratio suggests people like the library and are not having problems with it. (And if they do have problems those problems are addressed -- i.e. issues are getting closed.)
  - yorwba 8y ago
    Closed issues do not necessarily imply that problems are being addressed. Sometimes issues are closed because the maintainer has decided not to address the problem. Sometimes the project simply uses an external bug tracker instead of GitHub issues. Some projects do fix bugs reported in an issue, but they don't have a dedicated person managing issues, so they remain open long after the problem has been solved. I think a high star/open-issue ratio is only a very weak indicator, that you should only use when you have hundreds of repos to evaluate and no domain expertise to decide which actually solves your problem.
    - JesseAldridge 8y ago
      Closing issues without addressing them does indeed present a problem for this approach. The guy behind lodash is especially aggressive about this which is why lodash is the #1 repo on all of GitHub according to this metric (http://repoq.com/lodash/lodash/ http://repoq.com/lodash/lodash/). But I find that people closing issues without addressing them is actually a very rare thing. Far more common is for a project to be abandoned by the maintainer. And this heuristic is quite good at raising a red flag in that situation. Taking a deep dive into a library is of course the most reliable option. But in my experience, really getting an understanding of a large project is a multi-day or even multi-week affair, and this thing serves as an excellent first pass filter.
marcell 8y ago
Isn't more open issues just a sign of a older and/or more widely used library? It's not necessarily a bad thing.
- JesseAldridge 8y ago
  Indeed over time a project will accumulate more stars and issues. It's the ratio that's important. You can think of them as something like upvotes vs downvotes. Except the downvotes are much more meaningful because it takes a drastically greater effort to open a ticket than to star a project.
  - marcell 8y ago
    Sure but issues signify actual usage. You open an issue because you’re using the project and ran into a problem, possibly an edge case. A star can be just a low comittmwnt upvote. I’d almost consider open issue to be a good sign.
bllguo 8y ago
I certainly did not expect you to have built a project around quantifying repo quality. Interesting. Wonder if there are cross-cultural differences in starring/issue-opening frequencies, since the assumption here is that there are none
voxl 8y ago
The ratio does not convey quality. Stars can indicate interest where as issues are correlated with both usage and quality. A star is not nearly as strongly correlated with either usage or quality, it is most strongly correlated with interest. I star a thing because I think it is interesting and what to learn more about it. If I'm writing an issue on a thing I never even have to star it because I am either actively using it (and thus perhaps watching it) or I'm a drive-by issuer that just stumbled on this thing and thus will either star the project because it interests me or not. Stars to Issues is Interest to Usage, the more interesting a thing is an the least used it is the better it scores.
- JesseAldridge 8y ago
  In theory you are correct, but in my practical experience, I've found that the ratio does in fact convey quality quite strongly. The original motivation behind this project was this: I wrote a script to take a screenshot of my desktop every 10 seconds and upload them to S3. I used boto to do the uploading (https://github.com/boto/boto https://github.com/boto/boto). Boto leaked memory and eventually the background script would cause my laptop to crash. When I figured out the problem I noticed boto has a large number of open issues (http://repoq.com/boto/boto http://repoq.com/boto/boto). This indicates that the project is not well maintained. Obviously this is not a perfect heuristic. But it's a lot better than one might think.

techie128 8y ago

China has a population of 1.3B. Isn't it possible that a lot of them have starred it because it is in Mandarin and targeted primarily towards them? You're not accounting for bias.

JesseAldridge 8y ago
Presumeably the number of open issues should be proportional, no? A high star/open-issue ratio suggests people like the library and are not having problems with it. (And if they do have problems those problems are addressed -- i.e. issues are getting closed.)
- yorwba 8y ago
  Closed issues do not necessarily imply that problems are being addressed. Sometimes issues are closed because the maintainer has decided not to address the problem. Sometimes the project simply uses an external bug tracker instead of GitHub issues. Some projects do fix bugs reported in an issue, but they don't have a dedicated person managing issues, so they remain open long after the problem has been solved. I think a high star/open-issue ratio is only a very weak indicator, that you should only use when you have hundreds of repos to evaluate and no domain expertise to decide which actually solves your problem.
  - JesseAldridge 8y ago
    Closing issues without addressing them does indeed present a problem for this approach. The guy behind lodash is especially aggressive about this which is why lodash is the #1 repo on all of GitHub according to this metric (http://repoq.com/lodash/lodash/ http://repoq.com/lodash/lodash/). But I find that people closing issues without addressing them is actually a very rare thing. Far more common is for a project to be abandoned by the maintainer. And this heuristic is quite good at raising a red flag in that situation. Taking a deep dive into a library is of course the most reliable option. But in my experience, really getting an understanding of a large project is a multi-day or even multi-week affair, and this thing serves as an excellent first pass filter.

marcell 8y ago

Isn't more open issues just a sign of a older and/or more widely used library? It's not necessarily a bad thing.

JesseAldridge 8y ago
Indeed over time a project will accumulate more stars and issues. It's the ratio that's important. You can think of them as something like upvotes vs downvotes. Except the downvotes are much more meaningful because it takes a drastically greater effort to open a ticket than to star a project.
- marcell 8y ago
  Sure but issues signify actual usage. You open an issue because you’re using the project and ran into a problem, possibly an edge case. A star can be just a low comittmwnt upvote. I’d almost consider open issue to be a good sign.

bllguo 8y ago

I certainly did not expect you to have built a project around quantifying repo quality. Interesting. Wonder if there are cross-cultural differences in starring/issue-opening frequencies, since the assumption here is that there are none

voxl 8y ago

The ratio does not convey quality. Stars can indicate interest where as issues are correlated with both usage and quality. A star is not nearly as strongly correlated with either usage or quality, it is most strongly correlated with interest. I star a thing because I think it is interesting and what to learn more about it. If I'm writing an issue on a thing I never even have to star it because I am either actively using it (and thus perhaps watching it) or I'm a drive-by issuer that just stumbled on this thing and thus will either star the project because it interests me or not. Stars to Issues is Interest to Usage, the more interesting a thing is an the least used it is the better it scores.

JesseAldridge 8y ago
In theory you are correct, but in my practical experience, I've found that the ratio does in fact convey quality quite strongly. The original motivation behind this project was this: I wrote a script to take a screenshot of my desktop every 10 seconds and upload them to S3. I used boto to do the uploading (https://github.com/boto/boto https://github.com/boto/boto). Boto leaked memory and eventually the background script would cause my laptop to crash. When I figured out the problem I noticed boto has a large number of open issues (http://repoq.com/boto/boto http://repoq.com/boto/boto). This indicates that the project is not well maintained. Obviously this is not a perfect heuristic. But it's a lot better than one might think.