5 ms·

This is an interesting idea, but it suffers from one maor problem: Github's language detection is rather inflexible, and sometimes misleading. For example: I h

This is an interesting idea, but it suffers from one maor problem: Github's language detection is rather inflexible, and sometimes misleading. For example: I have a number of small Go projects that are incorrectly categorized as CSS or Javascript. One such example is a project explicitly designed to be a Go webserver that you can just "go get" (or git clone) and run immediately, with no extra setup, sort of like Twitter Bootstrap, but for the backend: https://github.com/ChimeraCoder/go-server-bootstrap https://github.com/ChimeraCoder/go-server-bootstrap Because it self-hosts the Bootstrap files, and because the lines of Go code required are relatively small, less than 5% of this project is detected as being written in Go, even though I didn't write a single line of Javascript or CSS myself for the entire project. (Yes, I know I could pull in the Bootstrap, etc. from a CDN, but this is just to illustrate the limitations of an automatic language detection model that the developer has zero control over).

vezzy-fnord 13y ago
Linguist also has an odd tendency to miscategorize projects as being written in Prolog, thus inflating the repository rates of an already very niche language. Perhaps it has something to do with (at least in part) Prolog and Perl sharing the same extensions?
- draegtun 13y ago
  Same problem happens with Rebol which often get categorised as R on Github (both share .r extension). some ref: https://github.com/search?l=R&q=rebol https://github.com/search?l=R&q=rebol | https://github.com/search?l=R&q=rebol&type=Code https://github.com/search?l=R&q=rebol&type=Code
thedaniel 13y ago
> an automatic language detection model that the developer has zero control over Pull requests welcome :) https://github.com/github/linguist https://github.com/github/linguist
- chimeracoder 13y ago
  I've seen that, and this is the corresponding issue: https://help.github.com/articles/my-repository-is-marked-as-the-wrong-language https://help.github.com/articles/my-repository-is-marked-as-.... Looking at vendor.yml, I'm actually not sure why this is happening. The repository I linked to above only has two files that end in .js: bootstrap.js and bootstrap.min.js[0], and only one that ends in .css (bootstrap.css). Unless I'm misreading the regex in vendor.yml, it seems like they should be excluded[1] but they aren't, and this isn't the only project of mine for which this is happening. I'm fairly certain it's picking up those files and not miscategorizing the Go files for some reason, because when I delete those files and instead serve the CSS/Javascript from a public cache, it categorizes the repository correctly, as is the case with this project (based off of go-server-bootstrap): https://github.com/ChimeraCoder/pluto https://github.com/ChimeraCoder/pluto [0] https://github.com/ChimeraCoder/go-server-bootstrap/tree/master/public/static/bootstrap/js https://github.com/ChimeraCoder/go-server-bootstrap/tree/mas... [1]https://github.com/github/linguist/blob/master/lib/linguist/vendor.yml#L34 https://github.com/github/linguist/blob/master/lib/linguist/...
  - coyotebush 13y ago
    So the jQuery regex, for comparison, is (^|/)jquery([^.]*)(\.min)?\.js$ while the Bootstrap regex in question is (^|/)bootstrap([^.]*)(\.min)\.(js|css)$ with the ".min" not optional. (Looks like this was the intent [1]). The 68.9%/26.5% percentages match the 130k/50k ratio of bootstrap.css to bootstrap.js, so it looks like only bootstrap.min.js is being excluded? [1] https://github.com/github/linguist/commit/d5002ef06a9f3463919a3fe8919844c3aea46fc0 https://github.com/github/linguist/commit/d5002ef06a9f346391...
    - chimeracoder 13y ago
      Thanks for pointing that out. You're right - it does seem like it was intentional. I filed an issue for this though just in case, and perhaps they'll explain the reason: https://github.com/github/linguist/pull/856 https://github.com/github/linguist/pull/856

vezzy-fnord 13y ago

Linguist also has an odd tendency to miscategorize projects as being written in Prolog, thus inflating the repository rates of an already very niche language. Perhaps it has something to do with (at least in part) Prolog and Perl sharing the same extensions?

draegtun 13y ago
Same problem happens with Rebol which often get categorised as R on Github (both share .r extension). some ref: https://github.com/search?l=R&q=rebol https://github.com/search?l=R&q=rebol | https://github.com/search?l=R&q=rebol&type=Code https://github.com/search?l=R&q=rebol&type=Code

thedaniel 13y ago

> an automatic language detection model that the developer has zero control over Pull requests welcome :) https://github.com/github/linguist https://github.com/github/linguist

chimeracoder 13y ago
I've seen that, and this is the corresponding issue: https://help.github.com/articles/my-repository-is-marked-as-the-wrong-language https://help.github.com/articles/my-repository-is-marked-as-.... Looking at vendor.yml, I'm actually not sure why this is happening. The repository I linked to above only has two files that end in .js: bootstrap.js and bootstrap.min.js[0], and only one that ends in .css (bootstrap.css). Unless I'm misreading the regex in vendor.yml, it seems like they should be excluded[1] but they aren't, and this isn't the only project of mine for which this is happening. I'm fairly certain it's picking up those files and not miscategorizing the Go files for some reason, because when I delete those files and instead serve the CSS/Javascript from a public cache, it categorizes the repository correctly, as is the case with this project (based off of go-server-bootstrap): https://github.com/ChimeraCoder/pluto https://github.com/ChimeraCoder/pluto [0] https://github.com/ChimeraCoder/go-server-bootstrap/tree/master/public/static/bootstrap/js https://github.com/ChimeraCoder/go-server-bootstrap/tree/mas... [1]https://github.com/github/linguist/blob/master/lib/linguist/vendor.yml#L34 https://github.com/github/linguist/blob/master/lib/linguist/...
- coyotebush 13y ago
  So the jQuery regex, for comparison, is (^|/)jquery([^.]*)(\.min)?\.js$ while the Bootstrap regex in question is (^|/)bootstrap([^.]*)(\.min)\.(js|css)$ with the ".min" not optional. (Looks like this was the intent [1]). The 68.9%/26.5% percentages match the 130k/50k ratio of bootstrap.css to bootstrap.js, so it looks like only bootstrap.min.js is being excluded? [1] https://github.com/github/linguist/commit/d5002ef06a9f3463919a3fe8919844c3aea46fc0 https://github.com/github/linguist/commit/d5002ef06a9f346391...
  - chimeracoder 13y ago
    Thanks for pointing that out. You're right - it does seem like it was intentional. I filed an issue for this though just in case, and perhaps they'll explain the reason: https://github.com/github/linguist/pull/856 https://github.com/github/linguist/pull/856