6 ms·

Introducing Libscore: A PageRank for JavaScript Libraries

118 points by applecore 12y ago

elvis635 12y ago
This is awesome! Thank you for your work and Stripe and DigitalOcean for the sponsorship I've just noticed that the search is not very good, a part the requirement of having to write exactly the correct name. Even writing "Angular" (which is the name showed in the global data) doesn't work. A fuzzy, case insensitive and autocompleted search would be much more useful
- purpleturtle 12y ago
  Good points :) The problem was that fuzzy search would have been technically overwhelming to implement due to the size of our data sets (1 million sites * avg. # of leaked global variables). Also, it would have resulted in a lot of confusing matches because of how arbitrary JavaScript variable names are. Keeping it to one-for-one case sensitive lookups was the only way to clearly express searching behavior and return accurate data every time. The downside is that we force people to read our homepage how-to to learn how to use it :)
drsim 12y ago
Related: https://github.com/ElbertF/Wappalyzer https://github.com/ElbertF/Wappalyzer Open source, regularly updated detections.
hokkos 12y ago
I really like the Wappalyser chrome extension, it tells you with which technologies the site you browse uses.
zackify 12y ago
Where's ReactJS?
- purpleturtle 12y ago
  Search for "React", which is the variable name React exposes itself under. It's also on the trending data page.
brandedkeyword 12y ago
This sounds pretty similar to http://nerdydata.com http://nerdydata.com which lets you search the source code of webpages. Does this only look at the window variable on webpages?
alandarev 12y ago
I noticed that you actually had the source code in the repository, and later removed it. The commits history still contains all the code. Question: Why did you decide to go private? You might have your reasons, care to share? Otherwise it is worrying: helping open source community, yet not being open source.
- purpleturtle 12y ago
  Cleaning a few things up first :) Star the repo and stay tuned for an update.
floatrock 12y ago
A pagerank for npm would be interesting, arguably more so than just the "most depended on" lists. anyone know if it's been done yet?
- purpleturtle 12y ago
  npm actually publishes its own top lists. what's interesting about the work we've done on libscore is that it shows the end result -- whether a lib was actually ultimately used on a site. npm can tell you download stats, but that's where its data ends.
akamaka 12y ago
It's not working at the moment. All of my searches ended with no results, and it took me a while to notice that the request to the web service is returning a 502. Better error messages would be helpful.
mikejarema 12y ago
I've been using the http://www.ruby-toolbox.com http://www.ruby-toolbox.com popularity score as a similar type of proxy to usefulness, reliability and desirability for ruby gems. It attempts to sum up how widely used and active a particular gem is. Plus the site categorizes gems, so when looking for a solution to a new problem you don't have particular expertise in, you can see at a glance which gems may not only help you out, but also those you can trust will be maintained for awhile.
jack_jennings 12y ago
The score simply being the number of top sites feels pretty difficult to understand in isolation. For instance, knowing that React has a libscore of 203 doesn't mean much if I don't know both that this is out of one million sites polled and the score of some other library that I might already have a vague mental picture of its popularity (i.e. compared to jQuery's 600k+). Other than that, very interesting information to sift through.
- purpleturtle 12y ago
  Exactly. That's why we removed any mention of a "score" from the copywriting before launching. We realized how vague a score could be. We focus on site counts now, since they're raw/unfiltered data.
thomasfromcdnjs 12y ago
We've added Libscore badges to Cdnjs library pages -> http://cdnjs.com/libraries/jQuery http://cdnjs.com/libraries/jQuery You can also see where cdnjs is used -> http://libscore.com/#script:cdnjs.cloudflare.com http://libscore.com/#script:cdnjs.cloudflare.com
owenversteeg 12y ago
FYI: Your SSL is broken. As a result, the links on the Github repo are as well. Would it be possible to do this for CSS frameworks as well? Some CSS frameworks have no Javascript component and are thus undetectable. Thanks!
- thomasfromcdnjs 12y ago
  SSL will be on later tonight, we didn't have the time to make the change before launching today and worried that there might be a bit of down time. I imagine we can programatically pick up on CSS namespaces though...you should create an issue over here -> https://github.com/julianshapiro/libscore https://github.com/julianshapiro/libscore it's a good idea
mrmagooey 12y ago
Is there a way to filter out the jquery scripts?
mynegation 12y ago
It is not PageRank, it is Altavista for JavaScript libraries. PageRank would be computing PageRank on JavaScript library dependency graph.
- purpleturtle 12y ago
  Yeah, the submitter (unaffiliated) came up with that slogan on his/her own. Happy to have admins here change the title if they're willing?
- hyp0 12y ago
  I was thinking actual PageRank would be interesting, ranking libraries by who uses what, transitively. Might be nice to include weightings of app usage by users and downloads (because apps aren't used by other libraries), but just sticking to code-used-by-code could be interesting: So, "one lib/app, one vote" (weighted by how much it is used by other libs/apps), measures coders' evaluation, not app-users'. And not have the search part (which PageRank ranks), just the ranking - e.g. a top 10
- Kiro 12y ago
  Just FYI: The article doesn't even mention PageRank.
- egeozcan 12y ago
  A limited way to achieve PageRank-like statistics could be parsing all the js package repositories (like npm, bower, atmosphere, etc.)
seanconaty 12y ago
> The end result is that developers contribute to open source in a vacuum; they develop, hoping — but never knowing — whether their library is being used at-large. Is popularity is the main reason behind releasing and maintaining open source software?
- purpleturtle 12y ago
  Unlike celebrity culture, popularity in the open source world translates to actual impact on the web. As an author of a popular library, your code plays a direct part in how other developers structure their codebase, and -- depending on the library -- the end user experience. And, yeah, impact/change/popularity (whatever you want to call it) is certainly a main reason behind releasing and maintaining open source software. Perhaps other dominant reasons include giving users differently opionionated alternatives that better suit their workflow, advancing the technical know-how of a field, and simply experimenting for expressiveness' sake.
- pjmlp 12y ago
  Didn't you get the memo? Software development on the 21st century is pop culture driven. What counts is whatever everyone else is using this week on "top of the charts", not how technical good it is.
- gedrap 12y ago
  It's not the main reason, but it's natural to be curious if someone is using code you spent so much time on, and there's nothing wrong or anti-open source about that.
billyhoffman 12y ago
"Before Libscore, front-end developers only had Github star counts as a proxy for their library’s success." Uhhh no. Before Libscore, we only had Wapanalyzer and BuiltWith and Alexa and Ghostry and countless other "crawl and aggregate stats data" services. Cool idea and all, but kinda silly to make claims like this.
- purpleturtle 12y ago
  builtwith doesn't show all js libs. it just whitelists the top 20 i believe. looks like Wapanalyzer might be similar. alexa is also unrelated to broad js penetration detection. not sure how ghostery is relevant. the novelty of libscore is that it detects all js libs (even brand new ones with only 20 sites using it); doesn't use a dumb whitelist filter.
  - AdamTReineke 12y ago
    Great example is to look at Microsoft.com and see a bunch of tiny "MS" prefixed libraries that probably only exist on their web properties.
danr4 12y ago
Not very functional at the moment. Especially if you don't use jQuery - the top 200 are practically useless IMO. That being said, being able to search for a library im already using, and seeing what are the top libraries that are used in conjunction with it would probably help.

118 points by applecore 12y ago

This is awesome! Thank you for your work and Stripe and DigitalOcean for the sponsorship I've just noticed that the search is not very good, a part the requirement of having to write exactly the correct name. Even writing "Angular" (which is the name showed in the global data) doesn't work. A fuzzy, case insensitive and autocompleted search would be much more useful

purpleturtle 12y ago
Good points :) The problem was that fuzzy search would have been technically overwhelming to implement due to the size of our data sets (1 million sites * avg. # of leaked global variables). Also, it would have resulted in a lot of confusing matches because of how arbitrary JavaScript variable names are. Keeping it to one-for-one case sensitive lookups was the only way to clearly express searching behavior and return accurate data every time. The downside is that we force people to read our homepage how-to to learn how to use it :)

drsim 12y ago

Related: https://github.com/ElbertF/Wappalyzer https://github.com/ElbertF/Wappalyzer Open source, regularly updated detections.

hokkos 12y ago

I really like the Wappalyser chrome extension, it tells you with which technologies the site you browse uses.

zackify 12y ago

Where's ReactJS?

purpleturtle 12y ago
Search for "React", which is the variable name React exposes itself under. It's also on the trending data page.

brandedkeyword 12y ago

This sounds pretty similar to http://nerdydata.com http://nerdydata.com which lets you search the source code of webpages. Does this only look at the window variable on webpages?

alandarev 12y ago

I noticed that you actually had the source code in the repository, and later removed it. The commits history still contains all the code. Question: Why did you decide to go private? You might have your reasons, care to share? Otherwise it is worrying: helping open source community, yet not being open source.

purpleturtle 12y ago
Cleaning a few things up first :) Star the repo and stay tuned for an update.

floatrock 12y ago

A pagerank for npm would be interesting, arguably more so than just the "most depended on" lists. anyone know if it's been done yet?

purpleturtle 12y ago
npm actually publishes its own top lists. what's interesting about the work we've done on libscore is that it shows the end result -- whether a lib was actually ultimately used on a site. npm can tell you download stats, but that's where its data ends.

akamaka 12y ago

It's not working at the moment. All of my searches ended with no results, and it took me a while to notice that the request to the web service is returning a 502. Better error messages would be helpful.

mikejarema 12y ago

I've been using the http://www.ruby-toolbox.com http://www.ruby-toolbox.com popularity score as a similar type of proxy to usefulness, reliability and desirability for ruby gems. It attempts to sum up how widely used and active a particular gem is. Plus the site categorizes gems, so when looking for a solution to a new problem you don't have particular expertise in, you can see at a glance which gems may not only help you out, but also those you can trust will be maintained for awhile.

jack_jennings 12y ago

The score simply being the number of top sites feels pretty difficult to understand in isolation. For instance, knowing that React has a libscore of 203 doesn't mean much if I don't know both that this is out of one million sites polled and the score of some other library that I might already have a vague mental picture of its popularity (i.e. compared to jQuery's 600k+). Other than that, very interesting information to sift through.

purpleturtle 12y ago
Exactly. That's why we removed any mention of a "score" from the copywriting before launching. We realized how vague a score could be. We focus on site counts now, since they're raw/unfiltered data.

thomasfromcdnjs 12y ago

We've added Libscore badges to Cdnjs library pages -> http://cdnjs.com/libraries/jQuery http://cdnjs.com/libraries/jQuery You can also see where cdnjs is used -> http://libscore.com/#script:cdnjs.cloudflare.com http://libscore.com/#script:cdnjs.cloudflare.com

owenversteeg 12y ago

FYI: Your SSL is broken. As a result, the links on the Github repo are as well. Would it be possible to do this for CSS frameworks as well? Some CSS frameworks have no Javascript component and are thus undetectable. Thanks!

thomasfromcdnjs 12y ago
SSL will be on later tonight, we didn't have the time to make the change before launching today and worried that there might be a bit of down time. I imagine we can programatically pick up on CSS namespaces though...you should create an issue over here -> https://github.com/julianshapiro/libscore https://github.com/julianshapiro/libscore it's a good idea

mrmagooey 12y ago

Is there a way to filter out the jquery scripts?

mynegation 12y ago

It is not PageRank, it is Altavista for JavaScript libraries. PageRank would be computing PageRank on JavaScript library dependency graph.

purpleturtle 12y ago
Yeah, the submitter (unaffiliated) came up with that slogan on his/her own. Happy to have admins here change the title if they're willing?
hyp0 12y ago
I was thinking actual PageRank would be interesting, ranking libraries by who uses what, transitively. Might be nice to include weightings of app usage by users and downloads (because apps aren't used by other libraries), but just sticking to code-used-by-code could be interesting: So, "one lib/app, one vote" (weighted by how much it is used by other libs/apps), measures coders' evaluation, not app-users'. And not have the search part (which PageRank ranks), just the ranking - e.g. a top 10
Kiro 12y ago
Just FYI: The article doesn't even mention PageRank.
egeozcan 12y ago
A limited way to achieve PageRank-like statistics could be parsing all the js package repositories (like npm, bower, atmosphere, etc.)

seanconaty 12y ago

> The end result is that developers contribute to open source in a vacuum; they develop, hoping — but never knowing — whether their library is being used at-large. Is popularity is the main reason behind releasing and maintaining open source software?

purpleturtle 12y ago
Unlike celebrity culture, popularity in the open source world translates to actual impact on the web. As an author of a popular library, your code plays a direct part in how other developers structure their codebase, and -- depending on the library -- the end user experience. And, yeah, impact/change/popularity (whatever you want to call it) is certainly a main reason behind releasing and maintaining open source software. Perhaps other dominant reasons include giving users differently opionionated alternatives that better suit their workflow, advancing the technical know-how of a field, and simply experimenting for expressiveness' sake.
pjmlp 12y ago
Didn't you get the memo? Software development on the 21st century is pop culture driven. What counts is whatever everyone else is using this week on "top of the charts", not how technical good it is.
gedrap 12y ago
It's not the main reason, but it's natural to be curious if someone is using code you spent so much time on, and there's nothing wrong or anti-open source about that.

billyhoffman 12y ago

"Before Libscore, front-end developers only had Github star counts as a proxy for their library’s success." Uhhh no. Before Libscore, we only had Wapanalyzer and BuiltWith and Alexa and Ghostry and countless other "crawl and aggregate stats data" services. Cool idea and all, but kinda silly to make claims like this.

purpleturtle 12y ago
builtwith doesn't show all js libs. it just whitelists the top 20 i believe. looks like Wapanalyzer might be similar. alexa is also unrelated to broad js penetration detection. not sure how ghostery is relevant. the novelty of libscore is that it detects all js libs (even brand new ones with only 20 sites using it); doesn't use a dumb whitelist filter.
- AdamTReineke 12y ago
  Great example is to look at Microsoft.com and see a bunch of tiny "MS" prefixed libraries that probably only exist on their web properties.

danr4 12y ago

Not very functional at the moment. Especially if you don't use jQuery - the top 200 are practically useless IMO. That being said, being able to search for a library im already using, and seeing what are the top libraries that are used in conjunction with it would probably help.