5 ms·

Show HN: GitMap – Interactive map of top 10,000 repos clustered into ecosystems

79 points by oracleofnj 11y ago

masukomi 11y ago
I really want to be able to run this for a specified repository
andrewstuart2 11y ago
This is really cool, and props for making it at least dynamic-ish by rendering JSON data with D3. That said, some of the clustering (or at least the clustering visualization) seems strange, such as AngularJS within jQuery, PouchDB within React, EmberJS within Rails, redis-doc being included within Go, oh-my-zsh within homebrew, etc. FWIW, there's also a bit more info available here: https://github.com/oracleofnj/gitmap https://github.com/oracleofnj/gitmap
- oracleofnj 11y ago
  Thanks for the feedback! I have a few ideas for how the clustering could be improved at the higher levels. The re-clustering step I'm doing right now is pretty simplistic.
- deleted 11y ago
  [deleted]
- deleted 11y ago
  [deleted]
- deleted 11y ago
  [deleted]
wyclif 11y ago
I was surprised to see a giant Jekyll cluster.
andy_ppp 11y ago
Amazing to visually see React in such a short amount of time is nearly as large an ecosystem as Jquery? Seems so difficult to believe, I question if I'm reading the data correctly! The pace at which new things happen that you have to know is incredible.
- oracleofnj 11y ago
  I wouldn't take it too literally since as I mentioned in another comment, the higher-order clustering I'm doing is fairly simplistic. But interestingly enough, the initial reason I started working on this was to try and figure out what other ecosystems on GitHub had as active a community as React.
  - andy_ppp 11y ago
    The interesting thing to do would be the things that go from nothing to many quickly and then learn them :-) (hint, elixir)...
- karavelov 11y ago
  Yes, when you put all the scala/akka/play/spray/netty/typesafe/typelevel/gloovy/gradle/etc project inside the React cluster - I don't see the connection.
  - kami8845 11y ago
    Well, React.js _was_ made by Facebook.
    - karavelov 11y ago
      and not of the other mentioned technologies was made by Facebook, so I don't see the connection
CiPHPerCoder 11y ago
This is cool. TIL paragonie/awesome-appsec is in the top 10,000, but other projects like paragonie/random_compat aren't :(
agentgt 11y ago
For some other clustering algorithms (albeit based on terms and semantics ie document clustering and not necessarily relationships) one might be interested in Carrot2[1]. Carrot is neat because you can change/tune the clustering algorithms on the fly something I think would make this gitmap more interesting. It would be interesting to see document clustering applied to code (I have not seen it but it seems it might even be more reliable than affinity). That being said ... a lot of memory will be needed since most of the clustering algorithms implementations need to run in memory :) [1]: http://project.carrot2.org/ http://project.carrot2.org/
mc808 11y ago
It's kind of interesting that almost all of the top projects are programming tools. Programmers building better programming tools to help build better programming tools. I wonder what nice things we could have if we diverted half of that effort into end-user projects with the tools we've already got.
- andrewstuart2 11y ago
  I think that's simply an artifact of the fact that client projects using these open source projects are often not themselves open sourced, or hosted on GitHub.
- r3bl 11y ago
  Well who knows programmers better than programmers themselves? And, if programmers decide to give their result back to the community, who are they going to give it back to before giving back to their own community? There's plenty of open source projects that are not aiming for the programmers, but since they're not aiming them, how are they going to become popular on a site that's mainly for the programmers if the programmers themselves cannot use it in their own advantage?
- androidlives 11y ago
  I think this was one reason for PHP and MySQL's initial success. There were simply a lot of useful open source end user projects using it (WordPress, phpMyAdmin, MediaWiki, Drupal, SugarCRM, phpBB, Gallery, Magento) . Say what you will about the coding style and issues with these applications, but millions of people built real businesses on top of them. And I don't remember feeling like I was falling behind if I didn't write my application an entirely new way every few months. I was mostly working on solving a problem for a paying customer.
stared 11y ago
By any chance could you post repos/stars data? (I guess downloading it would be much faster than crawling it.) In any case - thank you a lot for sharing the source code! :)
- oracleofnj 11y ago
  I added a file "starcounts.json.gz" to the repository.
  - stared 11y ago
    Thanks, but I meant more "cached_repos.json.gz" and "cached_users.json.gz". It took 2 nights of fetching and it is nowhere near there (~2100 crawled repos, and users).

79 points by oracleofnj 11y ago

I really want to be able to run this for a specified repository

This is really cool, and props for making it at least dynamic-ish by rendering JSON data with D3. That said, some of the clustering (or at least the clustering visualization) seems strange, such as AngularJS within jQuery, PouchDB within React, EmberJS within Rails, redis-doc being included within Go, oh-my-zsh within homebrew, etc. FWIW, there's also a bit more info available here: https://github.com/oracleofnj/gitmap https://github.com/oracleofnj/gitmap

oracleofnj 11y ago
Thanks for the feedback! I have a few ideas for how the clustering could be improved at the higher levels. The re-clustering step I'm doing right now is pretty simplistic.
deleted 11y ago
[deleted]
deleted 11y ago
[deleted]
deleted 11y ago
[deleted]

wyclif 11y ago

I was surprised to see a giant Jekyll cluster.

andy_ppp 11y ago

Amazing to visually see React in such a short amount of time is nearly as large an ecosystem as Jquery? Seems so difficult to believe, I question if I'm reading the data correctly! The pace at which new things happen that you have to know is incredible.

oracleofnj 11y ago
I wouldn't take it too literally since as I mentioned in another comment, the higher-order clustering I'm doing is fairly simplistic. But interestingly enough, the initial reason I started working on this was to try and figure out what other ecosystems on GitHub had as active a community as React.
- andy_ppp 11y ago
  The interesting thing to do would be the things that go from nothing to many quickly and then learn them :-) (hint, elixir)...
karavelov 11y ago
Yes, when you put all the scala/akka/play/spray/netty/typesafe/typelevel/gloovy/gradle/etc project inside the React cluster - I don't see the connection.
- kami8845 11y ago
  Well, React.js _was_ made by Facebook.
  - karavelov 11y ago
    and not of the other mentioned technologies was made by Facebook, so I don't see the connection

CiPHPerCoder 11y ago

This is cool. TIL paragonie/awesome-appsec is in the top 10,000, but other projects like paragonie/random_compat aren't :(

agentgt 11y ago

For some other clustering algorithms (albeit based on terms and semantics ie document clustering and not necessarily relationships) one might be interested in Carrot2[1]. Carrot is neat because you can change/tune the clustering algorithms on the fly something I think would make this gitmap more interesting. It would be interesting to see document clustering applied to code (I have not seen it but it seems it might even be more reliable than affinity). That being said ... a lot of memory will be needed since most of the clustering algorithms implementations need to run in memory :) [1]: http://project.carrot2.org/ http://project.carrot2.org/

mc808 11y ago

It's kind of interesting that almost all of the top projects are programming tools. Programmers building better programming tools to help build better programming tools. I wonder what nice things we could have if we diverted half of that effort into end-user projects with the tools we've already got.

andrewstuart2 11y ago
I think that's simply an artifact of the fact that client projects using these open source projects are often not themselves open sourced, or hosted on GitHub.
r3bl 11y ago
Well who knows programmers better than programmers themselves? And, if programmers decide to give their result back to the community, who are they going to give it back to before giving back to their own community? There's plenty of open source projects that are not aiming for the programmers, but since they're not aiming them, how are they going to become popular on a site that's mainly for the programmers if the programmers themselves cannot use it in their own advantage?
androidlives 11y ago
I think this was one reason for PHP and MySQL's initial success. There were simply a lot of useful open source end user projects using it (WordPress, phpMyAdmin, MediaWiki, Drupal, SugarCRM, phpBB, Gallery, Magento) . Say what you will about the coding style and issues with these applications, but millions of people built real businesses on top of them. And I don't remember feeling like I was falling behind if I didn't write my application an entirely new way every few months. I was mostly working on solving a problem for a paying customer.

stared 11y ago

By any chance could you post repos/stars data? (I guess downloading it would be much faster than crawling it.) In any case - thank you a lot for sharing the source code! :)

oracleofnj 11y ago
I added a file "starcounts.json.gz" to the repository.
- stared 11y ago
  Thanks, but I meant more "cached_repos.json.gz" and "cached_users.json.gz". It took 2 nights of fetching and it is nowhere near there (~2100 crawled repos, and users).