5 ms·
Visualizing relationships between Python packages
- superbeario 10y agoThe chart looks amazing! Will be good to see the result of other programming languages such as scala and java.
- ameliaquining 10y agoI'd be interested to see what this looks like for npm.
- kozikow 10y agoI plan to do the analysis for other languages soon.
- infocollector 10y agoThis might be of interest as well: http://pypi.compgeom.com http://pypi.compgeom.com
- fhoffa 10y agoNice! You might be alyso interested to know that PyPI download stats are also available in BigQuery: https://mail.python.org/pipermail/distutils-sig/2016-May/028986.html https://mail.python.org/pipermail/distutils-sig/2016-May/028... (project recently unveiled by Donald Stufft) I also added your article to the compilation at https://medium.com/@hoffa/github-on-bigquery-analyze-all-the-code-b3576fd2b150 https://medium.com/@hoffa/github-on-bigquery-analyze-all-the..., thanks for your prolific series of analysis!
- kozikow 10y agoThanks! I will join this data with pypi stats.
- stared 10y agoIs there any way to combine downloads for a single user? I.e. to know that the pip downloaded both package X and package Y?
- kozikow 10y ago99% chance that it's not possible and it won't be possible soon. You can take a look at the table at https://bigquery.cloud.google.com/table/the-psf:pypi.downloads20160711 https://bigquery.cloud.google.com/table/the-psf:pypi.downloa... and check out tabs "Schema" and "Preview". Mix of country_code, details* and tls* could give you "rough" grouping, but there would probably be 100s of users in the same bucket. It is probably designed in a way to not let you target single user that way. It's likely that there is a pipeline scrubbing small buckets to avoid privacy leaks. I had to write pipelines like this myself on different datasets.
- elcapitan 10y agoHonest question: Is anybody actually able to extract valuable information out of that type of visualization? They always seem like a bag of wool that my cat would want to play with. Maybe some better plotting algorithm would help?
- kozikow 10y agoI will try out the http://graphistry.com/ http://graphistry.com/ today after work. Zooming and mouse over will help extract information. In the second section of the post I posted some information I extracted. It was hard to balance number of nodes well: - Too few: Interesting clusters like robotics, numpy and openstack start to disappear, as they get dominated by massive clusters like django - Too many: It's hard to see things.
- michaelscott 10y agoThis. Unfortunately there is nothing actionable from this visualisation and, at the risk of sounding like an asshole, it's virtually meaningless. The only piece of information this visualisation does provide is what the largest Python packages are. What's missing is the ability for a user to "separate" clusters into spaces that they care about. Ideally though, the graphic should be able to display the relationships in a way that surfaces visually at a glance.
- kozikow 10y agoI agree. I was planning to add more things, but the weekend ran out. By using graphistry.com I will add: - Zooming in and out. - Mouse over the package to highlight nodes with an edge. - Cluster the packages and let users remove clusters they are not interested in
- michaelscott 10y agoSounds awesome man, keen to see it!
- kozikow 10y ago
- stared 10y agoFor strength of co-occurrence please use pointwise mutual information (or its function), see https://github.com/stared/tagoverflow#methods-and-tricks https://github.com/stared/tagoverflow#methods-and-tricks. I explored quite a few, and only this one gave reasonable results, including when there was a big difference in the number of counts of A and B (see: http://p.migdal.pl/tagoverflow/ http://p.migdal.pl/tagoverflow/). Also, PMI shows up in many contexts, including word2vec. Your measure is at best hacky (no direct interpretation, leaves many small nodes without links). Otherwise, a nice work! But it would be great if visualized more interactively and aesthetically (yeah, I can be biased here).
- kozikow 10y agoLooks great! Today after work I plan to try out version with PMI and https://github.com/graphistry/pygraphistry https://github.com/graphistry/pygraphistry .
- stared 10y agoI don't know this one... when not D3 I was using Gephi or igraph (see http://kateto.net/network-visualization http://kateto.net/network-visualization, it has also Python bindings).
- kozikow 10y agoDoes any of those libraries would support: - Mouse over to highlight neighbours - Zoom in and out - Browser based - Ideally "hide this cluster" functionality I wanted to add some of those features using d3, but the weekend ran out. Also "mouse over" would require me to move the vis from canvas to svg.
- stared 10y agoFor interactive I use D3.js, on SVG. And I can add all functionalities I like. :) In particular, I am thinking about creating a general-purpose co-occurrence graph (after some success of TagOverflow), see e.g. http://p.migdal.pl/delab-matury/koincydencje/ http://p.migdal.pl/delab-matury/koincydencje/. If you like it, you can motivate me to making it a library :). But: there should be exports from igraph to D3.js, and from Gephi to http://sigmajs.org/ http://sigmajs.org/ (<- this is interactive, and higher level that D3.js; I haven't used it, though; with mouseover and zoom).
- Feneric 10y agoIf you pick a fringe package it just points way off center in empty space. It's not immediately clear what's happening. An extra dot for the center might be helpful.
- toolslive 10y agodoes it handle runtime python path trickery ?