6 ms·

Oh hey there, I'm the author of this project. Just wanted to say thank you for sharing this! Would be happy to answer any questions - graphs are my long time h

by anvaka 7y ago

Oh hey there, I'm the author of this project. Just wanted to say thank you for sharing this! Would be happy to answer any questions - graphs are my long time hobby, and I love them! PS: You can find more recent graphs and fun projects here: https://twitter.com/search?q=from%3Aanvaka%20min_retweets%3A20&src=typed_query https://twitter.com/search?q=from%3Aanvaka%20min_retweets%3A...

bobosha 7y ago
How does this infer subreddit similarity? For instance, I checked for r/AskHistorians and the results don't seem that relevant. edit: never mind, I just read your GitHub readme. But the question still stands as if "users posting in x, also posted in y" is a good way to infer similarity. Could comparing top-ranked posts be a better comparator?
- I_am_tiberius 7y ago
  I guess the amazon principle: Users who posted here, also posted there?
- catach 7y ago
  The method seems surprisingly effective for a lot of subs, but also bothers me.
- anvaka 7y ago
  I used my own metric that is based on jaccard similarity. Which in turn is based on "users who posted to X also posted to Y" metric. That said, there are a few subreddits that are too popular and similarity results were too saturated (/r/videos, /r/funny, etc.) so I did a manual override by looking into most commonly mentioned other subreddits, and sometimes into `about` blurb of subreddit). Please don't consider these recommendation as source of truth! It's just a fun way to discover other subreddits :). I'm also very open to change this metric to something else - please let me know if you have any recommendations! [1]: https://github.com/anvaka/sayit#the-data https://github.com/anvaka/sayit#the-data - describes the data, indexing scripts are here: https://github.com/anvaka/sayit/tree/master/scripts https://github.com/anvaka/sayit/tree/master/scripts [2]: Manual overrides can be found here https://github.com/anvaka/sayit-data#sayit---recommendation-data https://github.com/anvaka/sayit-data#sayit---recommendation-...
  - ketralnis 7y ago
    I work for reddit and here I’ve used a similar technique to your jaccard distance but with one twist: divide by the size of the smaller subreddit (in your case, the number of unique posters that you’ve recorded). That gives you a directional relatedness, that is programming->python but not necessarily python->programming. Used this way you account for the giant subreddit problem automatically but now the results are less “amitheasshole is related to askreddit” and more like “linguisticshumor is a more niche version of linguistics”. The great thing is that it’s actually more actionable as far as recommendations go! Everybody has already heard of the bigger version of this subreddit, but they probably haven’t heard of the smaller versions. And it’s self-correcting. As a subreddit gets bigger we are less likely to recommend it (which is great because it needs our help less)
    - jcims 7y ago
      Can people still make their upvotes/downvotes public? I found that useful for sussing out related subreddits when it was possible back in the day.
    - anvaka 7y ago
      This is super awesome, thank you for sharing! If you guys are interested in seeing how your recommendation work for the entire reddit, I'd be happy to build you a spaceship similar to this one https://github.com/anvaka/word2vec-graph https://github.com/anvaka/word2vec-graph . I couldn't find an easy way to download the entire recommendation graph, but it would be awesome if we could make it work. My email is the same as this account at gmail, and twitter is all open: https://twitter.com/anvaka https://twitter.com/anvaka
      - yantrams 7y ago
        In case you haven't come across it already, here is a very exhaustive list of distance measures for dealing with problems of this kind - http://www.iiisci.org/journal/CV$/sci/pdfs/GS315JG.pdf http://www.iiisci.org/journal/CV$/sci/pdfs/GS315JG.pdf I fooled around a bit with lastfm data for band recommendation and found this sheet quite helpful. If you are interested in learning more about asymmetrical similarity, here is a great primer by Tversky - http://www.cogsci.ucsd.edu/~coulson/203/tversky-features.pdf http://www.cogsci.ucsd.edu/~coulson/203/tversky-features.pdf
- varjag 7y ago
  Have to say the link between The_Donald and TwoXChromosomes was a surprising one.
  - Mountain_Skies 7y ago
    Probably due to both subreddits going on trolling raids on each other.
    - varjag 7y ago
      Reddit tries to keep a lid on brigading, it's actively policed. Surprising to see this as a major component.
      - MagnumOpus 7y ago
        Reddit execs give lip service to fighting brigading. It is only enforced in rare cases.
ptah 7y ago
any information about coverage? r/electribe is not in auto complete and generates a graph with only itself
stared 7y ago
Thank you for creating this project! I share your interest in creating graphs (vide some older project of mine, a map of Stack Exchange tags https://p.migdal.pl/tagoverflow/ https://p.migdal.pl/tagoverflow/). When I dived into various visualization of Subreddits (as a background reading for this side-project https://observablehq.com/@stared/tree-of-reddit-sex-life https://observablehq.com/@stared/tree-of-reddit-sex-life), your vis was the best for exploration (out of many, many). The only other one that I found and this level was a Sigma.js-based https://www.jacobsilterra.com/subreddit_map/network/ https://www.jacobsilterra.com/subreddit_map/network/ (beautiful, but more static).
- anvaka 7y ago
  Yay! Thank you for sharing the links and thank you for your kind words! I see you are doing something with quantum tensors, which is very impressive! I'm still struggling with concept of regular tensors, and would love to have a good intuition/visualization for them - do you have any pointers?
  - stared 7y ago
    It depends on what you are looking for. Tensors as in physics (typically with each dimension being related to space(time)), tensors for quantum information & computing, tensors for deep learning? I've found it illuminating to see tensor diagrams. - https://medium.com/@pmigdal/in-the-topic-of-diagrams-i-did-write-a-review-simple-diagrams-of-convoluted-neural-networks-6418a63f9281 https://medium.com/@pmigdal/in-the-topic-of-diagrams-i-did-w... - https://www.math3ma.com/blog/matrices-as-tensor-network-diagrams https://www.math3ma.com/blog/matrices-as-tensor-network-diag... Also, right now, we are developing a matrix visualization in https://github.com/Quantum-Game https://github.com/Quantum-Game. However, it is pretty much work in progress; for a slightly more mature one, go to Quantum Game 2 website and in the element encyclopedia there is one. I am always up for talking about tensors, so feel invited to drop me an email.
ddoeth 7y ago
I just saw your city roads project, that is so awesome! Thanks for doing that
flanbiscuit 7y ago
I love your npm package dependency tree graph, I've actually posted links to it very recently on both HN and Reddit. Awesome tool! https://npm.anvaka.com/#/ https://npm.anvaka.com/#/
- anvaka 7y ago
  Oh thank you! I'm very glad you liked it :)!
throw_m239339 7y ago
Thank you, it's quite a useful app.
hooande 7y ago
great UI for this. very novel and intuitive
- anvaka 7y ago
  Thank you! I'm very happy to hear this!
papln 7y ago
Small request: Can you turn off the initial jumble of nodes before the graph is built and laid out? It's distracting clutter that adds no information. Animations are fun, but only if they convey artistic or intellectual meaning. Also, the default zoom level is zoomed in too far (large text, not showing the wohle graph). perhaps this is because viewport resolution/size (phone vs desktop) is not taken into account when chooseing a zoom level (font size)
wsinks 7y ago
This is so cool! You've basically made a different, better version of r/findasubreddit. Which led me to think... https://anvaka.github.io/sayit/?query=findasubreddit https://anvaka.github.io/sayit/?query=findasubreddit Super interesting! So many small and unknown subreddits, and one of the main connections is another subreddit called "Somebody Make This". I'll take a peek under the hood later, but this on the surface is very cool.
- anvaka 7y ago
  Thank you :)! Under the hood it is all very naive counting of users who posted to A also posted to B.
chx 7y ago
Thanks, this is very interesting. Mind if I ask how you determine relatedness? https://anvaka.github.io/sayit/?query=UsbCHardware https://anvaka.github.io/sayit/?query=UsbCHardware This subreddit is very small, I am a very active poster there and yet I recognize none of the related ones.
- anvaka 7y ago
  Hi there. Here is a similar question and answer: https://news.ycombinator.com/item?id=22178373 https://news.ycombinator.com/item?id=22178373
- rewq4321 7y ago
  I think for smaller subreddit's there's not as much data, so results are more noise. Apparently Jaccard similarity was used, except for the really big subreddits: https://github.com/anvaka/sayit https://github.com/anvaka/sayit Unrelated, but had to say it somewhere in this discussion: Thanks @anvaka for your open source graph libs! Way more performant than anything else I've tried.
DLA 7y ago
anvaka: your work is just amazing! The viz you did of software repos is outstanding. I just wanted to express thanks to you for sharing as I am a big fan of all things graph as well.

by anvaka 7y ago

bobosha 7y ago

How does this infer subreddit similarity? For instance, I checked for r/AskHistorians and the results don't seem that relevant. edit: never mind, I just read your GitHub readme. But the question still stands as if "users posting in x, also posted in y" is a good way to infer similarity. Could comparing top-ranked posts be a better comparator?

I_am_tiberius 7y ago
I guess the amazon principle: Users who posted here, also posted there?
catach 7y ago
The method seems surprisingly effective for a lot of subs, but also bothers me.
anvaka 7y ago
I used my own metric that is based on jaccard similarity. Which in turn is based on "users who posted to X also posted to Y" metric. That said, there are a few subreddits that are too popular and similarity results were too saturated (/r/videos, /r/funny, etc.) so I did a manual override by looking into most commonly mentioned other subreddits, and sometimes into `about` blurb of subreddit). Please don't consider these recommendation as source of truth! It's just a fun way to discover other subreddits :). I'm also very open to change this metric to something else - please let me know if you have any recommendations! [1]: https://github.com/anvaka/sayit#the-data https://github.com/anvaka/sayit#the-data - describes the data, indexing scripts are here: https://github.com/anvaka/sayit/tree/master/scripts https://github.com/anvaka/sayit/tree/master/scripts [2]: Manual overrides can be found here https://github.com/anvaka/sayit-data#sayit---recommendation-data https://github.com/anvaka/sayit-data#sayit---recommendation-...
- ketralnis 7y ago
  I work for reddit and here I’ve used a similar technique to your jaccard distance but with one twist: divide by the size of the smaller subreddit (in your case, the number of unique posters that you’ve recorded). That gives you a directional relatedness, that is programming->python but not necessarily python->programming. Used this way you account for the giant subreddit problem automatically but now the results are less “amitheasshole is related to askreddit” and more like “linguisticshumor is a more niche version of linguistics”. The great thing is that it’s actually more actionable as far as recommendations go! Everybody has already heard of the bigger version of this subreddit, but they probably haven’t heard of the smaller versions. And it’s self-correcting. As a subreddit gets bigger we are less likely to recommend it (which is great because it needs our help less)
  - jcims 7y ago
    Can people still make their upvotes/downvotes public? I found that useful for sussing out related subreddits when it was possible back in the day.
  - anvaka 7y ago
    This is super awesome, thank you for sharing! If you guys are interested in seeing how your recommendation work for the entire reddit, I'd be happy to build you a spaceship similar to this one https://github.com/anvaka/word2vec-graph https://github.com/anvaka/word2vec-graph . I couldn't find an easy way to download the entire recommendation graph, but it would be awesome if we could make it work. My email is the same as this account at gmail, and twitter is all open: https://twitter.com/anvaka https://twitter.com/anvaka
    - yantrams 7y ago
      In case you haven't come across it already, here is a very exhaustive list of distance measures for dealing with problems of this kind - http://www.iiisci.org/journal/CV$/sci/pdfs/GS315JG.pdf http://www.iiisci.org/journal/CV$/sci/pdfs/GS315JG.pdf I fooled around a bit with lastfm data for band recommendation and found this sheet quite helpful. If you are interested in learning more about asymmetrical similarity, here is a great primer by Tversky - http://www.cogsci.ucsd.edu/~coulson/203/tversky-features.pdf http://www.cogsci.ucsd.edu/~coulson/203/tversky-features.pdf
varjag 7y ago
Have to say the link between The_Donald and TwoXChromosomes was a surprising one.
- Mountain_Skies 7y ago
  Probably due to both subreddits going on trolling raids on each other.
  - varjag 7y ago
    Reddit tries to keep a lid on brigading, it's actively policed. Surprising to see this as a major component.
    - MagnumOpus 7y ago
      Reddit execs give lip service to fighting brigading. It is only enforced in rare cases.

ptah 7y ago

any information about coverage? r/electribe is not in auto complete and generates a graph with only itself

stared 7y ago

Thank you for creating this project! I share your interest in creating graphs (vide some older project of mine, a map of Stack Exchange tags https://p.migdal.pl/tagoverflow/ https://p.migdal.pl/tagoverflow/). When I dived into various visualization of Subreddits (as a background reading for this side-project https://observablehq.com/@stared/tree-of-reddit-sex-life https://observablehq.com/@stared/tree-of-reddit-sex-life), your vis was the best for exploration (out of many, many). The only other one that I found and this level was a Sigma.js-based https://www.jacobsilterra.com/subreddit_map/network/ https://www.jacobsilterra.com/subreddit_map/network/ (beautiful, but more static).

anvaka 7y ago
Yay! Thank you for sharing the links and thank you for your kind words! I see you are doing something with quantum tensors, which is very impressive! I'm still struggling with concept of regular tensors, and would love to have a good intuition/visualization for them - do you have any pointers?
- stared 7y ago
  It depends on what you are looking for. Tensors as in physics (typically with each dimension being related to space(time)), tensors for quantum information & computing, tensors for deep learning? I've found it illuminating to see tensor diagrams. - https://medium.com/@pmigdal/in-the-topic-of-diagrams-i-did-write-a-review-simple-diagrams-of-convoluted-neural-networks-6418a63f9281 https://medium.com/@pmigdal/in-the-topic-of-diagrams-i-did-w... - https://www.math3ma.com/blog/matrices-as-tensor-network-diagrams https://www.math3ma.com/blog/matrices-as-tensor-network-diag... Also, right now, we are developing a matrix visualization in https://github.com/Quantum-Game https://github.com/Quantum-Game. However, it is pretty much work in progress; for a slightly more mature one, go to Quantum Game 2 website and in the element encyclopedia there is one. I am always up for talking about tensors, so feel invited to drop me an email.

ddoeth 7y ago

I just saw your city roads project, that is so awesome! Thanks for doing that

flanbiscuit 7y ago

I love your npm package dependency tree graph, I've actually posted links to it very recently on both HN and Reddit. Awesome tool! https://npm.anvaka.com/#/ https://npm.anvaka.com/#/

anvaka 7y ago
Oh thank you! I'm very glad you liked it :)!

throw_m239339 7y ago

Thank you, it's quite a useful app.

hooande 7y ago

great UI for this. very novel and intuitive

anvaka 7y ago
Thank you! I'm very happy to hear this!

papln 7y ago

Small request: Can you turn off the initial jumble of nodes before the graph is built and laid out? It's distracting clutter that adds no information. Animations are fun, but only if they convey artistic or intellectual meaning. Also, the default zoom level is zoomed in too far (large text, not showing the wohle graph). perhaps this is because viewport resolution/size (phone vs desktop) is not taken into account when chooseing a zoom level (font size)

wsinks 7y ago

This is so cool! You've basically made a different, better version of r/findasubreddit. Which led me to think... https://anvaka.github.io/sayit/?query=findasubreddit https://anvaka.github.io/sayit/?query=findasubreddit Super interesting! So many small and unknown subreddits, and one of the main connections is another subreddit called "Somebody Make This". I'll take a peek under the hood later, but this on the surface is very cool.

anvaka 7y ago
Thank you :)! Under the hood it is all very naive counting of users who posted to A also posted to B.

chx 7y ago

Thanks, this is very interesting. Mind if I ask how you determine relatedness? https://anvaka.github.io/sayit/?query=UsbCHardware https://anvaka.github.io/sayit/?query=UsbCHardware This subreddit is very small, I am a very active poster there and yet I recognize none of the related ones.

anvaka 7y ago
Hi there. Here is a similar question and answer: https://news.ycombinator.com/item?id=22178373 https://news.ycombinator.com/item?id=22178373
rewq4321 7y ago
I think for smaller subreddit's there's not as much data, so results are more noise. Apparently Jaccard similarity was used, except for the really big subreddits: https://github.com/anvaka/sayit https://github.com/anvaka/sayit Unrelated, but had to say it somewhere in this discussion: Thanks @anvaka for your open source graph libs! Way more performant than anything else I've tried.

DLA 7y ago

anvaka: your work is just amazing! The viz you did of software repos is outstanding. I just wanted to express thanks to you for sharing as I am a big fan of all things graph as well.