4 ms·

Author here. I wrote a description in the main file, so I don't have to add much here. Though, I am open to feedback (especially for the graphical presentation

by stared 8y ago

Author here. I wrote a description in the main file, so I don't have to add much here. Though, I am open to feedback (especially for the graphical presentation, as the taxonomy & subreddit list is not mine). Even though I like and use D3.js a lot, it is my first post using ObservableHQ. It seemed to me to be a nice choice for this kind of content. Considered bl.ocks.org or repo/GH-page, but I think ObservableHQ makes it easier to edit and share.

Chris2048 8y ago
How did you get the URLs given the reddit history search limit? Was it from the big-query dataset?
- stared 8y ago
  To be honest, I have no idea. See https://github.com/EBazarov/nsfw_data_source_urls/ https://github.com/EBazarov/nsfw_data_source_urls/ for the data source I used. There is a Reddit bulk downloader capable of downloading ~1000 images from a subreddot: https://github.com/aliparlakci/bulk-downloader-for-reddit https://github.com/aliparlakci/bulk-downloader-for-reddit (or just image links, if needed); I tested it and it works well. Though, I have no idea if it was used for the project.
lecarore 8y ago
This is neat, however (here comes the harsh critic) : - Rendering a simple tree like this would work just as well, if better, in text only format (like the output of the "tree" command on linux) : https://justpaste.it/6r9un https://justpaste.it/6r9un - The classification also seems a bit manual and arbitrary. Maybe giving each node a weight would make d3 worth using ? Maybe scrap the subscribers of each subreddit, and make a force directed "map of reddit nsfw" ? Your skills seem wasted on this if you're good at d3
- stared 8y ago
  Your text-based tree is nice, but it comes with pros and cons, as every datq visualization. In fact, it was my first idea, but I felt I need to create something better for exploring the high-level structure. But feel free to do a fork and a viz in this line: https://codepen.io/brendandougan/pen/PpEzRp https://codepen.io/brendandougan/pen/PpEzRp There are quite a few maps of Reddit according to co-participation in subreddits. Though it does not make it semantic. (Maybe semantic things can be extracted from the post content; though, I didn’t try.)
  - wongarsu 8y ago
    If you wanted to do more than just a straight tree: For any pair of subreddits draw a line between the subs weighted according to the number of commenters that commented in both subreddits. Maybe doing a similar graph with people that post in both subreddits would be interesting too. I expect both variants to yield a somewhat more natural grouping of subreddits than an artifically curated list, but making it readable and explorable would be a challenge. 0: http://files.pushshift.io/reddit/comments/ http://files.pushshift.io/reddit/comments/
    - stared 8y ago
      Well, if you want to do it, go for it. If I were to do so, I know more suitable methods, vide description from https://github.com/stared/tagoverflow/ https://github.com/stared/tagoverflow/. Without such normalization, it won't work given that there are networking with different orders of magnitude of users. Right now don't have much time for this project, and well - I am this kind of person that has a few dozen times for idea than time, not the other way. :)
- nsuser3 8y ago
  > The classification also seems a bit manual and arbitrary. That's because it was basically copied from /r/NSFW411's wiki.
  - stared 8y ago
    Thank you a lot, a mystery solved! https://www.reddit.com/r/NSFW411/wiki/index https://www.reddit.com/r/NSFW411/wiki/index
distant_hat 8y ago
It would be great if you could overlay the number of subscribers of a subreddit in it. It would be a neat way to see what is most prominent. Also, and I don't know if this information is possible to get, if you could get intersections of users in different subreddits, you could see a graph of which subreddits are most tightly connected.
- stared 8y ago
  There are such projects, and in fact - I mention two of them. If you want to obtain data, Related subreddits shared it (150GB or so): https://github.com/anvaka/sayit https://github.com/anvaka/sayit
joshstrange 8y ago
This is very interesting! I was wondering what the lack of more LGBT (specifically gay) subreddits is due to and why they are grouped the way they are? What I mean is I know there are are a number more gay-related subreddits and I found it interesting that while straight porn got broken down into it's categories all LGBT porn was lumped together under high-level categories (ie. gay/lesbian/transgender/etc). Edit: Look like I misunderstood what "nsfw_data_source_urls" really was (I thought it was a list of urls to specific images, not a list of subs broken down like it is). I thought it was used to train not the source of the categorization. Given that I understand the grouping more.
deleted 8y ago
[deleted]

by stared 8y ago

Chris2048 8y ago

How did you get the URLs given the reddit history search limit? Was it from the big-query dataset?

stared 8y ago
To be honest, I have no idea. See https://github.com/EBazarov/nsfw_data_source_urls/ https://github.com/EBazarov/nsfw_data_source_urls/ for the data source I used. There is a Reddit bulk downloader capable of downloading ~1000 images from a subreddot: https://github.com/aliparlakci/bulk-downloader-for-reddit https://github.com/aliparlakci/bulk-downloader-for-reddit (or just image links, if needed); I tested it and it works well. Though, I have no idea if it was used for the project.

lecarore 8y ago

This is neat, however (here comes the harsh critic) : - Rendering a simple tree like this would work just as well, if better, in text only format (like the output of the "tree" command on linux) : https://justpaste.it/6r9un https://justpaste.it/6r9un - The classification also seems a bit manual and arbitrary. Maybe giving each node a weight would make d3 worth using ? Maybe scrap the subscribers of each subreddit, and make a force directed "map of reddit nsfw" ? Your skills seem wasted on this if you're good at d3

stared 8y ago
Your text-based tree is nice, but it comes with pros and cons, as every datq visualization. In fact, it was my first idea, but I felt I need to create something better for exploring the high-level structure. But feel free to do a fork and a viz in this line: https://codepen.io/brendandougan/pen/PpEzRp https://codepen.io/brendandougan/pen/PpEzRp There are quite a few maps of Reddit according to co-participation in subreddits. Though it does not make it semantic. (Maybe semantic things can be extracted from the post content; though, I didn’t try.)
- wongarsu 8y ago
  If you wanted to do more than just a straight tree: For any pair of subreddits draw a line between the subs weighted according to the number of commenters that commented in both subreddits. Maybe doing a similar graph with people that post in both subreddits would be interesting too. I expect both variants to yield a somewhat more natural grouping of subreddits than an artifically curated list, but making it readable and explorable would be a challenge. 0: http://files.pushshift.io/reddit/comments/ http://files.pushshift.io/reddit/comments/
  - stared 8y ago
    Well, if you want to do it, go for it. If I were to do so, I know more suitable methods, vide description from https://github.com/stared/tagoverflow/ https://github.com/stared/tagoverflow/. Without such normalization, it won't work given that there are networking with different orders of magnitude of users. Right now don't have much time for this project, and well - I am this kind of person that has a few dozen times for idea than time, not the other way. :)
nsuser3 8y ago
> The classification also seems a bit manual and arbitrary. That's because it was basically copied from /r/NSFW411's wiki.
- stared 8y ago
  Thank you a lot, a mystery solved! https://www.reddit.com/r/NSFW411/wiki/index https://www.reddit.com/r/NSFW411/wiki/index

distant_hat 8y ago

It would be great if you could overlay the number of subscribers of a subreddit in it. It would be a neat way to see what is most prominent. Also, and I don't know if this information is possible to get, if you could get intersections of users in different subreddits, you could see a graph of which subreddits are most tightly connected.

stared 8y ago
There are such projects, and in fact - I mention two of them. If you want to obtain data, Related subreddits shared it (150GB or so): https://github.com/anvaka/sayit https://github.com/anvaka/sayit

joshstrange 8y ago

This is very interesting! I was wondering what the lack of more LGBT (specifically gay) subreddits is due to and why they are grouped the way they are? What I mean is I know there are are a number more gay-related subreddits and I found it interesting that while straight porn got broken down into it's categories all LGBT porn was lumped together under high-level categories (ie. gay/lesbian/transgender/etc). Edit: Look like I misunderstood what "nsfw_data_source_urls" really was (I thought it was a list of urls to specific images, not a list of subs broken down like it is). I thought it was used to train not the source of the categorization. Given that I understand the grouping more.

deleted 8y ago

[deleted]