4 ms·

I have emailed Reddit about the legality of this project. (Have not received a response yet) So far it seems likely that I will have to take down the site. So

by xpose2000 14y ago

I have emailed Reddit about the legality of this project. (Have not received a response yet) So far it seems likely that I will have to take down the site. So consider this a small preview I suppose. I plan to port this to a Chrome Extension. Do you guys find this format much more readable than the current completed IAmA?

mariocesar 14y ago
Absolutely, If you add link references, like related news, events or uncommon knowledge this will be perfect. I always imagine someone making a book about the IAMA, there is just so much value on the posts, there are an amazing human view on people. Sometimes amazing, sometimes heartbreaking mostly just fascinating. Great work!
citricsquid 14y ago
I've wanted to build something like this for a long time, the problem I encountered (which I couldn't solve) is how to effectively and automatically find not just the questions and answers, but the discussion that stem from them. A good AMA is one that creates discussion between the subject and commenter's so the majority of good content is buried underneath multiple comments. For example I did an AMA and it had a few hundred comments, the "top" question was pretty mediocre but it had a reply that made it a valuable question and that reply was the most "popular" (most replies, most votes) thing in the AMA. Your site would not have picked this up. If you can solve the problem of dealing with all the nested comments and trying to work out the value of each reply (taking into account votes, length, number of replies and possibly the "relevant" workds?) then your site will absolutely be worth using. Instead of just being a different way to display the data, create extra value. Good AMAs are not just Q&As, they're discussions. > So far it seems likely that I will have to take down the site Traditionally reddit is very good towards community projects, they pride themselves on their "different" approach to community, unless you're causing harm to reddit I don't see why they would require you to take this down. There are multiple tools that already exist that extend reddit and provide extra value, none of which have been taken down -- from what I've seen anyway. Specific to your current implementation of the idea, your font-size is way too low, the focus of the site is the words so make them bigger! The size you have it now isn't very readable. Check our some blogs for examples of what sizes to use for text.
- xpose2000 14y ago
  The problem of dealing with nested comments is very tricky as you mentioned. I did my best to grab the "best" comments, but I know that it does miss some in a deep thread. I will try my best to further enhance its accuracy and ability to continue a discussion. I will also work on the font sizes and perhaps even modify the font choices to further enhance readability. Thanks for your suggestions and feedback! :)
  - sandollars 14y ago
    It looks like you grabbed every comment by the OP along with the comment it was a response too. In at least one instance, I see a situation where someone asked a question, I (not the OP) provided an answer, and the OP then commented on my answer to confirm that my answer was correct. The original question doesn't appear on your website.
twistedanimator 14y ago
Why would scraping their site be illegal?
- jawns 14y ago
  Scraping itself isn't always illegal. But scraping a site and then republishing the content elsewhere often is, unless you have permission.
- xpose2000 14y ago
  I figured displaying their content in full is illegal.
  - twistedanimator 14y ago
    Ah, I hadn't clicked any of the links. I thought you had just linked back to their site. By the way, I like your site. It makes it easy to cut through the cruft that always appears in reddit threads. Do you just grab all responses by the poster and the parent comment?
  - CraigRood 14y ago
    It's an interesting debate, surely the content is owned by the writers?
  - BillSaysThis 14y ago
    If they have a full content RSS feed, then this use should be okay. But I hope it survives their scrutiny since this is vastly more readable then the native view.
Gnolfo 14y ago
I ran into the same problem. I started a pet project that involved scraping reddit (though for a different purpose than AMAs). Their robots.txt and an admin writeup from somewhere on their site made me realize that I'd probably just have to take it down and/or my scraper would just get blacklisted. It's a bummer because there's 1001 great ideas out there for filtering, categorizing, and viewing reddit's data in different ways. And it seems like they encourage 3rd party interaction to some extent with their API and all, yet scraping is kind of needed in most cases. I do like the format for sure. The only thing I would consider is maybe nesting the Q/A divs (.qitem) for threads because a lot of times the Q/A content is contextual to past Q/As. You already order them that way and that helps a lot but on one of the ones I was reading it got confusing on whether they were speaking in the context of a thread or if it was a fresh Q/A. Maybe set it as a view option to toggle or something (maybe have it be a carousel where each frame contains all the Q/A divs in a thread starting with the root level, and keep it displayed flat like they are now).
- pavel_lishin 14y ago
  Why were you scraping? They offer a pretty good API.
  - Gnolfo 14y ago
    User comments, but going by users rather than threads. That way you could get a profile where someone posts, or turn it around and see what prolific posters existed in a given subreddit. The thing is it wouldn't sweep everything. Instead a user would only get scraped if a request was made to my app, and I had a tool that would go through a request queue (storing to my own DB) in a metered way so that reddit only experienced a handful of requests from me per minute. Nonetheless it still breaks robots.txt and if I could dig it up admins have said in the past that don't want automated/batched requests hitting their site.
    - pavel_lishin 14y ago
      Were you using the API, or just scraping HTML?
ebun 14y ago
This is MUCH better than the reddit version. If they're okay with you doing this, I whole-heartedly support your efforts here and wish you the best of luck with this endeavor
bostonvaulter2 14y ago
Why not just use their API instead of scraping? https://github.com/reddit/reddit/wiki/API https://github.com/reddit/reddit/wiki/API
steve8918 14y ago
Very very awesome! I love IAmA's, but the signal to noise ratio is relatively low, so that I can only bear it for about 1 page. This does a fantastic job in separating out the great questions and answers!
redthrowaway 14y ago
I'm pretty sure you'll be fine. Erik's a Pretty Cool Guy(TM); I can't see him trying to take down a site whose express purpose is to make the content on reddit more accessible.
lukevdp 14y ago
Yes I much prefer this format, great job. The only thing thing that I don't like is that the questions are hard to read for me. Maybe because they're large blocks of text in bold.
deleted 14y ago
[deleted]
saurik 14y ago
Why not build this as a browser extension or Greasemonkey script?

mariocesar 14y ago

Absolutely, If you add link references, like related news, events or uncommon knowledge this will be perfect. I always imagine someone making a book about the IAMA, there is just so much value on the posts, there are an amazing human view on people. Sometimes amazing, sometimes heartbreaking mostly just fascinating. Great work!

citricsquid 14y ago

I've wanted to build something like this for a long time, the problem I encountered (which I couldn't solve) is how to effectively and automatically find not just the questions and answers, but the discussion that stem from them. A good AMA is one that creates discussion between the subject and commenter's so the majority of good content is buried underneath multiple comments. For example I did an AMA and it had a few hundred comments, the "top" question was pretty mediocre but it had a reply that made it a valuable question and that reply was the most "popular" (most replies, most votes) thing in the AMA. Your site would not have picked this up. If you can solve the problem of dealing with all the nested comments and trying to work out the value of each reply (taking into account votes, length, number of replies and possibly the "relevant" workds?) then your site will absolutely be worth using. Instead of just being a different way to display the data, create extra value. Good AMAs are not just Q&As, they're discussions. > So far it seems likely that I will have to take down the site Traditionally reddit is very good towards community projects, they pride themselves on their "different" approach to community, unless you're causing harm to reddit I don't see why they would require you to take this down. There are multiple tools that already exist that extend reddit and provide extra value, none of which have been taken down -- from what I've seen anyway. Specific to your current implementation of the idea, your font-size is way too low, the focus of the site is the words so make them bigger! The size you have it now isn't very readable. Check our some blogs for examples of what sizes to use for text.

xpose2000 14y ago
The problem of dealing with nested comments is very tricky as you mentioned. I did my best to grab the "best" comments, but I know that it does miss some in a deep thread. I will try my best to further enhance its accuracy and ability to continue a discussion. I will also work on the font sizes and perhaps even modify the font choices to further enhance readability. Thanks for your suggestions and feedback! :)
- sandollars 14y ago
  It looks like you grabbed every comment by the OP along with the comment it was a response too. In at least one instance, I see a situation where someone asked a question, I (not the OP) provided an answer, and the OP then commented on my answer to confirm that my answer was correct. The original question doesn't appear on your website.

twistedanimator 14y ago

Why would scraping their site be illegal?

jawns 14y ago
Scraping itself isn't always illegal. But scraping a site and then republishing the content elsewhere often is, unless you have permission.
xpose2000 14y ago
I figured displaying their content in full is illegal.
- twistedanimator 14y ago
  Ah, I hadn't clicked any of the links. I thought you had just linked back to their site. By the way, I like your site. It makes it easy to cut through the cruft that always appears in reddit threads. Do you just grab all responses by the poster and the parent comment?
- CraigRood 14y ago
  It's an interesting debate, surely the content is owned by the writers?
- BillSaysThis 14y ago
  If they have a full content RSS feed, then this use should be okay. But I hope it survives their scrutiny since this is vastly more readable then the native view.

Gnolfo 14y ago

I ran into the same problem. I started a pet project that involved scraping reddit (though for a different purpose than AMAs). Their robots.txt and an admin writeup from somewhere on their site made me realize that I'd probably just have to take it down and/or my scraper would just get blacklisted. It's a bummer because there's 1001 great ideas out there for filtering, categorizing, and viewing reddit's data in different ways. And it seems like they encourage 3rd party interaction to some extent with their API and all, yet scraping is kind of needed in most cases. I do like the format for sure. The only thing I would consider is maybe nesting the Q/A divs (.qitem) for threads because a lot of times the Q/A content is contextual to past Q/As. You already order them that way and that helps a lot but on one of the ones I was reading it got confusing on whether they were speaking in the context of a thread or if it was a fresh Q/A. Maybe set it as a view option to toggle or something (maybe have it be a carousel where each frame contains all the Q/A divs in a thread starting with the root level, and keep it displayed flat like they are now).

pavel_lishin 14y ago
Why were you scraping? They offer a pretty good API.
- Gnolfo 14y ago
  User comments, but going by users rather than threads. That way you could get a profile where someone posts, or turn it around and see what prolific posters existed in a given subreddit. The thing is it wouldn't sweep everything. Instead a user would only get scraped if a request was made to my app, and I had a tool that would go through a request queue (storing to my own DB) in a metered way so that reddit only experienced a handful of requests from me per minute. Nonetheless it still breaks robots.txt and if I could dig it up admins have said in the past that don't want automated/batched requests hitting their site.
  - pavel_lishin 14y ago
    Were you using the API, or just scraping HTML?

ebun 14y ago

This is MUCH better than the reddit version. If they're okay with you doing this, I whole-heartedly support your efforts here and wish you the best of luck with this endeavor

bostonvaulter2 14y ago

Why not just use their API instead of scraping? https://github.com/reddit/reddit/wiki/API https://github.com/reddit/reddit/wiki/API

steve8918 14y ago

Very very awesome! I love IAmA's, but the signal to noise ratio is relatively low, so that I can only bear it for about 1 page. This does a fantastic job in separating out the great questions and answers!

redthrowaway 14y ago

I'm pretty sure you'll be fine. Erik's a Pretty Cool Guy(TM); I can't see him trying to take down a site whose express purpose is to make the content on reddit more accessible.

lukevdp 14y ago

Yes I much prefer this format, great job. The only thing thing that I don't like is that the questions are hard to read for me. Maybe because they're large blocks of text in bold.

deleted 14y ago

[deleted]

saurik 14y ago

Why not build this as a browser extension or Greasemonkey script?