4 ms·

OR - Everyone will just run web scrapers increasing the load on reddit's servers even more An API actually REDUCES load and lets you manage it.

jollofricepeas 3y ago
Your solution isn’t a solution. Please answer the question: How is Reddit supposed to make money with increased load and demand for its content?
- dingledork69 3y ago
  Your question isn't the question you should be asking. Please answer the question: How is reddit supposed to prevent scraping when it is legal to do so, and they have incentive to appear in search engines? Considering that scraping is legal and WILL happen, why not opt to reduce the load by offering an API?
  - brabel 3y ago
    If consuming their content by scraping was easy, nobody would be complaining about the APIs being taken away... it's very easy to make web scraping impractical if they want. Don't be deluded.
    - dingledork69 3y ago
      The context of the thread is machine learning companies. You're delusional if you think they don't have a ton of web scrapers running 24/7 already.
- ndriscoll 3y ago
  The framing in the question is wrong. Reddit gets ~80 comments per second on average. An API to get comments with ids starting after n would let you reasonably scrape all new comments with 1 request per second (or one every 10 seconds if the API returned up to 1000 results, which would be entirely reasonable). They get ~10 posts per second, so add another 1 request for that. The load is trivial. Storing the content is also trivial; the entire history of reddit fits very comfortably on a single consumer-tier SSD. The infrastructure they provide is easily replicable at almost no cost (really it's just the bandwidth that costs any money at all). The community curation and moderation is done by volunteers. The content is all from the users. Reddit Inc is providing almost none of the value, and is just benefiting from network effects. People go there because people go there.
  - charcircuit 3y ago
    Reddit hosts images and videos
    - ddingus 3y ago
      Yes, and those are features most Reddit users could care less about and or can be linked easy enough. The real meat and potatoes is plain old text with Reddit style markup.
    - flangola7 3y ago
      Only recently. For most of its history of only stored text.
      - charcircuit 3y ago
        It was 7 years ago.

jollofricepeas 3y ago

Your solution isn’t a solution. Please answer the question: How is Reddit supposed to make money with increased load and demand for its content?

dingledork69 3y ago
Your question isn't the question you should be asking. Please answer the question: How is reddit supposed to prevent scraping when it is legal to do so, and they have incentive to appear in search engines? Considering that scraping is legal and WILL happen, why not opt to reduce the load by offering an API?
- brabel 3y ago
  If consuming their content by scraping was easy, nobody would be complaining about the APIs being taken away... it's very easy to make web scraping impractical if they want. Don't be deluded.
  - dingledork69 3y ago
    The context of the thread is machine learning companies. You're delusional if you think they don't have a ton of web scrapers running 24/7 already.
ndriscoll 3y ago
The framing in the question is wrong. Reddit gets ~80 comments per second on average. An API to get comments with ids starting after n would let you reasonably scrape all new comments with 1 request per second (or one every 10 seconds if the API returned up to 1000 results, which would be entirely reasonable). They get ~10 posts per second, so add another 1 request for that. The load is trivial. Storing the content is also trivial; the entire history of reddit fits very comfortably on a single consumer-tier SSD. The infrastructure they provide is easily replicable at almost no cost (really it's just the bandwidth that costs any money at all). The community curation and moderation is done by volunteers. The content is all from the users. Reddit Inc is providing almost none of the value, and is just benefiting from network effects. People go there because people go there.
- charcircuit 3y ago
  Reddit hosts images and videos
  - ddingus 3y ago
    Yes, and those are features most Reddit users could care less about and or can be linked easy enough. The real meat and potatoes is plain old text with Reddit style markup.
  - flangola7 3y ago
    Only recently. For most of its history of only stored text.
    - charcircuit 3y ago
      It was 7 years ago.