5 ms·

I believe Teddit uses scraping; I run a Teddit instance for myself and haven't needed to set up an API key.

KMnO4 3y ago
It would be great for someone to scrape Reddit and expose that information in a format compatible with the official API. So if you call /get-comments/1234 it scrapes post 1234 and returns the JSON object exactly as the official API does. Then third party clients can just point to this endpoint.
- moneywoes 3y ago
  Won’t they just get rate limited or hit with captchas
- rovr138 3y ago
  Currently, just add .json to the end of the url. I have no idea what will happen to this with the changes. If what you suggest is done, even a package, we could probably do a distributed pushshift[1] alternative to aggregate the data like the archiveteam's warrior[2] does and keep publishing the monthly data. [1] - reddit.com/r/pushshift [2] - https://wiki.archiveteam.org/index.php?title=ArchiveTeam_Warrior https://wiki.archiveteam.org/index.php?title=ArchiveTeam_War...
bentcorner 3y ago
From a very brief skim it doesn't appear to scrape: https://codeberg.org/teddit/teddit/src/branch/main/routes/home.js https://codeberg.org/teddit/teddit/src/branch/main/routes/ho... But I'm not really a web dev so I might be misreading things. Also: https://codeberg.org/teddit/teddit/issues/400 https://codeberg.org/teddit/teddit/issues/400
- rovr138 3y ago
  You can (currently?) add .json to any url on reddit and it'll provide a json object. - https://codeberg.org/teddit/teddit/src/commit/b0ce6c52a66987543707d3559d50e4dc2d35a3e8/routes/subreddit.js#L379 https://codeberg.org/teddit/teddit/src/commit/b0ce6c52a66987... - https://codeberg.org/teddit/teddit/src/commit/b0ce6c52a66987543707d3559d50e4dc2d35a3e8/routes/subreddit.js#L642 https://codeberg.org/teddit/teddit/src/commit/b0ce6c52a66987... - https://codeberg.org/teddit/teddit/src/commit/b0ce6c52a66987543707d3559d50e4dc2d35a3e8/routes/home.js#L190 https://codeberg.org/teddit/teddit/src/commit/b0ce6c52a66987...
  - imglorp 3y ago
    That's another feature that that seems on its way out, like locking up the api..
    - rovr138 3y ago
      Do you mean that they've said it or just makes sense? Curious if they've said anything about it.
      - ffpip 3y ago
        The json thing is an API request, so it will come under the new API rate limits.
girfan 3y ago
Right now it seems legal to scrape Reddit. But given their trajectory of making the API fairly expensive to use, do you think it's likely that they would also limit/prohibit scraping (assuming apps like Apollo start scraping as an alternative)?
- xur17 3y ago
  My understanding is that scraping of public websites is generally legal, isn't it?
  - vcanales 3y ago
    Legal, but probably against ToS
    - Semaphor 3y ago
      That ToS is meaningless if you scrape logged out.

It would be great for someone to scrape Reddit and expose that information in a format compatible with the official API. So if you call /get-comments/1234 it scrapes post 1234 and returns the JSON object exactly as the official API does. Then third party clients can just point to this endpoint.

moneywoes 3y ago
Won’t they just get rate limited or hit with captchas
rovr138 3y ago
Currently, just add .json to the end of the url. I have no idea what will happen to this with the changes. If what you suggest is done, even a package, we could probably do a distributed pushshift[1] alternative to aggregate the data like the archiveteam's warrior[2] does and keep publishing the monthly data. [1] - reddit.com/r/pushshift [2] - https://wiki.archiveteam.org/index.php?title=ArchiveTeam_Warrior https://wiki.archiveteam.org/index.php?title=ArchiveTeam_War...

bentcorner 3y ago

From a very brief skim it doesn't appear to scrape: https://codeberg.org/teddit/teddit/src/branch/main/routes/home.js https://codeberg.org/teddit/teddit/src/branch/main/routes/ho... But I'm not really a web dev so I might be misreading things. Also: https://codeberg.org/teddit/teddit/issues/400 https://codeberg.org/teddit/teddit/issues/400

rovr138 3y ago
You can (currently?) add .json to any url on reddit and it'll provide a json object. - https://codeberg.org/teddit/teddit/src/commit/b0ce6c52a66987543707d3559d50e4dc2d35a3e8/routes/subreddit.js#L379 https://codeberg.org/teddit/teddit/src/commit/b0ce6c52a66987... - https://codeberg.org/teddit/teddit/src/commit/b0ce6c52a66987543707d3559d50e4dc2d35a3e8/routes/subreddit.js#L642 https://codeberg.org/teddit/teddit/src/commit/b0ce6c52a66987... - https://codeberg.org/teddit/teddit/src/commit/b0ce6c52a66987543707d3559d50e4dc2d35a3e8/routes/home.js#L190 https://codeberg.org/teddit/teddit/src/commit/b0ce6c52a66987...
- imglorp 3y ago
  That's another feature that that seems on its way out, like locking up the api..
  - rovr138 3y ago
    Do you mean that they've said it or just makes sense? Curious if they've said anything about it.
    - ffpip 3y ago
      The json thing is an API request, so it will come under the new API rate limits.

girfan 3y ago

Right now it seems legal to scrape Reddit. But given their trajectory of making the API fairly expensive to use, do you think it's likely that they would also limit/prohibit scraping (assuming apps like Apollo start scraping as an alternative)?

xur17 3y ago
My understanding is that scraping of public websites is generally legal, isn't it?
- vcanales 3y ago
  Legal, but probably against ToS
  - Semaphor 3y ago
    That ToS is meaningless if you scrape logged out.