4 ms·

How does your non official API work? Is it using the official API and caching, redirecting to the website, and transforming it into json? I'm just curious.

dom96 3y ago
It basically scrapes reddit.com's HTML. Doesn't use the official API at all (that would defeat the purpose).
- programmarchy 3y ago
  Impressive feat! Does reddit have rate limiting, or other hurdles in place similar to the hoops youtube-dl has to jump through? Curious what your thoughts are about maintaining a project like that.
  - sam0x17 3y ago
    As history has shown, you can only do so much to stop this. If you perfectly mimic the GoogleBot and use google IP ranges by hosting on google cloud, they either take an SEO hit or let you bot them at the end of the day. GoogleBot looks like a DDoS attack a lot of the time too You can also go the route of looking like a pool of users, then it's just a game of cat and mouse and one providers don't really have time to play
    - thih9 3y ago
      > As history has shown, you can only do so much to stop this. History has shown you can stop this well enough. Try accessing e.g. instagram; bibliogram attempted this, the project is now discontinued.
      - sam0x17 3y ago
        This is true in the case of content that doesn't care about SEO. Reddit cares very very much about SEO, so it can never truly block bots.
    - moneywoes 3y ago
      The google scraper ips are very different no? > If you perfectly mimic the GoogleBot and use google IP ranges by hosting on google cloud
      - sam0x17 3y ago
        You are right, apparently they do publish the ranges https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot https://developers.google.com/search/docs/crawling-indexing/...
  - dom96 3y ago
    I've only done some rudimentary rate limiting checks and it doesn't seem like they do. Though I haven't pushed it far (~1000 rpm). In any case, my plan is to deal with it if it becomes a problem.
- kamranjon 3y ago
  That doesn't sound like it's going to be a stable API? I'm not sure there is much you can do though. Edit: yea after looking at this, it definitely appears to violate the TOS. Also this seems like a huge red flag just begging reddit to shut it down: "Unlike with the Reddit API you do not need to authenticate using OAuth."
  - tlonny 3y ago
    IANAL but my understanding is that violating the ToS is only ever a problem if your crawler decides to "sign-in", as that constitutes agreeing to the terms of the ToS. Further, old.reddit.com doesn't gate any content (even NSFW stuff) behind a sign-in page (at least for now)
    - TylerE 3y ago
      > IANAL but my understanding is that violating the ToS is only ever a problem if your crawler decides to "sign-in" Where on earth would you get that impression? From the reddit ToS: "To use certain features of our Services, you may be required to create a Reddit account (an “Account”) and provide us with a username, password, and certain other information about yourself as set forth in the Privacy Policy." NB: "Certain features", that clearly means that features are not gated by sign in, and that the ToS also applies to them. Also from the Tos: " Except and solely to the extent such a restriction is impermissible under applicable law, you may not, without our written agreement: license, sell, transfer, assign, distribute, host, or otherwise commercially exploit the Services or Content; modify, prepare derivative works of, disassemble, decompile, or reverse engineer any part of the Services or Content; or access the Services or Content in order to build a similar or competitive website, product, or service, except as permitted under the Reddit API Terms of Use." And later: "Access, search, or collect data from the Services by any means (automated or otherwise) except as permitted in these Terms or in a separate agreement with Reddit (we conditionally grant permission to crawl the Services in accordance with the parameters set forth in our robots.txt file, but scraping the Services without Reddit’s prior consent is prohibited)"
      - keernan 3y ago

dom96 3y ago

It basically scrapes reddit.com's HTML. Doesn't use the official API at all (that would defeat the purpose).

programmarchy 3y ago
Impressive feat! Does reddit have rate limiting, or other hurdles in place similar to the hoops youtube-dl has to jump through? Curious what your thoughts are about maintaining a project like that.
- sam0x17 3y ago
  As history has shown, you can only do so much to stop this. If you perfectly mimic the GoogleBot and use google IP ranges by hosting on google cloud, they either take an SEO hit or let you bot them at the end of the day. GoogleBot looks like a DDoS attack a lot of the time too You can also go the route of looking like a pool of users, then it's just a game of cat and mouse and one providers don't really have time to play
  - thih9 3y ago
    > As history has shown, you can only do so much to stop this. History has shown you can stop this well enough. Try accessing e.g. instagram; bibliogram attempted this, the project is now discontinued.
    - sam0x17 3y ago
      This is true in the case of content that doesn't care about SEO. Reddit cares very very much about SEO, so it can never truly block bots.
  - moneywoes 3y ago
    The google scraper ips are very different no? > If you perfectly mimic the GoogleBot and use google IP ranges by hosting on google cloud
    - sam0x17 3y ago
      You are right, apparently they do publish the ranges https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot https://developers.google.com/search/docs/crawling-indexing/...
- dom96 3y ago
  I've only done some rudimentary rate limiting checks and it doesn't seem like they do. Though I haven't pushed it far (~1000 rpm). In any case, my plan is to deal with it if it becomes a problem.
kamranjon 3y ago
That doesn't sound like it's going to be a stable API? I'm not sure there is much you can do though. Edit: yea after looking at this, it definitely appears to violate the TOS. Also this seems like a huge red flag just begging reddit to shut it down: "Unlike with the Reddit API you do not need to authenticate using OAuth."
- tlonny 3y ago
  IANAL but my understanding is that violating the ToS is only ever a problem if your crawler decides to "sign-in", as that constitutes agreeing to the terms of the ToS. Further, old.reddit.com doesn't gate any content (even NSFW stuff) behind a sign-in page (at least for now)
  - TylerE 3y ago
    > IANAL but my understanding is that violating the ToS is only ever a problem if your crawler decides to "sign-in" Where on earth would you get that impression? From the reddit ToS: "To use certain features of our Services, you may be required to create a Reddit account (an “Account”) and provide us with a username, password, and certain other information about yourself as set forth in the Privacy Policy." NB: "Certain features", that clearly means that features are not gated by sign in, and that the ToS also applies to them. Also from the Tos: " Except and solely to the extent such a restriction is impermissible under applicable law, you may not, without our written agreement: license, sell, transfer, assign, distribute, host, or otherwise commercially exploit the Services or Content; modify, prepare derivative works of, disassemble, decompile, or reverse engineer any part of the Services or Content; or access the Services or Content in order to build a similar or competitive website, product, or service, except as permitted under the Reddit API Terms of Use." And later: "Access, search, or collect data from the Services by any means (automated or otherwise) except as permitted in these Terms or in a separate agreement with Reddit (we conditionally grant permission to crawl the Services in accordance with the parameters set forth in our robots.txt file, but scraping the Services without Reddit’s prior consent is prohibited)"
    - keernan 3y ago