5 ms·

# Welcome to Reddit's robots.txt # Reddit believes in an open internet, but not the misuse of public content. # See https://support.reddithelp.com/hc/en-us/

# Welcome to Reddit's robots.txt # Reddit believes in an open internet, but not the misuse of public content. # See https://support.reddithelp.com/hc/en-us/articles/26410290525844-Public-Content-Policy Reddit's Public Content Policy for access and use restrictions to Reddit content. # See https://www.reddit.com/r/reddit4researchers/ for details on how Reddit continues to support research and non-commercial use. # policy: https://support.reddithelp.com/hc/en-us/articles/26410290525844-Public-Content-Policy User-agent: * Disallow: / Source: https://www.reddit.com/robots.txt https://www.reddit.com/robots.txt

will0 2y ago
Looks like it changed a month ago: https://old.reddit.com/r/redditdev/comments/1doc3pt/updating_our_robotstxt_file_and_upholding_our/ https://old.reddit.com/r/redditdev/comments/1doc3pt/updating...
immibis 2y ago
Nobody who wants to be successful obeys robots.txt. And I do mean nobody.
- chippiewill 2y ago
  They changed it to disallow so that scrapers can't just claim the robots.txt gave them permission.
  - toomuchtodo 2y ago
    Independent scrapers can launder the data between Reddit and AI consumers. The only folks this hurts is users seeking info via search engines and folks willing to kowtow to rules that are potentially low impact to evade. Next steps would be (from an adversarial perspective) browser extensions that stream back data for ingestion similar to Recap for Pacer [1]. [1] https://free.law/recap/faq https://free.law/recap/faq (full disclosure: assisting someone pursuing regulatory action against reddit in the EU for a separate issue from scraping, it's a valuable resource, but the folks who own and control it are meh)
    - whycome 2y ago
      Scrapings laundering. Do we have a term for this?
      - throwaway4pp24 2y ago
        Yes, right in the law - "fair use"
        account42 2y ago
        Even more basic, it's free speech. The data itself is public domain so your free speech is not restricted and you don't need fair use excemptions for those restrictions. On the the access through the official system is restricted.
  - deleted 2y ago
    [deleted]
  - tedivm 2y ago
    According to the US court systems the robots.txt file is meaningless. If they respond with a 200 status code giving you the access then you can legally scrape it all you want. If they require that you log in then you have to follow the terms you agree to when creating an account. Public means public though, and if Reddit doesn't want to make the content private (put it behind a login) then we can scrape away. Note that scraping, regardless of the level of permission, doesn't mean you can do anything you want with the content. Copyright still applies. But you can scrape it, and if your use falls under Fair Use or another caveat to the copyright laws then you can do ahead and do it without needing any permission from the authors.
    - sssilver 2y ago
      Fascinating. Where can one learn more about this?
      - datavirtue 2y ago
        The internet.
        deprecative 2y ago
        If you have nothing constructive to say why say anything?
        datavirtue 2y ago
        That was my answer, FFS.
      - neongreen 2y ago
        I liked the chapter on DMCA from the 5-volume E-Commerce & Internet Law. It was super detailed. I haven’t read volume 1, but apparently half of it is about data scraping, and I expect it to be similarly detailed. So if I were you, that’s where I’d start. Another option is looking for “robots.txt” at Google Scholar and trying various keywords like “legality”, “scraping”, “case law”, etc.
- hakdbha 2y ago
  [dead]
- JohnFen 2y ago
  Sadly true. That's why I gave up on robots.txt years ago and started blocking crawlers outright in .htaccess Of course, that became unsustainable so now I have everything behind a login wall.
- maxnevermind 2y ago
  Has not NYT tried to sue OpenAI because of them ignoring robots.txt or you mean it's impossible to prove and / or it's still more profitable to just ignore robots.txt?
- latexr 2y ago
  That’s a weird statement to be absolutist about. The majority of individuals and companies who want to be successful do not do so by scrapping websites, thus have no reason to disobey robots.txt. Most people in the world, ambitious or not, wouldn’t even understand what your sentence refers to.
  - wsve 2y ago
    OP is obviously talking about people whose area of research/development/product would involve web scraping... This feels like being purposefully obtuse
sunaookami 2y ago
They serve a different robots.txt to Google: https://merj.com/blog/investigating-reddits-robots-txt-cloaking-strategy https://merj.com/blog/investigating-reddits-robots-txt-cloak... You can see it here: https://search.google.com/test/rich-results/result?id=_mYoglsZF_-j2rZjg1yb7g https://search.google.com/test/rich-results/result?id=_mYogl... (click on "View Tested Page")
dogleash 2y ago
> # Reddit believes in an open internet, but not the misuse of public content. Calling it "public" content in the very act of exercising their ownership over it. The balls on whoever wrote that.
- pas 2y ago
  it's even worse. it's not theirs (it's the users'), they are merely hosting it and using it (ToS gives them a fancy irrevocable license I guess). so they can do whatever they want with it and the actual owners/authors have no chance to really influence Reddit at all to make it crawlable. (the GDPR-like data takeout is nice, but ... completely useless in these cases where the value is in the composition and aggregation with other users' content.)
  - deepfriedbits 2y ago
    On top of that, a sizable chunk of Reddit content is ripped from elsewhere, whether videos, images, etc.
  - visarga 2y ago
    > the GDPR like data takeout is nice Is there a way to export my history? How?
    - pas 2y ago
      https://www.reddit.com/settings/data-request https://www.reddit.com/settings/data-request (and there's some help article for it that I didn't read, but google found this first https://support.reddithelp.com/hc/en-us/articles/360043048352-How-do-I-request-a-copy-of-my-Reddit-data-and-information https://support.reddithelp.com/hc/en-us/articles/36004304835... that's how I got to the link)
      - visarga 2y ago
        Thank you, it worked
    - deleted 2y ago
raverbashing 2y ago
With the amount of crap in Reddit, cleaning it must be a very non-trivial problem. (I mean, it never is, but in the case of Reddit it's probably extra complicated)
Zuiii 2y ago
> We believe in something that we will now proceed to violate. I will never take a statement given by a company that blatantly lies like this at face value going forward. What a bunch of clowns.

will0 2y ago

Looks like it changed a month ago: https://old.reddit.com/r/redditdev/comments/1doc3pt/updating_our_robotstxt_file_and_upholding_our/ https://old.reddit.com/r/redditdev/comments/1doc3pt/updating...

immibis 2y ago

Nobody who wants to be successful obeys robots.txt. And I do mean nobody.

chippiewill 2y ago
They changed it to disallow so that scrapers can't just claim the robots.txt gave them permission.
- toomuchtodo 2y ago
  Independent scrapers can launder the data between Reddit and AI consumers. The only folks this hurts is users seeking info via search engines and folks willing to kowtow to rules that are potentially low impact to evade. Next steps would be (from an adversarial perspective) browser extensions that stream back data for ingestion similar to Recap for Pacer [1]. [1] https://free.law/recap/faq https://free.law/recap/faq (full disclosure: assisting someone pursuing regulatory action against reddit in the EU for a separate issue from scraping, it's a valuable resource, but the folks who own and control it are meh)
  - whycome 2y ago
    Scrapings laundering. Do we have a term for this?
    - throwaway4pp24 2y ago
      Yes, right in the law - "fair use"
      - account42 2y ago
        Even more basic, it's free speech. The data itself is public domain so your free speech is not restricted and you don't need fair use excemptions for those restrictions. On the the access through the official system is restricted.
- deleted 2y ago
  [deleted]
- tedivm 2y ago
  According to the US court systems the robots.txt file is meaningless. If they respond with a 200 status code giving you the access then you can legally scrape it all you want. If they require that you log in then you have to follow the terms you agree to when creating an account. Public means public though, and if Reddit doesn't want to make the content private (put it behind a login) then we can scrape away. Note that scraping, regardless of the level of permission, doesn't mean you can do anything you want with the content. Copyright still applies. But you can scrape it, and if your use falls under Fair Use or another caveat to the copyright laws then you can do ahead and do it without needing any permission from the authors.
  - sssilver 2y ago
    Fascinating. Where can one learn more about this?
    - datavirtue 2y ago
      The internet.
      - deprecative 2y ago
        If you have nothing constructive to say why say anything?
        datavirtue 2y ago
        That was my answer, FFS.
    - neongreen 2y ago
      I liked the chapter on DMCA from the 5-volume E-Commerce & Internet Law. It was super detailed. I haven’t read volume 1, but apparently half of it is about data scraping, and I expect it to be similarly detailed. So if I were you, that’s where I’d start. Another option is looking for “robots.txt” at Google Scholar and trying various keywords like “legality”, “scraping”, “case law”, etc.
hakdbha 2y ago
[dead]
JohnFen 2y ago
Sadly true. That's why I gave up on robots.txt years ago and started blocking crawlers outright in .htaccess Of course, that became unsustainable so now I have everything behind a login wall.
maxnevermind 2y ago
Has not NYT tried to sue OpenAI because of them ignoring robots.txt or you mean it's impossible to prove and / or it's still more profitable to just ignore robots.txt?
latexr 2y ago
That’s a weird statement to be absolutist about. The majority of individuals and companies who want to be successful do not do so by scrapping websites, thus have no reason to disobey robots.txt. Most people in the world, ambitious or not, wouldn’t even understand what your sentence refers to.
- wsve 2y ago
  OP is obviously talking about people whose area of research/development/product would involve web scraping... This feels like being purposefully obtuse

sunaookami 2y ago

They serve a different robots.txt to Google: https://merj.com/blog/investigating-reddits-robots-txt-cloaking-strategy https://merj.com/blog/investigating-reddits-robots-txt-cloak... You can see it here: https://search.google.com/test/rich-results/result?id=_mYoglsZF_-j2rZjg1yb7g https://search.google.com/test/rich-results/result?id=_mYogl... (click on "View Tested Page")

dogleash 2y ago

> # Reddit believes in an open internet, but not the misuse of public content. Calling it "public" content in the very act of exercising their ownership over it. The balls on whoever wrote that.

pas 2y ago
it's even worse. it's not theirs (it's the users'), they are merely hosting it and using it (ToS gives them a fancy irrevocable license I guess). so they can do whatever they want with it and the actual owners/authors have no chance to really influence Reddit at all to make it crawlable. (the GDPR-like data takeout is nice, but ... completely useless in these cases where the value is in the composition and aggregation with other users' content.)
- deepfriedbits 2y ago
  On top of that, a sizable chunk of Reddit content is ripped from elsewhere, whether videos, images, etc.
- visarga 2y ago
  > the GDPR like data takeout is nice Is there a way to export my history? How?
  - pas 2y ago
    https://www.reddit.com/settings/data-request https://www.reddit.com/settings/data-request (and there's some help article for it that I didn't read, but google found this first https://support.reddithelp.com/hc/en-us/articles/360043048352-How-do-I-request-a-copy-of-my-Reddit-data-and-information https://support.reddithelp.com/hc/en-us/articles/36004304835... that's how I got to the link)
    - visarga 2y ago
      Thank you, it worked
  - deleted 2y ago

raverbashing 2y ago

With the amount of crap in Reddit, cleaning it must be a very non-trivial problem. (I mean, it never is, but in the case of Reddit it's probably extra complicated)

Zuiii 2y ago

> We believe in something that we will now proceed to violate. I will never take a statement given by a company that blatantly lies like this at face value going forward. What a bunch of clowns.