https://www.reddit.com/robots.txt https://www.reddit.com/robots.txt has additional comments: # Welcome to Reddit's robots.txt # Reddit believes in an open

https://www.reddit.com/robots.txt https://www.reddit.com/robots.txt has additional comments: # Welcome to Reddit's robots.txt # Reddit believes in an open internet, but not the misuse of public content. # See https://support.reddithelp.com/hc/en-us/articles/26410290525844-Public-Content-Policy Reddit's Public Content Policy for access and use restrictions to Reddit content. # See https://www.reddit.com/r/reddit4researchers/ for details on how Reddit continues to support research and non-commercial use. # policy: https://support.reddithelp.com/hc/en-us/articles/26410290525844-Public-Content-Policy User-agent: * Disallow: /