3 ms·

Try “site:old.Reddit.com”

• oefrha 5y ago
  That's not going to work. $ curl https://old.reddit.com/robots.txt User-Agent: * Disallow: / Also, even if search engines are allowed, old.reddit.com pages are not canonical (<link rel="canonical"> points to the www.reddit.com version, which is actually reasonable behavior), so pages there would not be crawled as often or at all.
  • linuxfan2021 5y ago
    we need a search engine that doesn't parse robots.txt. NOW!