4 ms·

Reddit's robots.txt disallows all web crawlers

• rawfael 2y ago
  Seems that they might be showing a different robots.txt to Google: https://x.com/pedrodias/status/1808799403798409216 https://x.com/pedrodias/status/1808799403798409216