3 ms·

Related thread for the official blog post: https://news.ycombinator.com/item?id=40799275 https://news.ycombinator.com/item?id=40799275 Side note: They seem to

Related thread for the official blog post: https://news.ycombinator.com/item?id=40799275 https://news.ycombinator.com/item?id=40799275 Side note: They seem to serve other robots.txt for different User-Agents & IPs: https://merj.com/blog/investigating-reddits-robots-txt-cloaking-strategy https://merj.com/blog/investigating-reddits-robots-txt-cloak...

r721 2y ago
Another discussion: https://news.ycombinator.com/item?id=40873781 https://news.ycombinator.com/item?id=40873781 (15 comments)
g-b-r 2y ago
User-Agent: bender Disallow: /my_shiny_metal_ass xD
- crtasm 2y ago
  How am I supposed to bite it now?