17 ms·
http://news.ycombinator.com/robots.txt http://news.ycombinator.com/robots.txt User-Agent: * Disallow: /x? Disallow: /vote? Disallow: /reply? Disallow: /submi
by VBprogrammer 16y ago
http://news.ycombinator.com/robots.txt http://news.ycombinator.com/robots.txt
User-Agent: * Disallow: /x? Disallow: /vote? Disallow: /reply? Disallow: /submitted? Disallow: /submitlink? Disallow: /threads? Crawl-delay: 30