2 ms·
I dont' think that's true. https://news.ycombinator.com/robots.txt https://news.ycombinator.com/robots.txt User-Agent: * Disallow: /x? Disallow: /vote?
by sp332 13y ago
I dont' think that's true. https://news.ycombinator.com/robots.txt https://news.ycombinator.com/robots.txt
User-Agent: *
Disallow: /x?
Disallow: /vote?
Disallow: /reply?
Disallow: /submitted?
Disallow: /submitlink?
Disallow: /threads?
Crawl-delay: 30
Links in comments are marked nofollow but submissions are not.