3 ms·
Whoa there, everything important is still crawlable and indexable. Here's what robots.txt says right now: User-Agent: * Disallow: /x? Disallow: /vote? Disall
by brianr 17y ago
Whoa there, everything important is still crawlable and indexable. Here's what robots.txt says right now:
User-Agent: * Disallow: /x? Disallow: /vote? Disallow: /reply? Disallow: /submitted? Disallow: /threads?
This just disallows those pages... not the home page, and not the /item? action (note the url of this page).