3 ms·
It appears to have been changed, just seconds ago. Now it reads: User-Agent: * Disallow: /x? Disallow: /vote? Disallow: /reply? Disallow: /submitted
by chaosmachine 17y ago
It appears to have been changed, just seconds ago. Now it reads:
User-Agent: *
Disallow: /x?
Disallow: /vote?
Disallow: /reply?
Disallow: /submitted?
Disallow: /threads?
- Matt_Cutts 17y agoThat seems like a nice middle ground--it should save server load, but search engines can still get to useful content.