3 ms·
Robots.txt User-Agent: * Disallow: /x? Disallow: /vote? Disallow: /reply? Disallow: /submitted? Disallow: /submitlink? Disallow: /threads?
by thinkbohemian 16y ago
Robots.txt
User-Agent: * Disallow: /x? Disallow: /vote? Disallow: /reply? Disallow: /submitted? Disallow: /submitlink? Disallow: /threads?
- sorbus 16y agoOne assumes that a core characteristic of spammers and spambots is that they ignore Robots.txt - or at least I would be extremely surprised if disallowing them in robots.txt has any effect.