3 ms·
One assumes that a core characteristic of spammers and spambots is that they ignore Robots.txt - or at least I would be extremely surprised if disallowing them
by sorbus 16y ago
One assumes that a core characteristic of spammers and spambots is that they ignore Robots.txt - or at least I would be extremely surprised if disallowing them in robots.txt has any effect.