3 ms·
On bbot.org I return 410 on robots.txt, for tedious ideological reasons. Doesn't stop spiders requesting it, of course, since that check is probably hard-coded.
by sbierwagen 12y ago
On bbot.org I return 410 on robots.txt, for tedious ideological reasons. Doesn't stop spiders requesting it, of course, since that check is probably hard-coded.