2 ms·
Even if the spider doesn't respect robots.txt, the act of indexing is really obvious -- your webserver does not have to respond to their HTTP Requests if you do
by blasdel 17y ago
Even if the spider doesn't respect robots.txt, the act of indexing is really obvious -- your webserver does not have to respond to their HTTP Requests if you don't want it to.
It's not as facile as 'lock your doors' -- instead just not answering when they knock.