4 ms·
I would hope that google, of all companies, would be capable of writing a crawler that ignores robots.txt comments. Other than that the guy only has one rule.
by fein 14y ago
I would hope that google, of all companies, would be capable of writing a crawler that ignores robots.txt comments.
Other than that the guy only has one rule.