3 ms·
Seems strange that you wouldn't have just looked when asking that question (less characters to type at least). Lots of things disallowed. http://google.com/rob
by thorax 19y ago
Seems strange that you wouldn't have just looked when asking that question (less characters to type at least). Lots of things disallowed.
http://google.com/robots.txt http://google.com/robots.txt
- Hexstream 19y agoI think it was a rhetorical question.
- michaelneale 19y agoYes it was. But I should have been clearer for the type of literal argument starters that would tend to hang around here ;) I wonder if google always honour robots.txt
- Hexstream 19y agoI hope so because otherwise they could get stuck in one of those: http://en.wikipedia.org/wiki/Spider_trap http://en.wikipedia.org/wiki/Spider_trap