35 ms·
Ah, that's true, indeed. The page, though, will appear as a link without any contents, because the bot won't be able to index it.
by dchest 10y ago
Ah, that's true, indeed. The page, though, will appear as a link without any contents, because the bot won't be able to index it.
- laumars 9y agoExcept it has indexed it. It just hasn't crawled it. But content or not, the aim you were trying to achieve (namely your content not being indexed) has failed. Thus you are then once again dependant on other countermeasures that render the robots.txt irrelevant.