13 ms·
Even if they do both, if the bots always follow what is entered in robots.txt and humans do not, it won’t be long before that’s the primary factor.
by alasdair_ 7y ago
Even if they do both, if the bots always follow what is entered in robots.txt and humans do not, it won’t be long before that’s the primary factor.
- derimagia 7y agoAnd that's totally fine - if they add their articles (for example) into their robots.txt, it would cripple their SEO. It wouldn't happen.
- zaarn 7y agoThe Google bot mostly ignores the paths specified in robots.txt
- chmod775 7y agoI've seen stuff in the robots.txt get crawled anyways if enough people link to it. In Google's results it will still only show up without any contextual information though.
- londons_explore 7y agoGoogle won't crawl it, but they can still include the link in search results, usually with the title guessed from the way it was referred to on another page, and no description.