11 ms·
I've seen stuff in the robots.txt get crawled anyways if enough people link to it. In Google's results it will still only show up without any contextual informa
by chmod775 7y ago
I've seen stuff in the robots.txt get crawled anyways if enough people link to it. In Google's results it will still only show up without any contextual information though.
- londons_explore 7y agoGoogle won't crawl it, but they can still include the link in search results, usually with the title guessed from the way it was referred to on another page, and no description.