4 ms·
I understand the point of using a special user-agent to crawl webpages for indexing, but search engines should use a "regular" browser UA string (full JavaScrip
by adtac 7y ago
I understand the point of using a special user-agent to crawl webpages for indexing, but search engines should use a "regular" browser UA string (full JavaScript, etc. to simulate an actual browser) occasionally. From a different IP range too, of course. If the contents of the page are wildly different, penalise the site.
- wolco 7y agoThey do both or at least google does.
- alasdair_ 7y agoEven if they do both, if the bots always follow what is entered in robots.txt and humans do not, it won’t be long before that’s the primary factor.
- derimagia 7y agoAnd that's totally fine - if they add their articles (for example) into their robots.txt, it would cripple their SEO. It wouldn't happen.
- zaarn 7y agoThe Google bot mostly ignores the paths specified in robots.txt
- chmod775 7y agoI've seen stuff in the robots.txt get crawled anyways if enough people link to it. In Google's results it will still only show up without any contextual information though.
- londons_explore 7y agoGoogle won't crawl it, but they can still include the link in search results, usually with the title guessed from the way it was referred to on another page, and no description.
- sieabahlpark 7y agoWhat about sites which display different content depending on origin?