5 ms·
Alternatively, Google starts crawling the web in incognito mode; punishes websites that hide content.
by underwater 7y ago
Alternatively, Google starts crawling the web in incognito mode; punishes websites that hide content.
- adtac 7y agoI understand the point of using a special user-agent to crawl webpages for indexing, but search engines should use a "regular" browser UA string (full JavaScript, etc. to simulate an actual browser) occasionally. From a different IP range too, of course. If the contents of the page are wildly different, penalise the site.
- wolco 7y agoThey do both or at least google does.
- alasdair_ 7y agoEven if they do both, if the bots always follow what is entered in robots.txt and humans do not, it won’t be long before that’s the primary factor.
- derimagia 7y agoAnd that's totally fine - if they add their articles (for example) into their robots.txt, it would cripple their SEO. It wouldn't happen.
- zaarn 7y agoThe Google bot mostly ignores the paths specified in robots.txt
- chmod775 7y agoI've seen stuff in the robots.txt get crawled anyways if enough people link to it. In Google's results it will still only show up without any contextual information though.
- londons_explore 7y agoGoogle won't crawl it, but they can still include the link in search results, usually with the title guessed from the way it was referred to on another page, and no description.
- sieabahlpark 7y agoWhat about sites which display different content depending on origin?
- propogandist 7y agoGoogle is an advertising company and everything they do is to improve the core ad-business. If Google had it their way every website would be using AMP, allowing google to have more control over the web, allowing them to become the only ad-network.