3 ms·
The article states: >"In its submission to the UK competition inquiry, Microsoft said that websites prioritize access to Google's web crawler over Bing's. As a
by bogomipz 6y ago
The article states:
>"In its submission to the UK competition inquiry, Microsoft said that websites prioritize access to Google's web crawler over Bing's. As a result, Google is able to index these pages more thoroughly and deeply."
Could someone say how this crawling priority works exactly? What is the thing being prioritized, the number of concurrent crawls by their crawlers? A different robots.txt for only the google domain? Something else?
- jsnell 6y agoRobots.txt has a directive for crawl delay, and like all directives it can be applied to specific user agents. In addition to that, the webmaster console can be used to set the crawl speed, and that data would obviously not be readable to third parties.