Google's crawler gets preferential treatment on many sites. Not necessarily because of private deals, but because it drives traffic from search results. Quote

Google's crawler gets preferential treatment on many sites. Not necessarily because of private deals, but because it drives traffic from search results. Quote via [2] --- start quote --- When Mr. Maril started researching how sites treated Google’s crawler, he downloaded 17 million so-called robots.txt files — essentially rules of the road posted by nearly every website laying out where crawlers can go — and found many examples where Google had greater access than competitors. ScienceDirect, a site for peer-reviewed papers, permits only Google’s crawler to have access to links containing PDF documents. Only Google’s computers get access to listings on PBS Kids. On Alibaba.com, the U.S. site of the Chinese e-commerce giant Alibaba, only Google’s crawler is given access to pages that list products. --- end quote --- [1] https://www.nytimes.com/2020/12/14/technology/how-google-dominates.html https://www.nytimes.com/2020/12/14/technology/how-google-dom... [2] https://daringfireball.net/linked/2020/12/14/wakabayashi-google-search https://daringfireball.net/linked/2020/12/14/wakabayashi-goo...