3 ms·
poacher69, we crawl the public web. Anyone that blocks us out with robots.txt, we won't crawl. If you check bing.com/robots.txt, it has "Disallow: /search" . So
by Matt_Cutts 16y ago
poacher69, we crawl the public web. Anyone that blocks us out with robots.txt, we won't crawl. If you check bing.com/robots.txt, it has "Disallow: /search" . So no, we won't crawl Bing's search results pages. If anything, users tend to complain when search results from Lycos or wherever show up in Google.
- underdown 16y agohttp://www.bing.com/robots.txt http://www.bing.com/robots.txt User-agent: * Disallow: /search Funny thing: http://www.google.com/search?q=site%3Abing.com%2Fsearch%2F http://www.google.com/search?q=site%3Abing.com%2Fsearch%2F I was gonna call out Matt for crawling bing's search results but I'm guessing Microsoft hasn't realized they return results from the /Search/ folder. ;)
- nitrogen 16y agoOnce again Microsoft is bitten by expecting case insensitivity.
- poacher69 16y agomatt, how does google do competitive relevance evaluations without scraping Bing?