3 ms·
Which web search engines have their own index? Evidence from website visits
- IceWreck 4y agoIs it possible that other search engines are using GoogleBot to bot friendly results ? Ex: to bypass an anti bot firewall filter or something
- solardev 4y agoYeah, that's very common, and why Cloudflare etc. have specific rules to detect fake Googlebots
- solardev 4y agoThis methodology is pretty suspect. A few hundred hits on some random site and simple UA lookups isn't representative of much except who might've crawled this specific site.
- manv1 4y agoTheir own index? What does that mean? Is the idea that they're trying to figure out if search engine X is actually forwarding requests to search engine Y? Using the user-agent string is the naive, and probably worst, way to figure this out. But of course, it's the easiest conceptually. What they should be doing if they want to see if the indexes are unique is to query the index itself ie: send in query strings and see what comes out, then compare them. But then they'd have to figure out a way to compare the returned results and determine if they're "different enough." Using their methodology all they have to do is count.