3 ms·
A site could start blocking certain IPs and user agents as well if need be to prevent being indexed, I'm sure there's was for crawlers to get around it but a si
by robryan 17y ago
A site could start blocking certain IPs and user agents as well if need be to prevent being indexed, I'm sure there's was for crawlers to get around it but a site could certainly make it hard to index there content.
- bad_user 17y agoThen you could start sending requests via proxies ... and sooner or later the site would start blocking legitimate users. Either way, the majority of all new traffic comes from Google right now, and if a website has exclusivity for Bing, it would surely lose a lot visits. And it's definitely not worth it unless Microsoft pays really well for this privilege ... and I'm not sure they can do that, even if they are Microsoft.