3 ms·
Yeah I've seen a lot of sites that explicitly state that, as well as in their robots.txt All robots forbidden, except googlebot
by andai 9y ago
Yeah I've seen a lot of sites that explicitly state that, as well as in their robots.txt
All robots forbidden, except googlebot
- chrisweekly 9y ago... which rule would be obeyed only by legitimate, robots.txt-honoring crawlers. This reminds me of the anti-piracy messages shown (solely) to viewers of legally-purchased media. Similar "logic", similar (counter-productive) "effectiveness".
- kuschku 9y agoWhich is why some search engines simply ended up following only the directives given to googlebot, and ignoring the rest.
- josteink 9y ago> All robots forbidden, except googlebot After which people keep claiming google search is better, not just given special treatment.