3 ms·
This is actually an interesting point. If you implemented an effective scraper-detection API, you'd run risk of locking out search engines too. (Though I guess
by xg15 9y ago
This is actually an interesting point. If you implemented an effective scraper-detection API, you'd run risk of locking out search engines too.
(Though I guess the real-life solution would be both simple and depressing: make an exception for googlebot and don't care about anyone else)
- andai 9y agoYeah I've seen a lot of sites that explicitly state that, as well as in their robots.txt All robots forbidden, except googlebot
- chrisweekly 9y ago... which rule would be obeyed only by legitimate, robots.txt-honoring crawlers. This reminds me of the anti-piracy messages shown (solely) to viewers of legally-purchased media. Similar "logic", similar (counter-productive) "effectiveness".
- kuschku 9y agoWhich is why some search engines simply ended up following only the directives given to googlebot, and ignoring the rest.
- josteink 9y ago> All robots forbidden, except googlebot After which people keep claiming google search is better, not just given special treatment.