3 ms·
That's assuming the scrapers will respect robots.txt. Some of them would, and that would help. Assuming google does take into account who was first, a similar
by 6ren 16y ago
That's assuming the scrapers will respect robots.txt. Some of them would, and that would help.
Assuming google does take into account who was first, a similar solution is for Jeff to submit his content directly to google for indexing, immediately it's published.
EDIT I'm wrong; google only accepts top-level URLs for indexing, not new content: http://www.google.com/addurl/?continue=/addurl http://www.google.com/addurl/?continue=/addurl
- aamar 16y agoWell, you could ban misbehaving robots by IP address. That seems like a reasonable step for bad behavior and not in conflict with Jeff's general "sharing"/"openness" values.
- rabidsnail 16y agoYou can specify specific urls by submitting a sitemap (http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156184 http://www.google.com/support/webmasters/bin/answer.py?hl=en...).