3 ms·
Scraping/doing it semantically has copyright implications (Re: Europe) so realistically the Google-owned approach is the most viable one.
by Shooti 13y ago
Scraping/doing it semantically has copyright implications (Re: Europe) so realistically the Google-owned approach is the most viable one.
- 3825 13y agoDoes Europe think that the robots.txt approach is cumbersome for "content" owners? If someone doesn't want anyone to crawl, they'd put something like User-agent: * Disallow: /