3 ms·
I'm glad that all of the sites you target want your scraper to access them. The goal in many cases where one would use a scraper is to access information not pr
by mdaniel 12y ago
I'm glad that all of the sites you target want your scraper to access them. The goal in many cases where one would use a scraper is to access information not provided in an API or otherwise encased in HTML. Most of their robots.txt are "User-Agent: *\nDisallow: /\n"
- preinheimer 12y agoThen we have no right to scrape that content. Why is there an implied right to scrape?