4 ms·
Wouldn't web scraping be considered abusive usage as it is frowned upon by most websites?
by vidyesh 10y ago
Wouldn't web scraping be considered abusive usage as it is frowned upon by most websites?
- ovibos 10y agoIMO scraping is only abusive if the site explicitly asks to not be scraped (via the site's robots.txt)
- tedmiston 10y agoThere are plenty of ways to scrape responsibly.
- ovao 10y agoIt could be done in an abusive way, certainly.
- OwainX 10y agoScraping, when done responsibly (respecting robots.txt and not overloading a server with requests) wouldn't be considered abusive in most cases. While we will be strict in enforcing abuse policies, when it's obvious a user acted with good intentions we can work on a case by case basis.