3 ms·
>If you can see it, you can scrap it. It is important to be nice and do not overload site's systems. It depends on what you do with the content, but generally
by mrweasel 9y ago
>If you can see it, you can scrap it. It is important to be nice and do not overload site's systems.
It depends on what you do with the content, but generally I would agree.
The content doesn't become any less copyrighted just because you scraped it. Keep that in mind.
Be nice is important to. A site may not react well to you scrapping the content in 1000 parallel connections, so ramp up slowly, and back of if the site because unresponsive or slow. Google, Bing and others indexes sites all the time, and they play nice and almost never causes issues for site operators. Amateurs often causes issues and crash sites as do some wannabe search engines (I looking at you Eniro, answer your emails).
If you don't play nice expect to be blocked or get served 1GB junk files or images of beavers.
Things you should ALWAYS do when running a web scraper:
* Back off the target server becomes slow.
* Include a unique user agent.
* Include contact information or make easy to find your contact information, should problems arise. Sites may have an API or data feeds they would be willing to share instead.
* Respect robots.txt
* Publish your crawlers IP ranges, in case people REALLY want to block you.