3 ms·

If you can see it, you can scrap it. It is important to be nice and do not overload site's systems, but personally I don't see a difference if I view the site m

If you can see it, you can scrap it. It is important to be nice and do not overload site's systems, but personally I don't see a difference if I view the site myself or some script is doing it for me for later consumption. Technically, sites can do anything they want to make scraping more difficult. But from the moral (and, I hope, legal — at least in the future) standpoint, scraping should be your right.

mrweasel 9y ago
>If you can see it, you can scrap it. It is important to be nice and do not overload site's systems. It depends on what you do with the content, but generally I would agree. The content doesn't become any less copyrighted just because you scraped it. Keep that in mind. Be nice is important to. A site may not react well to you scrapping the content in 1000 parallel connections, so ramp up slowly, and back of if the site because unresponsive or slow. Google, Bing and others indexes sites all the time, and they play nice and almost never causes issues for site operators. Amateurs often causes issues and crash sites as do some wannabe search engines (I looking at you Eniro, answer your emails). If you don't play nice expect to be blocked or get served 1GB junk files or images of beavers. Things you should ALWAYS do when running a web scraper: * Back off the target server becomes slow. * Include a unique user agent. * Include contact information or make easy to find your contact information, should problems arise. Sites may have an API or data feeds they would be willing to share instead. * Respect robots.txt * Publish your crawlers IP ranges, in case people REALLY want to block you.
IncRnd 9y ago
There is clearly a difference between scraping and not scraping, or you wouldn't want to scrape a site. Why _should_ scraping be your right, morally? Your convenience doesn't remove ownership of data.
- atemerev 9y ago
  If the data is already published, it is effectively made public.

mrweasel 9y ago

>If you can see it, you can scrap it. It is important to be nice and do not overload site's systems. It depends on what you do with the content, but generally I would agree. The content doesn't become any less copyrighted just because you scraped it. Keep that in mind. Be nice is important to. A site may not react well to you scrapping the content in 1000 parallel connections, so ramp up slowly, and back of if the site because unresponsive or slow. Google, Bing and others indexes sites all the time, and they play nice and almost never causes issues for site operators. Amateurs often causes issues and crash sites as do some wannabe search engines (I looking at you Eniro, answer your emails). If you don't play nice expect to be blocked or get served 1GB junk files or images of beavers. Things you should ALWAYS do when running a web scraper: * Back off the target server becomes slow. * Include a unique user agent. * Include contact information or make easy to find your contact information, should problems arise. Sites may have an API or data feeds they would be willing to share instead. * Respect robots.txt * Publish your crawlers IP ranges, in case people REALLY want to block you.

IncRnd 9y ago

There is clearly a difference between scraping and not scraping, or you wouldn't want to scrape a site. Why _should_ scraping be your right, morally? Your convenience doesn't remove ownership of data.

atemerev 9y ago
If the data is already published, it is effectively made public.