Check the site robots.txt https://en.wikipedia.org/wiki/Robots_exclusion_standard https://en.wikipedia.org/wiki/Robots_exclusion_standard But basically scrapp

Check the site robots.txt https://en.wikipedia.org/wiki/Robots_exclusion_standard https://en.wikipedia.org/wiki/Robots_exclusion_standard But basically scrapping is a loophole, you are not doing nothing wrong because everything is accessible with browser. Still web admins get upset. For tools I use now selenium with phantomJS, but you can also use scrappy. As long as you don´t claim alot of bandwith that other users could use should be fine