There are plenty sites that do that. And if you're determined to scrape [a website], sometimes it needs proxies, rotating user agents and some rate limiting.

There are plenty sites that do that. And if you're determined to scrape [a website], sometimes it needs proxies, rotating user agents and some rate limiting. Using a full blown browser sometimes helps prevent you hitting those rate limits, but they're still there.