3 ms·
> most websites are now blacklisting almost all datacentre IPs including Amazon, Azure, etc “Most” sounds like an exaggeration. Wouldn’t this also create probl
by extra88 5y ago
> most websites are now blacklisting almost all datacentre IPs including Amazon, Azure, etc
“Most” sounds like an exaggeration. Wouldn’t this also create problems for virtual desktop services like Amazon Workspaces?
> It almost makes me feel that I am breaking the law when scraping a site
You might be violating their copyright, it depends what you do with it. If you overdo it, you could also degrade their service for actual users.
- slver 5y ago> You might be violating their copyright, it depends what you do with it. Google is not new to such complaints (news sites). Everything is very relative.
- superasn 5y agoA lot of websites are using cloudflare which does make scraping quite difficult (just by default I think). Spoofing your user agent is a must if you need to do anything nowadays. To your second point that same would be applicable to Google and Bing and any other search engine. Even if you follow robots.txt and consume an equal or lesser bandwidth it does not matter much if you aren't an established player.