5 ms·
> Aren't there efficient techniques to prevent web scraping? Not especially, since internet browsers make GET requests the same as scrapers. There are techniqu
by claudiulodro 5y ago
> Aren't there efficient techniques to prevent web scraping?
Not especially, since internet browsers make GET requests the same as scrapers. There are techniques around blocking user agents, rate limiting, etc. but that's an arms race against the scrapers.
The tricky part about blocking Google is that if you do, your site won't show up in Google search results. :)
- shaftway 5y agoGoogle in particular always uses an identifiable agent. So you could serve one set of content to Google and a separate set to actual visitors, preventing Google from scraping the content. They clearly want the benefit of being scraped (the traffic) without the cost (the loss of traffic from people who don't click past the headline to the article).