4 ms·

Potentially stupid questions: Aren't there efficient techniques to prevent web scraping? And if Google really scrapes web content from them without their permis

by 13415 5y ago

Potentially stupid questions: Aren't there efficient techniques to prevent web scraping? And if Google really scrapes web content from them without their permission, wouldn't that be commercial copyright infringement which falls under criminal law? That's the thing I don't understand about the current situation.

claudiulodro 5y ago
> Aren't there efficient techniques to prevent web scraping? Not especially, since internet browsers make GET requests the same as scrapers. There are techniques around blocking user agents, rate limiting, etc. but that's an arms race against the scrapers. The tricky part about blocking Google is that if you do, your site won't show up in Google search results. :)
- shaftway 5y ago
  Google in particular always uses an identifiable agent. So you could serve one set of content to Google and a separate set to actual visitors, preventing Google from scraping the content. They clearly want the benefit of being scraped (the traffic) without the cost (the loss of traffic from people who don't click past the headline to the article).
NikolaNovak 5y ago
1. Are there techniques to prevent web scraping? Yes. Are they "Efficient", and even more importantly, effective? Debatable 2. However, more to the point, what's the end game there? If there are 10,000 outlets out there; and arbitrary number, say 100 of them block scrapes; now the other 9,900 will show up in Google/Facebook search and shares, and those who implemented scraping protection will quietly fade into non-existence. It's a damned-if-you-do:damned-if-you-don't choice of modern media consumption model. I assume that most outlets have decided that the exposure, trickle of users and monetization they DO get is superior to the influx of users drying out. It's consequence of the oft-quoted platitude that these days, for most people Google and Facebook ARE "The Internet" :-/

by 13415 5y ago

claudiulodro 5y ago

> Aren't there efficient techniques to prevent web scraping? Not especially, since internet browsers make GET requests the same as scrapers. There are techniques around blocking user agents, rate limiting, etc. but that's an arms race against the scrapers. The tricky part about blocking Google is that if you do, your site won't show up in Google search results. :)

shaftway 5y ago
Google in particular always uses an identifiable agent. So you could serve one set of content to Google and a separate set to actual visitors, preventing Google from scraping the content. They clearly want the benefit of being scraped (the traffic) without the cost (the loss of traffic from people who don't click past the headline to the article).

NikolaNovak 5y ago

1. Are there techniques to prevent web scraping? Yes. Are they "Efficient", and even more importantly, effective? Debatable 2. However, more to the point, what's the end game there? If there are 10,000 outlets out there; and arbitrary number, say 100 of them block scrapes; now the other 9,900 will show up in Google/Facebook search and shares, and those who implemented scraping protection will quietly fade into non-existence. It's a damned-if-you-do:damned-if-you-don't choice of modern media consumption model. I assume that most outlets have decided that the exposure, trickle of users and monetization they DO get is superior to the influx of users drying out. It's consequence of the oft-quoted platitude that these days, for most people Google and Facebook ARE "The Internet" :-/