3 ms·
To those lamenting that they're scraping... Google is the biggest scraper of them all. Facebook, Amazon, Google, Microsoft. All the big boys scrape voraciously,
by randombytes6869 6y ago
To those lamenting that they're scraping... Google is the biggest scraper of them all. Facebook, Amazon, Google, Microsoft. All the big boys scrape voraciously, yet try their best to block themselves from being scraped. Scraping is vital for the functionality of the internet. The narrative that scraping is evil is what big companies want you to think.
When you block small scrapers from your site but permit giants like Googlebot and Bing all you're doing is locking in a monopoly that's bad for everyone
- occamrazor 6y agoGoogle has the (often implicit) permission of the website owner to scrape. OTOH, Google Flights explicitly disallows scraping results.
- sdinsn 6y agoNo, Google's scraping is opt-out only, which they offer to be friendly. Google does not need anyone's permission to scrape publicly accessible data, and they are not required to follow any opt-out requests.