3 ms·
Google has a unique position in that any site that wants to be found has to let Google bot index its content. Google does not build a derived product from your
by idibidiart 11y ago
Google has a unique position in that any site that wants to be found has to let Google bot index its content. Google does not build a derived product from your site's content that ends up competing with your site.
If someone needs to build a search site for real estate etc why couldn't they just scrape the Google search result, filter it (white list domains), extract the actual link and present it? In that case you'l need a Google specific scraper that can be based on open source scraping libraries.
Update:
Google actually will IP ban you if it thinks you're a bot trying to scrape for search results.
But they have an API:
https://developers.google.com/web-search/docs/?hl=en https://developers.google.com/web-search/docs/?hl=en
- dchuk 11y agoFirst of all, Google does take your content and make a product out of it by selling ad space on search result pages. Those pages would have no value for advertisers if it weren't for the content producers Google scraped to fill those pages up with. Second, you linked to an api page that clearly says it was deprecated 5 years ago, come on. No one should ever feel bad about scraping Google, considering Google is the world's largest scraper themselves.
- cookiecaper 11y agoA lot of scrapers aren't used to build a competing site. It doesn't matter to most companies. The normal policy is to shut them all down ASAP.