3 ms·

If everyone had such a negative attitude as that we never would have had Google. Countless sites depend on web scraping. You can scrape and be a good netizen,

• idibidiart 11y ago
  Google has a unique position in that any site that wants to be found has to let Google bot index its content. Google does not build a derived product from your site's content that ends up competing with your site. If someone needs to build a search site for real estate etc why couldn't they just scrape the Google search result, filter it (white list domains), extract the actual link and present it? In that case you'l need a Google specific scraper that can be based on open source scraping libraries. Update: Google actually will IP ban you if it thinks you're a bot trying to scrape for search results. But they have an API: https://developers.google.com/web-search/docs/?hl=en https://developers.google.com/web-search/docs/?hl=en
  • dchuk 11y ago
    First of all, Google does take your content and make a product out of it by selling ad space on search result pages. Those pages would have no value for advertisers if it weren't for the content producers Google scraped to fill those pages up with. Second, you linked to an api page that clearly says it was deprecated 5 years ago, come on. No one should ever feel bad about scraping Google, considering Google is the world's largest scraper themselves.
  • cookiecaper 11y ago
    A lot of scrapers aren't used to build a competing site. It doesn't matter to most companies. The normal policy is to shut them all down ASAP.
• aikah 11y ago
  Yeah , lol , try to scrap Google search results and see how Google has such a positive attitude. not.