4 ms·

Isn't it nice that millions of people made web pages that Google decided to scrape to harvest the work of others and run ads next to it for themselves? Now try

by ck2 12y ago

Isn't it nice that millions of people made web pages that Google decided to scrape to harvest the work of others and run ads next to it for themselves? Now try scraping Google and see what they do to you.

vijayr 12y ago
Those millions of people want google to scrape and harvest, in the hope that they will rank higher etc etc. If an unknown person tries to scrape, he/she will promptly get banned by those very same people (Google wouldn't like someone scraping their stuff either). Different players different rules, I guess.
- yutah 12y ago
  Which large website did you get banned from for scraping? I did some scrapping and never got banned... perhaps my scraping's rate was not too fast.
  - adventured 12y ago
    If you try to mass scrape almost any major site (millions of pages of content) they'll block you. For example, if you went one by one through Stack Overflow and sucked out every question and answer, your scraper bot would get banned (unless you're doing one request per minute, in which case you'll never finish). Or if you tried to scrape Twitter.
dm2 12y ago
You can ban GoogleBot easily, just put a line in your robots.txt file, but then people won't be able to easily find your site using Google services. If you provide value to Google they will make an API to allow accessing that data easier. By scraping do you mean scraping their search results? They offer this, which is nice: https://developers.google.com/custom-search/ https://developers.google.com/custom-search/ Many large sites don't allow scraping because of unnecessary server load (denial of service sometimes) so they'll offer an API where you can download content in a controlled (and monitorable) manner.
- frik 12y ago
  We need a robots.txt and "noindex" metatag standard for emails. If one sends an email to an GMail/Outlook.com/Yahoo email address, one should be able to opt-out of their email crawler, advertisement analysis, artificial intelligence analysis, etc.
  - dm2 12y ago
    The user who receives it can: https://support.google.com/ads/answer/2662922?hl=en https://support.google.com/ads/answer/2662922?hl=en I don't think they do too much storing of email details, they know that it's a sensitive area and that an employee will eventually blow the whistle and it will hurt user trust, which is a big part of their business.
    - frik 12y ago
      I meant it the other direction. 1) Alice sends an email to Bob (GMail user). 2) Bob receives the email, meanwhile Google scrapes the content and extracts its meaning to show Bob some ads and use the facts to improve Google's A.I. Alice wants a way to mark her email content as "no index". So that email service provider don't crawl through the content. Exactly like the robots.txt for domains or the "noindex" metatag in HTML head element!
      - lern_too_spel 12y ago
        Why should Alice get to restrict what Bob can do with his inbox?
        frik 12y ago
        It's not about Bob, he can do what he want. It's about the email service provider, that should stop analyze the email text to extract its meaning. Gmail uses it to display ads to Bob, builds a shadow profile for Alice (like Facebook) and trains an artificial intelligence (see headline link).
- 12y ago

by ck2 12y ago

vijayr 12y ago

Those millions of people want google to scrape and harvest, in the hope that they will rank higher etc etc. If an unknown person tries to scrape, he/she will promptly get banned by those very same people (Google wouldn't like someone scraping their stuff either). Different players different rules, I guess.

yutah 12y ago
Which large website did you get banned from for scraping? I did some scrapping and never got banned... perhaps my scraping's rate was not too fast.
- adventured 12y ago
  If you try to mass scrape almost any major site (millions of pages of content) they'll block you. For example, if you went one by one through Stack Overflow and sucked out every question and answer, your scraper bot would get banned (unless you're doing one request per minute, in which case you'll never finish). Or if you tried to scrape Twitter.

dm2 12y ago

You can ban GoogleBot easily, just put a line in your robots.txt file, but then people won't be able to easily find your site using Google services. If you provide value to Google they will make an API to allow accessing that data easier. By scraping do you mean scraping their search results? They offer this, which is nice: https://developers.google.com/custom-search/ https://developers.google.com/custom-search/ Many large sites don't allow scraping because of unnecessary server load (denial of service sometimes) so they'll offer an API where you can download content in a controlled (and monitorable) manner.

frik 12y ago
We need a robots.txt and "noindex" metatag standard for emails. If one sends an email to an GMail/Outlook.com/Yahoo email address, one should be able to opt-out of their email crawler, advertisement analysis, artificial intelligence analysis, etc.
- dm2 12y ago
  The user who receives it can: https://support.google.com/ads/answer/2662922?hl=en https://support.google.com/ads/answer/2662922?hl=en I don't think they do too much storing of email details, they know that it's a sensitive area and that an employee will eventually blow the whistle and it will hurt user trust, which is a big part of their business.
  - frik 12y ago
    I meant it the other direction. 1) Alice sends an email to Bob (GMail user). 2) Bob receives the email, meanwhile Google scrapes the content and extracts its meaning to show Bob some ads and use the facts to improve Google's A.I. Alice wants a way to mark her email content as "no index". So that email service provider don't crawl through the content. Exactly like the robots.txt for domains or the "noindex" metatag in HTML head element!
    - lern_too_spel 12y ago
      Why should Alice get to restrict what Bob can do with his inbox?
      - frik 12y ago
        It's not about Bob, he can do what he want. It's about the email service provider, that should stop analyze the email text to extract its meaning. Gmail uses it to display ads to Bob, builds a shadow profile for Alice (like Facebook) and trains an artificial intelligence (see headline link).
12y ago