3 ms·

You can ban GoogleBot easily, just put a line in your robots.txt file, but then people won't be able to easily find your site using Google services. If you pro

by dm2 12y ago

You can ban GoogleBot easily, just put a line in your robots.txt file, but then people won't be able to easily find your site using Google services. If you provide value to Google they will make an API to allow accessing that data easier. By scraping do you mean scraping their search results? They offer this, which is nice: https://developers.google.com/custom-search/ https://developers.google.com/custom-search/ Many large sites don't allow scraping because of unnecessary server load (denial of service sometimes) so they'll offer an API where you can download content in a controlled (and monitorable) manner.

frik 12y ago
We need a robots.txt and "noindex" metatag standard for emails. If one sends an email to an GMail/Outlook.com/Yahoo email address, one should be able to opt-out of their email crawler, advertisement analysis, artificial intelligence analysis, etc.
- dm2 12y ago
  The user who receives it can: https://support.google.com/ads/answer/2662922?hl=en https://support.google.com/ads/answer/2662922?hl=en I don't think they do too much storing of email details, they know that it's a sensitive area and that an employee will eventually blow the whistle and it will hurt user trust, which is a big part of their business.
  - frik 12y ago
    I meant it the other direction. 1) Alice sends an email to Bob (GMail user). 2) Bob receives the email, meanwhile Google scrapes the content and extracts its meaning to show Bob some ads and use the facts to improve Google's A.I. Alice wants a way to mark her email content as "no index". So that email service provider don't crawl through the content. Exactly like the robots.txt for domains or the "noindex" metatag in HTML head element!
    - lern_too_spel 12y ago
      Why should Alice get to restrict what Bob can do with his inbox?
      - frik 12y ago
        It's not about Bob, he can do what he want. It's about the email service provider, that should stop analyze the email text to extract its meaning. Gmail uses it to display ads to Bob, builds a shadow profile for Alice (like Facebook) and trains an artificial intelligence (see headline link).
rstupek 12y ago
It would be nice if there were a way to provide google, bing, et. al. the data so they were constantly crawling sites. A non-insignificant amount of our bandwidth is used up daily by search engines spidering sites.

by dm2 12y ago

frik 12y ago

We need a robots.txt and "noindex" metatag standard for emails. If one sends an email to an GMail/Outlook.com/Yahoo email address, one should be able to opt-out of their email crawler, advertisement analysis, artificial intelligence analysis, etc.

dm2 12y ago
The user who receives it can: https://support.google.com/ads/answer/2662922?hl=en https://support.google.com/ads/answer/2662922?hl=en I don't think they do too much storing of email details, they know that it's a sensitive area and that an employee will eventually blow the whistle and it will hurt user trust, which is a big part of their business.
- frik 12y ago
  I meant it the other direction. 1) Alice sends an email to Bob (GMail user). 2) Bob receives the email, meanwhile Google scrapes the content and extracts its meaning to show Bob some ads and use the facts to improve Google's A.I. Alice wants a way to mark her email content as "no index". So that email service provider don't crawl through the content. Exactly like the robots.txt for domains or the "noindex" metatag in HTML head element!
  - lern_too_spel 12y ago
    Why should Alice get to restrict what Bob can do with his inbox?
    - frik 12y ago
      It's not about Bob, he can do what he want. It's about the email service provider, that should stop analyze the email text to extract its meaning. Gmail uses it to display ads to Bob, builds a shadow profile for Alice (like Facebook) and trains an artificial intelligence (see headline link).

rstupek 12y ago

It would be nice if there were a way to provide google, bing, et. al. the data so they were constantly crawling sites. A non-insignificant amount of our bandwidth is used up daily by search engines spidering sites.