4 ms·

I notice they don't actually give a good reason that robots.txt isn't suitable. Change for the sake of it?

by dbetteridge 3y ago

strombofulous 3y ago
AI!
- asudosandwich 3y ago
  Indeed, AI is called out directly. Google’s laying groundwork for their own version of a regulatory moat.
Animats 3y ago
It doesn't require signing up with Google.
- 0x073 3y ago
  If it would be public the ai could read it and can develop countermeasures ;) .
vore 3y ago
To steelman this maybe, I think they’re angling for something like a mechanism to indicate content is OK to index but not OK to use as AI training data. Maybe you could fudge it today with user agents in robots.txt but who knows what the concrete idea of this is.
- varenc 3y ago
  robots.txt is already outmoded. It only can indicate that content can’t be crawled but a URL marked this way can still be indexed. As Google says “it is not a mechanism for keeping a web page out of Google” [0] You need to use other things besides robots.txt to preventing indexing. [0] https://developers.google.com/search/docs/crawling-indexing/robots/intro https://developers.google.com/search/docs/crawling-indexing/...
  - dazc 3y ago
    Indeed, having pages indexed which can't then be crawled is a great way of shooting yourself in the foot.
    - floomk 3y ago
      I think you meant it's a great way for google to punish you for not giving them full access
  - deleted 3y ago
    [deleted]
- Aerroon 3y ago
  This seems weird to me though, aren't search engines something very similar to AI, if not outright AI?
helsinkiandrew 3y ago
> I notice they don't actually give a good reason that robots.txt isn't suitable It's kind of implied: specifying sitemaps/allowance/copyright for different use cases: search, scraping, republishing, training etc. and perhaps adding some of the non standard extensions: Crawl-delay, default host, even sitemap isn't part of the robots.txt standard > We believe it’s time for the web and AI communities to explore additional machine-readable means for web publisher choice and control for emerging AI and research use cases.
h1fra 3y ago
Came here to say that, seems like nobody as the answer :/ Maybe they want to have finer details on page content, e.g: "you can index those pages but not those nodes" or "those nodes are also AI generated please ignore". Otherwise I don't know, robots.txt is not sexy but definitely does the job.
JohnFen 3y ago
I think robots.txt isn't suitable for this for the same reason it's not suitable for keeping other bots from crawling your site: adhering to what robots.txt says is optional, and plenty of bots opt to ignore it.

I notice they don't actually give a good reason that robots.txt isn't suitable. Change for the sake of it?

strombofulous 3y ago

AI!

asudosandwich 3y ago
Indeed, AI is called out directly. Google’s laying groundwork for their own version of a regulatory moat.

Animats 3y ago

It doesn't require signing up with Google.

0x073 3y ago
If it would be public the ai could read it and can develop countermeasures ;) .

vore 3y ago

To steelman this maybe, I think they’re angling for something like a mechanism to indicate content is OK to index but not OK to use as AI training data. Maybe you could fudge it today with user agents in robots.txt but who knows what the concrete idea of this is.

varenc 3y ago
robots.txt is already outmoded. It only can indicate that content can’t be crawled but a URL marked this way can still be indexed. As Google says “it is not a mechanism for keeping a web page out of Google” [0] You need to use other things besides robots.txt to preventing indexing. [0] https://developers.google.com/search/docs/crawling-indexing/robots/intro https://developers.google.com/search/docs/crawling-indexing/...
- dazc 3y ago
  Indeed, having pages indexed which can't then be crawled is a great way of shooting yourself in the foot.
  - floomk 3y ago
    I think you meant it's a great way for google to punish you for not giving them full access
- deleted 3y ago
  [deleted]
Aerroon 3y ago
This seems weird to me though, aren't search engines something very similar to AI, if not outright AI?

helsinkiandrew 3y ago

> I notice they don't actually give a good reason that robots.txt isn't suitable It's kind of implied: specifying sitemaps/allowance/copyright for different use cases: search, scraping, republishing, training etc. and perhaps adding some of the non standard extensions: Crawl-delay, default host, even sitemap isn't part of the robots.txt standard > We believe it’s time for the web and AI communities to explore additional machine-readable means for web publisher choice and control for emerging AI and research use cases.

h1fra 3y ago

Came here to say that, seems like nobody as the answer :/ Maybe they want to have finer details on page content, e.g: "you can index those pages but not those nodes" or "those nodes are also AI generated please ignore". Otherwise I don't know, robots.txt is not sexy but definitely does the job.

JohnFen 3y ago

I think robots.txt isn't suitable for this for the same reason it's not suitable for keeping other bots from crawling your site: adhering to what robots.txt says is optional, and plenty of bots opt to ignore it.