5 ms·
To steelman this maybe, I think they’re angling for something like a mechanism to indicate content is OK to index but not OK to use as AI training data. Maybe y
by vore 3y ago
To steelman this maybe, I think they’re angling for something like a mechanism to indicate content is OK to index but not OK to use as AI training data. Maybe you could fudge it today with user agents in robots.txt but who knows what the concrete idea of this is.
- varenc 3y agorobots.txt is already outmoded. It only can indicate that content can’t be crawled but a URL marked this way can still be indexed. As Google says “it is not a mechanism for keeping a web page out of Google” [0] You need to use other things besides robots.txt to preventing indexing. [0] https://developers.google.com/search/docs/crawling-indexing/robots/intro https://developers.google.com/search/docs/crawling-indexing/...
- Aerroon 3y agoThis seems weird to me though, aren't search engines something very similar to AI, if not outright AI?