3 ms·
Yes, but there's no way to say "allow indexing for search, but not for AI use", right?
by foota 1y ago
Yes, but there's no way to say "allow indexing for search, but not for AI use", right?
- warkdarrior 1y agoBut there is: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers https://developers.google.com/search/docs/crawling-indexing/... There is an user agent for search that you can control in robots.txt. user-agent: Googlebot There is another user agent for AI training. user-agent: Google-Extended
- foota 1y agoWow, I had no idea this page existed, thanks for the reference!