But there is: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers https://developers.google.com/search/docs/crawling-indexing/...

But there is: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers https://developers.google.com/search/docs/crawling-indexing/... There is an user agent for search that you can control in robots.txt. user-agent: Googlebot There is another user agent for AI training. user-agent: Google-Extended