Google does not use a separate crawler, but they do use the robots.txt semantics to control what's allowed in the AI "index" separately from the search index. F

Google does not use a separate crawler, but they do use the robots.txt semantics to control what's allowed in the AI "index" separately from the search index. From the docs: > Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity. https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers https://developers.google.com/search/docs/crawling-indexing/...