Google has a separate user-agent identifier in for robots.txt for AI products that can be blocked separately from the search crawler. Many sites block the AI cr

Google has a separate user-agent identifier in for robots.txt for AI products that can be blocked separately from the search crawler. Many sites block the AI crawler while allowing the google search crawler. AI content is controlled using googlebot-extended user-agent See https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers https://developers.google.com/search/docs/crawling-indexing/... In fact NYT does block google from using its content for GenAI products: User-agent: Google-Extended Disallow: / As well as chat gpt User-agent: GPTBot Disallow: / So idk why this would be unexpected. See https://www.nytimes.com/robots.txt https://www.nytimes.com/robots.txt