4 ms·
Do the AI features apply to content from sites that block their 'Google-Extended' crawler (which they use to index content for their LLMs)?[1] I'm not as worri
by johneth 2y ago
Do the AI features apply to content from sites that block their 'Google-Extended' crawler (which they use to index content for their LLMs)?[1]
I'm not as worried if that distinction still applies – I can just block 'Google-Extended' and let the normal Googlebot through. If not, though, that feels like the beginning of the end of people wanting to be indexed, eventually leading to Google eating itself.
[1] https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers https://developers.google.com/search/docs/crawling-indexing/...