3 ms·
> Instead of blocking, it would be neater to present and alternative version to the crawlers IIUC, if a site presents different (view of) content to the crawle
by amf12 3y ago
> Instead of blocking, it would be neater to present and alternative version to the crawlers
IIUC, if a site presents different (view of) content to the crawler than users, the site can get de-indexed.
- JohnFen 3y agoIf that's a concern, and you're only worried about AI crawlers, then that's not a problem. Only provide the bogus pages to the AI crawlers, not to the search engine crawlers. Assuming there's a difference, anyway. I suspect that with Google and Bing, there isn't.