> If the user asks about a particular page and Perplexity fetches only that page, then robots.txt has nothing to say about this and Perplexity shouldn’t even co

> If the user asks about a particular page and Perplexity fetches only that page, then robots.txt has nothing to say about this and Perplexity shouldn’t even consider it That's not what Perplexity own documentation[1] says though: "Webmasters can use the following robots.txt tags to manage how their sites and content interact with Perplexity Perplexity-User supports user actions within Perplexity. When users ask Perplexity a question, it might visit a web page to help provide an accurate answer and include a link to the page in its response. Perplexity-User controls which sites these user requests can access. It is not used for web crawling or to collect content for training AI foundation models." [1] https://docs.perplexity.ai/guides/bots https://docs.perplexity.ai/guides/bots