4 ms·
I've jyst asked perplexity ai itself: this is the answer In summary: Officially, Perplexity claims its bots honor robots.txt. In practice, outside investigator
by oriettaxx 1y ago
I've jyst asked perplexity ai itself: this is the answer
In summary: Officially, Perplexity claims its bots honor robots.txt. In practice, outside investigators and hosting providers document persistent circumvention of such directives by undeclared or disguised crawlers acting on Perplexity’s behalf, especially for real-time user queries