2 ms·
It is. They do go further than just ignoring robots.txt too. https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-cr
by Hamuko 1y ago
It is. They do go further than just ignoring robots.txt too.
https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/ https://blog.cloudflare.com/perplexity-is-using-stealth-unde...