For what its worth, this statement here regarding Perplexity-User: > Since a user requested the fetch, this fetcher generally ignores robots.txt rules. ...has

For what its worth, this statement here regarding Perplexity-User: > Since a user requested the fetch, this fetcher generally ignores robots.txt rules. ...has been added sometime between 30.01.2025[0] and 07.02.2025[1], and makes it sound like robots.txt was not respected by that bot anyways. [0]: https://web.archive.org/web/20250130164401/https://docs.perplexity.ai/guides/bots https://web.archive.org/web/20250130164401/https://docs.perp... [1]: https://web.archive.org/web/20250207113929/https://docs.perplexity.ai/guides/bots https://web.archive.org/web/20250207113929/https://docs.perp...