3 ms·
It looks to me like this is mainly about RAG - Perplexity answers user questions by running searches and then displaying content from those searches to users, a
by simonw 1y ago
It looks to me like this is mainly about RAG - Perplexity answers user questions by running searches and then displaying content from those searches to users, and the BBC are arguing that this content display violates their copyright.
Unsurprisingly this article confuses the issue somewhat by also talking about training models on content. I understand why that's in there - it's a hot topic, especially in the UK right now - but I don't think it's directly relevant to this complaint.
The note about robots.txt is interesting - "The BBC said in its letter that while it disallowed two of Perplexity's crawlers, the company "is clearly not respecting robots.txt".
Perplexity describe their user-agents here: https://docs.perplexity.ai/guides/bots https://docs.perplexity.ai/guides/bots
I had a look at https://www.bbc.com/robots.txt https://www.bbc.com/robots.txt and it does indeed block both PerplexityBot ("designed to surface and link websites in search results on Perplexity" - I think that's their search index crawler) and Perplexity-User ("When users ask Perplexity a question, it might visit a web page to help provide an accurate answer and include a link to the page in its response").
But... I checked the Internet Archive for a random earlier date - Feb 2025 - https://web.archive.org/web/20250208052005/https://www.bbc.com/robots.txt https://web.archive.org/web/20250208052005/https://www.bbc.c... - and back then the BBC were blocking PerplexityBot but not Perplexity-User.
- hadrien01 1y agoThey also write this: > Since a user requested the fetch, this fetcher generally ignores robots.txt rules.
- dabeeeenster 1y agoI mean, that's just not true.
- esskay 1y agoWhich part? It's widely established and known that many AI crawlers are ignoring the robots.txt file, perplexity being one of them [1] [1]https://www.tomshardware.com/tech-industry/artificial-intelligence/several-ai-companies-said-to-be-ignoring-robots-dot-txt-exclusion-scraping-content-without-permission-report https://www.tomshardware.com/tech-industry/artificial-intell...
- simonw 1y agoOh wow, I missed that! That's from the docs for that Perplexity‑User user-agent, at which point presumably there's no point in listing that in robots.txt at all?
- bitpush 1y ago> Since a user requested the fetch, this fetcher generally ignores robots.txt rules. Normally the expecation is that the user-agent faithfully presents the content it fetched. If I make a browser that fetches bbc.com, and strips away ads and presented it to users - I would expect BBC to not like it and block the user-agent from accessing it. It isnt a robots.txt thing. It is a user-agent thing.
- whilenot-dev 1y agoFor what its worth, this statement here regarding Perplexity-User: > Since a user requested the fetch, this fetcher generally ignores robots.txt rules. ...has been added sometime between 30.01.2025[0] and 07.02.2025[1], and makes it sound like robots.txt was not respected by that bot anyways. [0]: https://web.archive.org/web/20250130164401/https://docs.perplexity.ai/guides/bots https://web.archive.org/web/20250130164401/https://docs.perp... [1]: https://web.archive.org/web/20250207113929/https://docs.perplexity.ai/guides/bots https://web.archive.org/web/20250207113929/https://docs.perp...
- simonw 1y agoGreat catch there.