4 ms·
The lie is in their documentation - they claim to use the PerplexityBot string in their user-agent: https://docs.perplexity.ai/docs/perplexitybot https://docs.p
by sjm-lbm 2y ago
The lie is in their documentation - they claim to use the PerplexityBot string in their user-agent: https://docs.perplexity.ai/docs/perplexitybot https://docs.perplexity.ai/docs/perplexitybot.
- simonw 2y agoThat is for the crawler, which is used to collect data for their search index. I think it is OK to use a different user agent for page retrievals made on demand that a user specifically requested (not to include in the index, just to answer a question). But... I think that user agent should be documented and should not just be a browser default. OpenAI do this for their crawlers: they have GPTBot for their crawler and ChatGPT-User for the requests made by their ChatGPT browser mode.
- sjm-lbm 2y agoYeah, that seems reasonable to me as well. I'm honestly not sure if this is a "lie" in the most basic sense, or more information omission done in a way that feels intentionally dishonest. At the very least, I do think that having an entire page in your docs about the user-agent strings you use without mentioning that, sometimes, you don't use those user agents at all is fairly misleading.
- simonw 2y agoYeah, I agree with that.
- bombela 2y agoIt's not a lie. This is the agent string of the bot used for ingesting data for training the AI. In the blog post, this is not what is happening. It is merely feeding the webpage as context to the AI during inference. You are all confused here.
- cdme 2y agoWebsite owners should be able to block this behavior as well — OpenAI has two different agents and doesn't obscure the agent when a user initiates a fetch.