It seems to me there could be some confusion here. When providing a service such as Perplexity AI's, there are two use cases to consider for accessing web site

It seems to me there could be some confusion here. When providing a service such as Perplexity AI's, there are two use cases to consider for accessing web sites. One is the scraping use case for training, where a crawler is being used and it is gathering data in bulk. Hopefully in a way that doesn't hammer one site at a time, but spreads the requests around gently. The other is the use case for fulfilling a user's specific query in real time. The blog post seemed to be hitting this second use case. In this use case, the system component that retrieves the web page is not acting as a crawler, but more as a browser or something akin to a browser plugin that is retrieving the content on behalf of the actual human end user, on their request. It's appropriate that these two use cases have different norms for how they behave. The author may have been thinking of the first use case, but actually exercising the second use case, and mistakenly expecting it to behave according to how it should behave for the first use case.