3 ms·
That is for the crawler, which is used to collect data for their search index. I think it is OK to use a different user agent for page retrievals made on deman
by simonw 2y ago
That is for the crawler, which is used to collect data for their search index.
I think it is OK to use a different user agent for page retrievals made on demand that a user specifically requested (not to include in the index, just to answer a question).
But... I think that user agent should be documented and should not just be a browser default.
OpenAI do this for their crawlers: they have GPTBot for their crawler and ChatGPT-User for the requests made by their ChatGPT browser mode.
- sjm-lbm 2y agoYeah, that seems reasonable to me as well. I'm honestly not sure if this is a "lie" in the most basic sense, or more information omission done in a way that feels intentionally dishonest. At the very least, I do think that having an entire page in your docs about the user-agent strings you use without mentioning that, sometimes, you don't use those user agents at all is fairly misleading.
- simonw 2y agoYeah, I agree with that.