4 ms·

It seems self-evident to me that if a user tells a bot to go get a web page, robots.txt doesn't apply, and the bot shouldn't respect it. I understand others' co

It seems self-evident to me that if a user tells a bot to go get a web page, robots.txt doesn't apply, and the bot shouldn't respect it. I understand others' concerns that, like Apple's reader, and other similar tools, it's ethically debatable whether a site should be required to comply with the request, and spoofing an agent seems in dubious territory. I don't think a good answer has been proposed for this challenge, unfortunately.

lolinder 2y ago
> spoofing an agent seems in dubious territory. Just to clarify, Perplexity is not spoofing a user agent, they're legitimately using a headless Chrome to fetch the page. The author just misunderstood their docs [0]: when they say that "you can identify our web crawler by its user agent", they're talking about the crawler, not the browser they use for ad hoc requests. As you note, crawling is different. [0] https://docs.perplexity.ai/docs/perplexitybot https://docs.perplexity.ai/docs/perplexitybot
- mattigames 2y ago
  This is completely false, the user agent being used by Perplexity its _not_ the headless-chrome user agent, wich is close similar to this (emphasis on HeadlessChrome): Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/119.0.0.0 Safari/537.36 They are spoofing it to pretend to be a desktop Chrome one: Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36
  - lolinder 2y ago
    Ah, you're correct, my bad. I don't personally have a problem with spoofing user agents, but yeah, they're either spoofing or for some reason they're truly using a non-headless Chrome.
  - jcrawfordor 2y ago
    There's a difference here between "headless chrome" as a concept and "headless-chrome" the software. It's still pretty common to run browser automation with a full "headful" browser, in which case you would just get the normal user agent. headless-chrome is sort of an optimized option that comes with some downsides.

lolinder 2y ago

> spoofing an agent seems in dubious territory. Just to clarify, Perplexity is not spoofing a user agent, they're legitimately using a headless Chrome to fetch the page. The author just misunderstood their docs [0]: when they say that "you can identify our web crawler by its user agent", they're talking about the crawler, not the browser they use for ad hoc requests. As you note, crawling is different. [0] https://docs.perplexity.ai/docs/perplexitybot https://docs.perplexity.ai/docs/perplexitybot

mattigames 2y ago
This is completely false, the user agent being used by Perplexity its _not_ the headless-chrome user agent, wich is close similar to this (emphasis on HeadlessChrome): Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/119.0.0.0 Safari/537.36 They are spoofing it to pretend to be a desktop Chrome one: Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36
- lolinder 2y ago
  Ah, you're correct, my bad. I don't personally have a problem with spoofing user agents, but yeah, they're either spoofing or for some reason they're truly using a non-headless Chrome.
- jcrawfordor 2y ago
  There's a difference here between "headless chrome" as a concept and "headless-chrome" the software. It's still pretty common to run browser automation with a full "headful" browser, in which case you would just get the normal user agent. headless-chrome is sort of an optimized option that comes with some downsides.