6 ms·

I don’t think we should lump together “AI company scraping a website to train their base model” and “AI tool retrieving a web page because I asked it to”. At le

by wrs 2y ago

I don’t think we should lump together “AI company scraping a website to train their base model” and “AI tool retrieving a web page because I asked it to”. At least, those should be two different user agents so you have the option to block one and not the other.

sebzim4500 2y ago
More than this, I'd rather use a tool which lets me fake the user agent like I can in my browser.
JohnMakin 2y ago
Is it actually retrieving the page on the fly though? How do you know this? Even if it were - it’s not supposed to be able to.
- IAmGraydon 2y ago
  He literally showed a server log of it retrieving the page on the fly in the article.
- tommy_axle 2y ago
  What I gathered from the post was that one of the investigations was to ask what was on [some page url] and then check the logs moments later and saw it using a normal user agent.
- janalsncm 2y ago
  To steel man this, even though I think the article did a fine job already, maybe the author could’ve changed the content on the page so you would know if they were serving a cached response.
  - rknightuk 2y ago
    Author here. The page I asked it to summarize was posted after I implemented all blocking on the server (and robots.txt). So they should not have had any cached data.
- supriyo-biswas 2y ago
  You can just point it at a webserver and ask it a question like "Summarize the content at [URL]" with a sufficiently unique URL that no one would hit, maybe with an UUID. This is also explored on the very article itself. In my testing they're using crawlers on AWS and they do not parse Javascript or CSS, so it is sufficient to serve some kind of interstitial challenge page like the one on Cloudflare, or you can build your own.
- parasense 2y ago
  > Is it actually retrieving the page on the fly though? They are able to do so. > How do you know this? The access logs. > Even if it were - it’s not supposed to be able to. There is a distinction from data used to train a model, which is the indexing bot with the custom user-agent string, and the user-query input given to the aforementioned AI model. When you ask an AI some question, you normally input text into a form, and the text goes back to the AI model where the magic happens. In this scenario, instead of inputting a wall text into a form, the text is coming from a url. These forms of user input are equivilent, and yet distinctly different. Therefore it's intelectually dishonest for the OP to claim the AI is indexing them, when OP is asking the AI to fetch their website to augment or add context to the question being asked.
condiment 2y ago
If an AI agent is performing a search on behalf of a user, should its user agent be the same as that user’s?
- gumby 2y ago
  I think that’s the ideal as the server may provide different data depending on UA. Does anyone actually do this, though?
  - JoosToopit 2y ago
    I fake my UA the way I like.
    - compootr 2y ago
      exactly, web standards are simply a suggestion, you can work around them any way you want
    - gumby 2y ago
      And why shouldn’t you — it’s your computer! But my question should have been phrased, “are there any frameworks commonly in use these days that provide different js payloads to different clients? I’ve been out of that part of the biz for a very long time so this could be a naive question.
- Filligree 2y ago
  Users don’t have user agent strings, user agents do.
- lofaszvanitt 2y ago
  It should, erm sorry, must pass all the info it got from the user to you, so you would have an idea who wanted info from your site.
supriyo-biswas 2y ago
And yet, OpenAI blocks both of these activities if you happen to block either "GPTBot" (the ingest crawler) or "ChatGPT-User" (retrieval during chat).
JoosToopit 2y ago
[flagged]
KomoD 2y ago
I agree with that, but I also think that they should at least identify themselves instead of using a generic user agent.
- BriggyDwiggs42 2y ago
  I’d rather share less information than more to any site I visit. Why does a user want to share that info?
  - KomoD 2y ago
    What, users won't share anything? I said I wanted Perplexity to identify themselves in the user agent instead of using the generic "Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.3" they're using right now for the "non-scraper bot". How does that impact users at all?
    - TeMPOraL 2y ago
      I don't, because if it will, then someone like the author of the article will do the obnoxious thing and ban it. We've been there before, 30 years ago. That's why all browsers' user agent strings start with "Mozilla".
      - sensanaty 2y ago
        Why is the author here obnoxious, and not Perplexity? I don't want these scumbag AI companies making money off me, end of story.
        TeMPOraL 2y ago
        The "scumbag AI company" in question is making money by offering me a way to access information while skipping any and all attention economy bullshit you may have on your site, on top of being just plain more convenient. Note that the author is confusing crawling (which is done with documented User Agent and presumably obeys robots.txt) with browsing (which is done by working as one-off user agent for the user). As for why this behavior is obnoxious, I refer you to 30 years worth of arguing on this, as it's been discussed ever since User-Agent header was first added, and then used by someone to discriminate visitors based on their browsers.
mrweasel 2y ago
Personally I don't even think that the issue. I'd prefer correct user-agent, that just common decency and shouldn't be an issue for most. What I do expect the AI companies to do is to check the license of the content they scrape and follow that. Let's say I run a blog, and I have a CC BY-NC 4.0 license. You can train your AI and that content, as long as it's non-commercial. Otherwise you'd need to contact me an negotiate and appropriate license, for a fee. Or you can train your AI on my personal Github repo, where everything is ISC, that's fine, but for my work, which is GPLv3, then you have to ensure that the code your LLM returns is also under the GPLv3. Does any of the AI companies check that the license of ANYTHING?
- lolinder 2y ago
  > I'd prefer correct user-agent, that just common decency and shouldn't be an issue for most. Tell that to the Chrome team. And the Safari team. And the Opera team. [0] [0] https://webaim.org/blog/user-agent-string-history/ https://webaim.org/blog/user-agent-string-history/
xbar 2y ago
Why should I have to differentiate Perplexity's services?

by wrs 2y ago

sebzim4500 2y ago

More than this, I'd rather use a tool which lets me fake the user agent like I can in my browser.

JohnMakin 2y ago

Is it actually retrieving the page on the fly though? How do you know this? Even if it were - it’s not supposed to be able to.

IAmGraydon 2y ago
He literally showed a server log of it retrieving the page on the fly in the article.
tommy_axle 2y ago
What I gathered from the post was that one of the investigations was to ask what was on [some page url] and then check the logs moments later and saw it using a normal user agent.
janalsncm 2y ago
To steel man this, even though I think the article did a fine job already, maybe the author could’ve changed the content on the page so you would know if they were serving a cached response.
- rknightuk 2y ago
  Author here. The page I asked it to summarize was posted after I implemented all blocking on the server (and robots.txt). So they should not have had any cached data.
supriyo-biswas 2y ago
You can just point it at a webserver and ask it a question like "Summarize the content at [URL]" with a sufficiently unique URL that no one would hit, maybe with an UUID. This is also explored on the very article itself. In my testing they're using crawlers on AWS and they do not parse Javascript or CSS, so it is sufficient to serve some kind of interstitial challenge page like the one on Cloudflare, or you can build your own.
parasense 2y ago
> Is it actually retrieving the page on the fly though? They are able to do so. > How do you know this? The access logs. > Even if it were - it’s not supposed to be able to. There is a distinction from data used to train a model, which is the indexing bot with the custom user-agent string, and the user-query input given to the aforementioned AI model. When you ask an AI some question, you normally input text into a form, and the text goes back to the AI model where the magic happens. In this scenario, instead of inputting a wall text into a form, the text is coming from a url. These forms of user input are equivilent, and yet distinctly different. Therefore it's intelectually dishonest for the OP to claim the AI is indexing them, when OP is asking the AI to fetch their website to augment or add context to the question being asked.

condiment 2y ago

If an AI agent is performing a search on behalf of a user, should its user agent be the same as that user’s?

gumby 2y ago
I think that’s the ideal as the server may provide different data depending on UA. Does anyone actually do this, though?
- JoosToopit 2y ago
  I fake my UA the way I like.
  - compootr 2y ago
    exactly, web standards are simply a suggestion, you can work around them any way you want
  - gumby 2y ago
    And why shouldn’t you — it’s your computer! But my question should have been phrased, “are there any frameworks commonly in use these days that provide different js payloads to different clients? I’ve been out of that part of the biz for a very long time so this could be a naive question.
Filligree 2y ago
Users don’t have user agent strings, user agents do.
lofaszvanitt 2y ago
It should, erm sorry, must pass all the info it got from the user to you, so you would have an idea who wanted info from your site.

supriyo-biswas 2y ago

And yet, OpenAI blocks both of these activities if you happen to block either "GPTBot" (the ingest crawler) or "ChatGPT-User" (retrieval during chat).

JoosToopit 2y ago

[flagged]

KomoD 2y ago

I agree with that, but I also think that they should at least identify themselves instead of using a generic user agent.

BriggyDwiggs42 2y ago
I’d rather share less information than more to any site I visit. Why does a user want to share that info?
- KomoD 2y ago
  What, users won't share anything? I said I wanted Perplexity to identify themselves in the user agent instead of using the generic "Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.3" they're using right now for the "non-scraper bot". How does that impact users at all?
  - TeMPOraL 2y ago
    I don't, because if it will, then someone like the author of the article will do the obnoxious thing and ban it. We've been there before, 30 years ago. That's why all browsers' user agent strings start with "Mozilla".
    - sensanaty 2y ago
      Why is the author here obnoxious, and not Perplexity? I don't want these scumbag AI companies making money off me, end of story.
      - TeMPOraL 2y ago
        The "scumbag AI company" in question is making money by offering me a way to access information while skipping any and all attention economy bullshit you may have on your site, on top of being just plain more convenient. Note that the author is confusing crawling (which is done with documented User Agent and presumably obeys robots.txt) with browsing (which is done by working as one-off user agent for the user). As for why this behavior is obnoxious, I refer you to 30 years worth of arguing on this, as it's been discussed ever since User-Agent header was first added, and then used by someone to discriminate visitors based on their browsers.

mrweasel 2y ago

Personally I don't even think that the issue. I'd prefer correct user-agent, that just common decency and shouldn't be an issue for most. What I do expect the AI companies to do is to check the license of the content they scrape and follow that. Let's say I run a blog, and I have a CC BY-NC 4.0 license. You can train your AI and that content, as long as it's non-commercial. Otherwise you'd need to contact me an negotiate and appropriate license, for a fee. Or you can train your AI on my personal Github repo, where everything is ISC, that's fine, but for my work, which is GPLv3, then you have to ensure that the code your LLM returns is also under the GPLv3. Does any of the AI companies check that the license of ANYTHING?

lolinder 2y ago
> I'd prefer correct user-agent, that just common decency and shouldn't be an issue for most. Tell that to the Chrome team. And the Safari team. And the Opera team. [0] [0] https://webaim.org/blog/user-agent-string-history/ https://webaim.org/blog/user-agent-string-history/

xbar 2y ago

Why should I have to differentiate Perplexity's services?