3 ms·

Is an AI chatbot fetching a web page to answer a prompt a 'web scraping bot'? If there is a user actively promoting the LLM, isn't it more of a user agent? My m

Is an AI chatbot fetching a web page to answer a prompt a 'web scraping bot'? If there is a user actively promoting the LLM, isn't it more of a user agent? My mental model, even before LLMs, was that a human being present changes a bot into a user agent. I'm curious if others agree.

bgwalter 1y ago
The Register calls them "fetchers". They still reproduce the content of the original website without the website gaining anything but additional high load. I'm not sure how many websites are searched and discarded per query. Since it's the remote, proprietary LLM that initiates the search I would hesitate to call them agents. Maybe "fetcher" is the best term.
- ronsor 1y ago
  > The Register calls them "fetchers". They still reproduce the content of the original website without the website gaining anything but additional high load. So does my browser when I have uBlock Origin enabled.
danaris 1y ago
But they're (generally speaking) not being asked for the contents of one specific webpage, fetching that, and summarizing it for the user. They're going out and scraping everything, so that when they're asked a question, they can pull a plausible answer from their dataset and summarize the page they found it on. Even the ones that actively go out and search/scrape in response to queries aren't just scraping a single site. At best, they're scraping some subset of the entire internet that they have tagged as being somehow related to the query. So even if what they present to the user is a summary of a single webpage, that is rarely going to be the product of a single request to that single webpage. That request is going to be just one of many, most of which are entirely fruitless for that specific query: purely extra load for their servers, with no gain whatsoever.

3 ms·

Is an AI chatbot fetching a web page to answer a prompt a 'web scraping bot'? If there is a user actively promoting the LLM, isn't it more of a user agent? My m

by 8organicbits 1y ago

bgwalter 1y ago
The Register calls them "fetchers". They still reproduce the content of the original website without the website gaining anything but additional high load. I'm not sure how many websites are searched and discarded per query. Since it's the remote, proprietary LLM that initiates the search I would hesitate to call them agents. Maybe "fetcher" is the best term.
- ronsor 1y ago
  > The Register calls them "fetchers". They still reproduce the content of the original website without the website gaining anything but additional high load. So does my browser when I have uBlock Origin enabled.
danaris 1y ago
But they're (generally speaking) not being asked for the contents of one specific webpage, fetching that, and summarizing it for the user. They're going out and scraping everything, so that when they're asked a question, they can pull a plausible answer from their dataset and summarize the page they found it on. Even the ones that actively go out and search/scrape in response to queries aren't just scraping a single site. At best, they're scraping some subset of the entire internet that they have tagged as being somehow related to the query. So even if what they present to the user is a summary of a single webpage, that is rarely going to be the product of a single request to that single webpage. That request is going to be just one of many, most of which are entirely fruitless for that specific query: purely extra load for their servers, with no gain whatsoever.