4 ms·

> At this point, Wired jumped in, confirming a finding from Robb Knight: Perplexity’s scraping of Forbes’ work wasn’t an exception. In fact, Perplexity has been

by Ukv 2y ago

> At this point, Wired jumped in, confirming a finding from Robb Knight: Perplexity’s scraping of Forbes’ work wasn’t an exception. In fact, Perplexity has been ignoring the robots.txt code that explicitly asks web crawlers not to scrape the page There's a distinction that is being missed between: * Web crawlers automatically accessing pages, such as recursively following links to index them for search engines * A tool accessing a URL in direct response to a user request robots.txt is only intended for the former. For instance, archive.is state: > [Why does archive.is not obey robots.txt?] Because it is not a free-walking crawler, it saves only one page acting as a direct agent of the human user. Such services don't obey robots.txt (e.g. Google Feedfetcher, screenshot- or pdf-making services, isup.me, …) Perplexity do (as far as I've been able to find) respect robots.txt for their scraping. What the investigations confirmed it was ignored for was users entering a URL to summarise. There is still a conversation to be had. I think users should be able to browse the web with whatever tool they want to - whether that's a standard browser, a minimal reader mode that doesn't show ads, or a statistical summarisation tool - but it does pose a problem for sites that rely on standard browsers letting them show users advertisements, get them to sign up to mailing lists, collect data, etc. if users decide to choose tools that don't allow that.

tomp 2y ago
Exactly. AI that answers questions using online sources isn't a web crawler, but a web researchers.
- deleted 2y ago
  [deleted]
maypop 2y ago
There’s a bit of a middle ground where the user may say something like “tech articles from new site” which would require the agent to move around reading and aggregating information to return to the user. Is this the former or the latter?
- zamadatix 2y ago
  I think the dividing line is more "happens in anticipation there might be a user session which wants the data" and "happens as the result of a user session which wants the data" than it is about how it was done. E.g. prefetching content after a page load is done as a result of the user session getting to the page, it just as going around to any sites it can find to cache it on the chance the user might need them.
nerdjon 2y ago
I have been pretty adamant about the privacy issues and stealing of content issues with AI... but this particular case seems, honestly I am on the side of perplexity here. Mostly... Once we get to the point of this AI running locally, or you just have a browser extension that grabs the data and sends it to the server, what is the difference between what it is doing now and that. If I use an AI and I give it a specific URL I think the expectation is that it is acting like a user and not like a crawler so the robots.txt is meaningless. That doesn't mean that perplexity is free of any blame here, if they are getting around paywalls or then using this loophole to gather up the data to train later. That isnt ok. I am also concerned about the reports of them using third party crawlers to get around blocking them. But, purely on what this seems to be focused on and the example given. I don't see anything wrong with it when it is triggered based on a specific action of the user. And should be the expected behavior IMO. Headline grabbing articles like this I feel like undermine the valid criticism of AI because it can be used to paint a picture of a vendetta against AI or worse as an example of why these arguments are wrong as a way to discredit large swaths of criticism. We really need to be more mindefull about how we criticize the real problems with the technology.

by Ukv 2y ago

tomp 2y ago

Exactly. AI that answers questions using online sources isn't a web crawler, but a web researchers.

deleted 2y ago
[deleted]

maypop 2y ago

There’s a bit of a middle ground where the user may say something like “tech articles from new site” which would require the agent to move around reading and aggregating information to return to the user. Is this the former or the latter?

zamadatix 2y ago
I think the dividing line is more "happens in anticipation there might be a user session which wants the data" and "happens as the result of a user session which wants the data" than it is about how it was done. E.g. prefetching content after a page load is done as a result of the user session getting to the page, it just as going around to any sites it can find to cache it on the chance the user might need them.

nerdjon 2y ago

I have been pretty adamant about the privacy issues and stealing of content issues with AI... but this particular case seems, honestly I am on the side of perplexity here. Mostly... Once we get to the point of this AI running locally, or you just have a browser extension that grabs the data and sends it to the server, what is the difference between what it is doing now and that. If I use an AI and I give it a specific URL I think the expectation is that it is acting like a user and not like a crawler so the robots.txt is meaningless. That doesn't mean that perplexity is free of any blame here, if they are getting around paywalls or then using this loophole to gather up the data to train later. That isnt ok. I am also concerned about the reports of them using third party crawlers to get around blocking them. But, purely on what this seems to be focused on and the example given. I don't see anything wrong with it when it is triggered based on a specific action of the user. And should be the expected behavior IMO. Headline grabbing articles like this I feel like undermine the valid criticism of AI because it can be used to paint a picture of a vendetta against AI or worse as an example of why these arguments are wrong as a way to discredit large swaths of criticism. We really need to be more mindefull about how we criticize the real problems with the technology.