3 ms·

If the AI archives/caches all the results it accesses and enough people use it, doesn't it become a scraper? Just learn off the cached data. Being the man-in-th

If the AI archives/caches all the results it accesses and enough people use it, doesn't it become a scraper? Just learn off the cached data. Being the man-in-the-middle seems like a pretty easy way to scrape salient content while also getting signals about that content's value.

JimDabell 1y ago
No. The key difference is that if a user asks about a specific page, when Perplexity fetches that page, it is being operated by a human not acting as a crawler. It doesn’t matter how many times this happens or what they do with the result. If they aren’t recursively fetching pages, then they aren’t a crawler and robots.txt does not apply to them. robots.txt is not a generic access control mechanism, it is designed solely for automated clients.
- sbarre 1y ago
  I would only agree with this if we knew for sure that these on-demand human-initiated crawls didn't result in the crawled page being added to an overall index and scheduled for future automated crawls. Otherwise it's just adding an unwilling website to a crawl index, and showing the result of the first crawl as a byproduct of that action.
- fluidcruft 1y ago
  Many people don't want their data used for free/any training. AI developers have been so repeatedly unethical that the well-earned Baysian prior is high probability that you cannot trust AI developers to not cross the training/inference streams.
  - JimDabell 1y ago
    > Many people don't want their data used for free/any training. That is true. But robots.txt is not designed to give them the ability to prevent this.
    - gunalx 1y ago
      It is in the name, rules for the robots. Any scraping ai or not, and even mass recrsive or single page, should abide by the rules.
- glenstein 1y ago
  > It doesn’t matter how many times this happens or what they do with the result. That's where you lost me, as this is key to GP's point above and it takes more than a mere out-of-left-field declaration that "it doesn't matter" to settle the question of whether it matters. I think they raised an important point about using cached data to support functions beyond the scope of simple at-request page retrieval.
gruez 1y ago
>If the AI archives/caches all the results it accesses and enough people use it, doesn't it become a scraper? That's basically how many crowdsourced crawling/archive projects work. For instance, sci-hub and RECAP[1]. Do you think they should be shut down as well? In both cases there's even a stronger justification to shutting them down, because the original content is paywalled and you could plausibly argue there's lost revenue on the line. [1] https://en.wikipedia.org/wiki/Free_Law_Project#RECAP https://en.wikipedia.org/wiki/Free_Law_Project#RECAP
- fluidcruft 1y ago
  I didn't suggest Perplexity should be shut down, though. And yes, in your analogy sites are completely justified to take whatever actions they can to block people who are building those caches.

JimDabell 1y ago

No. The key difference is that if a user asks about a specific page, when Perplexity fetches that page, it is being operated by a human not acting as a crawler. It doesn’t matter how many times this happens or what they do with the result. If they aren’t recursively fetching pages, then they aren’t a crawler and robots.txt does not apply to them. robots.txt is not a generic access control mechanism, it is designed solely for automated clients.

sbarre 1y ago
I would only agree with this if we knew for sure that these on-demand human-initiated crawls didn't result in the crawled page being added to an overall index and scheduled for future automated crawls. Otherwise it's just adding an unwilling website to a crawl index, and showing the result of the first crawl as a byproduct of that action.
fluidcruft 1y ago
Many people don't want their data used for free/any training. AI developers have been so repeatedly unethical that the well-earned Baysian prior is high probability that you cannot trust AI developers to not cross the training/inference streams.
- JimDabell 1y ago
  > Many people don't want their data used for free/any training. That is true. But robots.txt is not designed to give them the ability to prevent this.
  - gunalx 1y ago
    It is in the name, rules for the robots. Any scraping ai or not, and even mass recrsive or single page, should abide by the rules.
glenstein 1y ago
> It doesn’t matter how many times this happens or what they do with the result. That's where you lost me, as this is key to GP's point above and it takes more than a mere out-of-left-field declaration that "it doesn't matter" to settle the question of whether it matters. I think they raised an important point about using cached data to support functions beyond the scope of simple at-request page retrieval.

gruez 1y ago

>If the AI archives/caches all the results it accesses and enough people use it, doesn't it become a scraper? That's basically how many crowdsourced crawling/archive projects work. For instance, sci-hub and RECAP[1]. Do you think they should be shut down as well? In both cases there's even a stronger justification to shutting them down, because the original content is paywalled and you could plausibly argue there's lost revenue on the line. [1] https://en.wikipedia.org/wiki/Free_Law_Project#RECAP https://en.wikipedia.org/wiki/Free_Law_Project#RECAP

fluidcruft 1y ago
I didn't suggest Perplexity should be shut down, though. And yes, in your analogy sites are completely justified to take whatever actions they can to block people who are building those caches.