4 ms·

But it's not scraping, it's retrieving the page on request from the user.

by ulrikrasmussen 2y ago

cdme 2y ago
With no benefit provided to the creator — they're not directing users out, they're pulling data in.
- deleted 2y ago
  [deleted]
- threecheese 2y ago
  They are directing users __in__ in some cases though, no? I’m a perplexity user, and their summaries are often way off which drives me to the references (attribution). The ratio of fetches to clickthroughs is what’s important now though; this new model (which we’ve not negotiated or really asked for) is driving that upward from 1, and not only are you paying more as a provider but your consumer is paying more ($ to perplexity and/or via ad backend) and you aren’t seeing any of it. And you pay those extra costs to indirectly finance the competitor who put you in this situation, who intends to drive that ratio as high as it can in order to get more money from more of your customers tomorrow. Yay.
JumpCrisscross 2y ago
> it's not scraping, it's retrieving the page on request from the user Search engines already tried it. It’s not retrieving on request because the user didn’t request the page, they requested a bot find specific content on any page.
- alexey-salmin 2y ago
  But it's not what happened here. It WAS retrieving on request. > I went into Perplexity and asked "What's on this page rknight.me/PerplexityBot?". Immediately I could see the log and just like Lewis, the user agent didn't include their custom user agent
  - JumpCrisscross 2y ago
    That was to test the user-agent hiding. The broader problem—Perplexity laundering attribution—is where the scraping vs retrieval question comes into play.
    - Dylan16807 2y ago
      Well the example in the post doesn't show any laundering. Do you have an example of it? Unless you mean the entire concept of training launders attribution, but that's basically unrelated to this post and the complaints inside it.
  - threecheese 2y ago
    In this case you are 100% correct, but I think it’s reasonable to assume that the “read me this web page” use case constitutes a small minority of perplexity’s fetches. I find it useful because of the attribution - more so its references - which I almost always navigate to because its summaries are frequently crap.
- SamBam 2y ago
  This is why this conversation is making me insane. How are people saying straight-faced that the user is requesting a specific page? They aren't, they're doing a search of the web. That's not at all the same as a browser visiting a page.
  - lolinder 2y ago
    Because that's literally what the author does in TFA and then complains about when Perplexity complies. > What is this post about https://rknight.me/blog/blocking-bots-with-nginx/ https://rknight.me/blog/blocking-bots-with-nginx/
    - manuelmoreale 2y ago
      Am I the only one that sees a difference between “show me page X” and “what is page X about”? The first is how browsers work. The second is what perplexity is doing. Those two are clearly different imo.
      - halostatue 2y ago
        You are not. Perplexity should always respect robots.txt, even for summarization requests. If I say that I don't want Perplexity crawling my site, I mean at all, and I explicitly would not want them "summarizing" my page. The response from Perplexity to such a request should be "The owner of this page/site does not permit Perplexity to process any data from this site." Period. LLMs can't summarize in any case: https://ea.rna.nl/2024/05/27/when-chatgpt-summarises-it-actually-does-nothing-of-the-kind/ https://ea.rna.nl/2024/05/27/when-chatgpt-summarises-it-actu...
        lolinder 2y ago
        > Perplexity should always respect robots.txt, even for summarization requests. If I say that I don't want Perplexity crawling my site, I mean at all Issuing a single HTTP request is definitionally not crawling, and the robots.txt spec is specifically for crawlers, which this is not. If you want a specific tool to exclude you from their web request feature you have to talk to them about it. The web was designed to maximize interop between tools, it correctly doesn't have a mechanism for blacklisting specific tools from your site.
- SpaghettiCthulu 2y ago
  That's not how it works in this case. The author asked the AI for information about a specific page.
  - anileated 2y ago
    The only way available to immediately test whether Perplexity pretends not to be Perplexity is by actively requesting a page. The fact that they mask their UA in that scenario makes it fairly obvious that they are not above bending rules and “working around” inconvenient for them public conventions. It seems safe to assume, until proven otherwise, that they would fake their bots’ user agents in every other case, such as when acquiring training data.

But it's not scraping, it's retrieving the page on request from the user.

cdme 2y ago

With no benefit provided to the creator — they're not directing users out, they're pulling data in.

deleted 2y ago
[deleted]
threecheese 2y ago
They are directing users __in__ in some cases though, no? I’m a perplexity user, and their summaries are often way off which drives me to the references (attribution). The ratio of fetches to clickthroughs is what’s important now though; this new model (which we’ve not negotiated or really asked for) is driving that upward from 1, and not only are you paying more as a provider but your consumer is paying more ($ to perplexity and/or via ad backend) and you aren’t seeing any of it. And you pay those extra costs to indirectly finance the competitor who put you in this situation, who intends to drive that ratio as high as it can in order to get more money from more of your customers tomorrow. Yay.

JumpCrisscross 2y ago

> it's not scraping, it's retrieving the page on request from the user Search engines already tried it. It’s not retrieving on request because the user didn’t request the page, they requested a bot find specific content on any page.

alexey-salmin 2y ago
But it's not what happened here. It WAS retrieving on request. > I went into Perplexity and asked "What's on this page rknight.me/PerplexityBot?". Immediately I could see the log and just like Lewis, the user agent didn't include their custom user agent
- JumpCrisscross 2y ago
  That was to test the user-agent hiding. The broader problem—Perplexity laundering attribution—is where the scraping vs retrieval question comes into play.
  - Dylan16807 2y ago
    Well the example in the post doesn't show any laundering. Do you have an example of it? Unless you mean the entire concept of training launders attribution, but that's basically unrelated to this post and the complaints inside it.
- threecheese 2y ago
  In this case you are 100% correct, but I think it’s reasonable to assume that the “read me this web page” use case constitutes a small minority of perplexity’s fetches. I find it useful because of the attribution - more so its references - which I almost always navigate to because its summaries are frequently crap.
SamBam 2y ago
This is why this conversation is making me insane. How are people saying straight-faced that the user is requesting a specific page? They aren't, they're doing a search of the web. That's not at all the same as a browser visiting a page.
- lolinder 2y ago
  Because that's literally what the author does in TFA and then complains about when Perplexity complies. > What is this post about https://rknight.me/blog/blocking-bots-with-nginx/ https://rknight.me/blog/blocking-bots-with-nginx/
  - manuelmoreale 2y ago
    Am I the only one that sees a difference between “show me page X” and “what is page X about”? The first is how browsers work. The second is what perplexity is doing. Those two are clearly different imo.
    - halostatue 2y ago
      You are not. Perplexity should always respect robots.txt, even for summarization requests. If I say that I don't want Perplexity crawling my site, I mean at all, and I explicitly would not want them "summarizing" my page. The response from Perplexity to such a request should be "The owner of this page/site does not permit Perplexity to process any data from this site." Period. LLMs can't summarize in any case: https://ea.rna.nl/2024/05/27/when-chatgpt-summarises-it-actually-does-nothing-of-the-kind/ https://ea.rna.nl/2024/05/27/when-chatgpt-summarises-it-actu...
      - lolinder 2y ago
        > Perplexity should always respect robots.txt, even for summarization requests. If I say that I don't want Perplexity crawling my site, I mean at all Issuing a single HTTP request is definitionally not crawling, and the robots.txt spec is specifically for crawlers, which this is not. If you want a specific tool to exclude you from their web request feature you have to talk to them about it. The web was designed to maximize interop between tools, it correctly doesn't have a mechanism for blacklisting specific tools from your site.
SpaghettiCthulu 2y ago
That's not how it works in this case. The author asked the AI for information about a specific page.
- anileated 2y ago
  The only way available to immediately test whether Perplexity pretends not to be Perplexity is by actively requesting a page. The fact that they mask their UA in that scenario makes it fairly obvious that they are not above bending rules and “working around” inconvenient for them public conventions. It seems safe to assume, until proven otherwise, that they would fake their bots’ user agents in every other case, such as when acquiring training data.