3 ms·

> (which are notably not the same as crawling) Is that a distinction without a difference? I think the robots.txt RFC was addressed specifically to crawlers;

> (which are notably not the same as crawling) Is that a distinction without a difference? I think the robots.txt RFC was addressed specifically to crawlers; so technically "ad hoc" requests generated automatically (i.e. by robots) aren't included. But the distinction operators would like to make is between humans and automata. Whether some automaton is a crawler or not isn't relevant.

lolinder 2y ago
Actually, no, the fact that it's a crawler is the most important fact. The reason why website operators care at all about robots accessing their site (as distinct from humans controlling a browser) is historically one of two reasons: * The pattern of requests can be very problematic. Impolite crawlers are totally capable of taking down a website by hitting it over and over and over again for hours in a way that humans won't. * Crawlers are generally used to build search indexes, so instructing them about URLs that would be inappropriate to have show up in a search is relevant. The behavior that OP is complaining about is that when the user pastes a URL into Perplexity, Perplexity fetches that URL. Neither the traffic pattern nor the persistence profile are remotely similar to typical crawler behavior. As far as I can see there's almost nothing to distinguish it from someone using Edge and then using Edge's built-in summarizer.
Dylan16807 2y ago
If explicitly telling it to access a URL is an access by automaton, then isn't every web browser load an access by automaton?
- BeefWellington 2y ago
  The flaw with that example is your web browser isn't between other users and the website, turning 500 views into one. And if we took the analogy to the other end, one could argue that all crawlers have to be kicked off manually at some point... The problem is here in reality the differentiation is somewhat more understood. The honor system web is going away, that's for sure.
  - lolinder 2y ago
    > your web browser isn't between other users and the website, turning 500 views into one. There are a lot of people making this assumption about the way Perplexity is working, but there is no evidence in TFA that Perplexity is caching its ad hoc requests. And even if they were, what's left unsaid is why it even would matter if 500 views turned into one. It matters either because of lost ad revenue or lost ability to track the users' behavior. Personally, I'm okay with moving past that phase of the internet's life and look forward to new business models that aren't built around getting large numbers of "views".
  - Dylan16807 2y ago
    > The flaw with that example is your web browser isn't between other users and the website, turning 500 views into one. So, a caching proxy? That has its own issues, but it's the opposite of access by automaton. One button press causes less than one access to the server. Though one button press still results in one user view, so it's only reducing loads in some ways. But also is that happening here? > And if we took the analogy to the other end, one could argue that all crawlers have to be kicked off manually at some point... One button press causing a million page loads is access by automaton. The distinction seems pretty simple to me.

lolinder 2y ago

Actually, no, the fact that it's a crawler is the most important fact. The reason why website operators care at all about robots accessing their site (as distinct from humans controlling a browser) is historically one of two reasons: * The pattern of requests can be very problematic. Impolite crawlers are totally capable of taking down a website by hitting it over and over and over again for hours in a way that humans won't. * Crawlers are generally used to build search indexes, so instructing them about URLs that would be inappropriate to have show up in a search is relevant. The behavior that OP is complaining about is that when the user pastes a URL into Perplexity, Perplexity fetches that URL. Neither the traffic pattern nor the persistence profile are remotely similar to typical crawler behavior. As far as I can see there's almost nothing to distinguish it from someone using Edge and then using Edge's built-in summarizer.

Dylan16807 2y ago

If explicitly telling it to access a URL is an access by automaton, then isn't every web browser load an access by automaton?

BeefWellington 2y ago
The flaw with that example is your web browser isn't between other users and the website, turning 500 views into one. And if we took the analogy to the other end, one could argue that all crawlers have to be kicked off manually at some point... The problem is here in reality the differentiation is somewhat more understood. The honor system web is going away, that's for sure.
- lolinder 2y ago
  > your web browser isn't between other users and the website, turning 500 views into one. There are a lot of people making this assumption about the way Perplexity is working, but there is no evidence in TFA that Perplexity is caching its ad hoc requests. And even if they were, what's left unsaid is why it even would matter if 500 views turned into one. It matters either because of lost ad revenue or lost ability to track the users' behavior. Personally, I'm okay with moving past that phase of the internet's life and look forward to new business models that aren't built around getting large numbers of "views".
- Dylan16807 2y ago
  > The flaw with that example is your web browser isn't between other users and the website, turning 500 views into one. So, a caching proxy? That has its own issues, but it's the opposite of access by automaton. One button press causes less than one access to the server. Though one button press still results in one user view, so it's only reducing loads in some ways. But also is that happening here? > And if we took the analogy to the other end, one could argue that all crawlers have to be kicked off manually at some point... One button press causing a million page loads is access by automaton. The distinction seems pretty simple to me.