3 ms·

What is a "visit"? TFA demonstrates that they got a hit on their site, that's how they got the logs. Is it necessary to load the JavaScript for it to count as

What is a "visit"? TFA demonstrates that they got a hit on their site, that's how they got the logs. Is it necessary to load the JavaScript for it to count as a visit? What if I access the site with noscript? Or is it only a visit if I see all your recommended content? I usually block those recommendations so that I don't get distracted from the article I actually came to read—is my visit a less legitimate visit than other people's? What exactly is Perplexity doing here that isn't okay that people don't already do with their local user agents?

alt227 2y ago
> TFA demonstrates that they got a hit on their site Whats stopping perplexity caching this info say for 24 hours, and then redisplaying it to the next few hundred people who request it?
- lolinder 2y ago
  Then they don't get the extra hits. So is that it—is a "visit" important because of the data that you're able to collect from the visit? Does this place HN's rampant use of archive.md on the same moral footing as Perplexity?
denton-scratch 2y ago
> What exactly is Perplexity doing here that isn't okay that people don't already do with their local user agents? It's in the title of TFA: they're being dishonest about who they are. PerplexityBot seems to understand that robots.txt is addressed to it. It's understood that site operators have a right to use the User-Agent to discriminate among visitors; that's why robots.txt is a standard. Crawlers that disrespect the standard have for many years been considered beyond the pale; thieves and snoopers. TFA's complaint is entirely justified.
- lolinder 2y ago
  > It's in the title of TFA: they're being dishonest about who they are. PerplexityBot seems to understand that robots.txt is addressed to it. First, I'm ignoring the output of Perplexity. I have no reason to believe that they gave the LLM any knowledge about its internal operations, it's just riffing off of what OP is saying. Second, PerplexityBot is the user agent that they use when crawling and indexing. They never claimed to use that user agent for ad hoc HTTP requests (which are notably not the same as crawling). Third, I disagree that anyone has an obligation to be honest in their User-Agent. Have you ever looked at Chrome's user agent? They're spoofing just about everyone, as is every browser. Crawlers should respect robots.txt, but I'd be totally content if we just got rid of the User-Agent string entirely.
  - denton-scratch 2y ago
    > (which are notably not the same as crawling) Is that a distinction without a difference? I think the robots.txt RFC was addressed specifically to crawlers; so technically "ad hoc" requests generated automatically (i.e. by robots) aren't included. But the distinction operators would like to make is between humans and automata. Whether some automaton is a crawler or not isn't relevant.
    - lolinder 2y ago
      Actually, no, the fact that it's a crawler is the most important fact. The reason why website operators care at all about robots accessing their site (as distinct from humans controlling a browser) is historically one of two reasons: * The pattern of requests can be very problematic. Impolite crawlers are totally capable of taking down a website by hitting it over and over and over again for hours in a way that humans won't. * Crawlers are generally used to build search indexes, so instructing them about URLs that would be inappropriate to have show up in a search is relevant. The behavior that OP is complaining about is that when the user pastes a URL into Perplexity, Perplexity fetches that URL. Neither the traffic pattern nor the persistence profile are remotely similar to typical crawler behavior. As far as I can see there's almost nothing to distinguish it from someone using Edge and then using Edge's built-in summarizer.
    - Dylan16807 2y ago
      If explicitly telling it to access a URL is an access by automaton, then isn't every web browser load an access by automaton?
      - BeefWellington 2y ago
        The flaw with that example is your web browser isn't between other users and the website, turning 500 views into one. And if we took the analogy to the other end, one could argue that all crawlers have to be kicked off manually at some point... The problem is here in reality the differentiation is somewhat more understood. The honor system web is going away, that's for sure.
baxtr 2y ago
A visit is a human reader. At the very least they get exposed to your website name. Notice your product/service if you get lucky. Become a customer at a later visit. We are talking about cutting the first step off so that everything which may come afterwards is cut off as well.
- lolinder 2y ago
  The behavior that TFA is complaining about is that when the user drops a link to a site into Perplexity it is able to summarize the content of that link. This isn't about the discoverability aspect of Perplexity, they're specifically complaining that the ad hoc "summarize this post" requests don't respect robots.txt [0]. That's what I'm arguing in favor of and that's the behavior that TFA is attacking. [0] Which, incidentally, is entirely normal. robots.txt is for the web crawler that indexes, not for ad hoc requests.
- TeMPOraL 2y ago
  In other words, content is bait, reward is a captured user whose attention - whose sanity, the finite amount of life - can be wasted or plain used against them. I'm more than happy to see all the websites with attention economy business models to shut down. Yes, that might be 90% of the Internet. That would be the 90% that is poisonous shit.
  - baxtr 2y ago
    The attention economy will never die. Attention will only shift. From websites to aggregators like perplexity.
    - TeMPOraL 2y ago
      Perplexity isn't playing in the attention economy unless they upsell you, advertise to you, or put any other kind of bullshit between you and your goal. Attention economy is (as the name suggests) about monetizing attention; it does so through friction.
      - 2y ago

alt227 2y ago

> TFA demonstrates that they got a hit on their site Whats stopping perplexity caching this info say for 24 hours, and then redisplaying it to the next few hundred people who request it?

lolinder 2y ago
Then they don't get the extra hits. So is that it—is a "visit" important because of the data that you're able to collect from the visit? Does this place HN's rampant use of archive.md on the same moral footing as Perplexity?

denton-scratch 2y ago

> What exactly is Perplexity doing here that isn't okay that people don't already do with their local user agents? It's in the title of TFA: they're being dishonest about who they are. PerplexityBot seems to understand that robots.txt is addressed to it. It's understood that site operators have a right to use the User-Agent to discriminate among visitors; that's why robots.txt is a standard. Crawlers that disrespect the standard have for many years been considered beyond the pale; thieves and snoopers. TFA's complaint is entirely justified.

lolinder 2y ago
> It's in the title of TFA: they're being dishonest about who they are. PerplexityBot seems to understand that robots.txt is addressed to it. First, I'm ignoring the output of Perplexity. I have no reason to believe that they gave the LLM any knowledge about its internal operations, it's just riffing off of what OP is saying. Second, PerplexityBot is the user agent that they use when crawling and indexing. They never claimed to use that user agent for ad hoc HTTP requests (which are notably not the same as crawling). Third, I disagree that anyone has an obligation to be honest in their User-Agent. Have you ever looked at Chrome's user agent? They're spoofing just about everyone, as is every browser. Crawlers should respect robots.txt, but I'd be totally content if we just got rid of the User-Agent string entirely.
- denton-scratch 2y ago
  > (which are notably not the same as crawling) Is that a distinction without a difference? I think the robots.txt RFC was addressed specifically to crawlers; so technically "ad hoc" requests generated automatically (i.e. by robots) aren't included. But the distinction operators would like to make is between humans and automata. Whether some automaton is a crawler or not isn't relevant.
  - lolinder 2y ago
    Actually, no, the fact that it's a crawler is the most important fact. The reason why website operators care at all about robots accessing their site (as distinct from humans controlling a browser) is historically one of two reasons: * The pattern of requests can be very problematic. Impolite crawlers are totally capable of taking down a website by hitting it over and over and over again for hours in a way that humans won't. * Crawlers are generally used to build search indexes, so instructing them about URLs that would be inappropriate to have show up in a search is relevant. The behavior that OP is complaining about is that when the user pastes a URL into Perplexity, Perplexity fetches that URL. Neither the traffic pattern nor the persistence profile are remotely similar to typical crawler behavior. As far as I can see there's almost nothing to distinguish it from someone using Edge and then using Edge's built-in summarizer.
  - Dylan16807 2y ago
    If explicitly telling it to access a URL is an access by automaton, then isn't every web browser load an access by automaton?
    - BeefWellington 2y ago
      The flaw with that example is your web browser isn't between other users and the website, turning 500 views into one. And if we took the analogy to the other end, one could argue that all crawlers have to be kicked off manually at some point... The problem is here in reality the differentiation is somewhat more understood. The honor system web is going away, that's for sure.

baxtr 2y ago

A visit is a human reader. At the very least they get exposed to your website name. Notice your product/service if you get lucky. Become a customer at a later visit. We are talking about cutting the first step off so that everything which may come afterwards is cut off as well.

lolinder 2y ago
The behavior that TFA is complaining about is that when the user drops a link to a site into Perplexity it is able to summarize the content of that link. This isn't about the discoverability aspect of Perplexity, they're specifically complaining that the ad hoc "summarize this post" requests don't respect robots.txt [0]. That's what I'm arguing in favor of and that's the behavior that TFA is attacking. [0] Which, incidentally, is entirely normal. robots.txt is for the web crawler that indexes, not for ad hoc requests.
TeMPOraL 2y ago
In other words, content is bait, reward is a captured user whose attention - whose sanity, the finite amount of life - can be wasted or plain used against them. I'm more than happy to see all the websites with attention economy business models to shut down. Yes, that might be 90% of the Internet. That would be the 90% that is poisonous shit.
- baxtr 2y ago
  The attention economy will never die. Attention will only shift. From websites to aggregators like perplexity.
  - TeMPOraL 2y ago
    Perplexity isn't playing in the attention economy unless they upsell you, advertise to you, or put any other kind of bullshit between you and your goal. Attention economy is (as the name suggests) about monetizing attention; it does so through friction.
    - 2y ago