8 ms·

There are two different questions at play here, and we need to be careful what we wish for. The first concern is the most legitimate one: can I stop an LLM fro

There are two different questions at play here, and we need to be careful what we wish for. The first concern is the most legitimate one: can I stop an LLM from training itself on my data? This should be possible and Perplexity should absolutely make it easy to block them from training. The second concern, though, is can Perplexity do a live web query to my website and present data from my website in a format that the user asks for? Arguing that we should ban this moves into very dangerous territory. Everything from ad blockers to reader mode to screen readers do exactly the same thing that Perplexity is doing here, with the only difference being that they tend to be exclusively local. The very nature of a "user agent" is to be an automated tool that manipulates content hosted on the internet according to the specifications given to the tool by the user. I have a hard time seeing an argument against Perplexity using this data in this way that wouldn't apply equally to countless tools that we already all use and which companies try with varying degrees of success to block. I don't want to live in a world where website owners can use DRM to force me to display their website in exactly the way that their designers envisioned it. I want to be able to write scripts to manipulate the page and present it in a way that's useful for me. I don't currently use llms this way, but I'm uncomfortable with arguing that it's unethical for them to do that so long as they're citing the source.

neycoda 2y ago
AI scraping against permission could allow corporations to formulate a loophole where Congress argues that it's impossible to enforce a law against and that it's easier to just make laws to allow corporations to close-source their websites (yes, HTML, CSS, and JavaScript, etc). I think what's most likely to happen is nothing will fundamentally change, and browsers will continue showing page source, and AI will continue scraping source content without permission.
deleted 2y ago
[deleted]
gpm 2y ago
> The second concern, though, is can perplexity do a live web query to my website and present data from my website in a format that the user asks for? Arguing that we should ban this moves into very dangerous territory. This feels like the fundamental core component of what copyright allows you to forbid. > Everything from ad blockers to reader mode to screen readers do exactly the same thing that Perplexity is doing here, with the only difference being that they tend to be exclusively local Which is a huge difference. The latter is someone asking for a copy of my content (from someone with a valid license, myself), and manipulating it to display it (not creating new copies, broadly speaking allowed by copyright). The former adds in the criminal step of "and redistributing (modified, but that doesn't matter) versions of it to users without permission". I mean, I'm all for getting rid of copyright, but I also know that's an incredibly unpopular position to take, and I don't see how this isn't just copyright infringement if you aren't advocating for repealing copyright law all together.
- lolinder 2y ago
  I'm curious to know where you draw the line for what constitutes legitimate manipulation by a person and when it becomes distribution. I'm assuming that if I write code by hand for every part of the TCP/IP and HTTP stack I'm safe. What if I use libraries written by other people for the TCP/IP and HTTP part? What if I use a whole FOSS web browser? What about a paid local web browser? What if I run a script that I wrote on a cloud server? What if I then allow other people to download and use that script on their own cloud servers? What if I decide to offer that script as a service for free to friends and family, who can use my cloud server? What if I offer it for free to the general public? What if I start accepting money for that service, but I guarantee that only the one person who asked for the site sees the output? Can you help me to understand where exactly I crossed the line?
  - gpm 2y ago
    Obviously not legal advice and I doubt it's entirely settled law, but probably this step > What if I decide to offer that script as a service for free to friends and family, who can use my cloud server? You're allowed to make copies and adaptations in order to utilize the program (website), which probably covers a cloud server you yourself are controlling. You aren't allowed to do other things with those copies though, like distribute them to other people. Payment only matters if we're getting into "free use" arguments, and I don't think any really apply here. I think you're probably already in trouble with just offering it to family and friends, but if you take the next step offering it to the public that adds more issues because the copyright act includes definitions like "To perform or display a work “publicly” means (1) to perform or display it at a place open to the public or at any place where a substantial number of persons outside of a normal circle of a family and its social acquaintances is gathered; or (2) to transmit or otherwise communicate a performance or display of the work to a place specified by clause (1) or to the public, by means of any device or process, whether the members of the public capable of receiving the performance or display receive it in the same place or in separate places and at the same time or at different times."
    - lolinder 2y ago
putlake 2y ago
It's funny I posted the inverse of this. As a web publisher, I am fine with folks using my content to train their models because this training does not directly steal any traffic. It's the "train an AI by reading all the books in the world" analogy. But what Perplexity is doing when they crawl my content in response to a user question is that they are decreasing the probability that this user would come to by content (via Google, for example). This is unacceptable. A tool that runs on-device (like Reader mode) is different because Perplexity is an aggregator service that will continue to solidify its position as a demand aggregator and I will never be able to get people directly on my content. There are many benefits to having people visit your content on a property that you own. e.g., say you are a SaaS company and you have a bunch of Help docs. You can analyze traffic in this section of your website to get insights to improve your business: what are the top search queries from my users, this might indicate to me where they are struggling or what new features I could build. In a world where users ask Perplexity these Help questions about my SaaS, Perplexity may answer them and I would lose all the insights because I never get any traffic.
- epolanski 2y ago
  > they are decreasing the probability that this user would come to by content (via Google, for example). Google has been providing summaries of stuff and hijacking traffic for ages. I kid you not, in the tourism sector this has been a HUGE issue, we have seen 50%+ decrease in views when they started doing it. We paid gazzilions to write quality content for tourists about the most different places just so Google could put it on their homepage. It's just depressing. I'm more and more convinced that the age of regulations and competition is gone, US does want to have unkillable monopolies in the tech sector and we are all peons.
  - itsoktocry 2y ago
    >We paid gazzilions to write quality content for tourists about the most different places just so Google could put it on their homepage. It's just depressing It's a legitimate complaint, and it sucks for your business. But I think this demonstrates that the sort of quality content you were producing doesn't actually have much value.
buro9 2y ago
The companies will scrape and internalise the "customer asked for this" requests... and slowly turn the latter into the former, or just their own tool as the scraper. No, easier to just ask a simple question: Does the company respect the access rules communicated via a web standard? No? In that case hard deny access to that company. These companies don't need to be given an inch.
- lolinder 2y ago
  > Does the company respect the access rules communicated via a web standard? No? In that case hard deny access to that company. So should Firefox not allow changing the user agent in order to bypass websites that erroneously claim to not work on Firefox?
  - rezonant 2y ago
    Similarly, for sites which configure robots.txt to disallow all bots except Googlebot, I don't lose sleep about new search engines taking that with a grain of salt.
- elicksaur 2y ago
  This is exactly the concern and there’s a lot of comments just completely ignoring it or willfully conflating. Ad block isn’t the same problem because it doesn’t and can’t steal the creator’s data.
  - lolinder 2y ago
    > Ad block isn’t the same problem because it doesn’t and can’t steal the creator’s data. Arguably it does. That topic has been debated endlessly and there are plenty of people on HN who are willing to fiercely argue that adblock is theft. I happen to agree with you that adblock doesn't steal data, but I'm also completely unsure why interacting with a tool over a network suddenly turns what would be acceptable on my local computer into theft.
  - Dylan16807 2y ago
    If that's the concern, then ask for a line in the terms and conditions that explicitly says a user-initiated request will not be saved or used for training. Don't act like the access itself is an affront.
deleted 2y ago
[deleted]
baxtr 2y ago
What will happen if: Website owners decide to stop publishing because it’s not rewarded by a real human visit anymore? Then perplexity and the like won’t have new information to train their models on and no sites to answer the questions. I think there is a real content dilemma here at work. The incentives of Google and website owners were more or less aligned. This is not the case with perplexity.
- lolinder 2y ago
  What is a "visit"? TFA demonstrates that they got a hit on their site, that's how they got the logs. Is it necessary to load the JavaScript for it to count as a visit? What if I access the site with noscript? Or is it only a visit if I see all your recommended content? I usually block those recommendations so that I don't get distracted from the article I actually came to read—is my visit a less legitimate visit than other people's? What exactly is Perplexity doing here that isn't okay that people don't already do with their local user agents?
  - alt227 2y ago
    > TFA demonstrates that they got a hit on their site Whats stopping perplexity caching this info say for 24 hours, and then redisplaying it to the next few hundred people who request it?
    - lolinder 2y ago
      Then they don't get the extra hits. So is that it—is a "visit" important because of the data that you're able to collect from the visit? Does this place HN's rampant use of archive.md on the same moral footing as Perplexity?
  - denton-scratch 2y ago
    > What exactly is Perplexity doing here that isn't okay that people don't already do with their local user agents? It's in the title of TFA: they're being dishonest about who they are. PerplexityBot seems to understand that robots.txt is addressed to it. It's understood that site operators have a right to use the User-Agent to discriminate among visitors; that's why robots.txt is a standard. Crawlers that disrespect the standard have for many years been considered beyond the pale; thieves and snoopers. TFA's complaint is entirely justified.
lofaszvanitt 2y ago
Citing the source doesn't bring you, the owner of the site, valuable data. When was your data accessed, who accessed it, from where, at what time, what device, etc. It brings data to the LLM's owner, and you get N O T H I N G. Could you change the way printed news magazines showed their content? No. Then, why is that a problem? Btw nobody clicks on sources. NOBODY.
- bluish29 2y ago
  > Btw nobody clicks on sources. NOBODY. I always click on sources to verify what an LLM in this case says. I also hear the claim that a lot about people not reading sources (before LLM it was video content with references) but I always visited the sources. Is there a statistics or studies that actually support this claim? Or is it just a personal experience, of people (including me) enforcing it as generic behavior of all people?
  - lofaszvanitt 2y ago
    That's you, because you are a researcher or coder or someone who uses his brain much more than average, hence not an average joe. I ran a news site for 15 years and the stats showed that from 10000 views on an article, only a miniscule amount of clicks were made on the source links. Average people do not care where the info is coming from. Also perplexity shows the videos on their site, you cannot go to youtube, you have to start it on their site, and then you have to click on the youtube player's logo in the lower right to get to the site. Perplexity is getting greedy.
    - SpaghettiCthulu 2y ago
      You said "NOBODY" (pretty sure the all caps means it's extra true).
      - lofaszvanitt 2y ago
        Well, that's the reality. The tech savvy people here are the exception, and represent only a very minor percentage of users.
insane_dreamer 2y ago
> I don't want to live in a world where website owners can use DRM to force me to display their website in exactly the way that their designers envisioned it. I'm okay with this world, as a tradeoff. I'm not sure users should have _the right_ to reformat others' content.
- zzo38computer 2y ago
  Users should have the right to reformat their own copy of others content (automatically as well as manually). However, if they then redistribute the reformatted copy, then they should not be allowed to claim that it is the same formatting as the original, because it is not the same as the original.
nradov 2y ago
Why should it be possible to stop an LLM from training itself on your data? If you want to restrict access to data then don't post it on a public website. It's easy enough to require registration and agreement to licensing terms for access. It seems like some website owners want to have their cake and eat it too. They want their content indexed by Google and other crawlers in order to drive search traffic but they don't want their content used to train AI models that benefit other companies. At some point they're going to have to make a choice.
- marcus0x62 2y ago
  Because if I run a server - at my own expense - I get to use information provided by the client to determine what, if any, response to provide? This isn’t a very difficult concept to grasp.
  - nradov 2y ago
    I'm having difficulty grasping the concept. Only a fool would trust any HTTP headers such as User-Agent sent by a random unauthenticated client. Your expenses are your problem.
    - marcus0x62 2y ago
      … and I have absolutely no obligation to provide any particular response to any particular client. Parsing, rendering, and trusting that the payload is consistent from request to request is your problem. You can connect to my server, or not. I really don’t care. What you cannot do is dictate how my server responds to your request.
      - nradov 2y ago
        Sure. So just return an HTTP 4XX response to requests you don't like. What's the problem?
        marcus0x62 2y ago
dheera 2y ago
Personally I think AI is a major win for accessibility and we should not be preventing people to access information in the way that is best suited for them. Accessibility can mean everything from a blind person wanting to interacting with a website using voice, to someone recovering from a surgery and wanting something to reduce unnecessary popups and clicks on a website to get to the information they need. Accessibility is in the eye of the accessor, and AI is what enables them to achieve it. The way I see it, AI is not a robot and doesn't need to look at robots.txt. Rather, AI is my low-cost secretary.
- danlitt 2y ago
  > The way I see it, AI is not a robot and doesn't need to look at robots.txt I don't think you are seeing it very clearly then. Your secretary can also be a robot. What do you think an AI is if not a robot?? It doesn't "need" to look at robots.txt because nothing does.
gcanyon 2y ago
It seems self-evident to me that if a user tells a bot to go get a web page, robots.txt doesn't apply, and the bot shouldn't respect it. I understand others' concerns that, like Apple's reader, and other similar tools, it's ethically debatable whether a site should be required to comply with the request, and spoofing an agent seems in dubious territory. I don't think a good answer has been proposed for this challenge, unfortunately.
- lolinder 2y ago
  > spoofing an agent seems in dubious territory. Just to clarify, Perplexity is not spoofing a user agent, they're legitimately using a headless Chrome to fetch the page. The author just misunderstood their docs [0]: when they say that "you can identify our web crawler by its user agent", they're talking about the crawler, not the browser they use for ad hoc requests. As you note, crawling is different. [0] https://docs.perplexity.ai/docs/perplexitybot https://docs.perplexity.ai/docs/perplexitybot
  - mattigames 2y ago
    This is completely false, the user agent being used by Perplexity its _not_ the headless-chrome user agent, wich is close similar to this (emphasis on HeadlessChrome): Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/119.0.0.0 Safari/537.36 They are spoofing it to pretend to be a desktop Chrome one: Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36
    - lolinder 2y ago
      Ah, you're correct, my bad. I don't personally have a problem with spoofing user agents, but yeah, they're either spoofing or for some reason they're truly using a non-headless Chrome.
    - jcrawfordor 2y ago
      There's a difference here between "headless chrome" as a concept and "headless-chrome" the software. It's still pretty common to run browser automation with a full "headful" browser, in which case you would just get the normal user agent. headless-chrome is sort of an optimized option that comes with some downsides.
cal85 2y ago
> can I stop an LLM from training itself on my data? This should be possible and Perplexity should absolutely make it easy to block them from training. I’m not saying you’re wrong, but why? And what do you mean by “your data” here?
- __loam 2y ago
  By "my data" he means the data a site spent time and money to create and publish.
- bhelkey 2y ago
  > And what do you mean by “your data” here? The website that they created.
treyd 2y ago
Yeah if people get to extensive about blocking then we're going to end up with a scenario where the web request functionality is implementing by telling the chatbot's users's browser to make the fetch and submit it back to the server for processing, making it largely indistinguishable from the user making the query themselves. If CORS gets in the way they can just prompt users to install a browser extension to use the web request functionality.
__loam 2y ago
The problem that Perplexity has that ad blockers don't is that they're an independent site that is publishing content based on work they didn't produce. That runs afoul of both copyright laws and section 230 which let's sites like Google and Facebook operate. That's pretty different from an ad blocker running on your local machine. The ad blocker isn't publishing the page it edited for you.
- lolinder 2y ago
  > they're an independent site that is publishing content based on work they didn't produce. What distinguishes these two situations? * User asks proprietary web browser to fetch content and render it a specific way, which it does * User asks proprietary web service to fetch content and render it a specific way, which it does The technical distinction is that there's a network involved in the second scenario. What is the moral distinction? Why is it that a proprietary web service manipulating content on behalf of a user is "publishing" content illegally, while a proprietary web browser doing the exact same kind of transformations is not? Assume that in both cases the proprietary software fetches the data upon request, does not cache it, and does not make the transformed content available to other users.
  - quectophoton 2y ago
    I don't have a horse in this race, but: > * User asks proprietary web service to fetch content and render it a specific way, which it does That sounds like Google Translate to me, when pasting a URL. Bonus points if instead of pasting a URL directly, it is submitted to one of the Internet Archive-like sites; and then submit that archive URL to Google Translate. That would be download and adaptation (by Google Translate) of the download and adaptation[1] (by Internet Archive) of the original content. [1]: These archive sites usually present the content in a slightly different way. Granted, it's usually just adding stuff around the page, e.g. to let you move around different snapshots, but that's still showing stuff that was originally not there.
daft_pink 2y ago
The other question is once the user directs the ai to read the website instead of crawling will the site then be fair game for training?
zzo38computer 2y ago
If the user specifically asks for a file and asks a computer program to process it in a specific way, it should be permitted, regardless of user-agent spoofing (although user-agent spoofing should (normally) ideally only be done when the user specifically requests it; it should not do so automatically). However, this is better when using FOSS and/or local programs (or if the user is accessing them through a proxy, VPN, Tor, etc). Furthermore, any company that provides such services should not use unethical business practices, false advertising, etc, to do so. If the company wants a copy of the files for your own use, then that is a bit different. When accessing large number of files at once, robots.txt is useful to block it. If they can get a copy of the files in a different way (assuming the files are intended to be public anyways), then they might do so. However, even in this case, still they should not use unethical business practices, false advertising, etc; and, they should also avoid user-agent spoofing. (In this case, the reason for the user-agent spoofing does not seem to be deliberate, since it uses a headless browser. They should still change it though; probably by keeping the user-agent string but adding on a extra part such as "Perplexity", to indicate that it is what it is, in addition to the headless browser.)
- EGreg 2y ago
  A user-agent requests the file using your credentials, eg a cookie or public key signature. It is transforming the content for you, an authorized party. That is not the same as then making derivative copies and distributing the information to others without paying. For example, if I bought a ticket to a show, taped it and then distributed it to everyone, disregarding that the show prohibited this. If I shared my Netflix password with up to 5 others, at least I can argue that they are part of my “family” or something. But to unlimited numbers of people? Why would they pay for netflix, and how would the shows get made? I am not necessarily endorsing government force enforcing copyright, which is why I have been building a solution to enforce it at the tech level: https://Qbix.com/ecosystem https://Qbix.com/ecosystem
  - zzo38computer 2y ago
    Well, I am opposed to copyright. If it is publicly available, then you can make a copy, and even a modified version (as long as you do not claim that it is the same as the original). However, what you say about credentials is still valid in the case of private data; this is why you should run the program locally and not use some other company's remote service for this use. (Well, it is one reason why. Other reason is all of the other bad stuff they do with the service.) It is also valid about credentials, even if it is published but requires a password to access using that service; but even then, if you would ignore copyright, you can just use a different copy of the same file (which you might make by yourself). None of this is meaning that you cannot pay for it, if they accept payment. It is also not meaning that whoever made it is required to give it away for free. What it is meaning, is that if you have a copy, you do not have to worry about copyright and other legal mess; you can just to do it; a license is not required. However, it is also another issue how much power big companies are wasting with your data, whether they are authorized to access it or not. This is potentially a reason to disallow some uses, but that is independent from copyright (which is bad, anyways).
immibis 2y ago
You can poison all your images with Glaze and Nightshade. Then you don't have to stop them from using them - they have to stop themselves from using them or their image generator will be useless. I don't know if there's a comparable system for text. If there was, it would probably be noticeable to humans.
reissbaker 2y ago
To follow onto this: If what Perplexity is doing is illegal, is it illegal to run an open-source LLM on your own machine, and have it do the same thing? If so, how are ad blockers or Reader Modes or screen readers legal? And if it's legal to run an open-source LLM on your own machine, is it legal to run an open-source LLM on a rented server (e.g. because you need more GPUs)? And if that's legal, why is it illegal to run a closed-source LLM on servers? Could Perplexity simply release the model weights and keep doing what they're doing?
EGreg 2y ago
Let’s differentiate between: 1) a user-agent which makes an authenticated and authorized request for data, and delivers to the user 2) a user who then turns around and distributes the data or its derivatives to users in an unauthorized manner A “dumber” example would be whether I can indefinitely cache and index most of information via the Google Places API, as long as my users request each item at least once. Can I duplicate all that map or streetview photo information that google paid cars to go around and photograph? Or how about the info that Google users entered as user-generated content? THE REQUIREMENT TO OPEN SOURCE WEIGHTS Legally, if I had a Creative Commons Share-Alike license on my data, and the LLM was trained on it and then served unlimited requests to others, without making the weights available… …that would be almost exactly like if I had made my code available with Affero GPL license, someone would take my code but then incorporated it into a backend software hosting a social network or something, without making their own entire social network source code available. Technically this should be enforceable via a court order compelling the open sourcing to the public. (Alternatively, they’d have to pay damages in a class action lawsuit and stop using the tainted backend software or weights when serving all those people.) TECHNICAL ANALYSIS The key, as many here have missed, is authentication and authorization. You may have authorization to log in and view movies on Netflix. Not to rebroadcast them. Even the question of a VCR for personal use was debated in the past. Distributing your scripts and software to process data is not the same as distributing arbitrary data the user agent found on the internet for which you don’t have a license. If someone wrote an article, your reader transforms it based on your authenticated request, and your user would have an authorized subscription. LEGAL ANALYSIS Much of the content published on the Web isn’t secured with subscriptions and micropayments, which is why the whole thing becomes a legal battle as silly as “exceeding authorized access” which landed someone like Aaron Swartz in jail. In other words, it is the question of “piracy”, which has acquired a new character only in that the AI is trained on your data and transforms it before it republishes it. There was also a lawsuit aboot scraping LinkedIn, which was settled as follows: https://natlawreview.com/article/hiq-and-linkedin-reach-proposed-settlement-landmark-scraping-case https://natlawreview.com/article/hiq-and-linkedin-reach-prop... Legally, you can grant access to people subject to a certain license (eg Creative Commons Share Alike) and then any derived content must have its weights opened. Similar to, say, Affero GPL license for derivative software.
deleted 2y ago
[deleted]

neycoda 2y ago

AI scraping against permission could allow corporations to formulate a loophole where Congress argues that it's impossible to enforce a law against and that it's easier to just make laws to allow corporations to close-source their websites (yes, HTML, CSS, and JavaScript, etc). I think what's most likely to happen is nothing will fundamentally change, and browsers will continue showing page source, and AI will continue scraping source content without permission.

deleted 2y ago

[deleted]

gpm 2y ago

> The second concern, though, is can perplexity do a live web query to my website and present data from my website in a format that the user asks for? Arguing that we should ban this moves into very dangerous territory. This feels like the fundamental core component of what copyright allows you to forbid. > Everything from ad blockers to reader mode to screen readers do exactly the same thing that Perplexity is doing here, with the only difference being that they tend to be exclusively local Which is a huge difference. The latter is someone asking for a copy of my content (from someone with a valid license, myself), and manipulating it to display it (not creating new copies, broadly speaking allowed by copyright). The former adds in the criminal step of "and redistributing (modified, but that doesn't matter) versions of it to users without permission". I mean, I'm all for getting rid of copyright, but I also know that's an incredibly unpopular position to take, and I don't see how this isn't just copyright infringement if you aren't advocating for repealing copyright law all together.

lolinder 2y ago
I'm curious to know where you draw the line for what constitutes legitimate manipulation by a person and when it becomes distribution. I'm assuming that if I write code by hand for every part of the TCP/IP and HTTP stack I'm safe. What if I use libraries written by other people for the TCP/IP and HTTP part? What if I use a whole FOSS web browser? What about a paid local web browser? What if I run a script that I wrote on a cloud server? What if I then allow other people to download and use that script on their own cloud servers? What if I decide to offer that script as a service for free to friends and family, who can use my cloud server? What if I offer it for free to the general public? What if I start accepting money for that service, but I guarantee that only the one person who asked for the site sees the output? Can you help me to understand where exactly I crossed the line?
- gpm 2y ago
  Obviously not legal advice and I doubt it's entirely settled law, but probably this step > What if I decide to offer that script as a service for free to friends and family, who can use my cloud server? You're allowed to make copies and adaptations in order to utilize the program (website), which probably covers a cloud server you yourself are controlling. You aren't allowed to do other things with those copies though, like distribute them to other people. Payment only matters if we're getting into "free use" arguments, and I don't think any really apply here. I think you're probably already in trouble with just offering it to family and friends, but if you take the next step offering it to the public that adds more issues because the copyright act includes definitions like "To perform or display a work “publicly” means (1) to perform or display it at a place open to the public or at any place where a substantial number of persons outside of a normal circle of a family and its social acquaintances is gathered; or (2) to transmit or otherwise communicate a performance or display of the work to a place specified by clause (1) or to the public, by means of any device or process, whether the members of the public capable of receiving the performance or display receive it in the same place or in separate places and at the same time or at different times."
  - lolinder 2y ago

putlake 2y ago

It's funny I posted the inverse of this. As a web publisher, I am fine with folks using my content to train their models because this training does not directly steal any traffic. It's the "train an AI by reading all the books in the world" analogy. But what Perplexity is doing when they crawl my content in response to a user question is that they are decreasing the probability that this user would come to by content (via Google, for example). This is unacceptable. A tool that runs on-device (like Reader mode) is different because Perplexity is an aggregator service that will continue to solidify its position as a demand aggregator and I will never be able to get people directly on my content. There are many benefits to having people visit your content on a property that you own. e.g., say you are a SaaS company and you have a bunch of Help docs. You can analyze traffic in this section of your website to get insights to improve your business: what are the top search queries from my users, this might indicate to me where they are struggling or what new features I could build. In a world where users ask Perplexity these Help questions about my SaaS, Perplexity may answer them and I would lose all the insights because I never get any traffic.

epolanski 2y ago
> they are decreasing the probability that this user would come to by content (via Google, for example). Google has been providing summaries of stuff and hijacking traffic for ages. I kid you not, in the tourism sector this has been a HUGE issue, we have seen 50%+ decrease in views when they started doing it. We paid gazzilions to write quality content for tourists about the most different places just so Google could put it on their homepage. It's just depressing. I'm more and more convinced that the age of regulations and competition is gone, US does want to have unkillable monopolies in the tech sector and we are all peons.
- itsoktocry 2y ago
  >We paid gazzilions to write quality content for tourists about the most different places just so Google could put it on their homepage. It's just depressing It's a legitimate complaint, and it sucks for your business. But I think this demonstrates that the sort of quality content you were producing doesn't actually have much value.

buro9 2y ago

The companies will scrape and internalise the "customer asked for this" requests... and slowly turn the latter into the former, or just their own tool as the scraper. No, easier to just ask a simple question: Does the company respect the access rules communicated via a web standard? No? In that case hard deny access to that company. These companies don't need to be given an inch.

lolinder 2y ago
> Does the company respect the access rules communicated via a web standard? No? In that case hard deny access to that company. So should Firefox not allow changing the user agent in order to bypass websites that erroneously claim to not work on Firefox?
- rezonant 2y ago
  Similarly, for sites which configure robots.txt to disallow all bots except Googlebot, I don't lose sleep about new search engines taking that with a grain of salt.
elicksaur 2y ago
This is exactly the concern and there’s a lot of comments just completely ignoring it or willfully conflating. Ad block isn’t the same problem because it doesn’t and can’t steal the creator’s data.
- lolinder 2y ago
  > Ad block isn’t the same problem because it doesn’t and can’t steal the creator’s data. Arguably it does. That topic has been debated endlessly and there are plenty of people on HN who are willing to fiercely argue that adblock is theft. I happen to agree with you that adblock doesn't steal data, but I'm also completely unsure why interacting with a tool over a network suddenly turns what would be acceptable on my local computer into theft.
- Dylan16807 2y ago
  If that's the concern, then ask for a line in the terms and conditions that explicitly says a user-initiated request will not be saved or used for training. Don't act like the access itself is an affront.

deleted 2y ago

[deleted]

baxtr 2y ago

What will happen if: Website owners decide to stop publishing because it’s not rewarded by a real human visit anymore? Then perplexity and the like won’t have new information to train their models on and no sites to answer the questions. I think there is a real content dilemma here at work. The incentives of Google and website owners were more or less aligned. This is not the case with perplexity.

lolinder 2y ago
What is a "visit"? TFA demonstrates that they got a hit on their site, that's how they got the logs. Is it necessary to load the JavaScript for it to count as a visit? What if I access the site with noscript? Or is it only a visit if I see all your recommended content? I usually block those recommendations so that I don't get distracted from the article I actually came to read—is my visit a less legitimate visit than other people's? What exactly is Perplexity doing here that isn't okay that people don't already do with their local user agents?
- alt227 2y ago
  > TFA demonstrates that they got a hit on their site Whats stopping perplexity caching this info say for 24 hours, and then redisplaying it to the next few hundred people who request it?
  - lolinder 2y ago
    Then they don't get the extra hits. So is that it—is a "visit" important because of the data that you're able to collect from the visit? Does this place HN's rampant use of archive.md on the same moral footing as Perplexity?
- denton-scratch 2y ago
  > What exactly is Perplexity doing here that isn't okay that people don't already do with their local user agents? It's in the title of TFA: they're being dishonest about who they are. PerplexityBot seems to understand that robots.txt is addressed to it. It's understood that site operators have a right to use the User-Agent to discriminate among visitors; that's why robots.txt is a standard. Crawlers that disrespect the standard have for many years been considered beyond the pale; thieves and snoopers. TFA's complaint is entirely justified.

lofaszvanitt 2y ago

Citing the source doesn't bring you, the owner of the site, valuable data. When was your data accessed, who accessed it, from where, at what time, what device, etc. It brings data to the LLM's owner, and you get N O T H I N G. Could you change the way printed news magazines showed their content? No. Then, why is that a problem? Btw nobody clicks on sources. NOBODY.

bluish29 2y ago
> Btw nobody clicks on sources. NOBODY. I always click on sources to verify what an LLM in this case says. I also hear the claim that a lot about people not reading sources (before LLM it was video content with references) but I always visited the sources. Is there a statistics or studies that actually support this claim? Or is it just a personal experience, of people (including me) enforcing it as generic behavior of all people?
- lofaszvanitt 2y ago
  That's you, because you are a researcher or coder or someone who uses his brain much more than average, hence not an average joe. I ran a news site for 15 years and the stats showed that from 10000 views on an article, only a miniscule amount of clicks were made on the source links. Average people do not care where the info is coming from. Also perplexity shows the videos on their site, you cannot go to youtube, you have to start it on their site, and then you have to click on the youtube player's logo in the lower right to get to the site. Perplexity is getting greedy.
  - SpaghettiCthulu 2y ago
    You said "NOBODY" (pretty sure the all caps means it's extra true).
    - lofaszvanitt 2y ago
      Well, that's the reality. The tech savvy people here are the exception, and represent only a very minor percentage of users.

insane_dreamer 2y ago

> I don't want to live in a world where website owners can use DRM to force me to display their website in exactly the way that their designers envisioned it. I'm okay with this world, as a tradeoff. I'm not sure users should have _the right_ to reformat others' content.

zzo38computer 2y ago
Users should have the right to reformat their own copy of others content (automatically as well as manually). However, if they then redistribute the reformatted copy, then they should not be allowed to claim that it is the same formatting as the original, because it is not the same as the original.

nradov 2y ago

Why should it be possible to stop an LLM from training itself on your data? If you want to restrict access to data then don't post it on a public website. It's easy enough to require registration and agreement to licensing terms for access. It seems like some website owners want to have their cake and eat it too. They want their content indexed by Google and other crawlers in order to drive search traffic but they don't want their content used to train AI models that benefit other companies. At some point they're going to have to make a choice.

marcus0x62 2y ago
Because if I run a server - at my own expense - I get to use information provided by the client to determine what, if any, response to provide? This isn’t a very difficult concept to grasp.
- nradov 2y ago
  I'm having difficulty grasping the concept. Only a fool would trust any HTTP headers such as User-Agent sent by a random unauthenticated client. Your expenses are your problem.
  - marcus0x62 2y ago
    … and I have absolutely no obligation to provide any particular response to any particular client. Parsing, rendering, and trusting that the payload is consistent from request to request is your problem. You can connect to my server, or not. I really don’t care. What you cannot do is dictate how my server responds to your request.
    - nradov 2y ago
      Sure. So just return an HTTP 4XX response to requests you don't like. What's the problem?
      - marcus0x62 2y ago

dheera 2y ago

Personally I think AI is a major win for accessibility and we should not be preventing people to access information in the way that is best suited for them. Accessibility can mean everything from a blind person wanting to interacting with a website using voice, to someone recovering from a surgery and wanting something to reduce unnecessary popups and clicks on a website to get to the information they need. Accessibility is in the eye of the accessor, and AI is what enables them to achieve it. The way I see it, AI is not a robot and doesn't need to look at robots.txt. Rather, AI is my low-cost secretary.

danlitt 2y ago
> The way I see it, AI is not a robot and doesn't need to look at robots.txt I don't think you are seeing it very clearly then. Your secretary can also be a robot. What do you think an AI is if not a robot?? It doesn't "need" to look at robots.txt because nothing does.

gcanyon 2y ago

It seems self-evident to me that if a user tells a bot to go get a web page, robots.txt doesn't apply, and the bot shouldn't respect it. I understand others' concerns that, like Apple's reader, and other similar tools, it's ethically debatable whether a site should be required to comply with the request, and spoofing an agent seems in dubious territory. I don't think a good answer has been proposed for this challenge, unfortunately.

lolinder 2y ago
> spoofing an agent seems in dubious territory. Just to clarify, Perplexity is not spoofing a user agent, they're legitimately using a headless Chrome to fetch the page. The author just misunderstood their docs [0]: when they say that "you can identify our web crawler by its user agent", they're talking about the crawler, not the browser they use for ad hoc requests. As you note, crawling is different. [0] https://docs.perplexity.ai/docs/perplexitybot https://docs.perplexity.ai/docs/perplexitybot
- mattigames 2y ago
  This is completely false, the user agent being used by Perplexity its _not_ the headless-chrome user agent, wich is close similar to this (emphasis on HeadlessChrome): Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/119.0.0.0 Safari/537.36 They are spoofing it to pretend to be a desktop Chrome one: Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36
  - lolinder 2y ago
    Ah, you're correct, my bad. I don't personally have a problem with spoofing user agents, but yeah, they're either spoofing or for some reason they're truly using a non-headless Chrome.
  - jcrawfordor 2y ago
    There's a difference here between "headless chrome" as a concept and "headless-chrome" the software. It's still pretty common to run browser automation with a full "headful" browser, in which case you would just get the normal user agent. headless-chrome is sort of an optimized option that comes with some downsides.

cal85 2y ago

> can I stop an LLM from training itself on my data? This should be possible and Perplexity should absolutely make it easy to block them from training. I’m not saying you’re wrong, but why? And what do you mean by “your data” here?

__loam 2y ago
By "my data" he means the data a site spent time and money to create and publish.
bhelkey 2y ago
> And what do you mean by “your data” here? The website that they created.

treyd 2y ago

Yeah if people get to extensive about blocking then we're going to end up with a scenario where the web request functionality is implementing by telling the chatbot's users's browser to make the fetch and submit it back to the server for processing, making it largely indistinguishable from the user making the query themselves. If CORS gets in the way they can just prompt users to install a browser extension to use the web request functionality.

__loam 2y ago

The problem that Perplexity has that ad blockers don't is that they're an independent site that is publishing content based on work they didn't produce. That runs afoul of both copyright laws and section 230 which let's sites like Google and Facebook operate. That's pretty different from an ad blocker running on your local machine. The ad blocker isn't publishing the page it edited for you.

lolinder 2y ago
> they're an independent site that is publishing content based on work they didn't produce. What distinguishes these two situations? * User asks proprietary web browser to fetch content and render it a specific way, which it does * User asks proprietary web service to fetch content and render it a specific way, which it does The technical distinction is that there's a network involved in the second scenario. What is the moral distinction? Why is it that a proprietary web service manipulating content on behalf of a user is "publishing" content illegally, while a proprietary web browser doing the exact same kind of transformations is not? Assume that in both cases the proprietary software fetches the data upon request, does not cache it, and does not make the transformed content available to other users.
- quectophoton 2y ago
  I don't have a horse in this race, but: > * User asks proprietary web service to fetch content and render it a specific way, which it does That sounds like Google Translate to me, when pasting a URL. Bonus points if instead of pasting a URL directly, it is submitted to one of the Internet Archive-like sites; and then submit that archive URL to Google Translate. That would be download and adaptation (by Google Translate) of the download and adaptation[1] (by Internet Archive) of the original content. [1]: These archive sites usually present the content in a slightly different way. Granted, it's usually just adding stuff around the page, e.g. to let you move around different snapshots, but that's still showing stuff that was originally not there.

daft_pink 2y ago

The other question is once the user directs the ai to read the website instead of crawling will the site then be fair game for training?

zzo38computer 2y ago

If the user specifically asks for a file and asks a computer program to process it in a specific way, it should be permitted, regardless of user-agent spoofing (although user-agent spoofing should (normally) ideally only be done when the user specifically requests it; it should not do so automatically). However, this is better when using FOSS and/or local programs (or if the user is accessing them through a proxy, VPN, Tor, etc). Furthermore, any company that provides such services should not use unethical business practices, false advertising, etc, to do so. If the company wants a copy of the files for your own use, then that is a bit different. When accessing large number of files at once, robots.txt is useful to block it. If they can get a copy of the files in a different way (assuming the files are intended to be public anyways), then they might do so. However, even in this case, still they should not use unethical business practices, false advertising, etc; and, they should also avoid user-agent spoofing. (In this case, the reason for the user-agent spoofing does not seem to be deliberate, since it uses a headless browser. They should still change it though; probably by keeping the user-agent string but adding on a extra part such as "Perplexity", to indicate that it is what it is, in addition to the headless browser.)

EGreg 2y ago
A user-agent requests the file using your credentials, eg a cookie or public key signature. It is transforming the content for you, an authorized party. That is not the same as then making derivative copies and distributing the information to others without paying. For example, if I bought a ticket to a show, taped it and then distributed it to everyone, disregarding that the show prohibited this. If I shared my Netflix password with up to 5 others, at least I can argue that they are part of my “family” or something. But to unlimited numbers of people? Why would they pay for netflix, and how would the shows get made? I am not necessarily endorsing government force enforcing copyright, which is why I have been building a solution to enforce it at the tech level: https://Qbix.com/ecosystem https://Qbix.com/ecosystem
- zzo38computer 2y ago
  Well, I am opposed to copyright. If it is publicly available, then you can make a copy, and even a modified version (as long as you do not claim that it is the same as the original). However, what you say about credentials is still valid in the case of private data; this is why you should run the program locally and not use some other company's remote service for this use. (Well, it is one reason why. Other reason is all of the other bad stuff they do with the service.) It is also valid about credentials, even if it is published but requires a password to access using that service; but even then, if you would ignore copyright, you can just use a different copy of the same file (which you might make by yourself). None of this is meaning that you cannot pay for it, if they accept payment. It is also not meaning that whoever made it is required to give it away for free. What it is meaning, is that if you have a copy, you do not have to worry about copyright and other legal mess; you can just to do it; a license is not required. However, it is also another issue how much power big companies are wasting with your data, whether they are authorized to access it or not. This is potentially a reason to disallow some uses, but that is independent from copyright (which is bad, anyways).

immibis 2y ago

You can poison all your images with Glaze and Nightshade. Then you don't have to stop them from using them - they have to stop themselves from using them or their image generator will be useless. I don't know if there's a comparable system for text. If there was, it would probably be noticeable to humans.

reissbaker 2y ago

To follow onto this: If what Perplexity is doing is illegal, is it illegal to run an open-source LLM on your own machine, and have it do the same thing? If so, how are ad blockers or Reader Modes or screen readers legal? And if it's legal to run an open-source LLM on your own machine, is it legal to run an open-source LLM on a rented server (e.g. because you need more GPUs)? And if that's legal, why is it illegal to run a closed-source LLM on servers? Could Perplexity simply release the model weights and keep doing what they're doing?

EGreg 2y ago

Let’s differentiate between: 1) a user-agent which makes an authenticated and authorized request for data, and delivers to the user 2) a user who then turns around and distributes the data or its derivatives to users in an unauthorized manner A “dumber” example would be whether I can indefinitely cache and index most of information via the Google Places API, as long as my users request each item at least once. Can I duplicate all that map or streetview photo information that google paid cars to go around and photograph? Or how about the info that Google users entered as user-generated content? THE REQUIREMENT TO OPEN SOURCE WEIGHTS Legally, if I had a Creative Commons Share-Alike license on my data, and the LLM was trained on it and then served unlimited requests to others, without making the weights available… …that would be almost exactly like if I had made my code available with Affero GPL license, someone would take my code but then incorporated it into a backend software hosting a social network or something, without making their own entire social network source code available. Technically this should be enforceable via a court order compelling the open sourcing to the public. (Alternatively, they’d have to pay damages in a class action lawsuit and stop using the tainted backend software or weights when serving all those people.) TECHNICAL ANALYSIS The key, as many here have missed, is authentication and authorization. You may have authorization to log in and view movies on Netflix. Not to rebroadcast them. Even the question of a VCR for personal use was debated in the past. Distributing your scripts and software to process data is not the same as distributing arbitrary data the user agent found on the internet for which you don’t have a license. If someone wrote an article, your reader transforms it based on your authenticated request, and your user would have an authorized subscription. LEGAL ANALYSIS Much of the content published on the Web isn’t secured with subscriptions and micropayments, which is why the whole thing becomes a legal battle as silly as “exceeding authorized access” which landed someone like Aaron Swartz in jail. In other words, it is the question of “piracy”, which has acquired a new character only in that the AI is trained on your data and transforms it before it republishes it. There was also a lawsuit aboot scraping LinkedIn, which was settled as follows: https://natlawreview.com/article/hiq-and-linkedin-reach-proposed-settlement-landmark-scraping-case https://natlawreview.com/article/hiq-and-linkedin-reach-prop... Legally, you can grant access to people subject to a certain license (eg Creative Commons Share Alike) and then any derived content must have its weights opened. Similar to, say, Affero GPL license for derivative software.

deleted 2y ago

[deleted]