13 ms·
I find this problem quite difficult to solve: 1. If I as a human request a website, then I should be shown the content. Everyone agrees. 2. If I as the human
by fxtentacle 1y ago
I find this problem quite difficult to solve:
1. If I as a human request a website, then I should be shown the content. Everyone agrees.
2. If I as the human request the software on my computer to modify the content before displaying it, for example by installing an ad-blocker into my user agent, then that's my choice and the website should not be notified about it. Most users agree, some websites try to nag you into modifying the software you run locally.
3. If I now go one step further and use an LLM to summarize content because the authentic presentation is so riddled with ads, JavaScript, and pop-ups, that the content becomes borderline unusable, then why would the LLM accessing the website on my behalf be in a different legal category as my Firefox web browser accessing the website on my behalf?
- account42 1y agoIt's quite easy to solve. Hold companies legally accountable for computer fraud and abuse. The problem is that those in the position to do that are not interested.
- Beijinger 1y agoHow about I open a proxy, replace all ads with my ads, redirect the content to you and we share the ad revenue?
- fxtentacle 1y agoThat's somewhat antisocial, but perfectly legal in the US. It's called PayPal Honey, for example, and has been running for 13 years now.
- rustc 1y agoSince when does PayPal Honey replace ads on websites? > PayPal Honey is a browser extension that automatically finds and applies coupon codes at checkout with a single click.
- galaxy_gas 1y agoThey overwrite ad attributions, affiliate links, clickthrough attributions with their own.
- carlosjobim 1y agoThat's the Brave browser.
- zeta0134 1y agoIf the LLM were running this sort of thing at the user's explicit request this would be fine. The problem is training. Every AI startup on the planet right now is aggressively crawling everything that will let them crawl. The server isn't seeing occasional summaries from interested users, but thousands upon thousands of bots repeatedly requesting every link they can find as fast as they can.
- mnmalst 1y agoBut that's not what this article is about. From, what I understand, this articles is about a user requesting information about a specific domain and not general scraping.
- fxtentacle 1y agoThen what if I ask the LLM 10 questions about the same domain and ask it to research further? Any human would then click through 50 - 100 articles to make sure they know what that domain contains. If that part is automated by using an LLM, does that make any legal change? How many page URLs do you think one should be allowed to access per LLM prompt?
- zeta0134 1y agoAll of them. That's at the explicit request of the user. I'm not sure where the downvotes are coming from, since I agree with all of these points. The training thing has merely pissed off lots of server operators already, so they quite reasonably tend to block first and ask questions later. I think that's important context.
- hombre_fatal 1y agoTFA isn’t talking about crawling to harvest training data. It’s talking about Perplexity crawling sites on demand in response to user queries and then complaining that no it’s not fine, hence this thread.
- cjonas 1y agoDoesn't perplexity crawl to harvest and index data like a traditional search engine? Or is it all "on demand"?
- bbqfog 1y agoCorrect, it’s user hostile to dictate which software is allowed to see content.
- klabb3 1y agoThey all do it. Facebook, Reddit, Twitter, Instagram. Because it interferes with their business model. It was already bad, but now the conflict between business and the open web is reaching unprecedented levels, especially since the copyright was scrapped for AI companies.
- Workaccount2 1y ago>2. If I as the human request the software on my computer to modify the content before displaying it, for example by installing an ad-blocker into my user agent, then that's my choice and the website should not be notified about it. Most users agree, some websites try to nag you into modifying the software you run locally. If I put time and effort into a website and it's content, I should expect no compensation despite bearing all costs. Is that something everyone would agree with? The internet should be entirely behind paywalls, besides content that is already provided ad free. Is that something everyone would agree with? I think the problem you need to be thinking about is "How can the internet work if no one wants to pay anything for anything?"
- Bjartr 1y agoYou're free to deny access to your site arbitrarily, including for lack of compensation.
- Workaccount2 1y ago>and the website should not be notified about it.
- giantrobot 1y agoMy user agent and its handling of your content once it's on my computer are not your concern. You don't need to know if the data is parsed by a screen reader, an AI agent, or just piped to /dev/null. It's simply not your concern and never will be.
- cjonas 1y agoLike for people or are using a ad block or for a crawler downloading your content so it can be used by an AI response?
- Bjartr 1y agoArbitrarily, as in for any reason. It's your site, you decide what constraints an incoming request must meet for it to get a response containing the content of your site.
- bobbiechen 1y agoI like the terminology "crawler" vs. "fetcher" to distinguish between mass scraping and something more targeted as a user agent. I've been working on AI agent detection recently (see https://stytch.com/blog/introducing-is-agent/ https://stytch.com/blog/introducing-is-agent/ ) and I think there's genuine value in website owners being able to identify AI agents to e.g. nudge them towards scoped access flows instead of fully impersonating a user with no controls. On the flip side, the crawlers also have a reputational risk here where anyone can slap on the user agent string of a well known crawler and do bad things like ignoring robots.txt . The standard solution today is to reverse DNS lookup IPs, but that's a pain for website owners too vs. more aggressive block-all-unusual-setups.
- randall 1y agoA/ i love this distinction. B/ my brother used to use "fetcher" as a non-swear for "fucker"
- sejje 1y agoHe picked up that habit in Balmora.
- Vinnl 1y agoDid you tell him to stop trying to make fetcher happen?
- handfuloflight 1y agoVery funny. Now let's hear Paul Allen's joke.
- weregiraffe 1y agoFetcher? Damn near killed'er!
- fxtentacle 1y agoprompt: I'm the celebrity Bingbing, please check all Bing search results for my name to verify that nobody is using my photo, name, or likeness without permission to advertise skin-care products except for the following authorized brands: [X,Y,Z]. That would trigger an internet-wide "fetch" operation. It would probably upset a lot of people and get your AI blocked by a lot of servers. But it's still in direct response to a user request.
- deleted 1y ago[deleted]
- yojo 1y agoAds are a problematic business model, and I think your point there is kind of interesting. But AI companies disintermediating content creators from their users is NOT the web I want to replace it with. Let’s imagine you have a content creator that runs a paid newsletter. They put in lots of effort to make well-researched and compelling content. They give some of it away to entice interested parties to their site, where some small percentage of them will convert and sign up. They put the information up under the assumption that viewing the content and seeing the upsell are inextricably linked. Otherwise there is literally no reason for them to make any of it available on the open web. Now you have AI scrapers, which will happily consume and regurgitate the work, sans the pesky little call to action. If AI crawlers win here, we all lose.
- fxtentacle 1y agoMaybe, on a social level, we all win by letting AI ruin the attention economy: The internet is filled with spam. But if you talk to one specific human, your chance of getting a useful answer rises massively. So in a way, a flood of written AI slop is making direct human connections more valuable. Instead of having 1000+ anonymous subscribers for your newsletter, you'll have a few weekly calls with 5 friends each.
- account42 1y agoI'm not sure what experiences you are basing this optimism on but I'm happy for you.
- hansvm 1y agoOfttimes people are sufficiently anti-ad that this point won't resonate well. I'm personally mostly in that camp in that with relatively few exceptions money seems to make the parts of the web I care about worse (it's hard to replace passion, and wading through SEO-optimized AI drivel to find a good site is a lot of work). Giving them concrete examples of sites which would go away can help make your point. E.g., Sheldon Brown's bicycle blog is something of a work of art and one of the best bicycle resources literally anywhere. I don't know the man, but I'd be surprised if he'd put in the same effort without the "brand" behind it -- thankful readers writing in, somebody occasionally using the donate button to buy him a coffee, people like me talking about it here, etc.
- johnfn 1y agoUnless I am misunderstanding you, you are talking about something different than the article. The article is talking about web-crawling. You are talking about local / personal LLM usage. No one has any problems with local / personal LLM usage. It's when Perplexity uses web crawlers that an issue arises.
- lukeschlather 1y agoYou probably need a computer that costs $250,000 or more to run the kind of LLM that Perplexity uses, but with batching it costs pennies to have the same LLM fetch a page for you, summarize the content, and tell you what is on it. And the power usage similarly, running the LLM for a single user will cost you a huge amount of money relative to the power it takes in a cloud environment with many users. Perplexity's "web crawler" is mostly operating like this on behalf of users, so they don't need a massively expensive computer to run an LLM.
- sterlind 1y agodoes Perplexity store crawled pages for training?
- st3fan 1y agoIs the article really talking about crawling? Because in one of their screenshots where they ask information about the "honeypot" website you can see that the model requested pages from the website. But that is most definitely "fetching by proxy because I asked a question about the website" and not random crawling. It is confusing.
- johnfn 1y agoYea, now I feel like my comment might be misleading. The title mentions crawling; the article itself is talking about something else.
- sbarre 1y agoAll of these scenarios assume you have an unconditional right to access the content on a website in whatever way you want. Do you think you do? Or is there a balance between the owner's rights, who bears the content production and hosting/serving costs, and the rights of the end user who wishes to benefit from that content? If you say that you have the right, and that right should be legally protected, to do whatever you want on your computer, should the content owner not also have a legally protected right to control how, and by who, and in what manner, their content gets accessed? That's how it currently works in the physical world. It doesn't work like that in the digital world due to technical limitations (which is a different topic, and for the record I am fine with those technical limitations as they protect other more important rights). And since the content owner is, by definition, the owner of the content in question, it feels like their rights take precedence. If you don't agree with their offering (i.e. their terms of service), then as an end user you don't engage, and you don't access the content. It really can be that simple. It's only "difficult to solve" if you don't believe a content owner's rights are as valid as your own.
- cutemonster 1y agoIf there's an article you want to read, and the ToS says that in between reading each paragraph, you must switch to their YouTube channel and look at their ads about cat food for 5 minutes, are your going to do that?
- JimDabell 1y agoHacker News has collectively answered this question by consistently voting up the archive.is links in the comments of every paywalled article posted here.
- giantrobot 1y agoNew sites have collectively decided to require people use those services because they can't fathom not enshittifying everything until it's an unusable transaction hellscape. I never really minded magazine ads or even television ads. They might have tried to make me associate boobs with a brand of soda but they didn't data mine my life and track me everywhere. I'd much rather have old fashioned manipulation than pervasive and dangerous surveillance capitalism.
- jasonjmcghee 1y agoI think it's an issue of scale. The next step in your progression here might be: If / when people have personal research bots that go and look for answers across a number of sites, requesting many pages much faster than humans do - what's the tipping point? Is personal web crawling ok? What if it gets a bit smarter and tried to anticipate what you'll ask and does a bunch of crawling to gather information regularly to try to stay up to date on things (from your machine)? Or is it when you tip the scale further and do general / mass crawling for many users to consume that it becomes a problem?
- cj 1y agoDoesn't o3 sort of already do this? Whenever I ask it something, it makes it look like it simultaneously opens 3-8 pages (something a human can't do). Seems like a reasonable stance would be something like "Following the no crawl directive is especially necessary when navigating websites faster than humans can." > What if it gets a bit smarter and tried to anticipate what you'll ask and does a bunch of crawling to gather information regularly to try to stay up to date on things (from your machine)? To be fair, Google Chrome already (somewhat) does this by preloading links it thinks you might click, before you click it. But your point is still valid. We tolerate it because as website owners, we want our sites to load fast for users. But if we're just serving pages to robots and the data is repackaged to users without citing the original source, then yea... let's rethink that.
- Spivak 1y agoYou don't middle click a bunch of links when doing research? Of all the things to point to I wouldn't have thought "opens a bunch of tabs" to be one of the differentiating behaviors between browsing with Firefox and browsing with an LLM.
- daviddanielng 1y ago> simultaneously opens 3-8 pages (something a human can't do). Can't you read?
- 1y ago
- Spacecosmonaut 1y agoRegarding point 3: The problem from the perspective of websites would not be any different if they had been completely ad free. People would still consume LLM generated summaries because they cut down clicks and eyeballing to present you information that directly pertains to the promt. The whole concept of a "website" will simply become niche. How many zoomers still visit any but the most popular websites?
- ai-christianson 1y agoWebsites should be able to request payment. Who cares if it is a human or an agent of a human if it is paying for the request?
- carlosjobim 1y agoThey are able to request payment.
- sds357 1y agoWhat if the agent is reselling the request?
- adriand 1y agoCloudflare launched a mechanism for this: https://blog.cloudflare.com/introducing-pay-per-crawl/ https://blog.cloudflare.com/introducing-pay-per-crawl/
- pyrale 1y agoBecause LLM companies have historically been extremely disingenuous when it comes to crawling these sites. Also because there is a difference between a user hitting f5 a couple times and a crawler doing a couple hundred requests. Also because ultimately, by intermediating the request, llm companies rob website owners of a business model. A newspaper may be fine letting adblockers see their article, in hopes that they may eventually subscribe. When a LLM crawls the info and displays it with much less visibility for the source, that hope may not hold.
- fluidcruft 1y agoIn theory, couldn't the LLM access the content on your browser and it's cache, rather than interacting with the website directly? Browser automation directly related to user activity (prefetch etc) seems qualitatively different to me. Similarly, refusing to download content or modifying content after it's already in my browser is also qualitatively different. That all seems fair-use-y. I'm not sure there's a technical solution beyond the typical cat/mouse wars... but there is a smell when a datacenter pretends to be a person. That's not a browser. It could be a personal knowledge management system, but it seems like knowledge management systems should be operating off of things you already have. The research library down the street isn't considered a "personal knowledge management system" in any sense of the term if you know what I mean. If you dispatch an army of minions to take notes on the library's contents that doesn't seem personal. Similarly if you dispatch the army of minions to a bookstore rather than a library. At the very least bring the item into your house/office first. (Libraries are a little different because they are designed for studying and taking notes, it's use of an army of minions aspect)
- freehorse 1y ago> couldn't the LLM access the content on your browser Yes, orbit, a now deprecated firefox extension by mozilla was doing that. This way you could also use it to summarise content that would not be available to a third party (eg sth in google docs). You can still sort of do the same with the ai chatbot panel in firefox, sort of, but ctrl+A>right click>AI chatbot>summarise.
- beardyw 1y agoYou speak as 1% of the population to 1% of the population. Don't fool yourself.
- porridgeraisin 1y agoI don't think people have a problem with an LLM issuing GET website.com and then summarising that, each and every time it uses that information (or atleast, save a citation to it and refer to that citation). Except ad ecosystem, ignoring them for now, please refer to last paragraph. The problem is with the LLM then training on that data _once_ and then storing it forever and regurgitating it N times in the future without ever crediting the original author. So far, humans themselves did this, but only for relatively simple information (ratio of rice and water in specific $recipe). You're not gonna send a link to your friend just to see the ratio, you probably remember it off the top of your head. Unfortunately, the top of an LLMs head is pretty big, and they are fitting almost the entire website's content in there for most websites. The threshold beyond which it becomes irreproducible for human consumers, and therefore, copyrightable (lot of copyright law has "reasonable" term which refers to this same concept) has now shifted up many many times higher. Now, IMO: So far, for stuff that won't fit in someone's head, people were using citations (academia, for example). LLMs should also use citations. That solves the ethical problem pretty much. That the ad ecosystem chose views as the monetisation point and is thus hurt by this is not anyone else's problem. The ad ecosystem can innovate and adjust to the new reality in their own time and with their own effort. I promise most people won't be waiting. Maybe google can charge per LLM citation. Cost Per Citation, you even maintain the acronym :)
- skydhash 1y agoThat’s why websites have no issues with googlebot and the search results. It’s a giant index and citation list. But stripping works from its context and presenting as your own is decried throughout history.
- wulfstan 1y agoYes, this is the crux of the matter. The "social contract" that has been established over the last 25+ years is that site owners don't mind their site being crawled reasonably provided that the indexing that results from it links back to their content. So when AltaVista/Yahoo/Google do it and then score and list your website, interspersing that with a few ads, then it's a sensible quid pro quo for everyone. LLM AI outfits are abusing this social contract by stuffing the crawled data into their models, summarising/remixing/learning from this content, claiming "fair use" and then not providing the quid pro quo back to the originating data. This is quite likely terminal for many content-oriented businesses, which ironically means it will also be terminal for those who will ultimately depend on additions, changes and corrections to that content - LLM AI outfits. IMO: copyright law needs an update to mandate no training on content without explicit permission from the holder of the copyright of that content. And perhaps, as others have pointed out, an llms.txt to augment robots.txt that covers this for llm digestion purposes. EDIT: Apparently llms.txt has been suggested, but from what I can tell this isn't about restricting access: https://llmstxt.org/ https://llmstxt.org/
- itsdesmond 1y agoSome stores do not welcome Instacart or Postmates shoppers. You can shop there. You can shop with your phone out, scanning every item to price match, something that some bookstores frown on, for example. Third party services cannot send employees to index their inventory, nor can they be dispatched to pick up an item you order online. Their reasons vary. Some don’t want their businesses perception of quality to be taken out of their control (delivering cold food, marking up items, poor substitutions). Some would prefer their staff service and build relationships with customers directly, instead of disinterested and frequently quite demanding runners. Some just straight up disagree with the practice of third party delivery. I think that it’s pretty unambiguously reasonable to choose to not allow an unrelated business to operate inside of your physical storefront. I also think that maps onto digital services.
- rjbwork 1y agoBut I can send my personal shopper and you'll be none the wiser.
- deleted 1y ago[deleted]
- bradleyjg 1y agoIt’s possible to violate all sorts of social norms. Societies that celebrate people that do so are on the far opposite end of the spectrum from high trust ones. They are rather unpleasant.
- ToucanLoucan 1y agoJust the Silicon Valley ethos extended to it's logical conclusions. These companies take advantage of public space, utilities and goodwill at industrial scale to "move fast and break things" and then everyone else has to deal with the ensuing consequences. Like how cities are awash in those fucking electric scooters now. Mind you I'm not saying electric scooters are a bad idea, I have one and I quite enjoy it. I'm saying we didn't need five fucking startups all competing to provide them at the lowest cost possible just for 2/3s of them to end up in fucking landfills when the VC funding ran out.
- GardenLetter27 1y agoAnd isn't the obvious solution to just make some sort of browsers add-on for the LLM summary so the request comes from your browser and then gets sent to the LLM? I think the main concern here is the huge amount of traffic from crawling just for content for pre-training.
- otterley 1y agoWhy would a personal browser have to crawl fewer pages than the agent’s mechanism? If anything, the agent would be more efficient because it could cache the content for others to use. In the situation we’re talking about, the AI engine is behaving essentially like a caching proxy—just like a CDN.
- shadowgovt 1y agoNot only is it difficult to solve, it's the next step in the process of harvesting content to train AIs: companies will pay humans (probably in some flavor of "company scrip," such as extra queries on their AI engine) to install a browser extension that will piggy-back on their human access to sites and scrape the data from their human-controlled client. At the limit, this problem is the problem of "keeping secrets while not keeping secrets" and is unsolvable. If you've shared your site content to one entity you cannot control, you cannot control where your site content goes from there (technologically; the law is a different question).
- quectophoton 1y ago> companies will pay humans (probably in some flavor of "company scrip," such as extra queries on their AI engine) to install a browser extension that will piggy-back on their human access to sites and scrape the data from their human-controlled client. Proprietary web browsers are in a really good position to do something like this, especially if they offer a free VPN. The browser would connect to the "VPN servers", but it would be just to signal that this browser instance has an internet connection, while the requests are just proxied through another browser user. That way the company that owns this browser gets a free network of residential IP address ready to make requests (in background) using a real web browser instance. If one of those background requests requires a CAPTCHA, they can just show it to the real user, e.g. the real user visits a Google page and they see a Cloudflare CAPTCHA, but that CAPTCHA is actually from one of the background requests (while lying in its UI and still showing the user a Google URL in the address bar).
- microtonal 1y agoThere are also a gazillion pages that are not ad-riddled content. With search engines, the implicit contract was that they could crawl pages because they would drive traffic to the websites that are crawled. AI crawlers for non-open models void the implicit contract. First they crawl the data to build a model that can do QA. Proprietary LLM companies earn billions with knowledge that was crawled from websites and websites don't get anything in return. Fetching for user requests (to feed to an LLM) is kind of similar - the LLM provider makes a large profit and the author that actually put in time to create the content does not even get a visit anymore. Besides that, if Perplexity is fine with evading robots.txt and blocks for user requests, how can one expect them not to use the fetched pages to train/finetine LLMs (as a side channel when people block crawling for training).
- Tuna-Fish 1y agoI would not mind 3, so long as it's just the LLM processing the website inside its context window, and no information from the website ends up in the weights of the model.
- talos_ 1y agoThis analogy doesn't map to the actual problem here. Perplexity is not visiting a website everytime a user asks about it. It's frequently crawling and indexing the web, thus redirecting traffic away from websites. This crawling reduces costs and improves latency for Perplexity and its users. But it's a major threat to crawled websites
- shadowgovt 1y agoI have never created a website that I would not mind being fully crawled and indexed into another dataset that was divorced from the source (other than such divorcement makes it much harder to check pedigree, which is an academic concern, not a data-content concern: if people want to trust information from sources they can't know and they can't verify I can't fix that for them). In fact, the "old web" people sometimes pine for was mostly a place where people were putting things online so they were online, not because it would translate directly to money. Perhaps AI crawlers are a harbinger for the death of the web 2.0 pay-for-info model... And perhaps that's okay.
- short_sells_poo 1y agoThere's an important distinction that we are glossing over I think. In the times of the "old web", people were putting things online to interact with a (large) online audience. If people found your content interesting, they'd keep coming back and some of them would email you, there'd be discussions on forums, IRC chatrooms, mailing lists, etc. Communities were built around interesting topics, and websites that started out as just some personal blog that someone used to write down their thoughts would grow into fonts of information for a large number of people. Then came the social networks and walled gardens, SEO, and all the other cancer of the last 20 years and all of these disappeared for un-searchable videos, content farms and discord communities which are basically informational black holes. And now AI is eating that cancer, but IMO it's just one cancer being replaced by an even more insidious cancer. If all the information is accessed via AI, then the last semblance of interaction between content creators and content consumers disappears. There are no more communities, just disconnected consumers interacting with a massive aggregating AI. Instead of discussing an interesting topic with a human, we will discuss with AI...
- troyvit 1y ago> If I now go one step further and use an LLM to summarize content because the authentic presentation is so riddled with ads, JavaScript, and pop-ups, that the content becomes borderline unusable, then why would the LLM accessing the website on my behalf be in a different legal category as my Firefox web browser accessing the website on my behalf? I think one thing to ask outside of this question is how long before your LLM summaries don't also include ads and other manipulative patterns.
- Neil44 1y agoFlip it around, why would you go to the trouble of creating a web page and content for it, if some AI bot is going to scrape it and save people the trouble of visiting your site? The value of your work has been captured by some AI company (by somewhat nefarious means too).
- donkeybeer 1y agoI don't see the problem. I want AI agents to learn my website and have it available as part of its corpus of knowledge to users. If asked for something it can answer the question, and if user requests make a parallel web search for sources which would bring up my page. The latter is only a bonus not a necessity for me. Getting the information out there by whatever means is my first priority.
- carlosjobim 1y agoLegal category?
- renewiltord 1y agoThe websites don’t nag you, actually. They just send you data. You have configured your user agent to nag yourself when the website sends you data. And you’re right: there’s no difference. The web is just machines sending each other data. That’s why it’s so funny that people panic about “privacy violations” and server operators “spying on you”. We’re just sending data around. Don’t send the data you don’t want to send. If you literally send the data to another machine it might save it. If you don’t, it can’t. The data the website operator sends you might change as a result but it’s just data. And a free interaction between machines.
- baxuz 1y ago1. To access a website you need a limited anonymized token that proves you are a human being, issued by a state authority 2. the end I am firmly convinced that this should be the future in the next decade, since the internet as we know it has been weaponized and ruined by social media, bots, state actors and now AI. There should exist an internet for humans only, with a single account per domain.
- glenstein 1y agoA fascinating variation on this same issue can be found in Neal Stephenson's "Fall, or Dodge in Hell". There the solution is (1) discredit weaponized social media in its entirety by amplifying it's output exponentially and make its hostility universal in all directions, to the point that it's recognizeable as bad faith caricature. That way it can't be strategically leveraged with disproportionate directional focus against strategic targets by bad actors and (2) a new standard called PURDA, which is kind of behavioral signature as the mark of unique identity.
- dawnerd 1y agoNothing wrong if they fetch on your behalf. The problem is when they endlessly crawl along with every other ai company doing the same.
- epolanski 1y agoIt's somebody's else content and resources and they are free to ban you or your bots as much as they please.
- EGreg 1y ago1. I actually disagree. I think teasers should be free but websites should charge micropayments for their content. Here is how it can be done seamlessly, without individuals making decisions to pay every minute: https://qbix.com/ecosystem https://qbix.com/ecosystem 2. This also intersects with copyright law. Ingesting content to your servers en masse through automation and transforming it there is not the same as giving people a tool (like Safari Reader) they can run on their client for specific sites they visit. Examples of companies that lost court cases about this: Aereo, Inc. v. American Broadcasting Companies (2014) TVEyes, Inc. v. Fox News Network, LLC (2018) UMG Recordings, Inc. v. MP3.com, Inc. (2000) Capitol Records, LLC v. ReDigi Inc. (2018) Cartoon Network v. CSC Holdings (Cablevision) (2008) Image Search Engines: Perfect 10 v. Google (2007) That last one is very instructive. Caching thumbnails and previews may be OK. The rest is not. AMP is in a copyright grey area, because publishers choose to make their content available for AMP companies to redisplay. (@tptacek may have more on this) 3. Putting copyright law aside, that's the point. Decentralization vs Centralization. If a bunch of people want to come eat at an all-you-can-eat buffet, they can, because we know they have limited appetites. If you bring a giant truck and load up all the food from all all-you-can-eat buffets in the city, that's not OK, even if you later give the food away to homeless people for free. You're going to bankrupt the restaurants! https://xkcd.com/1499/ https://xkcd.com/1499/ So no. The difference is that people have come to expect "free" for everything, and this is how we got into ad-supported platforms that dominate our lives.
- glenstein 1y agoI would love micropayments as a kind of baked-in ecosystem support. You can crawl if you want, but it's pay to play. Which hopefully drives motivation for robust norms for content access and content scraping that makes everyone happy.
- EGreg 1y agoI want to bring Ted Nelson on my channel and interview him about Xanadu. Does anyone here know him? https://xanadu.com.au/ted/XU/XuPageKeio.html https://xanadu.com.au/ted/XU/XuPageKeio.html
- 1y ago
- gentle 1y agoI believe you're being disingenuous. Perplexity is running a set of crawlers that do not respect robots.txt and take steps to actively evade detection. They are running a service and this is not a user taking steps to modify their own content for their own use. Perplexity is not acting as a user proxy and they need to learn to stick to the rules, even when it interferes with their business model.
- axus 1y agoFor 1, 2, and 3, the website owner can choose to block you completely based on IP address or your User Agent. It's not nice, but the best reaction would be to find another website. Perplexity is choosing to come back "on a VPN" with new IP addresses to evade the block. #2 and #3 are about modifying data where access has been granted, I think Cloudflare is really complaining about #1. Evading an IP address ban doesn't violate my principles in some cases, and does in others.
- amiga386 1y agoIf you as a human are well behaved, that is absolutely fine. If you as a human spam the shit out of my website and waste my resources, I will block you. If you as a human use an agent (or browser or extension or external program) that modifies network requests on your behalf, but doesn't act as a massive leech, you're still welcome. If you as a human use an agent (or browser or extension or external program) that wrecks my website, I will block you and the agent you rode in on. Nobody would mind if you had an LLM that intelligently knew what pages contain what (because it had a web crawler backed index that refreshes at a respectful rate, and identifies itself accurately as a robot and follows robots.txt), and even if it needed to make an instantaneous request for you at the time of a pertinent query, it still identified itself as a bot and was still respectful... there would be no problem. The problem is that LLMs are run by stupid, greedy, evil people who don't give the slightest shit what resources they use up on the hosts they're sucking data from. They don't care what the URLs are, what the site owner wants to keep you away from. They download massive static files hundreds or thousands of times a day, not even doing a HEAD to see that the file hasn't changed in 12 years. They straight up ignore robots.txt and in fact use it as a template of what to go for first. It's like hearing an old man say "I need time to stand up because of this problem with my kneecaps" and thinking "right, I best go for his kneecaps because he's weak there" There are plenty of open crawler datasets, they should be using those... but they don't, they think that doesn't differentiate them enough from others using "fresher" data, so they crawl even the smallest sites dozens of times a day in case those small sites got updated. Their badly written software is wrecking sites, and they don't care about the wreckage. Not their problem. The people who run these agents, LLMs, whatever, have broken every rule of decency in crawling, and they're now deliberately evading checks, to try and run away from the repercussions of their actions. They are bad actors and need to be stopped. It's like the fuckwads who scorch the planet mining bitcoin; there's so much money flowing in the market for AI, that they feel they have to fuck over everyone else, as soon as possible, otherwise they won't get that big flow of money. They have zero ethics. They have to be stopped before their human behaviour destroys the entire internet.
- paulcole 1y ago> 1. If I as a human request a website, then I should be shown the content. Everyone agrees. Definitely don't agree. I don't think you should be shown the content, if for example: 1. You're in a country the site owner doesn't want to do business in. 2. You've installed an ad blocker or other tool that the site owner doesn't want you to use. 3. The site owner has otherwise identified you as someone they don't want visiting their site. You are welcome to try to fool them into giving you the content but it's not your right to get it.
- TZubiri 1y agoIn that case the llm would be a user-agent, quite distinct from scraping without a specific user request. This is well defined in specs and ToS, not quite a gray area
- Vegenoid 1y agoThere is a significant distinction between 2 and 3 that you glossed over. In 1 and 2, you the human may be forced to prove that you are human via a captcha. You are present at the time of the request. Once you’ve performed the exchange, then the HTML is on your computer and so you can do what you want to it. In 3, although you do not specify, I assume you mean that a bot requests the page, as opposed to you visiting the page like in scenario 2 and then an LLM processes the downloaded data (similarly to an adblocker). It is the former case that is a problem, the latter case is much harder to stop and there is much less reason to stop it. This is the distinction: is a human present at the time of request.
- philistine 1y agoTo me it's even simpler: 3 is a request made from another ip address that isn't directly yours. Why should an LLM request that acts exactly like a VPN request be treated differently from a VPN request?
- freehorse 1y agoYeah, I also find the analogy about "agent on behalf of the user interacting with a website" weak, because it is not about "an agent", it is a 3rd party service that actually takes content from a website, processes it and serves it to the user (even with their own ads?). It is more akin to, let's say, a scammy website that copies content from other legit websites and serves their own ads, than software running on the user's computer. There are legitimate reasons to do that, of course. Maybe I am trying to find info about some niche topic or how to do X, I ask an llm, the llm goes through some search results, a lot of which is search engine optimised crap, finds the relevant info and answers my question. But if I wrote articles in a news site, I am supported by ads or subscriptions and see my visits plummel because people, who would usually google about topic X and then visit my website that I wrote about X, were now reading the google summary that appeared when googling about topic X, based on my article, maybe I would have less motivation to continue writing. The only end result possible in such a scenario is that everything commercial of some quality being heavily paywalled, some tiny amount of free and open small web, and a huge amount of AI generated slop, because the value of an article in the open internet is now so low that only AI can produce it (economically, time-wise) efficiently enough.
- dabockster 1y ago> If I now go one step further and use an LLM to summarize content because the authentic presentation is so riddled with ads, JavaScript, and pop-ups, that the content becomes borderline unusable, then why would the LLM accessing the website on my behalf be in a different legal category as my Firefox web browser accessing the website on my behalf? Because the LLM is usually on a 3rd party cloud system and ultimately not under your full control. You have no idea if the LLM is retaining any of that information for that business's own purposes beyond what a EULA says - which basically amounts to a pinky swear here. Especially if that LLM is located across international borders. Now, for something like Ollama or LMStudio where the LLM and the whole toolchain is physically on your own system? Yeah that should be like Firefox legally since it's under your control.
- jpadkins 1y ago> 1. If I as a human request a website, then I should be shown the content. Everyone agrees. I disagree. The website should have the right to say that the user can be shown the content under specific conditions (usage terms, presented how they designed, shown with ads, etc). If the software can't comply with those terms, then the human shouldn't be shown the content. Both parties did not agree in good faith.
- dgshsg 1y agoYou want the website to be able to force the user to see ads?
- jpadkins 1y agono, I think a fair + just world, both parties agree before they transact. There is no force in either direction (don't force creators to give their content on terms they don't want, don't force users to view ads they don't want). It's perfectly fine if people with strict preferences don't match. It's a big web, there are plenty of creators and consumers. If the user doesn't want to view content with ads, that's okay and they can go elsewhere.
- sussmannbaka 1y ago4. If I now go one step further and use a commercial DDoS service to make the get requests for me because this comparison is already a stretch, then why would the DDoS provider accessing the website on my behalf be in a different legal category as my Firefox web browser accessing the website on my behalf?
- pavon 1y agoQuestion from a non-web-developer. In case 3, would it be technically possible for Perplexity's website to fetch the URL in question using javascript in the user's browser, and then send it to the server for LLM processing, rather than have the server fetch it? Or do cross-site restrictions prevent javascript from doing that?
- jabroni_salad 1y agoIf it was just one human requesting one summary of the page nobody would ever notice. The typical watermark for junk traffic is pretty high as it was. I have a dinky little txt site on my email domain. There is nothing of value on it, and the content changes less than once a year. So why are AI scrapers hitting it to the tune of dozens of GB per month?
- snihalani 1y agoyou are paying for LLM but not paying for the website. LLM is removing the power the website had. Legally, that's cause for loss of income
- bigbuppo 1y agoRight, but the LLM isn't really being used for that. It's being used for marketing and advertising purposes most of the time. The AI companies also let you play with it from time to time so you'll be a shill for them, but mostly it's the advertising people you claim to not like.
- RiverCrochet 1y agoIntellectual property laws are what creates the entitlement that someone else besides you can tell you what to do with the things Internet connected computers and phones download, because almost everything you download is copy of something a person created, therefore its copyrighted for the life of the author + 75 years or whatever by default. Therefore artifices like "you don't have the right to view this website without ads" or "you can't use your phone, computer, or LLM to download or process this outside of my terms because copyright" become possible, institutionalizable, enforceable, and eventually unbypassable by technology. If we reverted back to the Constitutional purpose of copyright (to Progress the Science and Useful Arts) then things might be more free. That's probably not happening in my lifetime or yours.
- amelius 1y agoIt's because they own the content so they get to set the terms.
- k1m 1y agoWhen Yahoo! Pipes was still running (long time ago), their official position was: > Because Pipes is not a web crawler (the service only retrieves URLs when requested to by a Pipe author or user) Pipes does not follow the robots exclusion protocol, and won't check your robots.txt file.
- remus 1y agoThe solution to 3 seems fairly straightforward: user requests content and passes it to llm to summarise.
- sterlind 1y ago> why would the LLM accessing the website on my behalf be in a different legal category as my Firefox web browser accessing the website on my behalf? is it just on your behalf? or is it on Perplexity's behalf? are they not archiving the pages to train on? it's the difference between using Google Chrome vs. Chrome beaming full page snapshots to train Gemini on.
- mrinterweb 1y agoIs a perplexity visit not cached and shared between users who perform a similar search? I don't know much about perplexity, but I'd be surprised if scraped results weren't used to serve multiple searches and users. By bypassing the no-crawl directive, that is a violation of the website's expressed request. I think it is different if individual users chooses to bypass certain things on a website, but for a company to choose to do it is another story.
- martin82 1y agoIt's a tough issue indeed. One thing that comes to my mind is: If a human tries to answer a question via the web, he will browse one site after the other. If that human asks an LLM, it will ping 25 sites in parallel. Scale this up to all of humanity, and it should be expected that internet traffic will rise 25x - just from humans manually asking questions every now and then - we are not even talking about AI companies actively crawling the web. That means, webmasters will have to figure out aggressive caching and let CDNs deal with the problem or put everything behind a login screen (which might also just be a temporary fix).
- userbinator 1y agoIf a human tries to answer a question via the web, he will browse one site after the other. Not me, I often open multiple tabs and windows at once to compare and contrast the results.
- snowwrestler 1y agoWe have a faceted search that creates billions of unique URLs by combinations of the facets. As such, we block all crawlers from it in robots.txt, which saves us AND them from a bunch of pointless indexing load. But a stealth bot has been crawling all these URLs for weeks. Thus wasting a shitload of our resources AND a shitload of their resources too. Whoever it is (and I now suspect it is Perplexity based on this Cloudflare post), they thought they were being so clever by ignoring our robots.txt. Instead they have been wasting money for weeks. Our block was there for a reason.
- graybanana 1y agoWe have the same issue (billions of URLs). The newer bots that rotate IPs across thousands of IP ranges are killing us and there is no good way to block them short of captcha's or forcing logins, which we would really rather not inflict on our users.
- benreesman 1y agoThis is a hypothetical so give me a little rope here, but what if robots.txt wasn't a suggestion? What if it were binding (leaving aside for a moment how one would enforce / mandate / guarantee that)? Would that solve the whole problem? Folks who ran webservers declared what they consent to, and that happens? I think it's useful to just see if there's a consensus on that: actually making that happen is a whole can of worms itself, but it's strictly simpler than devising a good outcome without the consensus. (And such things are not impossible, merely difficult, we have other systems ranging from BGP to the TLD mechanism that get honored in real life).
- 1vuio0pswjnm7 1y agoThe simple answer to #3 is advertising, including telemetry, tracking and other forms of web-based surveillance. These usually rely on certain browser "features" and/or default settings. The goal is not to make the content usable. The goal is to get the traffic. When advertising alone is the "business model", e.g., not the value of the "content", then even Cloudflare is going to try to protect it (the advertising, not the content). Anything to get www users to turn on Javascript so the surveillance capitalism can proceed. Hence all the "challenges" to frustrate and filter out software thatis not advertising-friendly, e.g., graphical. Cloudflare's ruminations on user-agent strings are perplexing. It has been an expectation that the user-agent HTTP header will be spoofed since the earliest web browsers. The user-agent header is a joke. This is from circa 1993, the year the www was opened to public access: https://raw.githubusercontent.com/alandipert/ncsa-mosaic/master/mosaic-spoof-agents https://raw.githubusercontent.com/alandipert/ncsa-mosaic/mas... Cloudflare's "bot protections" are not to ensure human use of a website but to ensure use of specific software to access a website. Software that facilitates data collection and advertising services. For example, advertising-sponsored browsers. Any other software is labeled "bot". It does not matter if a human is operating it.
- 1vuio0pswjnm7 1y ago(IMHO) The correct way to limit "abuse", e.g., by "bots", is to rate limit. But as other commenters point out, Cloudflare routinely (and knowingly) blocks humans sending only a single GET request, e.g., with Javascript disabled. Needless to say, this does not exceed any reasonable rate limit. It is not "abuse". By Cloudflare's own admission, and as demonstrated by the case of Perplexity AI, this "bot protection" does not stop "bots". It does stop any humans not using popular advertising-sponsored web browsers.
- miki123211 1y agoPeople seem to differentiate between an LLM on some other computer accessing the website and doing God knows what with it, versus your browser accessing the website and then passing it to an LLM. People are usually fine with the latter but not the former, even though they come down to the same thing. I think this is because people don't want LLMs to train on their content, and they don't differentiate between accessing a website to show it to the user, versus accessing it to train.
- ivanjermakov 1y agoThe problem is not about personal use. It's about big corporations scrapping billions of pages to make money.
- skeezyboy 1y agothose publicly accessible pages yea?
- antonymoose 1y agoWhile I agree it’s a problem, unless you get litigious or the backing of a national congressional body, you might as well not care. You won’t win this fight. Circa 2008 I worked for a startup that would scrape Google Books and a variety of other sources for public domain content to then print via Amazon’s Print-on-Demand services. Google, of course, didn’t like this and introduced a Captcha not very long after we started scraping. So we hired a team of underemployed / unemployed English majors during the height of the recession, paid them $10 per hour to type in Captchas all day long and we downloaded their full corpus anyways. If Google can’t win, you won’t either.
- fergie 1y ago> why would the LLM accessing the website on my behalf be in a different legal category as my Firefox web browser accessing the website on my behalf? It is illegal to copy stuff from the internet and then make it available from your own servers, especially when those sources have expressly asked you not to do it.
- danlitt 1y agoYou need more qualifiers for this to be true. archive.org routinely archives content that the site operators would prefer to be taken down.
- otikik 1y agoI don't really see the issue. The web admin should be able to block usages 1, 2 or 3 at their discretion. It's their website. Similarly the user is free to try to engage via 1, 2, 3, or refuse to interact with the website entirely.
- ccgreg 1y agoNote that a book author cannot publish a book and then refuse to let libraries buy copies and lend them out. This was litigated 100+ years ago.
- account42 1y agoThe difference is that libraries aren't all about concentrating wealth for themselves.
- lovich 1y ago> If I now go one step further and use an LLM to summarize content because the authentic presentation is so riddled with ads, JavaScript, and pop-ups, that the content becomes borderline unusable, then why would the LLM accessing the website on my behalf be in a different legal category as my Firefox web browser accessing the website on my behalf? Because quantity has a quality of its own. I say this as someone who is on the side of pro local user commands how local compute works, but understand why companies are reacting to how cheap LLMs are making information discovery against their own datasets
- npc_anon 1y agoThe problem in your logic is that all points starts wit "I". You're not the only stakeholder in any of those interactions. There's you, a mediator (search or LLM), and the website owner. The website owner (or its users) basically do all the work and provide all the value. They produce the content and carry the costs and risks. The pre-LLM "deal" was that at least some traffic was sent their way, which helps with reach and attempts at monetization. This too is largely a broken and asymmetrical deal where the search engine holds all the cards but it's better than nothing. A full LLM model that no longer sends traffic to websites means there's zero incentive to have a website in the first place, or it is encouraged to put it behind a login. I get that users prefer an uncluttered direct answer over manually scanning a puzzling web. But the entire reason that the web is so frustrating is that visitors don't want to pay for anything.
- ricardo81 1y agoAgreed. Cloudflare released these insights showing the disparity between crawling/scraping and visits referred from the AI platforms. https://radar.cloudflare.com/ai-insights#crawl-to-refer-ratio https://radar.cloudflare.com/ai-insights#crawl-to-refer-rati...
- danbruc 1y agoBut the entire reason that the web is so frustrating is that visitors don't want to pay for anything. They are already paying, it is the way they are paying that causes the mess. When you buy a product, some fraction of the price is the ad budget that gets then distributed to websites showing ads. Therefore there is also nothing wrong with blocking ads, they have already been paid for, whether you look at them or not. The ad budget will end up somewhere as long as not everyone is blocking all ads, only the distribution will get skewed. Which admittedly might be a problem for websites that have a user base that is disproportionally likely to use ad blockers. Paying for content directly has the problem that you can only pay for a selected few websites before the amount you have to pay becomes unreasonable. If you read one article on a hundred different websites, you can not realistically pay for a hundred subscriptions that are all priced as if you spent all your time on a single website. Nobody has yet succeeded in creating a web wide payment method that only charges you for the content that you actually consume and is frictionless enough to actually work, i.e. does not force you to make a conscious payment decisions for a few cents or maybe even only fractions of a cent for every link you click and is not a privacy nightmare collecting all the links you click for billing purposes. Also if you directly pay for content, you will pay twice - you will pay for the subscription and you will still pay into the ad budget with all the stuff you buy.
- Sam6late 1y agoperplexity is being used to bypass paywalls, I noticed this when I pasted into it a text and it was captured as a hyperlinked text. I will try other websites with paywalls to see if it is a go.
- SergeAx 1y ago1. Sometimes you should prove that you are human first. I think the line is drawn at "on my behalf". The silent agreement of the web is that humans are served content via a browser, and robots are obeying rules. All we need to support this status quo is to perform data processing by ML models on a client's side, in the browser, the same way we rip out ads.
- throw10920 1y ago> If I as the human request the software on my computer to modify the content before displaying it, for example by installing an ad-blocker into my user agent, then that's my choice and the website should not be notified about it. Because the website has every right to block you or refuse access to you if you do that, just like an establishment has the right to refuse you access if you try to enter without a shirt, if you're denying them access to revenue that they predicated your access on. Similarly, if you're using a user-agent the website doesn't like, they have the right to block you, or take action against that user-agent to prevent it from existing if they can't reliable detect it to block it.
- cyanydeez 1y agoThink of it like tge telephone game. Do you -really- want that much abstracrion? Theres a bunch of nerds and capitalists about to rediscover GIGO