7 ms·

I fear for the unauthenticated web

117 points by SethMLarson 2y ago

deleted 2y ago
[deleted]
jmclnx 2y ago
I would think all you need to do is add a copyright statement of some kind. Sad things are getting to this point. Maybe I should add this to my site :) (c) Copyright (my email), if used for any form of LLM processing, you must contact me and pay 1000USD per word from my site for each use.
- jasperr1 2y ago
  The reality is that a lot of these small websites have very permissive licenses. I really hope we don't get to the point where we must all make our licenses stricter.
  - krapp 2y ago
    The reality is that none of these LLM scrapers give a damn about copyright, because the entire AI industry is built on flagrant copyright violation, and the premise that they can be stopped by a magic string is laughable. You could sue, if you can afford it, meanwhile all of your data is already training their models.
    - jasonjayr 2y ago
      A class action, funded by their rivals could hurt quite a bit, especially for sites damaged monetarily by these LLM scrapers.
- JohnFen 2y ago
  Such a notice is legally meaningless, though. Doubly so if the courts rule that scraping for AI purposes counts as fair use.
- Aurornis 2y ago
  Copyright is for topics like redistribution of the source material. You can’t add arbitrary terms to a copyright claim that go beyond what copyright law supports. I think you’re confusing copyright with a EULA. You would need users to agree to the EULA terms before viewing the material. You can’t hide contractual obligations in the footer of your website and call it copyright.
  - 101008 2y ago
    What about if my index says "This are the EULA, by clicking "Next" or "Enter", you are accepting them", and a LLM scrapper "clicks" Next to fetch the rest of the content?
    - aaronbaugher 2y ago
      That's how the big software companies have been doing it to us for years, so it does seem like turnabout would be fair play.
      - WesolyKubeczek 2y ago
        "Ah, but for us, it's unenforceable, it's for you little people." "Copyright? Well if you are a big label, we probably need to talk. Little people? Oh fuck you, just give us your money and creative output."
- jefftk 2y ago
  It's reasonably likely, but not yet settled, that LLM training falls under fair use and doesn't require a license. This is what the https://githubcopilotlitigation.com/ https://githubcopilotlitigation.com/ class action (from 2022) is about, and its still making its way through the court. This prediction market has it at 12% likely to succeed, suggesting that courts will not agree with you: https://manifold.markets/JeffKaufman/will-the-github-copilot-litigation https://manifold.markets/JeffKaufman/will-the-github-copilot...
  - dingnuts 2y ago
    this isn't about copyright but about computer access. the CFAA is extremely broad; if you ban LLM companies from access on grounds of purpose you have every legal right to do so in theory that legislation has teeth, too. they are not allowed to access your system if you say they are not; authentication is irrelevant. every GET request to a system that doesn't permit access for training data is a felony
  - waveringana 2y ago
    why are we pretending that these gambling sites have any weight on anything
    - eudhxhdhsb32 2y ago
      What do you mean by weights? I'd certainly trust their predictions more than those given by most "experts".
  - maeln 2y ago
    > This prediction market has it at 12% likely to succeed Randos on the internet with a betting addiction are distinctively different from a court of law. I wish people would stop talking about prediction market as if they mattered.
    - eudhxhdhsb32 2y ago
      Participants in prediction market do not need to be experts for their collective input to be informative. There's a long history of economic research on the "wisdom of crowds" that backs up their value.
- jcranmer 2y ago
  The argument the AI companies are making is that training for LLMs is fair use which means a copyright statement means fuck all from their point of view. (Even if it does, assuming you're in the US, unless you register the copyright with the US copyright office, you can only sue for actual damages, which means the cost of filing a lawsuit against them--not even litigating, just the court fee for saying "I have a lawsuit"--would be more expensive than anything you could recover. Even if you did register and sued for statutory damages, the cost of litigation would probably exceed the recovery you could expect.) Of course, the big AI companies are already trying to get the government to codify AI training as fair use and sidestep the litigation which doesn't seem to be going entirely their way on this matter (cf. https://arstechnica.com/google/2025/03/google-agrees-with-openai-that-copyright-has-no-place-in-ai-development/ https://arstechnica.com/google/2025/03/google-agrees-with-op...).
  - tsumnia 2y ago
    In addition, we need to start paying attention to the growing legislation about AI and copyright law. There was an article on HN I think this week (or last) specifically where a judge ruled AI cannot own copyright on its generated materials. IANAL, but I do wonder how this ruling will be used as a point of reference whenever we finally ask the question "Does material produced by GenAI violate copyright laws?" Specifically if it cannot claim ownership, a right that we've awarded to trees and monkeys, how does it operate within ownership laws? And don't even get me ranting about HUMAN digital rights or Personified AIs.
  - tqwhite 2y ago
    Fair use requires transformation. LLM is as transformative as it gets. If I'm on the jury, you're going to have to make new copyright law for me to convict. I am personally happy to have everyone, people and LLM alike, learn from my wisdom.
    - jcranmer 2y ago
      > Fair use requires transformation. No, it doesn't. There are four factors for fair use, and whether the use is transformative is part of one of them. And you don't need to win on all four factors. > LLM is as transformative as it gets. The current ruling precedent for "transformative" is the Warhol decision, which effectively says that to look at whether or not something is transformative, you kind of have to start by analyzing its impact on the market (and if you're going "doesn't that import the fourth factor into the first?" the answer is "yes, I don't like it, but it's what SCOTUS said"). By that definition, LLMs are nowhere near "transformative." Even pre-Warhol, their role as "transformative" is sketchy, because you have to remember that this is using its legal definition, not its colloquial definition. > If I'm on the jury Fortunately, for this kind of question, the jury isn't going to be involved in determining fair use, so it doesn't matter what you think.
- jeffwask 2y ago
  Sure, because Meta certainly followed copyright law to the letter when they torrented thousands of copyrighted books from hundreds of published and known authors to train Lama. Forgive me if I doubt a text disclaimer on the page will slow them down.
- dspillett 2y ago
  Unfortunately copyright is no limit to these companies. Meta is stating in court that knowingly downloading pirated content is perfectly fine (ref https://news.ycombinator.com/item?id=43125840 https://news.ycombinator.com/item?id=43125840) so they for one would have absolutely no issue completely ignoring your copyright notice and stated licensing costs. Good luck affording a legal team to try force them to pay attention. Copyright is something for them to beat us with, not the other way around, apparently.
- kerkeslager 2y ago
  This is pretty naive. The only reason copyright is so strong in the US is that there are big players (Disney, Elsevier) who benefit from it. But gig tech is much bigger, and LLMs have created a situation where big tech has a vested interest in eroding copyright law. Both sides are gearing up for a war in the court systems, and it's definitely not a given who will win. But, if you try to enter the fray as an individual or small company, you definitely aren't going to win.
ToucanLoucan 2y ago
Yet another entry in the long and shameful history of Silicon Valley abusing the public square for their own profit (or in this case, fantasies of profit) and the rest of us just have to learn to live with it because the justice system simply will not even try and give us recourse. Move fast and break things apparently has a bonus clause for the things you break not being your responsibility to fix.
- Analemma_ 2y ago
  I don't think the justice system is the one to blame here. Right up until LLMs and their huge datamining operations appeared, everyone in tech was strongly for unrestricted scraping. Everybody here cheered the LinkedIn decision [0], saying "it's on the public web: if you didn't want it to be scraped, you should've put it behind authentication". LLMs change nothing about the legal landscape, they've just convinced everyone on an emotional level that unrestricted scraping is no longer an automatic good. It's not the justice system's job to react to such vibe shifts, the laws themselves have to be changed. [0]: https://news.ycombinator.com/item?id=21241395 https://news.ycombinator.com/item?id=21241395
  - ToucanLoucan 2y ago
    I'm not talking about the ethics of scraping itself. I think scraping is fine from an ethics perspective for exactly that reason. I think LLM companies, however, are scraping ineptly and with poorly implemented tooling, which is causing problems for the websites they're targeting, and that sucks ass and they should be held liable. On the legal end though, I do think there's a few things that should be done: * Scrapers should be CLEARLY, and CORRECTLY identified as what they are, and who they are being dispatched from. Changing user agents to get around blocks should not be permitted, ever. If you only get a certain amount of content or a certain subset of pages when you identify as a scraper, that is a choice the website operator is making and it should be respected. * Scrapers MUST OBEY robots.txt. We didn't create that for a fun hacker weekend. It's an important technical component of how we organize websites and how we want them crawled, if we want them crawled. It should be the first stop for any scraper on any website, and again, it should be respected. * Scrapers should always meter their traffic with respect to the website owner. Pounding an entire website's library of content request after request with only milliseconds between is, to put it bluntly, being a fucking asshole. And not just to the owner, but to anyone else attempting to use the site at the time. If a website operator configures their site incorrectly and pages they don't want scraped are, or pages they do want scraped aren't, then that is on them and they need to fix that. It is not in the scraper's purview to end-run around that configuration to "be real sure" they got everything they were meant to, and it's especially not that to get things the web operator has explicitly tried to not let the scraper have. And yes, all of these things should be legally actionable, with financial penalties attached and for serial offenders, we should have a registry of scraper bots that we disallow entirely because they are acting in bad faith.
Aurornis 2y ago
Rate limiting is the first step before cutting everything off behind forced logins. > This practice started with larger websites, ones that already had protection from malicious usage like denial-of-service and abuse in the form of services like Cloudflare or Fastly FYI Cloudflare has a very usable free tier that’s easy to set up. It’s not limited to large websites.
- blibble 2y ago
  I get the feeling that I'm going to read a blog post in a few years telling us that the CDN companies have been selling everything pulled through their cache to the AI companies since 2022
  - nottorp 2y ago
    And even if they don't, is everything depending on Cloudflare to stay online a good thing?
    - sshine 2y ago
      It’s a terrible thing. Cloudflare is the company I hate the most: I think (what I know of) their tech is done right, and they’re just too big to put my eggs in their basket.
    - koakuma-chan 2y ago
      Why is nobody building a better product?
      - nisegami 2y ago
        Because Cloudflare's products are very good. The issues people have with Cloudflare are more philosophical and won't be resolved by building a better product. GP's concerns would best be rectified by dozens of commodity-like competitors of similar quality.
        2y ago
JKCalhoun 2y ago
For some reason I am not really moved by a lot of the hand wringing I am seeing lately. It's a not a binary thing to me: LLMs are not god, but even without AGI, they have proven wildly useful to me. Calling them "shitty chat bots" doesn't sway me. Further I have always assumed that everything that I post to the web is publicly accessible to everyone/everything. We lost any battle we thought we could wage some 2+ decades ago when web crawlers started hoovering up data from our sites.
- gjsman-1000 2y ago
  I agree even if I have mixed feelings on it. To me this feels almost like the news complaining that they want a "link tax." Weren't their headlines and summaries used? It seems inconsistent to somehow say that AI and scraping is not okay; but that news companies should also not be entitled to their link tax. It's okay to index, but not that kind of index.
  - theamk 2y ago
    It seems pretty cut-and-dry to me: the website owner should opt-out if they don't like the deal (being indexed in this case). In the "link tax" case, there were plenty of trivial ways to opt out of headline usage - robots.txt, http headers, http tags. The problem was newspapers did not want to opt out (as they were benefiting from Google themselves), so they wanted a 3rd option. Which was pretty stupid of course - if you don't like the deal, don't take it; suing the offering party for a better deal is not a good long-term strategy. In the AI case, there is no opt-out. All those websites already indicated they want to opt-out via robots.txt, but the AI companies ignore robots.txt, change user-agent, fake IPs, and so on - do the things that are normally done by shady malwar-ish services rather than multi-billion-dollar companies. It really bothers me when people don't see the difference between those two cases.
- prosody 2y ago
  This article isn’t about that. It’s about the externalized costs that LLM companies are pushing onto webmasters because of their aggressive scraping. It’s one thing to believe that LLMs are a good thing, it’s another thing to believe that individuals and cooperative groups that run small internet services ought to be the ones to pay for that good.
cxr 2y ago
Perversely, this submission is essentially blogspam. The article linked in the second paragraph, to which this "1 minute" read adds almost nothing of value, is the main story: <https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies/ https://thelibre.news/foss-infrastructure-is-under-attack-by...> 394 comments. 645 points. Submitted 3 hours ago: <https://news.ycombinator.com/item?id=43422413 https://news.ycombinator.com/item?id=43422413>
- btown 2y ago
  But also ironically, it's almost heartwarming these days to see blogspam that's not machine-generated! A real live human cared enough about an article to write a brief (perhaps only barely substantial, but at least handwritten) take on it! It's reminiscent, perhaps, of the feel and motivation for Tumblr reblogs - and Tumblr continues to be vibrant by virtue of this culture: https://www.tumblr.com/engineering/189455858864/how-reblogs-work https://www.tumblr.com/engineering/189455858864/how-reblogs-... (2019) Now, is driving attention and reputation to their site (in the broadest senses) part of a blogspammer/reblogger's motivation? Absolutely! But should we be concerned about rewarding their act of curation, as long as there is at least some level of genuine curation intent? A world where that answer is categorically "no" would be antithetical, I think, to the concept of the participatory web.
  - dkkergoog 2y ago
    "heartwarming ... To see blogspam" the internet was a mistake
    - wongarsu 2y ago
      The internet was great, everything we did with it in the last 20 years with it was the mistake. Collimating in a comment that blogspam can now be one of the positive notes in the hellscape we are building. A very useful hellscape though, for all its flaws
      - 38 2y ago
0x1ceb00da 2y ago
Do I need to be worried about my bill if I've rented a simple EC2 instance without any fancy autoscaling stuff?
- simonw 2y ago
  Probably not. Keep an eye on bandwidth usage since you'll be charged for that but you would need to attract an incredible amount of bot traffic for that to add up to anything meaningful. The thing to watch out for is platforms like Vercel or Google Cloud Run where you get charged more for compute if you attract crawlers, potentially unbounded (make sure to set up spending limits if you can.)
hubraumhugo 2y ago
We should try separating good bots from bad bots: Good bots: search engine crawlers that help users find relevant information. These bots have been around since the early days of the internet and generally follow established best practices like robots.txt and rate limits. AI agents like OpenAI's Operator or Anthopic's Computer Use probably also fit into that bucket as they are offering useful automation without negative side effects. Bad bots: bots that have a negative affect website owners by causing higher costs, spam, or downtime (automated account creation, ad fraud, or DDoS). AI crawlers fit into that bucket as they disregard robots.txt and spoof user agent. They are creating a lot of headaches for developers responsible for maintaining heavily crawled sites. AI companies don't seem to care about any crawling best practices that the industry has developed over the past two decades. So the actual question is how good bots and humans can coexist on the web while we protect websites against abusive AI crawlers. It currently feels like an arms race without a winner.
- jsheard 2y ago
  Discriminating search engine bots is pretty straightforward, the big names provide bulletproof methods to validate whether a client claiming to be their bot is really their bot. It'll be an uphill battle for new search engines if everyone only trusts Googlebot and Bingbot though. https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot https://developers.google.com/search/docs/crawling-indexing/... https://www.bing.com/webmasters/help/verifying-that-bingbot-is-bingbot-3905dc26 https://www.bing.com/webmasters/help/verifying-that-bingbot-...
napolux 2y ago
> I suggest everyone that uses cloud infrastructure for hosting set-up a billing limit to avoid an unexpected bill in case they're caught in the cross-hairs of a negligent company. All the abusers anonymize their usage at this point, so good luck trying to get compensated for damages. This is scary
- jsheard 2y ago
  What's scarier is that most of the big clouds don't even let you set up a billing limit.
charcircuit 2y ago
Crawlers visiting every page on your website is not the main problem with the unauthenticated web. The amount of spam that happens when you let people freely post is a much bigger problem.
isoprophlex 2y ago
Everyone is (rightfully) outraged, but this is essentially nothing new. Asshat capitalists have been externalizing the costs of their asshat moneymaking schemes on the little guy since approximately forever. Deregulation is ultimately antithetical to our personal freedom. I just hope the spirit of the internet that I grew up with can be rescued, or reincarnated somehow...
MontgomeryPy 2y ago
Could an answer here be for smaller websites to convert their sites into chatbots which could prevent AI scrapers from slurping up all their content/drive up their hosting costs?
- cwmma 2y ago
  no
anovikov 2y ago
Pretty soon virtually everything will be paywalled. Ironically, it will provide us with a good metric that lets us find out whether AGI has arrive or not: when it does, paywalling will stop working because AGI could derive more value from accessing things and will thus outbid us.
hugs 2y ago
I might be naive, but I think it's time we seriously start implementing "HTTP status code 402: Payment Required" across the board. "L402" is an interesting proposal. Paying a fraction of a penny per request. https://github.com/l402-protocol/l402 https://github.com/l402-protocol/l402
- rambambram 2y ago
  I stumbled upon this status code last year - had never heard of it before - and I bookmarked it and then forgot about it. Thanks for the reminder.
- cwmma 2y ago
  this is basically what they are doing, but instead of charging actual money they are making visitors spin the CPU ideally in a proof of work problem, which has the same outcome from the crawlers perspective.
  - woah 2y ago
    This has existed for decades. The proof of CPU work is called "frontend frameworks"
  - fewsats 2y ago
    I've talked with tons of publishers and all say the same thing: "Hey, we'd happily give these companies clean data if they just paid us instead of building these scrapers." I think there is a psychological aspect that made micropayments never work for humans but machines may be better suited for it.
- SoftTalker 2y ago
  This is ultimately the answer. If something has value, users should pay for it. We haven't had a good way to do that on the web, so it has resulted in the complete shitshow that most websites are.
parliament32 2y ago
Linked in the article that this article links to is a project I found interesting for combatting this problem, a (non-crypto) proof-of-work challenge for new visitors https://github.com/TecharoHQ/anubis https://github.com/TecharoHQ/anubis Looks like the GNOME Gitlab instance implements it: https://gitlab.gnome.org/GNOME https://gitlab.gnome.org/GNOME
- kh_hk 2y ago
  For targeted scrapes, isn't proof of work trivial to bypass? 1. headless browser 2. get cookie 3. use cookie on subsequent plain requests
  - parliament32 2y ago
    It doesn't sound like the scrapers are that smart yet, but when they get there, presumably you'd just lower the cookie lifetime until the requests are down to an acceptable level. It takes a split-second in my browser so it shouldn't interfere much for human visitors.
kmeisthax 2y ago
> How long until scrapers start hammering Mastodon servers? Mastodon has AUTHORIZED_FETCH and DISALLOW_UNAUTHENTICATED_API_ACCESS which would at least stop these very naive scrapers from getting any data. Smarter scrapers could actually pretend to speak enough ActivityPub to scrape servers, though.
woah 2y ago
If you don't want someone to access your website, don't put it online
fewsats 2y ago
There's a real economic problem here: when someone scrapes your site, you're literally paying for them to use your stuff. That's messed up (and not sustainable) It seems like a good fit for micropayments. They never took off with people but machines may be better suited for them. L402 can help here. https://l402.org https://l402.org
- fewsats 2y ago
  The other obvious solution is a "web of trust" where Cloudflare just tells you "this request goes in, this one goes out". I think the paying approach is superior (after all you make money out of people using your service) but Cloudflare is a straight forward/simpler one.
- tqwhite 2y ago
  Aren't you paying for me to use the site, too? Or Google? Isn't the point of paying for a web hosting service to distribute information?
  - fewsats 2y ago
    Yes, but there is a "free lunch" problem. I can run a script that hits your page costing you X at a fraction of the cost for me (the user) *Edit: typo
    - tqwhite 2y ago
      I think the whole internet is a free lunch problem as far as that goes. I pay for web hosting because I consider the cost to be worth it to send my fabulous opinions into the ether. The premise of this thread is that somehow the LLM builders are reading too much. I bet it's less than google. I continue to believe, if you don't want everyone in the world to see and use your stuff, don't put it on the internet.
renegat0x0 2y ago
To be honest I feel that web2 is overrated. Most of content, blogs could be static sites. For mastodon, forums I think user validation is ok and a good way to go.
paranoidroid 2y ago
[dead]

117 points by SethMLarson 2y ago

deleted 2y ago

[deleted]

I would think all you need to do is add a copyright statement of some kind. Sad things are getting to this point. Maybe I should add this to my site :) (c) Copyright (my email), if used for any form of LLM processing, you must contact me and pay 1000USD per word from my site for each use.

jasperr1 2y ago
The reality is that a lot of these small websites have very permissive licenses. I really hope we don't get to the point where we must all make our licenses stricter.
- krapp 2y ago
  The reality is that none of these LLM scrapers give a damn about copyright, because the entire AI industry is built on flagrant copyright violation, and the premise that they can be stopped by a magic string is laughable. You could sue, if you can afford it, meanwhile all of your data is already training their models.
  - jasonjayr 2y ago
    A class action, funded by their rivals could hurt quite a bit, especially for sites damaged monetarily by these LLM scrapers.
JohnFen 2y ago
Such a notice is legally meaningless, though. Doubly so if the courts rule that scraping for AI purposes counts as fair use.
Aurornis 2y ago
Copyright is for topics like redistribution of the source material. You can’t add arbitrary terms to a copyright claim that go beyond what copyright law supports. I think you’re confusing copyright with a EULA. You would need users to agree to the EULA terms before viewing the material. You can’t hide contractual obligations in the footer of your website and call it copyright.
- 101008 2y ago
  What about if my index says "This are the EULA, by clicking "Next" or "Enter", you are accepting them", and a LLM scrapper "clicks" Next to fetch the rest of the content?
  - aaronbaugher 2y ago
    That's how the big software companies have been doing it to us for years, so it does seem like turnabout would be fair play.
    - WesolyKubeczek 2y ago
      "Ah, but for us, it's unenforceable, it's for you little people." "Copyright? Well if you are a big label, we probably need to talk. Little people? Oh fuck you, just give us your money and creative output."
jefftk 2y ago
It's reasonably likely, but not yet settled, that LLM training falls under fair use and doesn't require a license. This is what the https://githubcopilotlitigation.com/ https://githubcopilotlitigation.com/ class action (from 2022) is about, and its still making its way through the court. This prediction market has it at 12% likely to succeed, suggesting that courts will not agree with you: https://manifold.markets/JeffKaufman/will-the-github-copilot-litigation https://manifold.markets/JeffKaufman/will-the-github-copilot...
- dingnuts 2y ago
  this isn't about copyright but about computer access. the CFAA is extremely broad; if you ban LLM companies from access on grounds of purpose you have every legal right to do so in theory that legislation has teeth, too. they are not allowed to access your system if you say they are not; authentication is irrelevant. every GET request to a system that doesn't permit access for training data is a felony
- waveringana 2y ago
  why are we pretending that these gambling sites have any weight on anything
  - eudhxhdhsb32 2y ago
    What do you mean by weights? I'd certainly trust their predictions more than those given by most "experts".
- maeln 2y ago
  > This prediction market has it at 12% likely to succeed Randos on the internet with a betting addiction are distinctively different from a court of law. I wish people would stop talking about prediction market as if they mattered.
  - eudhxhdhsb32 2y ago
    Participants in prediction market do not need to be experts for their collective input to be informative. There's a long history of economic research on the "wisdom of crowds" that backs up their value.
jcranmer 2y ago
The argument the AI companies are making is that training for LLMs is fair use which means a copyright statement means fuck all from their point of view. (Even if it does, assuming you're in the US, unless you register the copyright with the US copyright office, you can only sue for actual damages, which means the cost of filing a lawsuit against them--not even litigating, just the court fee for saying "I have a lawsuit"--would be more expensive than anything you could recover. Even if you did register and sued for statutory damages, the cost of litigation would probably exceed the recovery you could expect.) Of course, the big AI companies are already trying to get the government to codify AI training as fair use and sidestep the litigation which doesn't seem to be going entirely their way on this matter (cf. https://arstechnica.com/google/2025/03/google-agrees-with-openai-that-copyright-has-no-place-in-ai-development/ https://arstechnica.com/google/2025/03/google-agrees-with-op...).
- tsumnia 2y ago
  In addition, we need to start paying attention to the growing legislation about AI and copyright law. There was an article on HN I think this week (or last) specifically where a judge ruled AI cannot own copyright on its generated materials. IANAL, but I do wonder how this ruling will be used as a point of reference whenever we finally ask the question "Does material produced by GenAI violate copyright laws?" Specifically if it cannot claim ownership, a right that we've awarded to trees and monkeys, how does it operate within ownership laws? And don't even get me ranting about HUMAN digital rights or Personified AIs.
- tqwhite 2y ago
  Fair use requires transformation. LLM is as transformative as it gets. If I'm on the jury, you're going to have to make new copyright law for me to convict. I am personally happy to have everyone, people and LLM alike, learn from my wisdom.
  - jcranmer 2y ago
    > Fair use requires transformation. No, it doesn't. There are four factors for fair use, and whether the use is transformative is part of one of them. And you don't need to win on all four factors. > LLM is as transformative as it gets. The current ruling precedent for "transformative" is the Warhol decision, which effectively says that to look at whether or not something is transformative, you kind of have to start by analyzing its impact on the market (and if you're going "doesn't that import the fourth factor into the first?" the answer is "yes, I don't like it, but it's what SCOTUS said"). By that definition, LLMs are nowhere near "transformative." Even pre-Warhol, their role as "transformative" is sketchy, because you have to remember that this is using its legal definition, not its colloquial definition. > If I'm on the jury Fortunately, for this kind of question, the jury isn't going to be involved in determining fair use, so it doesn't matter what you think.
jeffwask 2y ago
Sure, because Meta certainly followed copyright law to the letter when they torrented thousands of copyrighted books from hundreds of published and known authors to train Lama. Forgive me if I doubt a text disclaimer on the page will slow them down.
dspillett 2y ago
Unfortunately copyright is no limit to these companies. Meta is stating in court that knowingly downloading pirated content is perfectly fine (ref https://news.ycombinator.com/item?id=43125840 https://news.ycombinator.com/item?id=43125840) so they for one would have absolutely no issue completely ignoring your copyright notice and stated licensing costs. Good luck affording a legal team to try force them to pay attention. Copyright is something for them to beat us with, not the other way around, apparently.
kerkeslager 2y ago
This is pretty naive. The only reason copyright is so strong in the US is that there are big players (Disney, Elsevier) who benefit from it. But gig tech is much bigger, and LLMs have created a situation where big tech has a vested interest in eroding copyright law. Both sides are gearing up for a war in the court systems, and it's definitely not a given who will win. But, if you try to enter the fray as an individual or small company, you definitely aren't going to win.

ToucanLoucan 2y ago

Yet another entry in the long and shameful history of Silicon Valley abusing the public square for their own profit (or in this case, fantasies of profit) and the rest of us just have to learn to live with it because the justice system simply will not even try and give us recourse. Move fast and break things apparently has a bonus clause for the things you break not being your responsibility to fix.

Analemma_ 2y ago
I don't think the justice system is the one to blame here. Right up until LLMs and their huge datamining operations appeared, everyone in tech was strongly for unrestricted scraping. Everybody here cheered the LinkedIn decision [0], saying "it's on the public web: if you didn't want it to be scraped, you should've put it behind authentication". LLMs change nothing about the legal landscape, they've just convinced everyone on an emotional level that unrestricted scraping is no longer an automatic good. It's not the justice system's job to react to such vibe shifts, the laws themselves have to be changed. [0]: https://news.ycombinator.com/item?id=21241395 https://news.ycombinator.com/item?id=21241395
- ToucanLoucan 2y ago
  I'm not talking about the ethics of scraping itself. I think scraping is fine from an ethics perspective for exactly that reason. I think LLM companies, however, are scraping ineptly and with poorly implemented tooling, which is causing problems for the websites they're targeting, and that sucks ass and they should be held liable. On the legal end though, I do think there's a few things that should be done: * Scrapers should be CLEARLY, and CORRECTLY identified as what they are, and who they are being dispatched from. Changing user agents to get around blocks should not be permitted, ever. If you only get a certain amount of content or a certain subset of pages when you identify as a scraper, that is a choice the website operator is making and it should be respected. * Scrapers MUST OBEY robots.txt. We didn't create that for a fun hacker weekend. It's an important technical component of how we organize websites and how we want them crawled, if we want them crawled. It should be the first stop for any scraper on any website, and again, it should be respected. * Scrapers should always meter their traffic with respect to the website owner. Pounding an entire website's library of content request after request with only milliseconds between is, to put it bluntly, being a fucking asshole. And not just to the owner, but to anyone else attempting to use the site at the time. If a website operator configures their site incorrectly and pages they don't want scraped are, or pages they do want scraped aren't, then that is on them and they need to fix that. It is not in the scraper's purview to end-run around that configuration to "be real sure" they got everything they were meant to, and it's especially not that to get things the web operator has explicitly tried to not let the scraper have. And yes, all of these things should be legally actionable, with financial penalties attached and for serial offenders, we should have a registry of scraper bots that we disallow entirely because they are acting in bad faith.

Aurornis 2y ago

Rate limiting is the first step before cutting everything off behind forced logins. > This practice started with larger websites, ones that already had protection from malicious usage like denial-of-service and abuse in the form of services like Cloudflare or Fastly FYI Cloudflare has a very usable free tier that’s easy to set up. It’s not limited to large websites.

blibble 2y ago
I get the feeling that I'm going to read a blog post in a few years telling us that the CDN companies have been selling everything pulled through their cache to the AI companies since 2022
- nottorp 2y ago
  And even if they don't, is everything depending on Cloudflare to stay online a good thing?
  - sshine 2y ago
    It’s a terrible thing. Cloudflare is the company I hate the most: I think (what I know of) their tech is done right, and they’re just too big to put my eggs in their basket.
  - koakuma-chan 2y ago
    Why is nobody building a better product?
    - nisegami 2y ago
      Because Cloudflare's products are very good. The issues people have with Cloudflare are more philosophical and won't be resolved by building a better product. GP's concerns would best be rectified by dozens of commodity-like competitors of similar quality.
      - 2y ago

JKCalhoun 2y ago

For some reason I am not really moved by a lot of the hand wringing I am seeing lately. It's a not a binary thing to me: LLMs are not god, but even without AGI, they have proven wildly useful to me. Calling them "shitty chat bots" doesn't sway me. Further I have always assumed that everything that I post to the web is publicly accessible to everyone/everything. We lost any battle we thought we could wage some 2+ decades ago when web crawlers started hoovering up data from our sites.

gjsman-1000 2y ago
I agree even if I have mixed feelings on it. To me this feels almost like the news complaining that they want a "link tax." Weren't their headlines and summaries used? It seems inconsistent to somehow say that AI and scraping is not okay; but that news companies should also not be entitled to their link tax. It's okay to index, but not that kind of index.
- theamk 2y ago
  It seems pretty cut-and-dry to me: the website owner should opt-out if they don't like the deal (being indexed in this case). In the "link tax" case, there were plenty of trivial ways to opt out of headline usage - robots.txt, http headers, http tags. The problem was newspapers did not want to opt out (as they were benefiting from Google themselves), so they wanted a 3rd option. Which was pretty stupid of course - if you don't like the deal, don't take it; suing the offering party for a better deal is not a good long-term strategy. In the AI case, there is no opt-out. All those websites already indicated they want to opt-out via robots.txt, but the AI companies ignore robots.txt, change user-agent, fake IPs, and so on - do the things that are normally done by shady malwar-ish services rather than multi-billion-dollar companies. It really bothers me when people don't see the difference between those two cases.
prosody 2y ago
This article isn’t about that. It’s about the externalized costs that LLM companies are pushing onto webmasters because of their aggressive scraping. It’s one thing to believe that LLMs are a good thing, it’s another thing to believe that individuals and cooperative groups that run small internet services ought to be the ones to pay for that good.

cxr 2y ago

Perversely, this submission is essentially blogspam. The article linked in the second paragraph, to which this "1 minute" read adds almost nothing of value, is the main story: <https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies/ https://thelibre.news/foss-infrastructure-is-under-attack-by...> 394 comments. 645 points. Submitted 3 hours ago: <https://news.ycombinator.com/item?id=43422413 https://news.ycombinator.com/item?id=43422413>

btown 2y ago
But also ironically, it's almost heartwarming these days to see blogspam that's not machine-generated! A real live human cared enough about an article to write a brief (perhaps only barely substantial, but at least handwritten) take on it! It's reminiscent, perhaps, of the feel and motivation for Tumblr reblogs - and Tumblr continues to be vibrant by virtue of this culture: https://www.tumblr.com/engineering/189455858864/how-reblogs-work https://www.tumblr.com/engineering/189455858864/how-reblogs-... (2019) Now, is driving attention and reputation to their site (in the broadest senses) part of a blogspammer/reblogger's motivation? Absolutely! But should we be concerned about rewarding their act of curation, as long as there is at least some level of genuine curation intent? A world where that answer is categorically "no" would be antithetical, I think, to the concept of the participatory web.
- dkkergoog 2y ago
  "heartwarming ... To see blogspam" the internet was a mistake
  - wongarsu 2y ago
    The internet was great, everything we did with it in the last 20 years with it was the mistake. Collimating in a comment that blogspam can now be one of the positive notes in the hellscape we are building. A very useful hellscape though, for all its flaws
    - 38 2y ago

0x1ceb00da 2y ago

Do I need to be worried about my bill if I've rented a simple EC2 instance without any fancy autoscaling stuff?

simonw 2y ago
Probably not. Keep an eye on bandwidth usage since you'll be charged for that but you would need to attract an incredible amount of bot traffic for that to add up to anything meaningful. The thing to watch out for is platforms like Vercel or Google Cloud Run where you get charged more for compute if you attract crawlers, potentially unbounded (make sure to set up spending limits if you can.)

hubraumhugo 2y ago

We should try separating good bots from bad bots: Good bots: search engine crawlers that help users find relevant information. These bots have been around since the early days of the internet and generally follow established best practices like robots.txt and rate limits. AI agents like OpenAI's Operator or Anthopic's Computer Use probably also fit into that bucket as they are offering useful automation without negative side effects. Bad bots: bots that have a negative affect website owners by causing higher costs, spam, or downtime (automated account creation, ad fraud, or DDoS). AI crawlers fit into that bucket as they disregard robots.txt and spoof user agent. They are creating a lot of headaches for developers responsible for maintaining heavily crawled sites. AI companies don't seem to care about any crawling best practices that the industry has developed over the past two decades. So the actual question is how good bots and humans can coexist on the web while we protect websites against abusive AI crawlers. It currently feels like an arms race without a winner.

jsheard 2y ago
Discriminating search engine bots is pretty straightforward, the big names provide bulletproof methods to validate whether a client claiming to be their bot is really their bot. It'll be an uphill battle for new search engines if everyone only trusts Googlebot and Bingbot though. https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot https://developers.google.com/search/docs/crawling-indexing/... https://www.bing.com/webmasters/help/verifying-that-bingbot-is-bingbot-3905dc26 https://www.bing.com/webmasters/help/verifying-that-bingbot-...

napolux 2y ago

> I suggest everyone that uses cloud infrastructure for hosting set-up a billing limit to avoid an unexpected bill in case they're caught in the cross-hairs of a negligent company. All the abusers anonymize their usage at this point, so good luck trying to get compensated for damages. This is scary

jsheard 2y ago
What's scarier is that most of the big clouds don't even let you set up a billing limit.

charcircuit 2y ago

Crawlers visiting every page on your website is not the main problem with the unauthenticated web. The amount of spam that happens when you let people freely post is a much bigger problem.

isoprophlex 2y ago

Everyone is (rightfully) outraged, but this is essentially nothing new. Asshat capitalists have been externalizing the costs of their asshat moneymaking schemes on the little guy since approximately forever. Deregulation is ultimately antithetical to our personal freedom. I just hope the spirit of the internet that I grew up with can be rescued, or reincarnated somehow...

MontgomeryPy 2y ago

Could an answer here be for smaller websites to convert their sites into chatbots which could prevent AI scrapers from slurping up all their content/drive up their hosting costs?

cwmma 2y ago
no

anovikov 2y ago

Pretty soon virtually everything will be paywalled. Ironically, it will provide us with a good metric that lets us find out whether AGI has arrive or not: when it does, paywalling will stop working because AGI could derive more value from accessing things and will thus outbid us.

hugs 2y ago

I might be naive, but I think it's time we seriously start implementing "HTTP status code 402: Payment Required" across the board. "L402" is an interesting proposal. Paying a fraction of a penny per request. https://github.com/l402-protocol/l402 https://github.com/l402-protocol/l402

rambambram 2y ago
I stumbled upon this status code last year - had never heard of it before - and I bookmarked it and then forgot about it. Thanks for the reminder.
cwmma 2y ago
this is basically what they are doing, but instead of charging actual money they are making visitors spin the CPU ideally in a proof of work problem, which has the same outcome from the crawlers perspective.
- woah 2y ago
  This has existed for decades. The proof of CPU work is called "frontend frameworks"
- fewsats 2y ago
  I've talked with tons of publishers and all say the same thing: "Hey, we'd happily give these companies clean data if they just paid us instead of building these scrapers." I think there is a psychological aspect that made micropayments never work for humans but machines may be better suited for it.
SoftTalker 2y ago
This is ultimately the answer. If something has value, users should pay for it. We haven't had a good way to do that on the web, so it has resulted in the complete shitshow that most websites are.

parliament32 2y ago

Linked in the article that this article links to is a project I found interesting for combatting this problem, a (non-crypto) proof-of-work challenge for new visitors https://github.com/TecharoHQ/anubis https://github.com/TecharoHQ/anubis Looks like the GNOME Gitlab instance implements it: https://gitlab.gnome.org/GNOME https://gitlab.gnome.org/GNOME

kh_hk 2y ago
For targeted scrapes, isn't proof of work trivial to bypass? 1. headless browser 2. get cookie 3. use cookie on subsequent plain requests
- parliament32 2y ago
  It doesn't sound like the scrapers are that smart yet, but when they get there, presumably you'd just lower the cookie lifetime until the requests are down to an acceptable level. It takes a split-second in my browser so it shouldn't interfere much for human visitors.

kmeisthax 2y ago

> How long until scrapers start hammering Mastodon servers? Mastodon has AUTHORIZED_FETCH and DISALLOW_UNAUTHENTICATED_API_ACCESS which would at least stop these very naive scrapers from getting any data. Smarter scrapers could actually pretend to speak enough ActivityPub to scrape servers, though.

woah 2y ago

If you don't want someone to access your website, don't put it online

fewsats 2y ago

There's a real economic problem here: when someone scrapes your site, you're literally paying for them to use your stuff. That's messed up (and not sustainable) It seems like a good fit for micropayments. They never took off with people but machines may be better suited for them. L402 can help here. https://l402.org https://l402.org

fewsats 2y ago
The other obvious solution is a "web of trust" where Cloudflare just tells you "this request goes in, this one goes out". I think the paying approach is superior (after all you make money out of people using your service) but Cloudflare is a straight forward/simpler one.
tqwhite 2y ago
Aren't you paying for me to use the site, too? Or Google? Isn't the point of paying for a web hosting service to distribute information?
- fewsats 2y ago
  Yes, but there is a "free lunch" problem. I can run a script that hits your page costing you X at a fraction of the cost for me (the user) *Edit: typo
  - tqwhite 2y ago
    I think the whole internet is a free lunch problem as far as that goes. I pay for web hosting because I consider the cost to be worth it to send my fabulous opinions into the ether. The premise of this thread is that somehow the LLM builders are reading too much. I bet it's less than google. I continue to believe, if you don't want everyone in the world to see and use your stuff, don't put it on the internet.

renegat0x0 2y ago

To be honest I feel that web2 is overrated. Most of content, blogs could be static sites. For mastodon, forums I think user validation is ok and a good way to go.

paranoidroid 2y ago

[dead]