16 ms·
If you forget about the LLM aspect, and simply build a product out of (legally) scraped NYT articles, is that fair use? Let's say I host these, offer some inde
by rich_sasha 3y ago
If you forget about the LLM aspect, and simply build a product out of (legally) scraped NYT articles, is that fair use?
Let's say I host these, offer some indexing on it, and rewrite articles. Something like, summarise all articles on US-UK relationships over past 5 years. I charge money for it, and all I pay NYT is a monthly subscription fee. To keep things simple, let's say I never regurgitate chunks of verbatim NYT articles, maybe quite short snippets.
Is that fair use? IANAL, but doesn't sound like it. Typically I can't take a personal "tier" of a product and charge 3rd parties for derivatives of it. Say like VS Code.
A sibling comment mentions search engines. I think there's a big difference. A search engine doesn't replace the source, not at all. Rather it points me at it, and offers me the opportunity to pay for the article. Whereas either this or an LLM uses NYT content as an alternative to actually paying for an NYT subscription.
But then what do I know...
- logicchains 3y ago> Something like, summarise all articles on US-UK relationships over past 5 years. I charge money for it, and all I pay NYT is a monthly subscription fee. >Is that fair use? IANAL, but doesn't sound like it. If you pay someone to do the summarisation for you, then you publish the content and charge a fee for it, you're the one liable, not the person you paid to summarise it for you. Similarly if you ask GPT to do it for you, then publish it, you're liable for what you publish; GPT is just a summarisation tool.
- rich_sasha 3y agoThat's not the example. Here I proactively scrape NYT, summarise articles for a fee and sell that as a service. It's not people coming to me with some articles to summarise, and maybe then publishing it online. At some level it becomes a subversion of NYTs fees. First, say I subscribe and simply host the articles verbatim, for a fee. Clearly, that's not right. Suppose I change some spelling or word order, or use a synonym or two. That's still not ok. And if I substantially paraphrase the articles? I guess this is the relevant case. This is kind of what LLMs do. And also feels like not fair use.
- logicchains 3y ago>That's not the example. Here I proactively scrape NYT, summarise articles for a fee and sell that as a service. It's not people coming to me with some articles to summarise, and maybe then publishing it online. That's not what OpenAI is doing; it's not selling summarised articles as a service. Your example is a false equivalence. >This is kind of what LLMs do. And also feels like not fair use An LLM doesn't do this unless you ask it to. And if you then take that output and publish it as your own, you're breaching the copyright, not OpenAI.
- heavyset_go 3y ago> An LLM doesn't do this unless you ask it to. And if you then take that output and publish it as your own, you're breaching the copyright, not OpenAI. In this case, OpenAI is violating copyright by modifying, reproducing and distributing copyrighted content to its customer.
- 8note 3y agoHow far is this from what reddit does? I read a NYT article, then summarize it into a link title for reddit. Reddit then republishes the summary to all of its users.
- tsimionescu 3y agoThat's not true at all. If you pay someone to copy NYT articles for you verbatim, and then they give the copies to you, and then you publish them online, then you've both violated the copyright. You are never allowed to make copies of copyrighted works, even for private deals (making such copies for purely personal use, such as archival, falls under fair use - but you can't build a service out of that). So, if the summaries are derived works and not covered by fair use, then both you and the summarizee are separately breaking the NYT's copyrights. Otherwise, if this is covered by fair use, then you are both in the clear. Finally, GPT is not "a summarization tool" in this case. If you provide a copy of a NYT article as a prompt and then ask for summarization, then yes, it is clear that GPT is not doing anything wrong, even if it spits out the exact same text. But if you simply ask for a summary of a specific article by, say, just name and date, and you get a copy of it, it's clear that GPT is storing the original data in some way, and thus it has copied the NYT's protected works without permission.
- logicchains 3y ago>But if you simply ask for a summary of a specific article by, say, just name and date, and you get a copy of it, it's clear that GPT is storing the original data in some way, and thus it has copied the NYT's protected works without permission. In this particular case they were using it via Bing, which actively did a HTTP request to the particular article to extract the content. So GPT hadn't memorised it verbatim, instead it fetched it, much like a human using a search engine would.
- tsimionescu 3y agoThe article states that they used it initially through ChatGPT, but that seems to have been fixed in the meantime, at least for the very simplistic queries that used to work ("the first paragraph of the Carl Zimmer article on old DNA" in ChatGPT used to return the exact data from NYT, and "next paragraph" could then be used to get the following ones). Even if this has been fixed, it still proves that ChatGPT encodes exact copies of NYT articles in its weights, which may be a violation in itself, even if it is prevented from returning them directly. Especially if they ever started distributing the trained model. Additionally, even the use through Copilot is very debatable. They are not returning the NYT link, which requires a subscription, they are returning the contents of it even to non-subscribers. And they are doing this in a commercial product, not a non profit like the Internet Archive, which has some arguments for fair use.
- heavyset_go 3y agoAnother factor to consider is that neural nets can function as lossy compression, which becomes extremely evident when using models that are overfit. Sometimes they're so overfit that the compression isn't even lossy, and the data is encoded verbatim in the NN.
- TeMPOraL 3y agoYes, but this then hits against learning/understanding and compression being fundamentally the same thing. I can't think of a better way to argue in favor of "it's fine if human does it, therefore it's fine if LLM does it", than from the "lossy compression" angle.
- RandomLensman 3y agoWe can have different rules for humans than for machines. In fact, that happens all the time.
- cyborgx7 3y agoIt's fine for a human to remember it. It's not fine for a human redistribute it for money (legally speaking). That's copyright infringement.
- Robotbeat 3y agoCorrect, just like it’s infringement to reproduce an article from memory using pen and paper intentionally. The person deciding to do that bears responsibility. OpenAI would be liable IFF they were intentionally facilitating that, instead of it being an undesired artifact from overfitting.
- cyborgx7 3y agoI'm pretty sure if you reproduce a work from memory by accident, because you didn't notice your subconscious had just stored the entire article and is now reproducing it word for word, you'd still be guilty of copyright infringement.
- TeMPOraL 3y ago> Typically I can't take a personal "tier" of a product and charge 3rd parties for derivatives of it. Say like VS Code. Can't you, though? I'd thought in general, it's a very important for the market to be able to do just that, otherwise everything gets gummed up in webs of exclusive contractual dependencies between established companies.
- rich_sasha 3y agoAs I say, I don't really know. But then, this is exactly how SaaS licensing works. There may even be a free personal tier, where you can't sell products based on it, and a professional tier which may be very expensive indeed. Typically providers of online databases go to some effort to stop people from sharing logins. Even from that point or view, I can imagine scraping articles and providing paraphrases of it for a fee is fishy. All I'm saying, to some people it's obvious that the whole LLM on scraped Internet is fair use, to me it is not obvious.
- px43 3y agoFrom what I can tell, this has nothing to do with LLMs at all. In the example in the article, the user is asking Bing to go fetch the contents of an article directly from the website, and print it out, which it dutifully does. Seems like the "problem" is that NYT etc gives privileged access to search engines for indexing their content, but then get upset when snippets of the indexed content is being shown to users without the users having to fight the paywall or whatever. This article also claims that the screenshot is coming from ChatGPT when it clearly is not.
- rich_sasha 3y agoI suppose that's a relatively easy thing to fix, technically. It proves, however, that th underlying LLM is trained on copyrighted data. I'm not sure the problem goes away simply if the LLM in question (or any other one) gets some "no verbose regurgitation" filter.
- exitb 3y agoIn that case, the language model calls a search function and just repeats the result out its conversation context, not its training data. With that in mind it's not clear why it's ok for Bing itself to quote the source, but it stops being ok, when a chatbot does it.
- 3y ago
- 6stringmerc 3y ago[flagged]
- dang 3y agoCan you please make your substantive points thoughtfully and without snark or putdowns? Edit: it looks like you've unfortunately been breaking the site guidelines quite a bit lately. Can you please review them and stick to the intended use of the site? We'd appreciate it. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- sagarm 3y agoIsn't it totally normal to write articles / blog posts that effectively summarize, and often quote from, news articles?
- laborcontract 3y agoMy impression is that it’s not necessarily legal, but going after bloggers and proving damages based is just a huge waste of their time. OpenAI came by with their fat stack of funding and changed that.
- facu17y 3y agoIt is legal. Fair use. People have been doing it for ages. Almost every article you've ever read has some fair use of another article, book or news item, etc.
- lacrimacida 3y agoWhen it becomes a service where you make money but the source doesn’t is it still fair use?
- whythre 3y agoYeah. No one is out there suing the shit out of cliff notes because they published a summary of Catcher in the Rye.
- tkz1312 3y agothey might if cliff notes starting copy pasting parts of the source into their articles and passing it off as original writing though :)
- throwuwu 3y agoThe Tolkien estate should get busy suing all the fantasy writers, comic artists, game developers and board and card game companies. Lots of cash there.
- jojobas 3y agoCan you read all of NYT and other things, and answer others' questions based on your knowledge? I'd imagine you can. I'm afraid you can't sidestep the question whether an LLM is more like a person who's read a lot or an archive/index.
- papruapap 3y agoUsing similar logic NYT should pay all actors involved in their articles.
- BlueTemplar 3y agoThere's nothing wrong with scraping openly available data (including data openly available by mistake, as long as you are not aware of it, see the Bluetouff affair). So the demand to destroy those databases seems very dubious to me. Of course later violating fair use is another issue.
- oefrha 3y agoThe real answer is it totally depends on whether your product grows to $10,000,000,000, and whether you pays part of it back. Search engines pay with referral traffic.
- tauntz 3y ago> Is that fair use? As always, the answer is.. "it depends". I guess it depends mostly on the jurisdiction that applies to you. "Fair use" can have rather different legal meaning (or not exist at all) in different countries.
- madeofpalk 3y agoAlso “fair use” does not use/define precedent - each case is assessed individually which really can be a flip of the coin.
- hn_acker 3y agoFair use is specific to the US, as far as I'm aware. Moreover, Congress had to codify fair use (turn fair use common into statutory law in the form of 17 U.S. Code § 107) in order to make copyright statutes compatible with the First Amendment. Most other countries don't have freedom of expression and freedom of the press, so copyright law in a different country usually lacks a unifying exception test like fair use to supplement the specific enumerated exceptions.
- anticensor 3y ago> Most other countries don't have freedom of expression and freedom of the press This is demonstrably wrong. Many countries have both freedoms, albeit some have less strong protection than others.
- gls2ro 3y agoI have to point out that you did not refute the initial assertion. They said "Most other countries" and you replied with "Many countries" "Many" does not necessary include "most" but "most" does include "many".
- defrost 3y agoMost female Vice Presidents of the USofA would not agree that that there are many female US Vice Presidents.
- qeternity 3y ago> Typically I can't take a personal "tier" of a product and charge 3rd parties for derivatives of it. I think you’re confusing terms of service and copyright. IANAL but what you describe sounds exactly like fair use to me, irrespective of how much you are paying NYT.
- ks2048 3y agoIt would be nice to have a nice principled answer to this, but unfortunately, in our world, the answer is probably: if you start making LOTS of money doing this, they will come after you.
- doctorpangloss 3y agoThe best example is that sport scores, names and stats are not copyrightable by settled case law; however, you still have to go to the NBA and players union if you want to make a fantasy basketball game that has stats or names.
- bnralt 3y agoAs someone pointed out, plenty of blogs made money off of doing just that. Many people go to Reddit to read news article summaries (and often a comment just pastes the whole article verbatim), instead of paying a site like the New York Times. Twitter and other social media sites are full of people summarizing articles from the New York Times. Any late breaking news article from Wikipedia is going to be mostly summarizing information from reporters. I think people severely underestimate how much they've grown accustomed to this information being freely available. It's easy to say "Well it shouldn't be available with ChatGPT," but if we actually put everything back behind a paywall and stopped people from doing things like writing blogs or newsletters that summarize the news, people here would get angry very fast.
- superb-owl 3y ago> A sibling comment mentions search engines. I think there's a big difference. A search engine doesn't replace the source, not at all. Google has been accused for years of replacing sources with their "One Box"--the big answers at the top of the page, which are usually pulled from or corroborated by search results. They don't want you to leave the search results page (where the ads are).
- paxys 3y agoGoogle is very careful to license all the content that shows up in that interface. They even pay Wikipedia, despite legally not needing to at all.
- Kim_Bruning 3y agoWhile paying for CC content is not 100% anathema, it'd be pretty weird if that was what they were doing. Instead, I think they're paying for this: https://enterprise.wikimedia.com/ https://enterprise.wikimedia.com/
- carlosdp 3y agoWhat you described is entirely fair use, actually. Not only that, look at a few news articles from Tier 2 and down publications, and you'll realize that almost all of them are directly sourced from NYT and others. They'll say "so and so happened, according to The Times" (and usually link the article there)
- hn_throwaway_99 3y ago> What you described is entirely fair use, actually Just like during the pandemic how everyone became an epidemiologist, suddenly everyone's a copyright lawyer. I'll just dispute your assertion by saying: 1. Questions of fair use are famously gray, and anyone who declares something as "entirely fair use", with no caveats, is nearly always wrong except for the must obvious cases, which the given example is most definitely not. A judge has wide latitude in determining fair use. 2. People should familiarize themselves with the four factors of fair use determination. In particular, if a work is purely derivative of a source work and substantially negatively impacts the market for the original work, it's very likely to not be considered fair use. A great overview is https://fairuse.stanford.edu/overview/fair-use/four-factors/ https://fairuse.stanford.edu/overview/fair-use/four-factors/
- NegativeK 3y ago> suddenly everyone's a copyright lawyer Roll back 20+ years ago on Slashdot and you'll see the exact same thing. Copyright has been a hot button issue on the internet for decades. People end up thinking (rightly or wrongly) that they understand it without being a lawyer.
- Teever 3y agoOne of my biggest gripes is a somewhat adjacent issue where everyone thinks they're an American copyright lawyer and that American copyright law is universal. It's very possible that the example provided above is an example of fair use in some country, and that the website offering that service could be hosted there.
- 3y ago
- brookst 3y agoHow about if you read the paper every day and write opinion pieces about world events? Fair use?
- deleted 3y ago[deleted]
- modeless 3y ago> To keep things simple, let's say I never regurgitate chunks of verbatim NYT articles, maybe quite short snippets. You just described Google. When you think about it, it's surprising that Google is legal. However, it is well established that what Google does is perfectly legal. Remember that internally Google keeps and uses complete verbatim copies of every web page they index. Yes, Google offers a link to the source. If OpenAI did the same, even if only 0.1% of people clicked on the links and NYTimes hardly got any revenue from it, would that make it legal in your eyes? What if they implemented a system that detected when it was outputting a verbatim copy of something and simply paraphrased it? NYTimes clearly doesn't have copyright on paraphrased versions of their articles. I think it would be pretty silly if the government forced them to do that as it wouldn't make any practical difference to anyone.
- inetknght 3y ago> However, it is well established that what Google does is perfectly legal. Google has a wide range of products and shakedowns. Not all of them are "perfectly" legal: Google is being challenged in court over some of their shakedowns and products practices.
- modeless 3y agoI am clearly talking about the web search engine in the context of copyright. Other products or legal concerns like antitrust are completely irrelevant here.
- graeme 3y agoAny publisher can opt out of google. Publisher also have substantial control over titles and snippets shown in google, whether an article appears in google news, etc Paraphrasing is also known as cloning and is often a copyright violation
- modeless 3y agoCopyright law doesn't mention opt outs or search engine snippet controls. It's not clear to me that robots.txt is the singular thing that makes Google legal. In US copyright law facts cannot be copyrighted, so copyright on factual content like newspaper articles is limited. Simply replacing a few words wouldn't work, but I am certain that GPT-4 is capable of paraphrasing factual content at a level that would not be considered infringement if a human did it.
- oh_sigh 3y agoI agree with your IANAL take, but what about a situation with an extra level of indirection? So the service never reads actual NYT articles, but only reads blog/forum posts about NYT articles, and derives what is in the article from conversations about the article by people who have read it. Is that legal now?
- JCharante 3y agoBut is it legal for me to read the NY Times about a war, and then charge people to interview me as an "expert"?
- charcircuit 3y agoThis analogy fails to capture the transformative nature of these models. Hosting a derivative work that is also a news article is not transformative. Hosting a next word completer is very different than a news article and can't be used as a substitute.
- deleted 3y ago[deleted]
- amelius 3y ago> If you forget about the LLM aspect, and simply build a product out of (legally) scraped NYT articles, is that fair use? That's not a good question. If I look out of my window and see my neighbor go to the shop, that's fine. If I use cameras and track everybody I see on the street and put them in a database, then that's problematic and illegal in many places. Logic does not necessarily apply when scaling is involved.
- oivey 3y agoIt is a good question with a simple answer: no.
- amelius 3y agoIt depends. Google built a product out of scraping content (Google Search). But what I'm saying is that answering the question does not allow you to deduce anything about your rights; that's what I mean by "not a good question".
- rafabulsing 3y agoIt can allow you to deduce something, depending on the answer. If we want to establish whether scenario A is fair use or not, and we all agree that A is "worse" (regarding fair use status) than some other scenario B, then if we also agree that B is not fair use, A by definition isn't either. The opposite is not true, of course: B being fair use does not imply that A has to be as well. I find that kind of upper/lower bound logic can be pretty useful and I think it's what the parent comment was trying to do. On a related note, that same logic is why I think Godwin's law can be a bit misapplied now and then. Sometimes bringing up nazis/Hitler can be useful to establish some ground truth in a debate (instead of just a way to imply your opponent is actually a bad person, or, possibly, an actual nazi themselves). E.g. a conversation on the morality of violence is vastly different depending on whether you agree that violence against nazis is ok or not.
- oivey 3y ago
- paulddraper 3y ago> Something like, summarize all articles on US-UK relationships over past 5 years. So like....Wikipedia, CliffNotes, encyclopedias, etc? None of these pay royalties to original.
- canttosme 3y agohttps://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn : > Implications: The Ninth Circuit's declaration that selectively banning potential competitors from accessing and using data that is publicly available can be considered unfair competition under California law may have large implication for antitrust law. [citation needed] > Other countries with laws to prevent monopolistic practices or anti-trust laws may also see similar disputes and prospectively judgements hailing commercial use of publicly accessible information. While there is global precedence by virtue of large companies such as Thomson Reuters, Bloomberg or Google [or LexisNexis or Westlaw] effectively using web-scraping or crawling to aggregate information from disparate sources across the web, fundamentally the judgement by Ninth Circuit fortifies the lack of enforceability of browse-wrap agreements over conduct of trade using publicly available information.
- object-a 3y agoIANAL but aren't the key terms there "selectively banning" and "publicly available"? NYT articles are largely behind a paywall for everyone. That means they are not publicly available, and a competitor who was blocked from accessing or reproducing that content without a license would not be "selectively banned"
- urbandw311er 3y agoI believe that part of the law suit contends that the content wasn’t able to be scraped “legally” as you put it. Instead they show that ChatGPT will regurgitate verbatim excerpts from articles that are behind the paywall.
- deleted 3y ago[deleted]
- 1vuio0pswjnm7 3y agoIf there is payment then usually there is an agreement. An agreement can limit fair use. Can the NYT, via an agreement, e.g., "Terms of Use", limit what the subscriber does with the articles. There is not much precedent that suggests otherwise. Consider the analogy from libraries that want to do data mining. "Unfortunately, in licenses for digital scholarly content the majority of content acquired by research libraries publishers often include terms that prohibit certain uses that would otherwise be allowable under the Copyright Act. For instance, licenses may require libraries or individual researchers to negotiate for otherwise lawful activities, such as text and data mining, and to pay exorbitant fees on top of the cost of the content itself. While new regulations allow researchers to circumvent technological protection measures to access copyrighted materials, licenses for that content may include terms that explicitly prohibit this circumvention. In many cases, these activities might actually increase the value of published material; for instance, if a data-mining project yields new knowledge about a topic covered in a journal, it may very well spark new interest in that journals content. Libraries and publishers have often assumed that license terms that restrict copyright exceptions are enforceable under state contract law. There is, however, surprisingly little case law on this point." https://www.arl.org/wp-content/uploads/2022/07/Copyright-and-Contracts-Paper.pdf https://www.arl.org/wp-content/uploads/2022/07/Copyright-and... Putting some string in a robots.txt to try to stop data collection is an amusing "solution". Should copyright owners have "Terms of Use" that limit usage for commercial "AI" purposes.