5 ms·

Zuckerberg appeared to know Llama trained on Libgen

57 points by arresin 2y ago

neuroelectron 2y ago
Which is exactly why the want to shut it down, preventing competition in the Ai space.
ripped_britches 2y ago
Sometimes I just feel like these people overestimate how much they are actually owed from these training runs. It’s trained on 15T tokens. So how many did you provide that were genuinely novel? And how much money do you want? Like $5 from OpenAI? And $0 from meta since it’s open source? I personally hope we can all get on the same team with AI and treat its advancement as scientific research for the betterment of humanity.
- blibble 2y ago
  well US copyright law statutory damages are $30,000 per work infringed, and $150,000 if done deliberately so I think $150,000 per copyrighted work ingested is fair
- papercrane 2y ago
  Assuming the works have registered with the copyright office they're eligible for statutory damages. The range for that is huge though, it can be in the hundreds of dollars per work, or if the infringement is shown to be wilful then a judge can award up to $150,000 per work.
  - ripped_britches 2y ago
    Fair point, seems willful here
- logifail 2y ago
  > It’s trained on 15T tokens. So how many did you provide that were genuinely novel? Are we suggesting that we should ditch creators' rights and instead value intellectual property along the lines of "I should be able to copy all your stuff as long as long as I copy lots of other stuff too, and give it all away for free or almost free...?"
  - ripped_britches 2y ago
    No not at all, just that their damages are going to be measurably fairly low
  - llm_trw 2y ago
    The question if training an llm is fair use is one that will have to be answered by the courts.
    - yencabulator 2y ago
      The point (often) is to stop the practice, not to ask money for it. Like, you get fined for speeding, but if you keep speeding you'll get you're license revoked, and if you keep driving after that you get jail time. The payment required is punitive, but the point is to stop you.
      - llm_trw 2y ago
        Again, that an opinion that will need to be tested in the courts. You will have to explain why Hunter Thompson copying every word of every Hemingway novel isn't copyright infringement, but a computer doing the same is.
        yencabulator 2y ago
        Three things: 1. Humans are not machines; arguments saying because a human can learn LLMs must be allowed to copy is not interesting. 2. Did Thompson publish the work? It sounds like you're referring to an activity Thompson did in private, to improve his skills as an author. Meanwhile, lawsuits are alleging that LLM services reproduce copyrighted materials. 3. What can be fair use at small scale is no longer fair use at large scale.
        llm_trw 2y ago
        If you think that pretraining is copying then your opinions are irrelevant.
- Hammershaft 2y ago
  Hard to treat llms training aon your data at your expense as research for the betterment of humanity when it is specifically the private company who is imposing that cost on you that profits.
- lm28469 2y ago
  There is absolutely no logical pathway between the current flavor of hardcore free for all individualistic capitalism and what you describe here
- fourside 2y ago
  Does this goes both ways? Can I infringe of Disney’s IP on the grounds that their stories are so derivative that they aren’t actually that new? The betterment of humanity seems to involve some parties making a ton of money while the people who provided the data apparently just need to be grateful.
  - jncfhnb 2y ago
    You can absolutely use the story frameworks that Disney has done. You cannot make a story featuring Simba.
    - giantrobot 2y ago
      > You cannot make a story featuring Simba. Just use Kimba the White Lion.[0] [0] https://12tomatoes.com/kimba-similarity-lion-king/ https://12tomatoes.com/kimba-similarity-lion-king/
- skulk 2y ago
  > I personally hope we can all get on the same team with AI and treat its advancement as scientific research for the betterment of humanity. s/AI/capital/. It's painfully obvious that this is going to make material conditions worse for most people who use their minds to work instead of their hands. to these people, the "betterment of humanity" is a cruel joke.
  - idunnoman1222 2y ago
    Yeah, just like Google and stack overflow
- latexr 2y ago
  > Sometimes I just feel like these people overestimate how much they are actually owed from these training runs. It’s not about being paid for including their work, it’s about being compensated for having done so without permission. For crying out loud, they went out of their way to remove copyright notices from the pirated work. > It’s trained on 15T tokens. So how many did you provide that were genuinely novel? Then they can just take it out. And go ahead and take out every thing you didn’t have permission to include. What’s that? The model is now significantly worse? Yeah, these things compound. > And how much money do you want? Like $5 from OpenAI? And $0 from meta since it’s open source? No, they would’ve wanted for the work to not have been included without permission in the first place. Do you understand the world you’re advocating for? You’re arguing it’s OK for rich people to do whatever they want if they throw some scraps on the floor for you. Not everything is about money. Unfortunately there’s no other reasonable way (legal and non violent) to punish these infringers. > I personally hope we can all get on the same team with AI and treat its advancement as scientific research for the betterment of humanity. What you’re expressing is “I hope everyone will stop arguing and agree with me”. These moguls care about themselves, it is incredibly naive to believe they give a rat’s ass about “the betterment of humanity”.
- forgetfulness 2y ago
  LLM companies aren't being funded by the hundreds of billions because investors expect science to be advanced by text and image generators. I find it very unlikely that the commodification of knowledge work will be for the betterment humanity, I don't know if people are expecting here that just because the value of more people's labor becomes zero that we will do, what, do away with money? No, it will just mean that fewer people will have the chance to earn the right to use space and resources in a meaningful way.
  - inetknght 2y ago
    > I find it very unlikely that the commodification of knowledge work will be for the betterment humanity There's no law to force it. So of course it won't be. Even if there were a law to force it, how would you enforce it?
- deleted 2y ago
  [deleted]
- wat10000 2y ago
  The normal way to figure this out is to negotiate. We’ll either come to a mutually agreeable amount, or they’ll decide it’s not worth the cost to use my stuff. If I think I deserve $5 from OpenAI, then I’d suggest that, and they’d accept or come back with a counteroffer or tell me I’m nuts and move on. Probably that last one. But for some reason, these companies think they don’t need to bother, and can just use everyone’s stuff. Wait, I phrased that wrong. For a very good reason based on long precedent, these companies know that IP law is a tool to be used by big companies against individuals and sometimes other big companies, but never by individuals against company, so they know they don’t have to bother.
- jsheard 2y ago
  The fact they they were willing to risk significant legal exposure in order to use this dataset suggests it's worth considerably more than 5 dollars to them. Zuck isn't putting his ass on the line for a Big Mac.
- TruffleLabs 2y ago
  Stealing is still illegal
- sensanaty 2y ago
  So if the works they're stealing aren't worth anything, why do they need it so badly?
lvl155 2y ago
Of course he does. Heck most of us in early stages of LLM did the same thing. The data simply did not exists outside Google which is why it’s crazy that Google completely dropped the ball on AI this decade. They had such a huge lead in terms data access.
- logifail 2y ago
  Perhaps the more interesting question would be exactly how did they obtain their copy/copies of Libgen?
  - perfmode 2y ago
    Are you asking for a way to obtain a copy?
    - logifail 2y ago
      Nope, I have no need for any <whisper>further</whisper> copies. I'm more interested in how a for-profit corp decides to obtain a copy for development of a commercial product, and how they execute that ... whether they still have the data, and whether legal know about it :) It's exactly not the kind of thing you can say you "found on a USB stick lying around in the car park"...
      - thfuran 2y ago
        You should've seen the size of it. More of a USB baton really.
        selimthegrim 2y ago
        If Ryobi and DeWalt can make Bluetooth speakers, ASP can get into USB drives.
        logifail 2y ago
        > You should've seen the size of it. More of a USB baton really. <glances at shelf with many, many external USB drives hooked up to a Pi 400> Oh, really? :)
gausswho 2y ago
"The note observed that including the LibGen material would help them reach certain performance benchmarks, and alluded to industry rumors that other AI companies, including OpenAI and Mistral AI, are “using the library for their models.” A new chapter in "information wants to be free". Copyright was always an artificial restriction on human instinct. We now enter an age where piracy is keeping up with the Jones', and those who respect intellectual property choose irrelevance. Prosecution becomes impossible as the laundering grows ever more sophisticated. Adaptation and acceptance painful, but the only path forward .
- bayindirh 2y ago
  Tangential question: What's your take about Code AIs using GPL and source available code in their training sets, and breaching both licenses?
  - gausswho 2y ago
    Eschewing the ethics of such an action, my take would be that it happens (and will happen) regardless of license and no one can stop it unless those utilizing such code leave evidence that they disregarded the license.
    - bayindirh 2y ago
      The models can and will reproduce a lot of their training set given the correct prompt [0]. How you can know you're infringing a license if the model doesn't tell you from which repository it got "inspired" for generating that code? GitHub was working on a feature which supposedly tells you which repository the snippet you just got is copied from, or IOW, which repositories have similar code, sorted by date. Effectively pushing the blame further on you by making you spend the time you just saved by investigating which repository provided the code you just got from "AI". Supposedly, if the license is not friendly, you can delete the snippet and write your own version. :) [0]: https://x.com/docsparse/status/1581461734665367554 https://x.com/docsparse/status/1581461734665367554
savoyard 2y ago
Library Genesis is living up to its name.
keernan 2y ago
[flagged]
- dang 2y ago
  Could you please stop posting unsubstantive comments and flamebait? You've unfortunately been doing it repeatedly. It's not what this site is for, and destroys what it is for. If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.
  - keernan 2y ago
    I'll do my best to tone things down. However I struggle a bit with your observation. The posted article is about one of the wealthiest men in the world likely engaging in the largest copyright violation in history for his own financial benefit. Such antisocial behavior is almost certainly linked to male genetics. Something this same man just recently publicly stated he believes the world needs more, not less, of his anti-social aggressive tendencies. I would argue that you don't like what I have to say and are simply trying to squash my opinions. If you feel the post itself doesn't belong here, then I agree - and to that extent, I also agree to do my best to refrain from commenting upon submissions that themselves do not belong here. But, and here is my biggest objection: I have strong opinions regarding the dangers male genetics pose to world order and the future of humanity. And males unedcuated about the subject react in the manner they know best: aggressive anger; a reaction I am not inclined to back away from (having my own aggressive male genetics).
    - dang 2y ago
      "almost certainly" sounds like quite a stretch to me, but if you wanted to make a substantive point about male genetics that might be ok. Maybe. But it might also be a generic flamewar tangent (which we definitely don't want on HN), so a comment like that would need to be written in a particularly flame-retardant style. Either way, though, a dismissive one-liner like the GP comment is definitely against the site guidelines.
ChrisArchitect 2y ago
[dupe] https://news.ycombinator.com/item?id=42651007 https://news.ycombinator.com/item?id=42651007 https://news.ycombinator.com/item?id=42673628 https://news.ycombinator.com/item?id=42673628
deleted 2y ago
[deleted]

57 points by arresin 2y ago

Which is exactly why the want to shut it down, preventing competition in the Ai space.

Sometimes I just feel like these people overestimate how much they are actually owed from these training runs. It’s trained on 15T tokens. So how many did you provide that were genuinely novel? And how much money do you want? Like $5 from OpenAI? And $0 from meta since it’s open source? I personally hope we can all get on the same team with AI and treat its advancement as scientific research for the betterment of humanity.

blibble 2y ago
well US copyright law statutory damages are $30,000 per work infringed, and $150,000 if done deliberately so I think $150,000 per copyrighted work ingested is fair
papercrane 2y ago
Assuming the works have registered with the copyright office they're eligible for statutory damages. The range for that is huge though, it can be in the hundreds of dollars per work, or if the infringement is shown to be wilful then a judge can award up to $150,000 per work.
- ripped_britches 2y ago
  Fair point, seems willful here
logifail 2y ago
> It’s trained on 15T tokens. So how many did you provide that were genuinely novel? Are we suggesting that we should ditch creators' rights and instead value intellectual property along the lines of "I should be able to copy all your stuff as long as long as I copy lots of other stuff too, and give it all away for free or almost free...?"
- ripped_britches 2y ago
  No not at all, just that their damages are going to be measurably fairly low
- llm_trw 2y ago
  The question if training an llm is fair use is one that will have to be answered by the courts.
  - yencabulator 2y ago
    The point (often) is to stop the practice, not to ask money for it. Like, you get fined for speeding, but if you keep speeding you'll get you're license revoked, and if you keep driving after that you get jail time. The payment required is punitive, but the point is to stop you.
    - llm_trw 2y ago
      Again, that an opinion that will need to be tested in the courts. You will have to explain why Hunter Thompson copying every word of every Hemingway novel isn't copyright infringement, but a computer doing the same is.
      - yencabulator 2y ago
        Three things: 1. Humans are not machines; arguments saying because a human can learn LLMs must be allowed to copy is not interesting. 2. Did Thompson publish the work? It sounds like you're referring to an activity Thompson did in private, to improve his skills as an author. Meanwhile, lawsuits are alleging that LLM services reproduce copyrighted materials. 3. What can be fair use at small scale is no longer fair use at large scale.
        llm_trw 2y ago
        If you think that pretraining is copying then your opinions are irrelevant.
Hammershaft 2y ago
Hard to treat llms training aon your data at your expense as research for the betterment of humanity when it is specifically the private company who is imposing that cost on you that profits.
lm28469 2y ago
There is absolutely no logical pathway between the current flavor of hardcore free for all individualistic capitalism and what you describe here
fourside 2y ago
Does this goes both ways? Can I infringe of Disney’s IP on the grounds that their stories are so derivative that they aren’t actually that new? The betterment of humanity seems to involve some parties making a ton of money while the people who provided the data apparently just need to be grateful.
- jncfhnb 2y ago
  You can absolutely use the story frameworks that Disney has done. You cannot make a story featuring Simba.
  - giantrobot 2y ago
    > You cannot make a story featuring Simba. Just use Kimba the White Lion.[0] [0] https://12tomatoes.com/kimba-similarity-lion-king/ https://12tomatoes.com/kimba-similarity-lion-king/
skulk 2y ago
> I personally hope we can all get on the same team with AI and treat its advancement as scientific research for the betterment of humanity. s/AI/capital/. It's painfully obvious that this is going to make material conditions worse for most people who use their minds to work instead of their hands. to these people, the "betterment of humanity" is a cruel joke.
- idunnoman1222 2y ago
  Yeah, just like Google and stack overflow
latexr 2y ago
> Sometimes I just feel like these people overestimate how much they are actually owed from these training runs. It’s not about being paid for including their work, it’s about being compensated for having done so without permission. For crying out loud, they went out of their way to remove copyright notices from the pirated work. > It’s trained on 15T tokens. So how many did you provide that were genuinely novel? Then they can just take it out. And go ahead and take out every thing you didn’t have permission to include. What’s that? The model is now significantly worse? Yeah, these things compound. > And how much money do you want? Like $5 from OpenAI? And $0 from meta since it’s open source? No, they would’ve wanted for the work to not have been included without permission in the first place. Do you understand the world you’re advocating for? You’re arguing it’s OK for rich people to do whatever they want if they throw some scraps on the floor for you. Not everything is about money. Unfortunately there’s no other reasonable way (legal and non violent) to punish these infringers. > I personally hope we can all get on the same team with AI and treat its advancement as scientific research for the betterment of humanity. What you’re expressing is “I hope everyone will stop arguing and agree with me”. These moguls care about themselves, it is incredibly naive to believe they give a rat’s ass about “the betterment of humanity”.
forgetfulness 2y ago
LLM companies aren't being funded by the hundreds of billions because investors expect science to be advanced by text and image generators. I find it very unlikely that the commodification of knowledge work will be for the betterment humanity, I don't know if people are expecting here that just because the value of more people's labor becomes zero that we will do, what, do away with money? No, it will just mean that fewer people will have the chance to earn the right to use space and resources in a meaningful way.
- inetknght 2y ago
  > I find it very unlikely that the commodification of knowledge work will be for the betterment humanity There's no law to force it. So of course it won't be. Even if there were a law to force it, how would you enforce it?
deleted 2y ago
[deleted]
wat10000 2y ago
The normal way to figure this out is to negotiate. We’ll either come to a mutually agreeable amount, or they’ll decide it’s not worth the cost to use my stuff. If I think I deserve $5 from OpenAI, then I’d suggest that, and they’d accept or come back with a counteroffer or tell me I’m nuts and move on. Probably that last one. But for some reason, these companies think they don’t need to bother, and can just use everyone’s stuff. Wait, I phrased that wrong. For a very good reason based on long precedent, these companies know that IP law is a tool to be used by big companies against individuals and sometimes other big companies, but never by individuals against company, so they know they don’t have to bother.
jsheard 2y ago
The fact they they were willing to risk significant legal exposure in order to use this dataset suggests it's worth considerably more than 5 dollars to them. Zuck isn't putting his ass on the line for a Big Mac.
TruffleLabs 2y ago
Stealing is still illegal
sensanaty 2y ago
So if the works they're stealing aren't worth anything, why do they need it so badly?

lvl155 2y ago

Of course he does. Heck most of us in early stages of LLM did the same thing. The data simply did not exists outside Google which is why it’s crazy that Google completely dropped the ball on AI this decade. They had such a huge lead in terms data access.

logifail 2y ago
Perhaps the more interesting question would be exactly how did they obtain their copy/copies of Libgen?
- perfmode 2y ago
  Are you asking for a way to obtain a copy?
  - logifail 2y ago
    Nope, I have no need for any <whisper>further</whisper> copies. I'm more interested in how a for-profit corp decides to obtain a copy for development of a commercial product, and how they execute that ... whether they still have the data, and whether legal know about it :) It's exactly not the kind of thing you can say you "found on a USB stick lying around in the car park"...
    - thfuran 2y ago
      You should've seen the size of it. More of a USB baton really.
      - selimthegrim 2y ago
        If Ryobi and DeWalt can make Bluetooth speakers, ASP can get into USB drives.
      - logifail 2y ago
        > You should've seen the size of it. More of a USB baton really. <glances at shelf with many, many external USB drives hooked up to a Pi 400> Oh, really? :)

gausswho 2y ago

"The note observed that including the LibGen material would help them reach certain performance benchmarks, and alluded to industry rumors that other AI companies, including OpenAI and Mistral AI, are “using the library for their models.” A new chapter in "information wants to be free". Copyright was always an artificial restriction on human instinct. We now enter an age where piracy is keeping up with the Jones', and those who respect intellectual property choose irrelevance. Prosecution becomes impossible as the laundering grows ever more sophisticated. Adaptation and acceptance painful, but the only path forward .

bayindirh 2y ago
Tangential question: What's your take about Code AIs using GPL and source available code in their training sets, and breaching both licenses?
- gausswho 2y ago
  Eschewing the ethics of such an action, my take would be that it happens (and will happen) regardless of license and no one can stop it unless those utilizing such code leave evidence that they disregarded the license.
  - bayindirh 2y ago
    The models can and will reproduce a lot of their training set given the correct prompt [0]. How you can know you're infringing a license if the model doesn't tell you from which repository it got "inspired" for generating that code? GitHub was working on a feature which supposedly tells you which repository the snippet you just got is copied from, or IOW, which repositories have similar code, sorted by date. Effectively pushing the blame further on you by making you spend the time you just saved by investigating which repository provided the code you just got from "AI". Supposedly, if the license is not friendly, you can delete the snippet and write your own version. :) [0]: https://x.com/docsparse/status/1581461734665367554 https://x.com/docsparse/status/1581461734665367554

savoyard 2y ago

Library Genesis is living up to its name.

keernan 2y ago

[flagged]

dang 2y ago
Could you please stop posting unsubstantive comments and flamebait? You've unfortunately been doing it repeatedly. It's not what this site is for, and destroys what it is for. If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.
- keernan 2y ago
  I'll do my best to tone things down. However I struggle a bit with your observation. The posted article is about one of the wealthiest men in the world likely engaging in the largest copyright violation in history for his own financial benefit. Such antisocial behavior is almost certainly linked to male genetics. Something this same man just recently publicly stated he believes the world needs more, not less, of his anti-social aggressive tendencies. I would argue that you don't like what I have to say and are simply trying to squash my opinions. If you feel the post itself doesn't belong here, then I agree - and to that extent, I also agree to do my best to refrain from commenting upon submissions that themselves do not belong here. But, and here is my biggest objection: I have strong opinions regarding the dangers male genetics pose to world order and the future of humanity. And males unedcuated about the subject react in the manner they know best: aggressive anger; a reaction I am not inclined to back away from (having my own aggressive male genetics).
  - dang 2y ago
    "almost certainly" sounds like quite a stretch to me, but if you wanted to make a substantive point about male genetics that might be ok. Maybe. But it might also be a generic flamewar tangent (which we definitely don't want on HN), so a comment like that would need to be written in a particularly flame-retardant style. Either way, though, a dismissive one-liner like the GP comment is definitely against the site guidelines.

ChrisArchitect 2y ago

[dupe] https://news.ycombinator.com/item?id=42651007 https://news.ycombinator.com/item?id=42651007 https://news.ycombinator.com/item?id=42673628 https://news.ycombinator.com/item?id=42673628

deleted 2y ago

[deleted]