8 ms·

It's hilarious that big players in this space seem to think these are consistent views: - It's okay to train a model on arbitrary internet data without permiss

It's hilarious that big players in this space seem to think these are consistent views: - It's okay to train a model on arbitrary internet data without permission/license just because you can access it - It's not okay train a model on our model

whatshisface 3y ago
The belief that makes them consistent is that the authors of a million Reddit posts have no way to assert their rights while the big company that trained a Redditor model does.
- LastTrain 3y ago
  Sure they do, albeit a shitty one: it's called a class-action.
rahkiin 3y ago
Like google is allowed to scrape the whole internet but you’re not allowed to scrape google. Rules for thee but not for me
- kgwgk 3y ago
  What rules? Google won’t scrape your part of the internet if you don’t allow it, right?
  - makeitdouble 3y ago
    Google respects the "robot.txt" and asks you to use it to opt out of their crawling. Parent's point is if your own scaping army respects the "scaping.txt" and goes down on Google as they don't opt-out in their scraping.txt, it probably wouldn't fly.
    - kgwgk 3y ago
      I don't understand. What does "Rules for thee but not for me" mean if "google is allowed to scrape" whatever people allows Google to scrape but "you’re not allowed to scrape google" because using the same rules google.com/robots.txt says User-agent: * Disallow: /search ....
      - makeitdouble 3y ago
        There's an imbalance because the robot.txt rule is something Google pushed forward (didn't invent it, but made it standard) and is opt-out. So yes, Google made up their rules and won't let other people to make up their own self-beneficial rules in a similar way.
        kgwgk 3y ago
        > Google [...] won't let other people to make up their own self-beneficial rules in a similar way. What "other people"? If it's the "you" who is not allowed to scrape google in https://news.ycombinator.com/item?id=36817237 https://news.ycombinator.com/item?id=36817237 then you can make your own "google is not allowed to scrape my thing" rules if you think that's beneficial for you. If it's somehow related to LLM providers or users I doubt that's what the original comment was referring to. To be clear, I understand the original comment as LLM companies say "I can use your content and you cannot not prevent me from doing so, but I won't allow you to use the output of the LLM" just like Google says "I can scrape your content and you cannot not prevent me from doing so, but I won't allow you to scrape the output of the search engine" and that doesn't seem a valid analogy.
        makeitdouble 3y ago
        You should change "you cannot prevent me from doing so" into "you'll need to setup your ressources in the way that I defined if you don't want me to slurp them". I see it as the equivalent of the spam mail that require the user to login to disable them.
- rvnx 3y ago
  Also the main business model of Google (and of search engines in general) is to republish rearranged snippets of copyrighted content and even serve whole copies of the content (googleusercontent cache), without prior authorization of the copyright holders, and for-profit. It’s completely illegal if you think about it. So why LLMs who crawl the internet to present snippets and information should be treated differently from Google ? (who also reproduce verbatim the same content without paying any compensation to the copyright owners (all types: text, image, code)
  - az226 3y ago
    It wouldn’t. Facebook is delusional if they think the license can pass muster. Presumably you can’t build an LLM that is a competitor of LlaMA using its outputs. But AI weights are in legal gray zone for now. So it’s muddy waters and fair game for anyone who wants to take on the legal risks.
  - panzi 3y ago
    Not wanting to defend the likes of Google, but search engines link the original source (in contrast to LLMs). Their basic idea is to direct people to your content. There are countries where content companies didn't like what Google does: Google took them out of the index -> suddenly they where ok with it again so that Google put them in again. (extremely simplified story)
    - pyrale 3y ago
      > Their basic idea is to direct people to your content. This is less and less true, as evidenced by the progression of 0-click searchs. > There are countries where content companies didn't like what Google does: Google took them out of the index -> suddenly they where ok with it again so that Google put them in again. This story screams antitrust.
      - account42 3y ago
        > This story screams antitrust. You're right, the number of news publishers that share a common owner is something that should be of concern to antitrust enforcers.
realusername 3y ago
Yes, they have to pick one or the other. Until then I'm going to assume that the model licence doesn't apply since the first point would be invalid and the model could not be built in the first place.
torstenvl 3y ago
Those are perfectly consistent, despite what ideologically-driven people may want to believe. Copyright is literally the right to copy. Arbitrary Internet data that is not copied does not have any copyright implications. The difference is that LLaMa imposes additional contractual obligations that, for ideological reasons (Freedom #0), open source software does not. This issue reminds me of the FSF/AGPL situation. At some point you just have to accept that copyright law, in and of itself, is not sufficient to control what people do with your software. If you want to do that, you have to limit end-user freedom with an EULA. If someone uses LLaMa output to train models, it is unlikely they will be sued for copyright infringement. It is far more likely they will be sued for breach of contract.
- owenfi 3y ago
  But you can publish the output, right? And then a “third party” could train a different model on just that published material without copying it or ever agreeing to a EULA.
  - torstenvl 3y ago
    If you believe that courts will find your shell game convincing, you are free to try it and incur the legal risk. I recommend you consult with an attorney before doing so.
    - themoonisachees 3y ago
      You could simply train on the output straight up and nobody would ever be able to tell anyway.
- 6gvONxR4sf7o 3y ago
  One of the common elements of training sets for these models (including LLama) is the Books3 dataset, which is a huge number of pirated books from torrents. That's exactly what you described. Regardless, the lack of a license cannot give you more permission than a restrictive license. You're arguing that if take a book out of a bookstore without paying (or signing a contract), then I have more rights than if I sign a contract and then leave with the book.
lhnz 3y ago
It tells you that they think their moat is data quality/quantity.
taneq 3y ago
I can license my LLM however I want to... but I can't sail this ship to generally-intelligent-Tortooga on me lonesome. Savvy?
chii 3y ago
> think these are consistent views they are consistent, if they believe themselves to be "special" and deserves special treatment!

whatshisface 3y ago

The belief that makes them consistent is that the authors of a million Reddit posts have no way to assert their rights while the big company that trained a Redditor model does.

LastTrain 3y ago
Sure they do, albeit a shitty one: it's called a class-action.

rahkiin 3y ago

Like google is allowed to scrape the whole internet but you’re not allowed to scrape google. Rules for thee but not for me

kgwgk 3y ago
What rules? Google won’t scrape your part of the internet if you don’t allow it, right?
- makeitdouble 3y ago
  Google respects the "robot.txt" and asks you to use it to opt out of their crawling. Parent's point is if your own scaping army respects the "scaping.txt" and goes down on Google as they don't opt-out in their scraping.txt, it probably wouldn't fly.
  - kgwgk 3y ago
    I don't understand. What does "Rules for thee but not for me" mean if "google is allowed to scrape" whatever people allows Google to scrape but "you’re not allowed to scrape google" because using the same rules google.com/robots.txt says User-agent: * Disallow: /search ....
    - makeitdouble 3y ago
      There's an imbalance because the robot.txt rule is something Google pushed forward (didn't invent it, but made it standard) and is opt-out. So yes, Google made up their rules and won't let other people to make up their own self-beneficial rules in a similar way.
      - kgwgk 3y ago
        > Google [...] won't let other people to make up their own self-beneficial rules in a similar way. What "other people"? If it's the "you" who is not allowed to scrape google in https://news.ycombinator.com/item?id=36817237 https://news.ycombinator.com/item?id=36817237 then you can make your own "google is not allowed to scrape my thing" rules if you think that's beneficial for you. If it's somehow related to LLM providers or users I doubt that's what the original comment was referring to. To be clear, I understand the original comment as LLM companies say "I can use your content and you cannot not prevent me from doing so, but I won't allow you to use the output of the LLM" just like Google says "I can scrape your content and you cannot not prevent me from doing so, but I won't allow you to scrape the output of the search engine" and that doesn't seem a valid analogy.
        makeitdouble 3y ago
        You should change "you cannot prevent me from doing so" into "you'll need to setup your ressources in the way that I defined if you don't want me to slurp them". I see it as the equivalent of the spam mail that require the user to login to disable them.
rvnx 3y ago
Also the main business model of Google (and of search engines in general) is to republish rearranged snippets of copyrighted content and even serve whole copies of the content (googleusercontent cache), without prior authorization of the copyright holders, and for-profit. It’s completely illegal if you think about it. So why LLMs who crawl the internet to present snippets and information should be treated differently from Google ? (who also reproduce verbatim the same content without paying any compensation to the copyright owners (all types: text, image, code)
- az226 3y ago
  It wouldn’t. Facebook is delusional if they think the license can pass muster. Presumably you can’t build an LLM that is a competitor of LlaMA using its outputs. But AI weights are in legal gray zone for now. So it’s muddy waters and fair game for anyone who wants to take on the legal risks.
- panzi 3y ago
  Not wanting to defend the likes of Google, but search engines link the original source (in contrast to LLMs). Their basic idea is to direct people to your content. There are countries where content companies didn't like what Google does: Google took them out of the index -> suddenly they where ok with it again so that Google put them in again. (extremely simplified story)
  - pyrale 3y ago
    > Their basic idea is to direct people to your content. This is less and less true, as evidenced by the progression of 0-click searchs. > There are countries where content companies didn't like what Google does: Google took them out of the index -> suddenly they where ok with it again so that Google put them in again. This story screams antitrust.
    - account42 3y ago
      > This story screams antitrust. You're right, the number of news publishers that share a common owner is something that should be of concern to antitrust enforcers.

realusername 3y ago

Yes, they have to pick one or the other. Until then I'm going to assume that the model licence doesn't apply since the first point would be invalid and the model could not be built in the first place.

torstenvl 3y ago

Those are perfectly consistent, despite what ideologically-driven people may want to believe. Copyright is literally the right to copy. Arbitrary Internet data that is not copied does not have any copyright implications. The difference is that LLaMa imposes additional contractual obligations that, for ideological reasons (Freedom #0), open source software does not. This issue reminds me of the FSF/AGPL situation. At some point you just have to accept that copyright law, in and of itself, is not sufficient to control what people do with your software. If you want to do that, you have to limit end-user freedom with an EULA. If someone uses LLaMa output to train models, it is unlikely they will be sued for copyright infringement. It is far more likely they will be sued for breach of contract.

owenfi 3y ago
But you can publish the output, right? And then a “third party” could train a different model on just that published material without copying it or ever agreeing to a EULA.
- torstenvl 3y ago
  If you believe that courts will find your shell game convincing, you are free to try it and incur the legal risk. I recommend you consult with an attorney before doing so.
  - themoonisachees 3y ago
    You could simply train on the output straight up and nobody would ever be able to tell anyway.
6gvONxR4sf7o 3y ago
One of the common elements of training sets for these models (including LLama) is the Books3 dataset, which is a huge number of pirated books from torrents. That's exactly what you described. Regardless, the lack of a license cannot give you more permission than a restrictive license. You're arguing that if take a book out of a bookstore without paying (or signing a contract), then I have more rights than if I sign a contract and then leave with the book.

lhnz 3y ago

It tells you that they think their moat is data quality/quantity.

taneq 3y ago

I can license my LLM however I want to... but I can't sail this ship to generally-intelligent-Tortooga on me lonesome. Savvy?

chii 3y ago

> think these are consistent views they are consistent, if they believe themselves to be "special" and deserves special treatment!