3 ms·

These models have been explicitly nerfed since their first release due to copyright considerations. I've mentioned in two previous cases both for [1] code gener

These models have been explicitly nerfed since their first release due to copyright considerations. I've mentioned in two previous cases both for [1] code generation and [2] book summarizing. From my point of view, it is sad that these sort of socio-political constructs (copyright) are hindering innovation. The funny thing is that in say, 10 years, the "pirate" version of LLMs will be way more powerful and useful than the "corporate" versions. Once the RIAA/MPAA has sued all of them preventing them of using their audio/video; after the editorials have sued them to prevent them from using their books and articles, and after every internet site has sued them to prevent them from using their text. LLM models trained on SciHub, Library Genesis and Torrents will be amazing in comparison. I sincerely wish that some country would apply to information copyright a similar approach to what India does for medicine patents. [1] https://news.ycombinator.com/item?id=31852138 https://news.ycombinator.com/item?id=31852138 [2] https://news.ycombinator.com/item?id=36138930 https://news.ycombinator.com/item?id=36138930

robterrell 3y ago
> From my point of view, it is sad that these sort of socio-political constructs (copyright) are hindering innovation Why not pay authors of the data the LLM has ingested?
- SirMaster 3y ago
  Don't a lot of content creators expect payment per use? Like every time someone streams a song on sportify, the artist gets a few pennies. So should they be paid a few pennies every time the LLM spits out a response that "used" that training data? And I am pretty sure it's not even possible to really link the output back to training data anyways.
  - robterrell 3y ago
    > Like every time someone streams a song on sportify, the artist gets a few pennies You're thinking of broadcast radio. Streaming is a fraction of a penny! > So should they be paid a few pennies every time the LLM spits out a response that "used" that training data? That doesn't sound unreasonable! I understand that current LLMs have no way to report "this token came from this data" but that doesn't mean it's impossible to build. (Ack: this is a full-on proper dunning-kruger, having not looked into it & having zero knowledge of the field.) But I think it's probably more reasonable to simply split a % of the service revenue across everyone whose data was used. Or pay an up-fee for ingesting the data in the first place. Generally, it's crazy to name that some people think it's reasonable for these companies to pay for GPUs and CPUs and electricity to run them but not for the data that's the actual core of their service.
- epups 3y ago
  Since they are trained on all kinds of Internet content, might be a little tricky to manage paying >1B people
  - robterrell 3y ago
    I mean, figure something out? Didn't Sam want to create UBI? That's one way!
  - ChatGTP 3y ago
    I'm sure it would be easy to pay publishers who pass on the money? How do you think Amazon does things for kindle?
    - epups 3y ago
      Will you get a check for your posts on HN? Your tweets? No, right? So all this talk about paying people for their data is rubbish, and what is really at stake here are checks between huge corporations that already stole it from you in the first place.
      - ChatGTP 3y ago
        I'm not quite sure which side your on but yeah, I personally have stopped participating in social media since realizing I'm the product. HN I agree with you, but I feel like I'm getting enough back from this community to make my time worth it.
lukas099 3y ago
> I sincerely wish that some country would apply to information copyright a similar approach to what India does for medicine patents. Care to take this opportunity to explain India's approach to medicine patents, and why you think it's good?
ChatGTP 3y ago
Personally, I disagree with you. I have absolutely no idea why OpenAI should benefit from copyrighted material without paying authors for the usage. If everything OpenAI did was opensource, I'd probably feel differently about it, but it's totally not, so I really disagree with the view that they should deserve special treatment here. I remember seeing interviews and reading many comments saying that that the copyright data shouldn't be an issue because once the model is trained, it kind of "forgets" the copyrighted material and we're just left with pure, unfettered intelligence...it's not a fuzzy jpeg of the web etc... I think the copyright angle is on the money though, this is why Bard isn't as good, just as Adobe Firefly isn't as good as Stable Diffusion. The inputs aren't as good so the outputs aren't either. Google can't come out in public and accuse OpenAI of blatant copyright infringement for legal reasons, but I bet internally, they know what's up.

robterrell 3y ago

> From my point of view, it is sad that these sort of socio-political constructs (copyright) are hindering innovation Why not pay authors of the data the LLM has ingested?

SirMaster 3y ago
Don't a lot of content creators expect payment per use? Like every time someone streams a song on sportify, the artist gets a few pennies. So should they be paid a few pennies every time the LLM spits out a response that "used" that training data? And I am pretty sure it's not even possible to really link the output back to training data anyways.
- robterrell 3y ago
  > Like every time someone streams a song on sportify, the artist gets a few pennies You're thinking of broadcast radio. Streaming is a fraction of a penny! > So should they be paid a few pennies every time the LLM spits out a response that "used" that training data? That doesn't sound unreasonable! I understand that current LLMs have no way to report "this token came from this data" but that doesn't mean it's impossible to build. (Ack: this is a full-on proper dunning-kruger, having not looked into it & having zero knowledge of the field.) But I think it's probably more reasonable to simply split a % of the service revenue across everyone whose data was used. Or pay an up-fee for ingesting the data in the first place. Generally, it's crazy to name that some people think it's reasonable for these companies to pay for GPUs and CPUs and electricity to run them but not for the data that's the actual core of their service.
epups 3y ago
Since they are trained on all kinds of Internet content, might be a little tricky to manage paying >1B people
- robterrell 3y ago
  I mean, figure something out? Didn't Sam want to create UBI? That's one way!
- ChatGTP 3y ago
  I'm sure it would be easy to pay publishers who pass on the money? How do you think Amazon does things for kindle?
  - epups 3y ago
    Will you get a check for your posts on HN? Your tweets? No, right? So all this talk about paying people for their data is rubbish, and what is really at stake here are checks between huge corporations that already stole it from you in the first place.
    - ChatGTP 3y ago
      I'm not quite sure which side your on but yeah, I personally have stopped participating in social media since realizing I'm the product. HN I agree with you, but I feel like I'm getting enough back from this community to make my time worth it.

lukas099 3y ago

> I sincerely wish that some country would apply to information copyright a similar approach to what India does for medicine patents. Care to take this opportunity to explain India's approach to medicine patents, and why you think it's good?

ChatGTP 3y ago

Personally, I disagree with you. I have absolutely no idea why OpenAI should benefit from copyrighted material without paying authors for the usage. If everything OpenAI did was opensource, I'd probably feel differently about it, but it's totally not, so I really disagree with the view that they should deserve special treatment here. I remember seeing interviews and reading many comments saying that that the copyright data shouldn't be an issue because once the model is trained, it kind of "forgets" the copyrighted material and we're just left with pure, unfettered intelligence...it's not a fuzzy jpeg of the web etc... I think the copyright angle is on the money though, this is why Bard isn't as good, just as Adobe Firefly isn't as good as Stable Diffusion. The inputs aren't as good so the outputs aren't either. Google can't come out in public and accuse OpenAI of blatant copyright infringement for legal reasons, but I bet internally, they know what's up.