5 ms·

Not an opinion on the topic, but technically they could check for copyright issues before inserting it into the training data? How that would work, at those vol

Not an opinion on the topic, but technically they could check for copyright issues before inserting it into the training data? How that would work, at those volumes, or if it is feasible, no clue. Have they tried? I doubt it.

mewpmewp2 2y ago
There could be some methods they can employ, but I don't see any of them being remotely foolproof. E.g. if they ask NYT to provide them with a list of content hashes to check against or the content itself to match similarity, but it's really easy for plagiarisers to use already existing open source LLMs to slightly modify content in different places in automatic fashion. And they would be incentivised to do that as well. I don't see right now how this battle could be won in theory. There's going to be bunch of content farms, plagiarisers using their open source LLM automation tools, which essentially launder the information from other sources. Even if you denylist them, it's arbitrary for them to spawn further instances. Eventually the only way would be to not use the Internet at all, and only use allowlisted sources. But that's quite ridiculous to me.
pjc50 2y ago
They could never economically achieve the required volume of training data from licensed data alone. The model requires plagiarizing the Internet.
- Sakos 2y ago
  There are two conditions for AI working at all. 1) Access to all the internet's data without restriction, including all books and research papers. 2) The model isn't controlled and owned by a for-profit company, it has to be an intra-country initiative run for the benefit of mankind. Anything else is a wild misuse and bastardisation of AI. We're already seeing all the issues inherent in our current approaches.
- exitb 2y ago
  What if they do a "clean room approach" of sorts. One non-public LLM is trained on copyrighted material and creates its own articles about stuff. Then a public LLM is trained on these articles. It appears to work for people-based processes.
  - techjamie 2y ago
    The problem with that approach is that any biases the original model has will be amplified during the training process, and cause it to have issues.
  - tivert 2y ago
    > It appears to work for people-based processes. Machines aren't people, and workarounds based on the affordances given to people should not work for machines, because machines should not be given those affordances.
    - ben_w 2y ago
      I see no reason to care what affordances someone gives to an (artificial) machine that I don't own.

mewpmewp2 2y ago

There could be some methods they can employ, but I don't see any of them being remotely foolproof. E.g. if they ask NYT to provide them with a list of content hashes to check against or the content itself to match similarity, but it's really easy for plagiarisers to use already existing open source LLMs to slightly modify content in different places in automatic fashion. And they would be incentivised to do that as well. I don't see right now how this battle could be won in theory. There's going to be bunch of content farms, plagiarisers using their open source LLM automation tools, which essentially launder the information from other sources. Even if you denylist them, it's arbitrary for them to spawn further instances. Eventually the only way would be to not use the Internet at all, and only use allowlisted sources. But that's quite ridiculous to me.

pjc50 2y ago

They could never economically achieve the required volume of training data from licensed data alone. The model requires plagiarizing the Internet.

Sakos 2y ago
There are two conditions for AI working at all. 1) Access to all the internet's data without restriction, including all books and research papers. 2) The model isn't controlled and owned by a for-profit company, it has to be an intra-country initiative run for the benefit of mankind. Anything else is a wild misuse and bastardisation of AI. We're already seeing all the issues inherent in our current approaches.
exitb 2y ago
What if they do a "clean room approach" of sorts. One non-public LLM is trained on copyrighted material and creates its own articles about stuff. Then a public LLM is trained on these articles. It appears to work for people-based processes.
- techjamie 2y ago
  The problem with that approach is that any biases the original model has will be amplified during the training process, and cause it to have issues.
- tivert 2y ago
  > It appears to work for people-based processes. Machines aren't people, and workarounds based on the affordances given to people should not work for machines, because machines should not be given those affordances.
  - ben_w 2y ago
    I see no reason to care what affordances someone gives to an (artificial) machine that I don't own.