3 ms·

The training data would need to have a permissive license for this to be possible.

by kibae 1mo ago

ux266478 1mo ago
You could sidestep it by running non-permissibly licensed training data that you purchased through an LLM. Legal attitude so far seems to be that this is transformative as long as it's not 1:1. The question on whether or not the end result is copyrightable of course remains controversial and inconsistent, but that question is also fairly irrelevent. You don't get more libre than public domain. That's a fair amount of computational and labor overhead mind you, as you'll need to verify and prune the quality of your mountain of synthetic data, but certainly possible. Though this assumes the legal system is a rational actor playing by the set of rules it claims to. In fact, I highly suspect you could get very unlucky and get an unfavorable ruling against you, because you stepped on a big pile of money's toes in the process of doing this.
- alightsoul 1mo ago
  It can also be used to sidestep copyright like this forum, books and most websites even if the data was not purchased but is a website or book. Are LLMs what we need to make all data public domain? This way it could be used for that purpose
echelon 1mo ago
Eventually we'll just construct 100% synthetic training data that can reliably reproduce pretrains and fine tunes. The first broadly useful fully open source models will do this. We already have open data / open code / open weights for some domain-specific cases, such as audio models trained on large open datasets, eg. Tacotron / LJSpeech from waaay back in the day, though that is certainly not SOTA anymore. Distillation could possibly be considered an early case of this as raw AI outputs are themselves not copyrightable unless humans enrich, filter, or transform them. Granted, that does not handle the cases where the outputs are sufficiently similar to copyrighted original works.
- waffleiron 1mo ago
  Where does that synthetic data come from? Magically just started existing?
- chaosharmonic 1mo ago
  But how much of that synthetic data still ultimately derives from non-open sources? You'd still have to ask what a clean room implementation ultimately is, depending on how granular or aggressive a large publisher wanted to get about it. That said, I don't necessarily disagree with you. Talkie[1] presents an interesting case for it being at least possible to do this entirely on public domain material. But even that used Claude somewhere in the course of its training pipeline (it's listed as a contributor on their GitHub), so again, how granular you want to get with that is still a question. [1] https://talkie-lm.com/chat https://talkie-lm.com/chat
jjordan 1mo ago
Hear me out. Decentralized unstoppable storage, combined with decentralized unstoppable training, sorta like SETI for AI training. The seed of this tech already exists with IPFS and others like it. We know (some? all?) of the big labs have skirted copyright laws at one point or another. Truly open models would just build on what is publicly available.
- deleted 1mo ago
  [deleted]
- alightsoul 1mo ago
  Crypto bros took the idea with some blockchain shit and no one takes it seriously anymore so it died
  - embedding-shape 1mo ago
    If the LLM/AI ecosystem starts actually needing some Person-To-Person (or maybe Agent-To-Agent?) payment system because things actually get smart enough to be useful autonomously, they're gonna need some way to send money/currency around. Depending on how banks will react to this need, we might see another return of digital currencies from the current winter.
    - idiotsecant 1mo ago
      They used to say that cryptocurrency will be the dopamine layer of the first artificial intelligence
embedding-shape 1mo ago
Or, we just need to get this over with and declare any digital data findable via the internet to just be public property of everyone. Everything becomes public, besides stuff you keep locally, and there is no difference anymore, it's all just data anyone can use for whatever. A 1 year grace period for everyone to pull stuff off they don't want to be a part of this bright new open era, then we just scrap everything related to intellectual property, copyright and similar stupid stuff, and slap UBI on top of all of it for good measure.
- deleted 1mo ago
  [deleted]
- chme 1mo ago
  I'd prefer to stay within the [hacker ethics](https://www.ccc.de/en/hackerethics https://www.ccc.de/en/hackerethics), and protect private data. For non-private/personal data, sure. But individual people need their privacy protected.
  - embedding-shape 1mo ago
    Me too, I'm hacker ethics all the way, which is why I'm saying anything network connected should really realize the "All information should be free." dream, and then private data should be far away from the internet, on computers/drives not even connected to the internet. The whole E2E encryption is a ticking time bomb people rely to keep their data safe from others, but nothing that you don't physically have close to you can be truly secret forever, and even then it'll be hard.
    - Caracas288 29d ago
      Exactly, and right now we have the worst of both worlds with companies blatantly ignoring copyright, but individuals prosecuted for violating it.
    - chme 29d ago
      Well... If someone leaks private data on individuals online, those should be deleted.
ignoramous 1mo ago
UAE's IFM / LLM360 MO is indeed "fully open source" LLMs: https://www.llm360.ai/reports/LLM360-Towards-Fully-Transparent-Open-Source-LLMs.pdf https://www.llm360.ai/reports/LLM360-Towards-Fully-Transpare...
- yorwba 29d ago
  They do not appear to have published the training data yet, but if they do it like Olmo https://huggingface.co/datasets/allenai/dolma3_pool https://huggingface.co/datasets/allenai/dolma3_pool you get a license to the database, but not to its content, which they cannot license to you because it was scraped from the internet. E.g. have a look at the preamble of the ODC-By license https://opendatacommons.org/licenses/by/1-0/ https://opendatacommons.org/licenses/by/1-0/ which makes this distinction.

The training data would need to have a permissive license for this to be possible.

by kibae 1mo ago

The training data would need to have a permissive license for this to be possible.

You could sidestep it by running non-permissibly licensed training data that you purchased through an LLM. Legal attitude so far seems to be that this is transformative as long as it's not 1:1. The question on whether or not the end result is copyrightable of course remains controversial and inconsistent, but that question is also fairly irrelevent. You don't get more libre than public domain. That's a fair amount of computational and labor overhead mind you, as you'll need to verify and prune the quality of your mountain of synthetic data, but certainly possible. Though this assumes the legal system is a rational actor playing by the set of rules it claims to. In fact, I highly suspect you could get very unlucky and get an unfavorable ruling against you, because you stepped on a big pile of money's toes in the process of doing this.

alightsoul 1mo ago
It can also be used to sidestep copyright like this forum, books and most websites even if the data was not purchased but is a website or book. Are LLMs what we need to make all data public domain? This way it could be used for that purpose

echelon 1mo ago

Eventually we'll just construct 100% synthetic training data that can reliably reproduce pretrains and fine tunes. The first broadly useful fully open source models will do this. We already have open data / open code / open weights for some domain-specific cases, such as audio models trained on large open datasets, eg. Tacotron / LJSpeech from waaay back in the day, though that is certainly not SOTA anymore. Distillation could possibly be considered an early case of this as raw AI outputs are themselves not copyrightable unless humans enrich, filter, or transform them. Granted, that does not handle the cases where the outputs are sufficiently similar to copyrighted original works.

waffleiron 1mo ago
Where does that synthetic data come from? Magically just started existing?
chaosharmonic 1mo ago
But how much of that synthetic data still ultimately derives from non-open sources? You'd still have to ask what a clean room implementation ultimately is, depending on how granular or aggressive a large publisher wanted to get about it. That said, I don't necessarily disagree with you. Talkie[1] presents an interesting case for it being at least possible to do this entirely on public domain material. But even that used Claude somewhere in the course of its training pipeline (it's listed as a contributor on their GitHub), so again, how granular you want to get with that is still a question. [1] https://talkie-lm.com/chat https://talkie-lm.com/chat

jjordan 1mo ago

Hear me out. Decentralized unstoppable storage, combined with decentralized unstoppable training, sorta like SETI for AI training. The seed of this tech already exists with IPFS and others like it. We know (some? all?) of the big labs have skirted copyright laws at one point or another. Truly open models would just build on what is publicly available.

deleted 1mo ago
[deleted]
alightsoul 1mo ago
Crypto bros took the idea with some blockchain shit and no one takes it seriously anymore so it died
- embedding-shape 1mo ago
  If the LLM/AI ecosystem starts actually needing some Person-To-Person (or maybe Agent-To-Agent?) payment system because things actually get smart enough to be useful autonomously, they're gonna need some way to send money/currency around. Depending on how banks will react to this need, we might see another return of digital currencies from the current winter.
  - idiotsecant 1mo ago
    They used to say that cryptocurrency will be the dopamine layer of the first artificial intelligence

embedding-shape 1mo ago

Or, we just need to get this over with and declare any digital data findable via the internet to just be public property of everyone. Everything becomes public, besides stuff you keep locally, and there is no difference anymore, it's all just data anyone can use for whatever. A 1 year grace period for everyone to pull stuff off they don't want to be a part of this bright new open era, then we just scrap everything related to intellectual property, copyright and similar stupid stuff, and slap UBI on top of all of it for good measure.

deleted 1mo ago
[deleted]
chme 1mo ago
I'd prefer to stay within the [hacker ethics](https://www.ccc.de/en/hackerethics https://www.ccc.de/en/hackerethics), and protect private data. For non-private/personal data, sure. But individual people need their privacy protected.
- embedding-shape 1mo ago
  Me too, I'm hacker ethics all the way, which is why I'm saying anything network connected should really realize the "All information should be free." dream, and then private data should be far away from the internet, on computers/drives not even connected to the internet. The whole E2E encryption is a ticking time bomb people rely to keep their data safe from others, but nothing that you don't physically have close to you can be truly secret forever, and even then it'll be hard.
  - Caracas288 29d ago
    Exactly, and right now we have the worst of both worlds with companies blatantly ignoring copyright, but individuals prosecuted for violating it.
  - chme 29d ago
    Well... If someone leaks private data on individuals online, those should be deleted.

ignoramous 1mo ago

UAE's IFM / LLM360 MO is indeed "fully open source" LLMs: https://www.llm360.ai/reports/LLM360-Towards-Fully-Transparent-Open-Source-LLMs.pdf https://www.llm360.ai/reports/LLM360-Towards-Fully-Transpare...

yorwba 29d ago
They do not appear to have published the training data yet, but if they do it like Olmo https://huggingface.co/datasets/allenai/dolma3_pool https://huggingface.co/datasets/allenai/dolma3_pool you get a license to the database, but not to its content, which they cannot license to you because it was scraped from the internet. E.g. have a look at the preamble of the ODC-By license https://opendatacommons.org/licenses/by/1-0/ https://opendatacommons.org/licenses/by/1-0/ which makes this distinction.