5 ms·

Sorry, where exactly is "data" in "source"? I can understand not wanting to call it open source if they don't give you the algorithm and software used for trai

Sorry, where exactly is "data" in "source"? I can understand not wanting to call it open source if they don't give you the algorithm and software used for training, but wanting the training data too? That's completely different

croes 2mo ago
For LLMs the training data is the source of their weights. You can‘t reproduce the LLM without the same data
- anon373839 2mo ago
  That’s not really true. The models have source code defining their architecture and it is open-source. People keep trying to shoehorn OSS concepts onto model weights, but the concepts don’t fit because the weights aren’t software. They aren’t compiled code. They are learned parameters to use with a (very big) function that itself is expressed in the code. So they’re a very valuable asset that complements the code, but they are not the code. You could use randomly initialized weights and the software will work - it will output tokens. They just won’t have useful patterns. I don’t think OSS definitions have ever required that assets have their source included. For example, artwork is very important to a game, but nobody thinks a game is not open source if it doesn’t come with sketches and a copy of Adobe Illustrator to recreate the artwork from scratch. Edit: The distinction I would draw between models like Deepseek and models like OLMo is whether they are open science. With a model like OLMo, they have published everything you need to replicate the training experiment. Whereas Deepseek does share a lot of knowledge, but keeps a lot proprietary too.
  - pocketarc 2mo ago
    Your game example is a great one: If a game didn’t come with its 3D models, sound, level files, game data, or assets of any kind, but its engine was open source, I don’t think anyone would say “hey this game is open source”. They’d say the engine is, but that’s it. All of the above things are not code, but they are part of a game and yes, expected to be included in an open source game.
    - anon373839 2mo ago
      I think this misses the point I was trying to make. Both an open source game and an open weights LLM include the necessary assets. But people seem to be saying, “you need everything that was used to create the assets”. For game art, that would include sketches, prior art used as inspiration, project files that aren’t used by the game engine but were used during the design process, and so on - if you apply the same standard. But I don’t think anyone does.
      - gpm 2mo ago
        Similar to this I've never seen a project request anyone license their github PR comments under any license whatsoever even though they're undoubtedly copyrighted and used in the creation of the actual source code.
    - BoxOfRain 2mo ago
      Doom pioneered this model, the Doom engine was open-sourced while the assets remained proprietary. This is where projects like Freedoom exist, to create an entirely free software equivalent.
    - gpm 2mo ago
      People refer to games with proprietary assets as open source all the time. The source code is open - the assets are not. For one example: https://news.ycombinator.com/item?id=28201265 https://news.ycombinator.com/item?id=28201265 / https://github.com/beyond-all-reason/Beyond-All-Reason/blob/master/license_bitmaps.txt#L54 https://github.com/beyond-all-reason/Beyond-All-Reason/blob/... Source as a word has a meaning that doesn't include non-source things.
      - SV_BubbleTime 2mo ago
        That’s an open source engine, we already have a word to define what that is. And we have Open Weights to define what those are.
        gpm 2mo ago
- ungovernableCat 2mo ago
  My understanding was that there was a lot of non determinism in the training process because of many factors: how GPUs optimise floating point arithmetic, how the compute is distributed between the thousands of GPUs etc I'm not sure how static training data is either (or how you'd distribute it considering its size and nevermind the legality of sharing copyrighted things). You'd likely get a model with very similar behaviour but the weights would be different. Please someone correct me if I'm wrong.

croes 2mo ago

For LLMs the training data is the source of their weights. You can‘t reproduce the LLM without the same data

anon373839 2mo ago
That’s not really true. The models have source code defining their architecture and it is open-source. People keep trying to shoehorn OSS concepts onto model weights, but the concepts don’t fit because the weights aren’t software. They aren’t compiled code. They are learned parameters to use with a (very big) function that itself is expressed in the code. So they’re a very valuable asset that complements the code, but they are not the code. You could use randomly initialized weights and the software will work - it will output tokens. They just won’t have useful patterns. I don’t think OSS definitions have ever required that assets have their source included. For example, artwork is very important to a game, but nobody thinks a game is not open source if it doesn’t come with sketches and a copy of Adobe Illustrator to recreate the artwork from scratch. Edit: The distinction I would draw between models like Deepseek and models like OLMo is whether they are open science. With a model like OLMo, they have published everything you need to replicate the training experiment. Whereas Deepseek does share a lot of knowledge, but keeps a lot proprietary too.
- pocketarc 2mo ago
  Your game example is a great one: If a game didn’t come with its 3D models, sound, level files, game data, or assets of any kind, but its engine was open source, I don’t think anyone would say “hey this game is open source”. They’d say the engine is, but that’s it. All of the above things are not code, but they are part of a game and yes, expected to be included in an open source game.
  - anon373839 2mo ago
    I think this misses the point I was trying to make. Both an open source game and an open weights LLM include the necessary assets. But people seem to be saying, “you need everything that was used to create the assets”. For game art, that would include sketches, prior art used as inspiration, project files that aren’t used by the game engine but were used during the design process, and so on - if you apply the same standard. But I don’t think anyone does.
    - gpm 2mo ago
      Similar to this I've never seen a project request anyone license their github PR comments under any license whatsoever even though they're undoubtedly copyrighted and used in the creation of the actual source code.
  - BoxOfRain 2mo ago
    Doom pioneered this model, the Doom engine was open-sourced while the assets remained proprietary. This is where projects like Freedoom exist, to create an entirely free software equivalent.
  - gpm 2mo ago
    People refer to games with proprietary assets as open source all the time. The source code is open - the assets are not. For one example: https://news.ycombinator.com/item?id=28201265 https://news.ycombinator.com/item?id=28201265 / https://github.com/beyond-all-reason/Beyond-All-Reason/blob/master/license_bitmaps.txt#L54 https://github.com/beyond-all-reason/Beyond-All-Reason/blob/... Source as a word has a meaning that doesn't include non-source things.
    - SV_BubbleTime 2mo ago
      That’s an open source engine, we already have a word to define what that is. And we have Open Weights to define what those are.
      - gpm 2mo ago
ungovernableCat 2mo ago
My understanding was that there was a lot of non determinism in the training process because of many factors: how GPUs optimise floating point arithmetic, how the compute is distributed between the thousands of GPUs etc I'm not sure how static training data is either (or how you'd distribute it considering its size and nevermind the legality of sharing copyrighted things). You'd likely get a model with very similar behaviour but the weights would be different. Please someone correct me if I'm wrong.