3 ms·

Linux is open source and is mostly C code. You cannot run C code directly, you have to compile it and produce binaries. But it's the C code, not binary form, wh

by nine_k 2y ago

Linux is open source and is mostly C code. You cannot run C code directly, you have to compile it and produce binaries. But it's the C code, not binary form, where the collaboration happens. With LLMs, weights are the binary code: it's how you run the model. But to be able to train the model from scratch, or to collaborate on new approaches, you have to operate at a the level of architecture, methods, and training data sets. They are the source code.

verdverm 2y ago
Analogies are always going to fall short. With LLM weights, you can modify them (quant, fine-tuning) to get something different, which is not something you do with compiled binaries. There are ample areas for collaboration even without being able to reproduce from scratch, which takes $X Millions of dollars, also something that a typical binary does not have as a feature.
- piperswe 2y ago
  You can absolutely modify compiled binaries to get something different. That's how lots of video game modding and ROM hacks work.
  - krisoft 2y ago
    And we would absolutely do it more often if compiling would cost as much as training of an LLM costs now.
  - verdverm 2y ago
    I considered adding "normally" to the binary modifications expecting a response like this. The concepts are still worlds apart Weights aren't really a binary in the same sense that a compiler produces, they lack instructions and are more just a bunch of floating point values. Nor can you run model weights without separate code to interpret them correctly. In this sense, they are more like a JPEG or 3d model
    - the8thbit 2y ago
      JPEGs and 3D models are also executable binaries. They, like model weights, contain domain specific instructions that execute in a domain specific and turing incomplete environment. The model weights are the instructions, and those instructions are interpreted by the inference harness to produce outputs.

by nine_k 2y ago

verdverm 2y ago

Analogies are always going to fall short. With LLM weights, you can modify them (quant, fine-tuning) to get something different, which is not something you do with compiled binaries. There are ample areas for collaboration even without being able to reproduce from scratch, which takes $X Millions of dollars, also something that a typical binary does not have as a feature.

piperswe 2y ago
You can absolutely modify compiled binaries to get something different. That's how lots of video game modding and ROM hacks work.
- krisoft 2y ago
  And we would absolutely do it more often if compiling would cost as much as training of an LLM costs now.
- verdverm 2y ago
  I considered adding "normally" to the binary modifications expecting a response like this. The concepts are still worlds apart Weights aren't really a binary in the same sense that a compiler produces, they lack instructions and are more just a bunch of floating point values. Nor can you run model weights without separate code to interpret them correctly. In this sense, they are more like a JPEG or 3d model
  - the8thbit 2y ago
    JPEGs and 3D models are also executable binaries. They, like model weights, contain domain specific instructions that execute in a domain specific and turing incomplete environment. The model weights are the instructions, and those instructions are interpreted by the inference harness to produce outputs.