3 ms·
Analogies are always going to fall short. With LLM weights, you can modify them (quant, fine-tuning) to get something different, which is not something you do w
by verdverm 2y ago
Analogies are always going to fall short. With LLM weights, you can modify them (quant, fine-tuning) to get something different, which is not something you do with compiled binaries. There are ample areas for collaboration even without being able to reproduce from scratch, which takes $X Millions of dollars, also something that a typical binary does not have as a feature.
- piperswe 2y agoYou can absolutely modify compiled binaries to get something different. That's how lots of video game modding and ROM hacks work.
- krisoft 2y agoAnd we would absolutely do it more often if compiling would cost as much as training of an LLM costs now.
- verdverm 2y agoI considered adding "normally" to the binary modifications expecting a response like this. The concepts are still worlds apart Weights aren't really a binary in the same sense that a compiler produces, they lack instructions and are more just a bunch of floating point values. Nor can you run model weights without separate code to interpret them correctly. In this sense, they are more like a JPEG or 3d model
- the8thbit 2y agoJPEGs and 3D models are also executable binaries. They, like model weights, contain domain specific instructions that execute in a domain specific and turing incomplete environment. The model weights are the instructions, and those instructions are interpreted by the inference harness to produce outputs.