4 ms·
The weights are the modifiable representation. You modify them with gradient updates.
by WithinReason 2mo ago
The weights are the modifiable representation. You modify them with gradient updates.
- HPsquared 2mo agoThey are a cooked stew; the ingredients have already been chopped and mixed together. You can add things, yes, but you can't inspect the ingredients.
- ungovernableCat 2mo agoIf we start with the same ingredients and independently make our stews they're likely still going to taste a bit different due to the non deterministic nature of the process. Isn't it similar with training LLMs at scale? Do you get identical weights if you do multiple runs?
- HPsquared 2mo agoThey don't even say what the ingredients are.
- ungovernableCat 2mo agoYeah I understand that. I mean just the methods they use to acquire the training data has already had a shit ton of drama, there's no way the western labs are ever going to be public about the ingredients.
- halJordan 2mo agoI can also open broadcom's antenna firmware in any hex editor. Why are you arguing that makes it open source?
- anon373839 2mo agoThe antenna firmware is software. Model weights are essentially learned settings, and they are not software. Note also that software is copyrightable and model weights aren’t (in the US anyway).
- croemer 2mo agoNot sure why it matters whether weights are software or not. One can equally argue that firmware is just settings for the CPU. If weights aren't copyrightable then doesn't this argue against the idea of them being able to have the quality of open-sourceness? Meanwhile the training code could be copyrightable.
- anon373839 2mo agoWell, my position is that the software that defines/runs the model is the open-source part. The weights are the asset that makes it useful, but I don’t think 40T tokens of pre-training data should be required to call the model itself open source when you can inspect every line of code in the model without that, as well as instantiate and run the model with randomly initialized weights.
- inigyou 2mo agoYou can hand-patch a binary, too.