4 ms·

The model weights ARE the preferred form for modification

tonyarkles 3y ago
Unless you want to try modifying the model structure, in which case the weights aren’t necessarily valid anymore and will need to be retrained.
PeterisP 3y ago
As long-time a 'practitioner' of machine learning models I strongly disagree, the preferred form for model modification is by retraining the model with a tweak to the parameters or the training algorithm or the model structure or data selection or length of training. You can get some effects by fine tuning, and in that case it may be preferable as it's cheaper, but in general if I want to have a different or better model, that involves retraining.
- jncfhnb 3y ago
  I don’t really believe your long time practitioning is aligned to the kind of models being discussed
anothernewdude 3y ago
Yeah, that's why data scientists are out there editing the weights rather than cleaning up datasets and rerunning training with different settings.
- jncfhnb 3y ago
  If that was supposed to be clever it just sounds naive. There’s a ton of work going on fine tuning open source models
  - nullc 3y ago
    > There’s a ton of work going on fine tuning ... models provided in weights only form. (mostly!) I believe the preferred form would be the whole kit and caboodle: the collection and filtering scripts, the data to the extent that it's non-public, the training routine, and the model weights... because sometimes you'll perform changes at any of those stages.
    - jncfhnb 3y ago
      Do you actually do this for a living? Do you have experience doing this and have credibility talking about what’s preferred? I do.
      - Trapais 3y ago
        OK. Where is your reproduction of Pythia trained from scratch? Or MPT? Or Amber? Shall we play a game where you give paper regarding pretraining (and we are not taling about puny models based on wikitext2) I give you a paper based around finetuning and we'll see who run out of papers first?
        jncfhnb 3y ago
        Reproduction is not the goal! Making papers is not the goal! Making useful models is the goal. And having open source models is by an enormous degree more useful thing. I see you’re someone else, so I’ll ask you too. Do you actually have any experience doing this? Have you ever fine tuned models or tried to change architecture or put a piece of one model into another?
        hn_acker 3y ago
        > Reproduction is not the goal! Making papers is not the goal! Making useful models is the goal. 1. The thread is about the requirements of calling a model open source. The goal of making the models is separate from the requirements of the open source definition. 2. Suppose that author A of a model prefers working exclusively with the model weights to modify the model. Author A's preferred form of modification includes the model weights - and whatever scripts are needed to generate a running model from the weights - but does not include the training set and initial training scripts. Suppose that author B of an unrelated model prefers to retrain the model as part of the process of modifying the model. If author B changes the training set and/or changes the training scripts, then the training set is part of the preferred form of modifying the model. The training set and the training scripts are both necessary for turning the training set into a running model, so I think author B would have to include the training set even if author B changes only the training scripts. Correct me if I'm wrong.) jncfhnb, you're like author A, so if you were to release an open source model then you would need to include the weights but not the training data. Trapais and nullc, don't assume that every model author is author B. For personal reference, here is the relevant excerpt from the open source definition from the Open Source Initiative [A1]: > The source code must be the preferred form in which a programmer would modify the program. Deliberately obfuscated source code is not allowed. Intermediate forms such as the output of a preprocessor or translator are not allowed. Open source software is not the same as free software, but here is the relevant excerpt from the free software definition explainer from the Free Software Foundation [A2]: > Source code is defined as the preferred form of the program for making changes in. Thus, whatever form a developer changes to develop the program is the source code of that developer's version. [A1] https://opensource.org/osd/ https://opensource.org/osd/ [A2] https://www.gnu.org/philosophy/free-sw.html https://www.gnu.org/philosophy/free-sw.html

Unless you want to try modifying the model structure, in which case the weights aren’t necessarily valid anymore and will need to be retrained.

PeterisP 3y ago

As long-time a 'practitioner' of machine learning models I strongly disagree, the preferred form for model modification is by retraining the model with a tweak to the parameters or the training algorithm or the model structure or data selection or length of training. You can get some effects by fine tuning, and in that case it may be preferable as it's cheaper, but in general if I want to have a different or better model, that involves retraining.

jncfhnb 3y ago
I don’t really believe your long time practitioning is aligned to the kind of models being discussed

anothernewdude 3y ago

Yeah, that's why data scientists are out there editing the weights rather than cleaning up datasets and rerunning training with different settings.

jncfhnb 3y ago
If that was supposed to be clever it just sounds naive. There’s a ton of work going on fine tuning open source models
- nullc 3y ago
  > There’s a ton of work going on fine tuning ... models provided in weights only form. (mostly!) I believe the preferred form would be the whole kit and caboodle: the collection and filtering scripts, the data to the extent that it's non-public, the training routine, and the model weights... because sometimes you'll perform changes at any of those stages.
  - jncfhnb 3y ago
    Do you actually do this for a living? Do you have experience doing this and have credibility talking about what’s preferred? I do.
    - Trapais 3y ago
      OK. Where is your reproduction of Pythia trained from scratch? Or MPT? Or Amber? Shall we play a game where you give paper regarding pretraining (and we are not taling about puny models based on wikitext2) I give you a paper based around finetuning and we'll see who run out of papers first?
      - jncfhnb 3y ago
        Reproduction is not the goal! Making papers is not the goal! Making useful models is the goal. And having open source models is by an enormous degree more useful thing. I see you’re someone else, so I’ll ask you too. Do you actually have any experience doing this? Have you ever fine tuned models or tried to change architecture or put a piece of one model into another?
        hn_acker 3y ago
        > Reproduction is not the goal! Making papers is not the goal! Making useful models is the goal. 1. The thread is about the requirements of calling a model open source. The goal of making the models is separate from the requirements of the open source definition. 2. Suppose that author A of a model prefers working exclusively with the model weights to modify the model. Author A's preferred form of modification includes the model weights - and whatever scripts are needed to generate a running model from the weights - but does not include the training set and initial training scripts. Suppose that author B of an unrelated model prefers to retrain the model as part of the process of modifying the model. If author B changes the training set and/or changes the training scripts, then the training set is part of the preferred form of modifying the model. The training set and the training scripts are both necessary for turning the training set into a running model, so I think author B would have to include the training set even if author B changes only the training scripts. Correct me if I'm wrong.) jncfhnb, you're like author A, so if you were to release an open source model then you would need to include the weights but not the training data. Trapais and nullc, don't assume that every model author is author B. For personal reference, here is the relevant excerpt from the open source definition from the Open Source Initiative [A1]: > The source code must be the preferred form in which a programmer would modify the program. Deliberately obfuscated source code is not allowed. Intermediate forms such as the output of a preprocessor or translator are not allowed. Open source software is not the same as free software, but here is the relevant excerpt from the free software definition explainer from the Free Software Foundation [A2]: > Source code is defined as the preferred form of the program for making changes in. Thus, whatever form a developer changes to develop the program is the source code of that developer's version. [A1] https://opensource.org/osd/ https://opensource.org/osd/ [A2] https://www.gnu.org/philosophy/free-sw.html https://www.gnu.org/philosophy/free-sw.html