3 ms·
I’ve disputed that fact before. It depends what part of your data is generated from GPT4 if I have high quality code and I’m using gpt4 to synthetically generat
by ianbutler 3y ago
I’ve disputed that fact before. It depends what part of your data is generated from GPT4 if I have high quality code and I’m using gpt4 to synthetically generate variations on how some could ask for the code to be written it’s entirely possible because of the high quality code for the model to be better. It’s not all or nothing if you’re mixing synthetic data with quality sources.
In this case even though parts of the dataset are synthetic the bound is on the code not necessarily the 50 ways I got gpt4 to say “write me a script to do x” or modeled other interactions with that code data source.
- PrayagBhakar 3y agoIf we’re talking about model distillation[0] I don’t think the student can ever be better than the teacher as optimising for speed and smaller model sizes inherently means that there will be precision loss. Even if the student is as big as the teacher, there is still data loss. [0] https://arxiv.org/pdf/2210.17332.pdf https://arxiv.org/pdf/2210.17332.pdf