3 ms·
Agreed. Using LLM to generate or curate training sets for other generations seems like a cool approach. Maybe if you trained a small base model to know it does
by philipswood 3y ago
Agreed. Using LLM to generate or curate training sets for other generations seems like a cool approach.
Maybe if you trained a small base model to know it doesn't know in general and THEN trained it on the entire web with embedded not-knowing preserving training examples, it would work?
- philipswood 3y agoReminded of this approach where a tiny model was trained children's stories generated by a larger model: https://www.quantamagazine.org/tiny-language-models-thrive-with-gpt-4-as-a-teacher-20231005/ https://www.quantamagazine.org/tiny-language-models-thrive-w...