3 ms·
> I've wondered if one could train a LLM on a closed set of curated knowledge. Then include training data that models the behaviour of not knowing. To the point
by aidenn0 3y ago
> I've wondered if one could train a LLM on a closed set of curated knowledge. Then include training data that models the behaviour of not knowing. To the point that it could generalize to being able to represent its own not knowing.
The problem is that curating data is slow and expensive and downloading the entire web is fast and cheap.
See also https://en.wikipedia.org/wiki/Cyc https://en.wikipedia.org/wiki/Cyc
- philipswood 3y agoAgreed. Using LLM to generate or curate training sets for other generations seems like a cool approach. Maybe if you trained a small base model to know it doesn't know in general and THEN trained it on the entire web with embedded not-knowing preserving training examples, it would work?
- philipswood 3y agoReminded of this approach where a tiny model was trained children's stories generated by a larger model: https://www.quantamagazine.org/tiny-language-models-thrive-with-gpt-4-as-a-teacher-20231005/ https://www.quantamagazine.org/tiny-language-models-thrive-w...