3 ms·
When LLMs are trained on text, are the words annotated to indicate the semantic meaning, or is the LLM training process expected to disambiguate the possibly hu
by Merrill 3y ago
When LLMs are trained on text, are the words annotated to indicate the semantic meaning, or is the LLM training process expected to disambiguate the possibly hundreds of semantic meanings of an individual common word such as "run"?
- OkayPhysicist 3y agoThe task LLMs are trained on is "predict the next word", which elegantly is included for free in your training set of text. Typically no annotation is provided, since that would involve a ton of human labor doing the annotations.
- Merrill 3y agoCould you ask a specialized AI to define each of the words in a block of text to automate the meaning extraction?
- IanCal 3y agoProbably. There's good work showing impressively performing small models that were trained on more "text book" like data rather than just loads of text - but where the "text books" were either wholly or largely created by another AI model. Using models to generate/score/rank/modify data to be more useful as training data is a very interesting angle.