4 ms·
Does training an LLM on a variety of different languages improve its reasoning and understanding in other languages? For example, if GPT-5 (or whatever next ge
by crakenzak 3y ago
Does training an LLM on a variety of different languages improve its reasoning and understanding in other languages?
For example, if GPT-5 (or whatever next gen model) was trained on a staggering amount of high quality Norwegian tokens that talk about medicine, city life, romance, maths, etc... would it be able to apply those concepts & learnings to other languages?
I'm trilingual and have ~100% of transfer of knowledge/reading I learn/read in one language into the others, curious to see if LLMs are like that.
- vidarh 3y agoI think how well it handles Norwegian despite the small number of Norwegian tokens present for GPT3 (I'm not aware of up to date overviews of token distributions in the training data) is a good indicator that a lot of it does transfer.
- huac 3y agothe 'low resource languages jailbreak gpt-4' paper (https://arxiv.org/abs/2310.02446v1 https://arxiv.org/abs/2310.02446v1) shows that the models, when prompted in a low resource language, return highly coherent and sophisticated responses. it's possible that RLHF does not do a great job here but the base models are clearly cross-applying intelligence from high resource languages to low resource languages.