4 ms·

I wasn't talking about "state of the art LLMs," I am aware that commercial offerings are much better trained in Spanish. This was a thought experiment based on

I wasn't talking about "state of the art LLMs," I am aware that commercial offerings are much better trained in Spanish. This was a thought experiment based on comments from people testing GPT-3.5 with Swahili. > You can definitely query knowledge only learnt in one language in other languages. Do you have a source on that? I believe this is simply not true, unless maybe the pretraining data has enough context-specific "bridge translations." And I am not sure how on earth you would verify that any major LLM only learnt something in one language. What if the pretraining data includes machine translations? Frustratingly, just a few months ago I read a paper describing how LLMs excessively rely on English-language representations of ideas, but now I can't find it. So I can't really criticize you if you don't have a source :) The argument was essentially what I said above: since LLMs associate tokens by related tokens, not ideas by related ideas, the emergent conceptual relations formed around the token "cat" do not have any means of transferring to conceptual relations around the token "gato."

famouswaffles 2y ago
>I wasn't talking about "state of the art LLMs," I am aware that commercial offerings are much better trained in Spanish. This was a thought experiment based on comments from people testing GPT-3.5 with Swahili. A thought experiment from other people comments on another language. So...No. Fabricating failure modes from their personally constructed ideas about how LLMs work seems to be a frustratingly common occurrence in these kinds of discussions. >Frustratingly, just few months ago I read a paper describing how LLMs excessively rely on English-language representations of ideas, but now I can't find it. Most LLMs are trained on English overwhelmingly. GPT-3 had a 92.6% English dataset. https://github.com/openai/gpt-3/blob/master/dataset_statistics/languages_by_word_count.csv https://github.com/openai/gpt-3/blob/master/dataset_statisti... That the models are as proficient as they are is evidence enough of knowledge transfer clearly happening. https://arxiv.org/abs/2108.13349 https://arxiv.org/abs/2108.13349. If you trained a model on the Catalan tokens GPT-3 was trained on alone, you'd just get a GPT-2 level gibberish model at best. I don't doubt you, i just don't think it means what you think it means. As for papers, these are some interesting ones. How do languages influence each other? Studying cross-lingual data sharing during LLM fine-tuning - https://arxiv.org/pdf/2305.13286 https://arxiv.org/pdf/2305.13286 Teaching Llama a New Language Through Cross-Lingual Knowledge Transfer - https://arxiv.org/abs/2404.04042 https://arxiv.org/abs/2404.04042 Zero-shot cross-lingual transfer in instruction tuning of large language models - https://arxiv.org/abs/2402.14778 https://arxiv.org/abs/2402.14778 Multilingual LLMs are Better Cross-lingual In-context Learners with Alignment - https://arxiv.org/abs/2305.05940 https://arxiv.org/abs/2305.05940 It's not like there is perfect transfer but the idea that there's none at all seemed so ridiculous to me (and why i asked the first question). Models would be utterly useless in multilingual settings if that were really the case. Like the 3rd paper, GPT's instruction tuning is done basically only in English - https://mobile.twitter.com/janleike/status/1625207251630960640?t=Z3RCi1T9TFAY9ylf8Y0vXA&s=19 https://mobile.twitter.com/janleike/status/16252072516309606...

famouswaffles 2y ago

>I wasn't talking about "state of the art LLMs," I am aware that commercial offerings are much better trained in Spanish. This was a thought experiment based on comments from people testing GPT-3.5 with Swahili. A thought experiment from other people comments on another language. So...No. Fabricating failure modes from their personally constructed ideas about how LLMs work seems to be a frustratingly common occurrence in these kinds of discussions. >Frustratingly, just few months ago I read a paper describing how LLMs excessively rely on English-language representations of ideas, but now I can't find it. Most LLMs are trained on English overwhelmingly. GPT-3 had a 92.6% English dataset. https://github.com/openai/gpt-3/blob/master/dataset_statistics/languages_by_word_count.csv https://github.com/openai/gpt-3/blob/master/dataset_statisti... That the models are as proficient as they are is evidence enough of knowledge transfer clearly happening. https://arxiv.org/abs/2108.13349 https://arxiv.org/abs/2108.13349. If you trained a model on the Catalan tokens GPT-3 was trained on alone, you'd just get a GPT-2 level gibberish model at best. I don't doubt you, i just don't think it means what you think it means. As for papers, these are some interesting ones. How do languages influence each other? Studying cross-lingual data sharing during LLM fine-tuning - https://arxiv.org/pdf/2305.13286 https://arxiv.org/pdf/2305.13286 Teaching Llama a New Language Through Cross-Lingual Knowledge Transfer - https://arxiv.org/abs/2404.04042 https://arxiv.org/abs/2404.04042 Zero-shot cross-lingual transfer in instruction tuning of large language models - https://arxiv.org/abs/2402.14778 https://arxiv.org/abs/2402.14778 Multilingual LLMs are Better Cross-lingual In-context Learners with Alignment - https://arxiv.org/abs/2305.05940 https://arxiv.org/abs/2305.05940 It's not like there is perfect transfer but the idea that there's none at all seemed so ridiculous to me (and why i asked the first question). Models would be utterly useless in multilingual settings if that were really the case. Like the 3rd paper, GPT's instruction tuning is done basically only in English - https://mobile.twitter.com/janleike/status/1625207251630960640?t=Z3RCi1T9TFAY9ylf8Y0vXA&s=19 https://mobile.twitter.com/janleike/status/16252072516309606...