5 ms·

I have had this question. How much better would common LLMs (Llama, GPTN) be if they were only trained in one language? I have to assume they would perform bett

I have had this question. How much better would common LLMs (Llama, GPTN) be if they were only trained in one language? I have to assume they would perform better, but I might be wrong.

fermuch 2y ago
Just like adding code to textual models helps the model develop its reasoning capabilities, it seems like adding more languages helps in other areas too. What is needed is more good quality data to train on...
- nickpsecurity 2y ago
  We also see humans get worse at specific things when they learn too much in general. There is a cut-off point to how many concepts we can learn with what skill. To be most effective, we have to specialize in the right things while continuing to acquire generalist knowledge. It’s a balancing act. These architectures are less capable than brains in many ways. So, we should expect them to have such trade-offs. An efficient one should work fine on English, mathematical notation, and a programming language. Maybe samples of others that illustrate unique concepts. I’m also curious how many languages or concepts you can add to a given architecture before its effectiveness starts dropping.
- worldsayshi 2y ago
  I guess you mean non-textual data then because the amount of text data they are being trained on ought to be enough for agi by now? Some kind of diminishing returns asymptote from text volume alone must have been hit a long time ago.
  - imtringued 2y ago
    It's not the amount that is wrong, it's how the model is trained. The model is trained for zero and few shot tasks. It is not surprising that it is performing well when you ask for that.
- darby_eight 2y ago
  > its reasoning capabilities To be clear, LLMs are not capable of reasoning.
  - whimsicalism 2y ago
    imo this is an uninteresting debate over semantics/metaphysics
    - ganzuul 2y ago
      Would you say a deontologist reasons? Evolution survives, but does it reason? Is it reasonable to show interest in something you call uninteresting? Was Gödel a reasonable man, starving to death in fear of being poisoned?
coffeebeqn 2y ago
Perform better how? Knowing more languages gives you more data and different points of view rather than just using the English corpus and culture. When I ask chatgpt for a translation it seems to understand the meaning behind the words and finds the closest thing in the other language. The datasets seem to merge in some way
- ClarityJones 2y ago
  Fair, but there may be overhead that doesn't need to exist. Certainly - for the limited compute my brain can accomplish - I could gain a deeper understanding of physics, if I focused on learning physics and didn't also have to simultaneously learn French.
  - NhanH 2y ago
    Human is not limited by computational power of brain (or rather, it is not the limitation we encounter). We are limited by time and the fact that our machinery degrades with time (aging).
  - olddustytrail 2y ago
    In the short term. In the longer term you'll understand concepts better when you're multilingual.
  - staticman2 2y ago
    Wouldn't a better metaphor be if a child growing up in a bilingual household would be worse at physics as an adult? My guess would be growing up bilingual would have no impact.
    - fikama 2y ago
      This hypotetical kid would have the same size of brain/number of neurons anyway. In case of LLMs one could create a model that could be smaller thakns to not including the knowlegde about unecessary languages. A problem though could be with lacking traing data in other languages.
whimsicalism 2y ago
they would perform worse, i promise you
- rangerelf 2y ago
  "I promise you"? This is Hackernews, I would have expected data, not promises.
- ClarityJones 2y ago
  I think this makes sense to the extent that an understanding of the differences between language helps separate out language from the underlying meaning. However... the models that are used receive input (i.e. translate from language), and to learn / understand, and to output information (i.e. re-encode into language), do not all have to be the same.
richdougherty 2y ago
I can't track down the citation (either Google or DeepMind I think), but I remember reading research from a year or two ago how adding extra languages (French, German) improved English language performance. There may have also been an investigation about multi modality too, which found that adding vision or audio helped with text as well.
phlip9 2y ago
Interesting thought. Maybe an LLM would build deeper insight with only one training language. On the other hand, the model might overfit with just one language -- maybe multilingual models generalize better?

I have had this question. How much better would common LLMs (Llama, GPTN) be if they were only trained in one language? I have to assume they would perform better, but I might be wrong.

fermuch 2y ago

Just like adding code to textual models helps the model develop its reasoning capabilities, it seems like adding more languages helps in other areas too. What is needed is more good quality data to train on...

nickpsecurity 2y ago
We also see humans get worse at specific things when they learn too much in general. There is a cut-off point to how many concepts we can learn with what skill. To be most effective, we have to specialize in the right things while continuing to acquire generalist knowledge. It’s a balancing act. These architectures are less capable than brains in many ways. So, we should expect them to have such trade-offs. An efficient one should work fine on English, mathematical notation, and a programming language. Maybe samples of others that illustrate unique concepts. I’m also curious how many languages or concepts you can add to a given architecture before its effectiveness starts dropping.
worldsayshi 2y ago
I guess you mean non-textual data then because the amount of text data they are being trained on ought to be enough for agi by now? Some kind of diminishing returns asymptote from text volume alone must have been hit a long time ago.
- imtringued 2y ago
  It's not the amount that is wrong, it's how the model is trained. The model is trained for zero and few shot tasks. It is not surprising that it is performing well when you ask for that.
darby_eight 2y ago
> its reasoning capabilities To be clear, LLMs are not capable of reasoning.
- whimsicalism 2y ago
  imo this is an uninteresting debate over semantics/metaphysics
  - ganzuul 2y ago
    Would you say a deontologist reasons? Evolution survives, but does it reason? Is it reasonable to show interest in something you call uninteresting? Was Gödel a reasonable man, starving to death in fear of being poisoned?

coffeebeqn 2y ago

Perform better how? Knowing more languages gives you more data and different points of view rather than just using the English corpus and culture. When I ask chatgpt for a translation it seems to understand the meaning behind the words and finds the closest thing in the other language. The datasets seem to merge in some way

ClarityJones 2y ago
Fair, but there may be overhead that doesn't need to exist. Certainly - for the limited compute my brain can accomplish - I could gain a deeper understanding of physics, if I focused on learning physics and didn't also have to simultaneously learn French.
- NhanH 2y ago
  Human is not limited by computational power of brain (or rather, it is not the limitation we encounter). We are limited by time and the fact that our machinery degrades with time (aging).
- olddustytrail 2y ago
  In the short term. In the longer term you'll understand concepts better when you're multilingual.
- staticman2 2y ago
  Wouldn't a better metaphor be if a child growing up in a bilingual household would be worse at physics as an adult? My guess would be growing up bilingual would have no impact.
  - fikama 2y ago
    This hypotetical kid would have the same size of brain/number of neurons anyway. In case of LLMs one could create a model that could be smaller thakns to not including the knowlegde about unecessary languages. A problem though could be with lacking traing data in other languages.

whimsicalism 2y ago

they would perform worse, i promise you

rangerelf 2y ago
"I promise you"? This is Hackernews, I would have expected data, not promises.
ClarityJones 2y ago
I think this makes sense to the extent that an understanding of the differences between language helps separate out language from the underlying meaning. However... the models that are used receive input (i.e. translate from language), and to learn / understand, and to output information (i.e. re-encode into language), do not all have to be the same.

richdougherty 2y ago

I can't track down the citation (either Google or DeepMind I think), but I remember reading research from a year or two ago how adding extra languages (French, German) improved English language performance. There may have also been an investigation about multi modality too, which found that adding vision or audio helped with text as well.

phlip9 2y ago

Interesting thought. Maybe an LLM would build deeper insight with only one training language. On the other hand, the model might overfit with just one language -- maybe multilingual models generalize better?