4 ms·

> LLMs do not learn to translate through explicit examples of parallel text. Of course they do, lots of that is in the training data.

andrewla 3y ago
For general-purpose LLMs, there is no indication given during training that texts are parallel -- while there are parallel texts in the training data, the model is not trained on generating one text from another. That is, it does not get any feedback during training about translation, only prediction in the current token stream, not parallel ones. There are several completely unsupervised language translation systems that predate LLMs, but the performance is middling. For LLMs the translation behavior is largely an emergent property that is not completely understood; if we can tokenize whale language in a useful way, it is entirely possible that the LLM can derive a weak or approximate translation of some of the language structure.
- janalsncm 3y ago
  If you have a parallel corpus in your training data you don’t need to explicitly give feedback about translation. The loss function is the feedback. Maybe a better example is in the multimodal context. Drawing images from words is a type of “translation”. But for this task we need captions, i.e. a parallel corpus.
  - famouswaffles 3y ago
    It's still not necessary to have bilingual data https://news.ycombinator.com/item?id=38574712 https://news.ycombinator.com/item?id=38574712
    - janalsncm 3y ago
      First of all, thank you for finding this paper, it seems to be addressing the question directly. However, I believe you are overstating the results of the paper. If anything, the paper demonstrated massive reductions in zero shot translation capabilities after removing translation. And for languages with no cognates with English, BLEU scores are pretty abysmal. So the paper suggests that parallel corpuses still very important. For whale languages, you won’t have any cognates with English, and we don’t even know if they have a grammar that is remotely human.
      - famouswaffles 3y ago
        Language Models trained at the size and data they tested are pretty terrible to begin with, translation or not. Like I said, the crucial thing is that it looks to be a diminishing gap. I'm not saying parallel corpora is doing nothing or is unimportant but that's a clue that monolingual competency of the languages in question is the biggest factor here. The extreme positive transfer language models exhibit in terms of multilingual competency is another clue. https://arxiv.org/abs/2108.13349 https://arxiv.org/abs/2108.13349 Parallel corpora may be a crutch with fading relevance.
  - andrewla 3y ago
    > The loss function is the feedback Please expand -- as I understand it the LLMs use cross entropy on next token prediction as their loss function and I fail to see how this gives any feedback about translation, even given parallel texts. Predicting the next token in a French language text and predicting the next token in an English language text are not obviously more interesting than dealing with non-parallel texts in both languages. It's totally possible that these parallel texts are critical in a sense to the translation capabilities but this is not obvious. It would be incredibly difficult to establish empirically because reducing training data reduces the effectiveness at all tasks. Translation, like almost all of the capabilities of LLMs, is an emergent behavior that we don't fully understand yet.

andrewla 3y ago

For general-purpose LLMs, there is no indication given during training that texts are parallel -- while there are parallel texts in the training data, the model is not trained on generating one text from another. That is, it does not get any feedback during training about translation, only prediction in the current token stream, not parallel ones. There are several completely unsupervised language translation systems that predate LLMs, but the performance is middling. For LLMs the translation behavior is largely an emergent property that is not completely understood; if we can tokenize whale language in a useful way, it is entirely possible that the LLM can derive a weak or approximate translation of some of the language structure.

janalsncm 3y ago
If you have a parallel corpus in your training data you don’t need to explicitly give feedback about translation. The loss function is the feedback. Maybe a better example is in the multimodal context. Drawing images from words is a type of “translation”. But for this task we need captions, i.e. a parallel corpus.
- famouswaffles 3y ago
  It's still not necessary to have bilingual data https://news.ycombinator.com/item?id=38574712 https://news.ycombinator.com/item?id=38574712
  - janalsncm 3y ago
    First of all, thank you for finding this paper, it seems to be addressing the question directly. However, I believe you are overstating the results of the paper. If anything, the paper demonstrated massive reductions in zero shot translation capabilities after removing translation. And for languages with no cognates with English, BLEU scores are pretty abysmal. So the paper suggests that parallel corpuses still very important. For whale languages, you won’t have any cognates with English, and we don’t even know if they have a grammar that is remotely human.
    - famouswaffles 3y ago
      Language Models trained at the size and data they tested are pretty terrible to begin with, translation or not. Like I said, the crucial thing is that it looks to be a diminishing gap. I'm not saying parallel corpora is doing nothing or is unimportant but that's a clue that monolingual competency of the languages in question is the biggest factor here. The extreme positive transfer language models exhibit in terms of multilingual competency is another clue. https://arxiv.org/abs/2108.13349 https://arxiv.org/abs/2108.13349 Parallel corpora may be a crutch with fading relevance.
- andrewla 3y ago
  > The loss function is the feedback Please expand -- as I understand it the LLMs use cross entropy on next token prediction as their loss function and I fail to see how this gives any feedback about translation, even given parallel texts. Predicting the next token in a French language text and predicting the next token in an English language text are not obviously more interesting than dealing with non-parallel texts in both languages. It's totally possible that these parallel texts are critical in a sense to the translation capabilities but this is not obvious. It would be incredibly difficult to establish empirically because reducing training data reduces the effectiveness at all tasks. Translation, like almost all of the capabilities of LLMs, is an emergent behavior that we don't fully understand yet.