5 ms·

> 1. GPT-3 can speak native-level french despite the language only being 1.8% of it's training corpus. I personally think 1.8% is a lot. If I lived in France f

by geocar 4y ago

> 1. GPT-3 can speak native-level french despite the language only being 1.8% of it's training corpus. I personally think 1.8% is a lot. If I lived in France for 1.8% of my life, and aggressively consumed only French for that time, I would probably speak French too: I learned English faster than that. > 2. GPT-3 can follow instruction in other languages just fine despite the instruct tuning data being almost entirely in English. Almost every language in Europe shares a lot of words and idioms, and English took every one it could, so training on English is training it on those other languages. > 3. LLMs trained on code demonstrably reason better. What? No. Nonsense. I haven't seen any demonstration of reasoning. LLM's are more like the cargo-cult tribes watching things they don't understand and mimicking those things; Sure, they built an airport, but it doesn't work, and that makes all the difference. > It's fairly obvious pre-trained LLMs aren't really operating at just the word/token level. What's obvious is that there's a relationship between words/tokens and ideas. What is not obvious is how far that relationship goes.

famouswaffles 4y ago
>personally think 1.8% is a lot. It's not lol. If the model was only trained on that content, it wouldn't be anywhere near as good at french. That's a fact. And French is one thing but they're are languages it's fluent in with less data. >Almost every language in Europe shares a lot of words and idioms, and English took every one it could, so training on English is training it on those other languages. First of all, what you're saying is straight up nonsensical. Knowing English doesn't make you even close to competent on the vast majority of European languages. Secondly, it has nothing to do with European languages. It's excellent following instructions in Mandarin (0.1%) and Korean(0.02%), languages that had much less than 1.8% data. >What? No. Nonsense. I haven't seen any demonstration of reasoning. Many benchmarks that test reasoning. LLMs trained on code perform better here. Simple as that. >What is not obvious is how far that relationship goes. Only if you're being obtuse lol.
- PheonixPharts 4y ago
  I really think you need to research a bit more about how statistical language models work. The reason this models are able to learn other languages so quickly is that they're using the majority language to do much of the heavy lifting in defining the overall latent language space. Once this is established, adding new languages requires much less information since you only have to learn to encode and slightly tweak the latent presentation you learn. Again the view of statistical NLP, from LSA to GPT-N is that the words we write are generated from transformation of a latent space. For LSA this transformation is linear and the space is low dimensional, and the model only learn to encode words to the space. For GPT-N this transformation is non-linear, the latent space is very high dimensional and the model learns both the encoder and a decoder allowing it to generate language from the latent space. If you have enough language to model this latent space well, then learning additional languages requires very little training data since the space itself is well modeled, you just need to learn the details of the encoder/decoder. That is you've learned the abstract point in high dimensional space that represents the English word "cat", you just need to learn that "猫" points to the same place. Of course these models aren't just representing words/tokens without context to the latent space (like word vector models had done in the past) but an entire sequence of words. Interestingly enough even with word vectors you can learn word vectors in language A and also language B and then manually align the latent spaces in order to get simple translation for close to free. So given the training data size of GPT-3+ 0.02% is a lot of language.
  - famouswaffles 4y ago
    I know how language models work. It's funny how you paint this obvious in retrospect. Guess Jan Leike doesn't know how statistical language models work. https://mobile.twitter.com/janleike/status/1625207251630960640?t=Z3RCi1T9TFAY9ylf8Y0vXA&s=19 https://mobile.twitter.com/janleike/status/16252072516309606...
    - PheonixPharts 4y ago
      Jan Leike's point is a bit more subtle there. He's not surprised it can learn translation with a relatively small number of training examples, but surprised that it seems to know some instruction with no examples. And just a few Tweets later speculates it's likely because there are examples in the training data, and the success of the model to follow instructions in a non-English language from "nothing" is proportional to the presence of those examples in the training data [0]. I don't think he would disagree with anything in my statement. 0. https://mobile.twitter.com/janleike/status/1625215091988926464 https://mobile.twitter.com/janleike/status/16252150919889264...
- geocar 4y ago
  > > Almost every language in Europe shares a lot of words and idioms, and English took every one it could, so training on English is training it on those other languages > Knowing English doesn't make you even close to competent on the vast majority of European languages. I cannot fathom the mind that thinks these two statements occurred in the same conversation.
  - famouswaffles 4y ago
    Uh..Okay ? Lol

by geocar 4y ago

famouswaffles 4y ago

>personally think 1.8% is a lot. It's not lol. If the model was only trained on that content, it wouldn't be anywhere near as good at french. That's a fact. And French is one thing but they're are languages it's fluent in with less data. >Almost every language in Europe shares a lot of words and idioms, and English took every one it could, so training on English is training it on those other languages. First of all, what you're saying is straight up nonsensical. Knowing English doesn't make you even close to competent on the vast majority of European languages. Secondly, it has nothing to do with European languages. It's excellent following instructions in Mandarin (0.1%) and Korean(0.02%), languages that had much less than 1.8% data. >What? No. Nonsense. I haven't seen any demonstration of reasoning. Many benchmarks that test reasoning. LLMs trained on code perform better here. Simple as that. >What is not obvious is how far that relationship goes. Only if you're being obtuse lol.

PheonixPharts 4y ago
I really think you need to research a bit more about how statistical language models work. The reason this models are able to learn other languages so quickly is that they're using the majority language to do much of the heavy lifting in defining the overall latent language space. Once this is established, adding new languages requires much less information since you only have to learn to encode and slightly tweak the latent presentation you learn. Again the view of statistical NLP, from LSA to GPT-N is that the words we write are generated from transformation of a latent space. For LSA this transformation is linear and the space is low dimensional, and the model only learn to encode words to the space. For GPT-N this transformation is non-linear, the latent space is very high dimensional and the model learns both the encoder and a decoder allowing it to generate language from the latent space. If you have enough language to model this latent space well, then learning additional languages requires very little training data since the space itself is well modeled, you just need to learn the details of the encoder/decoder. That is you've learned the abstract point in high dimensional space that represents the English word "cat", you just need to learn that "猫" points to the same place. Of course these models aren't just representing words/tokens without context to the latent space (like word vector models had done in the past) but an entire sequence of words. Interestingly enough even with word vectors you can learn word vectors in language A and also language B and then manually align the latent spaces in order to get simple translation for close to free. So given the training data size of GPT-3+ 0.02% is a lot of language.
- famouswaffles 4y ago
  I know how language models work. It's funny how you paint this obvious in retrospect. Guess Jan Leike doesn't know how statistical language models work. https://mobile.twitter.com/janleike/status/1625207251630960640?t=Z3RCi1T9TFAY9ylf8Y0vXA&s=19 https://mobile.twitter.com/janleike/status/16252072516309606...
  - PheonixPharts 4y ago
    Jan Leike's point is a bit more subtle there. He's not surprised it can learn translation with a relatively small number of training examples, but surprised that it seems to know some instruction with no examples. And just a few Tweets later speculates it's likely because there are examples in the training data, and the success of the model to follow instructions in a non-English language from "nothing" is proportional to the presence of those examples in the training data [0]. I don't think he would disagree with anything in my statement. 0. https://mobile.twitter.com/janleike/status/1625215091988926464 https://mobile.twitter.com/janleike/status/16252150919889264...
geocar 4y ago
> > Almost every language in Europe shares a lot of words and idioms, and English took every one it could, so training on English is training it on those other languages > Knowing English doesn't make you even close to competent on the vast majority of European languages. I cannot fathom the mind that thinks these two statements occurred in the same conversation.
- famouswaffles 4y ago
  Uh..Okay ? Lol