7 ms·

In the grand scheme of things GPT isn't learning "meaning" in any different sense than Latent Semantic Analysis (LSA) learns meaning (which is just using "perce

In the grand scheme of things GPT isn't learning "meaning" in any different sense than Latent Semantic Analysis (LSA) learns meaning (which is just using "percent chance that some token should show up next" using covariance). For those without a background in NLP, LSA is literally SVD/PCA (which is ultimately just based on the covariance matrix) on the term frequency matrix for a corpus of text. However LSA is well known to learn certain meanings such as "pet" by projecting words like "dog" and "cat" onto the same principle component. Semantics is learned by exploring the "loadings" in PCA terminology. Now LSA is just a linear transformation, but the idea is that same as GPT that uses a wildly more sophisticated non-linear transformation: the assumption is that words (or tokens in the case of GPT) are caused by some latent variables (in the case of LSA, for GPT high dimensional embedding) and most importantly the patterns of word/token usage allow us to infer what those latent variables are. So it's not absurd and it's how statistical NLP has always worked, only now we have much, much more sophisticated ways to model that latent space. LSA, HMM, and Transformers all assume that the patterns of text usage can be used to model an unobserved latent space that generates the text, which is what we traditionally call "meaning". Ultimately the limits of GPT-N (unless there are none and it does become AGI) will be valuable in showing the limitations of this fundamental model of semantics.

knome 4y ago
When I said it operates on a level of meaning, I refer to its ability to perform transformations like this: H: translate the following sentence to instead use olympic curling related metaphors ( do not explicitly mention curling, and do not use similes in forming the translation ): man:"I've been spinning my tires all day without getting anywhere" woman:"running out of gas?" man:"nah, I'm gassed up, geared up, oiled and revving. I just can't seem to get in the right lane, you know?" woman:"anything knocking?" man:"nah, I'm tuned just fine. I can't seem to get my GPS to connect, you know?" woman:"take a few laps without a destination, see if you can figure out the lay of the land" man:"you know, that might help. thanks barbara" R: man: "I've been sweeping the ice all day without hitting the target." . woman: "Losing momentum?" . man: "Nah, I've got the energy, the technique, and the focus. I just can't seem to find the right path, you know?" . woman: "Anything coming in your way?" . man: "Nah, I'm on point with my aim. I just can't seem to get my strategy to align, you know?" . woman: "Maybe glide a few times without aiming specifically, see if you can figure out the ice conditions." . man: "You know, that might help. Thanks, Barbara." I don't think you can perform this kind of translation by merely comparing tokens statistically tending to be near one another. I understand that this is how the data was encoded and the output judged. However, the neural net between those two points is free to encode whatever interconnecting values it happens to as its training drives it through to whatever multidimensional local maximum it manages to settle upon. In short, I expect that it has to translate to and from an abstract meaning in order to manipulate metaphors in such a manner. Most human language is metaphorical, as discussed in Lakoff & Johnson's "Metaphors We Live By". We have a core of experiential terms, but near the whole of the rest of our language is metaphoric in nature. The better a model gets at "understanding" English, the closer to a metaphor engine it must become, needing to be capable of applying arbitrary metaphors and partial metaphors in order to properly manipulate the language. I'm not saying your position is wrong, but it reminds me of the contrasting viewpoints given in Hofstadter's "I am a Strange Loop" as he discussed his "careenium" and "simm balls" analogy for the rise of complex epiphenomena arising from simple rule bound systems.
- YeGoblynQueenne 4y ago
  That's amazing and all, but it's also speculation, yes? It's how you imagine, or guess, that things are happening. It's not like you, or anyone else, knows that this kind of thing is really happening. Moreover, there isn't any clear reason why you need to speculate all that, in order to explain the observed behaviour of Large Language Models (LLMs). You say you don't think LLMs "can perform this kind of translation by merely comparing tokens statistically tending to be near one another". And yet, that's what we know that LLMs do, because that's what they are made to do [edit: they do what you describe after the "by merely"]. Why do we need to imagine some other, so far unobserved, process at play? What is the motivation? What is the justification? Why do you think LLM's can't "perform this kind of translation by merely comparing tokens statistically tending to be near one another". It's clear what you are saying, but, why? Why are you saying that? Why should we assume that the well-known and understood design and functionality of LLMs is not enough to explain their behaviour? Is it because we haven't observed such behaviour before and some people are surprised by it? Well, we didn't have as large language models before, so obviously we didn't have a chance to observe their behaviour first-hand, but we also didn't have cars that could go 100 miles an hour, until we did, and making a faster car is no reason to be surprised, or wonder "how does it do it if it's just a car?".
  - knome 4y ago
    >That's amazing and all It earnestly is amazing. >Why do you think LLM's can't "perform this kind of translation by merely comparing tokens statistically tending to be near one another". The implication I gather from those dismissing this technology as "merely comparing tokens statistically tending to be near one another" is that they think the whole of the LLM is nothing but a trick, a sleight of hand where it generates output token by token Markov style after consuming enough text, just using some percent chance of tokens following on after another. In my example, it generated the metaphor "sweeping the ice" as a replacement for "spinning my tires". Both are using in context to imply a level of busywork associated with the respective contexts. To generate such a transformation requires understanding this inner meaning and associating it. It requires a level of creativity to determine that "sweeping the ice" in curling is an appropriate replacement. You cannot rearrange "spinning my tires" into "sweeping the ice" without having derived and translated the underlying meaning. There is nothing to connect the two sets of tokens except the metaphoric intent. >Why should we assume that the well-known and understood design and functionality of LLMs is not enough to explain their behaviour? Obviously LLMs are capable of this as it is an LLM that is producing it. I am not denying that the inputs and outputs are tokens marked up with distance information, nor that the process of generating tokens is just a bunch of vector multiplication to generate neuron values from weights across the layers of the network. I am suggesting that with sufficient training, layers and parameters, that the neural net must be forced into a configuration that allows it to perform this meaning/intent based processing as gradient descent pushes and pulls its the network along the twelve thousand or so dimensions GPT-3 uses. Undoubtedly GPT-4 and soon 5 use even more. LLMs have repeatedly been found to have developed novel and unintended capabilities as their complexity has increased. Why do you think that LLMs shouldn't be capable of manipulating on the level of meanings and intents rather than just rearranging tokens?
famouswaffles 4y ago
1. GPT-3 can speak native-level french despite the language only being 1.8% of it's training corpus. 2. GPT-3 can follow instruction in other languages just fine despite the instruct tuning data being almost entirely in English. 3. LLMs trained on code demonstrably reason better. It's fairly obvious pre-trained LLMs aren't really operating at just the word/token level.
- geocar 4y ago
  > 1. GPT-3 can speak native-level french despite the language only being 1.8% of it's training corpus. I personally think 1.8% is a lot. If I lived in France for 1.8% of my life, and aggressively consumed only French for that time, I would probably speak French too: I learned English faster than that. > 2. GPT-3 can follow instruction in other languages just fine despite the instruct tuning data being almost entirely in English. Almost every language in Europe shares a lot of words and idioms, and English took every one it could, so training on English is training it on those other languages. > 3. LLMs trained on code demonstrably reason better. What? No. Nonsense. I haven't seen any demonstration of reasoning. LLM's are more like the cargo-cult tribes watching things they don't understand and mimicking those things; Sure, they built an airport, but it doesn't work, and that makes all the difference. > It's fairly obvious pre-trained LLMs aren't really operating at just the word/token level. What's obvious is that there's a relationship between words/tokens and ideas. What is not obvious is how far that relationship goes.
  - famouswaffles 4y ago
    >personally think 1.8% is a lot. It's not lol. If the model was only trained on that content, it wouldn't be anywhere near as good at french. That's a fact. And French is one thing but they're are languages it's fluent in with less data. >Almost every language in Europe shares a lot of words and idioms, and English took every one it could, so training on English is training it on those other languages. First of all, what you're saying is straight up nonsensical. Knowing English doesn't make you even close to competent on the vast majority of European languages. Secondly, it has nothing to do with European languages. It's excellent following instructions in Mandarin (0.1%) and Korean(0.02%), languages that had much less than 1.8% data. >What? No. Nonsense. I haven't seen any demonstration of reasoning. Many benchmarks that test reasoning. LLMs trained on code perform better here. Simple as that. >What is not obvious is how far that relationship goes. Only if you're being obtuse lol.

knome 4y ago

When I said it operates on a level of meaning, I refer to its ability to perform transformations like this: H: translate the following sentence to instead use olympic curling related metaphors ( do not explicitly mention curling, and do not use similes in forming the translation ): man:"I've been spinning my tires all day without getting anywhere" woman:"running out of gas?" man:"nah, I'm gassed up, geared up, oiled and revving. I just can't seem to get in the right lane, you know?" woman:"anything knocking?" man:"nah, I'm tuned just fine. I can't seem to get my GPS to connect, you know?" woman:"take a few laps without a destination, see if you can figure out the lay of the land" man:"you know, that might help. thanks barbara" R: man: "I've been sweeping the ice all day without hitting the target." . woman: "Losing momentum?" . man: "Nah, I've got the energy, the technique, and the focus. I just can't seem to find the right path, you know?" . woman: "Anything coming in your way?" . man: "Nah, I'm on point with my aim. I just can't seem to get my strategy to align, you know?" . woman: "Maybe glide a few times without aiming specifically, see if you can figure out the ice conditions." . man: "You know, that might help. Thanks, Barbara." I don't think you can perform this kind of translation by merely comparing tokens statistically tending to be near one another. I understand that this is how the data was encoded and the output judged. However, the neural net between those two points is free to encode whatever interconnecting values it happens to as its training drives it through to whatever multidimensional local maximum it manages to settle upon. In short, I expect that it has to translate to and from an abstract meaning in order to manipulate metaphors in such a manner. Most human language is metaphorical, as discussed in Lakoff & Johnson's "Metaphors We Live By". We have a core of experiential terms, but near the whole of the rest of our language is metaphoric in nature. The better a model gets at "understanding" English, the closer to a metaphor engine it must become, needing to be capable of applying arbitrary metaphors and partial metaphors in order to properly manipulate the language. I'm not saying your position is wrong, but it reminds me of the contrasting viewpoints given in Hofstadter's "I am a Strange Loop" as he discussed his "careenium" and "simm balls" analogy for the rise of complex epiphenomena arising from simple rule bound systems.

YeGoblynQueenne 4y ago
That's amazing and all, but it's also speculation, yes? It's how you imagine, or guess, that things are happening. It's not like you, or anyone else, knows that this kind of thing is really happening. Moreover, there isn't any clear reason why you need to speculate all that, in order to explain the observed behaviour of Large Language Models (LLMs). You say you don't think LLMs "can perform this kind of translation by merely comparing tokens statistically tending to be near one another". And yet, that's what we know that LLMs do, because that's what they are made to do [edit: they do what you describe after the "by merely"]. Why do we need to imagine some other, so far unobserved, process at play? What is the motivation? What is the justification? Why do you think LLM's can't "perform this kind of translation by merely comparing tokens statistically tending to be near one another". It's clear what you are saying, but, why? Why are you saying that? Why should we assume that the well-known and understood design and functionality of LLMs is not enough to explain their behaviour? Is it because we haven't observed such behaviour before and some people are surprised by it? Well, we didn't have as large language models before, so obviously we didn't have a chance to observe their behaviour first-hand, but we also didn't have cars that could go 100 miles an hour, until we did, and making a faster car is no reason to be surprised, or wonder "how does it do it if it's just a car?".
- knome 4y ago
  >That's amazing and all It earnestly is amazing. >Why do you think LLM's can't "perform this kind of translation by merely comparing tokens statistically tending to be near one another". The implication I gather from those dismissing this technology as "merely comparing tokens statistically tending to be near one another" is that they think the whole of the LLM is nothing but a trick, a sleight of hand where it generates output token by token Markov style after consuming enough text, just using some percent chance of tokens following on after another. In my example, it generated the metaphor "sweeping the ice" as a replacement for "spinning my tires". Both are using in context to imply a level of busywork associated with the respective contexts. To generate such a transformation requires understanding this inner meaning and associating it. It requires a level of creativity to determine that "sweeping the ice" in curling is an appropriate replacement. You cannot rearrange "spinning my tires" into "sweeping the ice" without having derived and translated the underlying meaning. There is nothing to connect the two sets of tokens except the metaphoric intent. >Why should we assume that the well-known and understood design and functionality of LLMs is not enough to explain their behaviour? Obviously LLMs are capable of this as it is an LLM that is producing it. I am not denying that the inputs and outputs are tokens marked up with distance information, nor that the process of generating tokens is just a bunch of vector multiplication to generate neuron values from weights across the layers of the network. I am suggesting that with sufficient training, layers and parameters, that the neural net must be forced into a configuration that allows it to perform this meaning/intent based processing as gradient descent pushes and pulls its the network along the twelve thousand or so dimensions GPT-3 uses. Undoubtedly GPT-4 and soon 5 use even more. LLMs have repeatedly been found to have developed novel and unintended capabilities as their complexity has increased. Why do you think that LLMs shouldn't be capable of manipulating on the level of meanings and intents rather than just rearranging tokens?

famouswaffles 4y ago

1. GPT-3 can speak native-level french despite the language only being 1.8% of it's training corpus. 2. GPT-3 can follow instruction in other languages just fine despite the instruct tuning data being almost entirely in English. 3. LLMs trained on code demonstrably reason better. It's fairly obvious pre-trained LLMs aren't really operating at just the word/token level.

geocar 4y ago
> 1. GPT-3 can speak native-level french despite the language only being 1.8% of it's training corpus. I personally think 1.8% is a lot. If I lived in France for 1.8% of my life, and aggressively consumed only French for that time, I would probably speak French too: I learned English faster than that. > 2. GPT-3 can follow instruction in other languages just fine despite the instruct tuning data being almost entirely in English. Almost every language in Europe shares a lot of words and idioms, and English took every one it could, so training on English is training it on those other languages. > 3. LLMs trained on code demonstrably reason better. What? No. Nonsense. I haven't seen any demonstration of reasoning. LLM's are more like the cargo-cult tribes watching things they don't understand and mimicking those things; Sure, they built an airport, but it doesn't work, and that makes all the difference. > It's fairly obvious pre-trained LLMs aren't really operating at just the word/token level. What's obvious is that there's a relationship between words/tokens and ideas. What is not obvious is how far that relationship goes.
- famouswaffles 4y ago
  >personally think 1.8% is a lot. It's not lol. If the model was only trained on that content, it wouldn't be anywhere near as good at french. That's a fact. And French is one thing but they're are languages it's fluent in with less data. >Almost every language in Europe shares a lot of words and idioms, and English took every one it could, so training on English is training it on those other languages. First of all, what you're saying is straight up nonsensical. Knowing English doesn't make you even close to competent on the vast majority of European languages. Secondly, it has nothing to do with European languages. It's excellent following instructions in Mandarin (0.1%) and Korean(0.02%), languages that had much less than 1.8% data. >What? No. Nonsense. I haven't seen any demonstration of reasoning. Many benchmarks that test reasoning. LLMs trained on code perform better here. Simple as that. >What is not obvious is how far that relationship goes. Only if you're being obtuse lol.