3 ms·

Counterpoint: the more networks internalize the patterns behind engineered prompts, the closer they get to being general reasoners.

mjburgess 2y ago
Statistical patterns in text tokens have nothing to do with reasoning. They lack such a property to learn in the first place. This would be more obvious, I guess, if the tokens were in an alien language. Consider a translation to 1,000 different alien languages of any given novel.. there is no distributional property they would share. Statistical AI is just a way of sampling from a historical dataset with a similarity metric. It only works to answer questions if you're sampling from a (Q, A) database in the same language the user already understands. The question was answered by a reasoner, it is now answered by a system which replays answers.
- FeepingCreature 2y ago
  This is just chinese room. But also, I disagree that the languages would not share properties. A novel is too small. Consider a network trained on GPT-3 scale datasets in 1000 alien languages. The shared structures behind the sentences will be the same, even if the grammar is completely different. Stars will be stars, moons will be moons. I'd bet you that the model would be able to translate shared concepts between those languages even if it had not been trained on the same novels, same as GPT can translate terms that it has not seen in dictionary pairs. (I have no idea if it can! But I'm confident enough that I'm willing to just say it can, at risk of being proven wrong. If GPT can't do that, I'm fundamentally misunderstanding how it works - that is, I don't have a paper offhand showing that GPT shares concept neurons between languages, but I'm willing to bet I could find one if I went looking.) In other words, if you co-trained GPT-4 on Earth Internet and Alien Internet, there's a good chance it'd end up able to translate English to Alienese, purely as an emergent ability, if it had the concept of translation at all. Intelligence is compression. With sufficient abstraction (layers) and sufficient volume (dataset), any description of the same reality will assume the same structure. And no learning algo worth its salt will keep two identical structures around.
  - mjburgess 2y ago
    Almost every predicate in natural languages is an arbitrary association of properties in the world. There isn't any property a person has "bald", nor are there "tree"s in the world. What bundles of properties the vast majority of language names are a product of historical and contingent associations we've made for practical reasons. Likewise, languages do not have the same distributional structure. There is no reason an alien language would use discrete tokens to name properties, nor co-locate tokens by linear position in a 'sentence'. Historical human languages did not; using, eg., the full 2D structure of the clay tablet. To suppose that it is the glyphs and their colocation which somehow bare meaning is a nonesensical superstition. The world is what our words mean, and it is we, in that world, who provide them their meaning. We can do so with arbitrary linguistic structures.
    - FeepingCreature 2y ago
      Right, but neural networks at a certain level of scale begin to compress the perception described by the text rather than the text per se. The lexical features are abstracted away and the network begins to understand meaning directly. Of course, you'd still have differentiation by different perceptions and focus on different parts of reality, but I see no reason why that should not likewise abstract out at even higher levels. Ultimately, the whole point of language is that it's about reality, and there is only one reality no matter how many different cultures you feed in. The network isn't doing magic - it's just doing exactly the same thing that we're doing with language. The reason why language is useful is the same reason it's learnable.
      - mjburgess 2y ago
        If you want to understand how statistical AI systems work, then you should understand the basics of formalism. They are curve-fitting algorithms that approximate the probability distribution of a hisotrical dataset. That's all they are. By analogising to any animal or human mind, you aren't describing anythign that acutally exists. A neutral network algorithm isnt neural and it isnt a network. It's a statistical curve-fitting algorith. One oughtnt study trees to understand a decision tree either. This language is entirely metaphorical. There are no neurones in an NN, there are just summation entires in a matrix. This matrix comprises weights, which define the orientation and scale of the line pieces which form the curve being fit to the data.
- empath75 2y ago
  Have you ever actually bothered testing your assumption that chatgpt can't reason. And I don't mean that it fails to reason properly about certain questions, that's trivially easy to show, but is less interesting than many people think it is, because humans can't reason properly about many questions (see, for example, the Monty Hall problem). Test your assumption that it can never reason properly about any question not in its data set. It's way easier, IME, to find examples where it does give the correct answer to novel problems than it is to trip it up with a difficult problem.
  - mjburgess 2y ago
    It's data set is, approximately, everything ever written. We have no access to it. And this is widely studied. You can find trivial mistakes in apparent reasoning. There are two hypotheses: H1, the structure of a response from any given prompt is computed using distributional properties of historical data; H2: the response is computed via deduction from premises to conclusions of agent employing the semantics of the terms, their logical connections, and connections of relevance. In many cases a prompt/reply will confirm both hypotheses, hence confirmation bias and why we dont bother "confirming" any hypothesis. Rather to choose betweeen them, if you wanted to use data, you just find cases where reasoning fails in such a way that H1 is the more plausible answer. Such cases are easy to find, and across the literature. This whole thing is pointless however, because it's blindingly obvious from what a statistical AI algorithm does, which is empirical function fiting on historical datasets to approximate conditional probability distributions. This process is extremely well-understood, and we know necessarily that it is just approximating a historical distribution.
    - stevenhuang 2y ago
      > This process is extremely well-understood, and we know necessarily that it is just approximating a historical distribution. Prove that this precludes formation of mind. Prove that human minds do not work this way. Until such premises are established, your argument is not sound. This leaves open the possibility what we're seeing is in fact a degree of reasoning ability in LLMs.

mjburgess 2y ago

Statistical patterns in text tokens have nothing to do with reasoning. They lack such a property to learn in the first place. This would be more obvious, I guess, if the tokens were in an alien language. Consider a translation to 1,000 different alien languages of any given novel.. there is no distributional property they would share. Statistical AI is just a way of sampling from a historical dataset with a similarity metric. It only works to answer questions if you're sampling from a (Q, A) database in the same language the user already understands. The question was answered by a reasoner, it is now answered by a system which replays answers.

FeepingCreature 2y ago
This is just chinese room. But also, I disagree that the languages would not share properties. A novel is too small. Consider a network trained on GPT-3 scale datasets in 1000 alien languages. The shared structures behind the sentences will be the same, even if the grammar is completely different. Stars will be stars, moons will be moons. I'd bet you that the model would be able to translate shared concepts between those languages even if it had not been trained on the same novels, same as GPT can translate terms that it has not seen in dictionary pairs. (I have no idea if it can! But I'm confident enough that I'm willing to just say it can, at risk of being proven wrong. If GPT can't do that, I'm fundamentally misunderstanding how it works - that is, I don't have a paper offhand showing that GPT shares concept neurons between languages, but I'm willing to bet I could find one if I went looking.) In other words, if you co-trained GPT-4 on Earth Internet and Alien Internet, there's a good chance it'd end up able to translate English to Alienese, purely as an emergent ability, if it had the concept of translation at all. Intelligence is compression. With sufficient abstraction (layers) and sufficient volume (dataset), any description of the same reality will assume the same structure. And no learning algo worth its salt will keep two identical structures around.
- mjburgess 2y ago
  Almost every predicate in natural languages is an arbitrary association of properties in the world. There isn't any property a person has "bald", nor are there "tree"s in the world. What bundles of properties the vast majority of language names are a product of historical and contingent associations we've made for practical reasons. Likewise, languages do not have the same distributional structure. There is no reason an alien language would use discrete tokens to name properties, nor co-locate tokens by linear position in a 'sentence'. Historical human languages did not; using, eg., the full 2D structure of the clay tablet. To suppose that it is the glyphs and their colocation which somehow bare meaning is a nonesensical superstition. The world is what our words mean, and it is we, in that world, who provide them their meaning. We can do so with arbitrary linguistic structures.
  - FeepingCreature 2y ago
    Right, but neural networks at a certain level of scale begin to compress the perception described by the text rather than the text per se. The lexical features are abstracted away and the network begins to understand meaning directly. Of course, you'd still have differentiation by different perceptions and focus on different parts of reality, but I see no reason why that should not likewise abstract out at even higher levels. Ultimately, the whole point of language is that it's about reality, and there is only one reality no matter how many different cultures you feed in. The network isn't doing magic - it's just doing exactly the same thing that we're doing with language. The reason why language is useful is the same reason it's learnable.
    - mjburgess 2y ago
      If you want to understand how statistical AI systems work, then you should understand the basics of formalism. They are curve-fitting algorithms that approximate the probability distribution of a hisotrical dataset. That's all they are. By analogising to any animal or human mind, you aren't describing anythign that acutally exists. A neutral network algorithm isnt neural and it isnt a network. It's a statistical curve-fitting algorith. One oughtnt study trees to understand a decision tree either. This language is entirely metaphorical. There are no neurones in an NN, there are just summation entires in a matrix. This matrix comprises weights, which define the orientation and scale of the line pieces which form the curve being fit to the data.
empath75 2y ago
Have you ever actually bothered testing your assumption that chatgpt can't reason. And I don't mean that it fails to reason properly about certain questions, that's trivially easy to show, but is less interesting than many people think it is, because humans can't reason properly about many questions (see, for example, the Monty Hall problem). Test your assumption that it can never reason properly about any question not in its data set. It's way easier, IME, to find examples where it does give the correct answer to novel problems than it is to trip it up with a difficult problem.
- mjburgess 2y ago
  It's data set is, approximately, everything ever written. We have no access to it. And this is widely studied. You can find trivial mistakes in apparent reasoning. There are two hypotheses: H1, the structure of a response from any given prompt is computed using distributional properties of historical data; H2: the response is computed via deduction from premises to conclusions of agent employing the semantics of the terms, their logical connections, and connections of relevance. In many cases a prompt/reply will confirm both hypotheses, hence confirmation bias and why we dont bother "confirming" any hypothesis. Rather to choose betweeen them, if you wanted to use data, you just find cases where reasoning fails in such a way that H1 is the more plausible answer. Such cases are easy to find, and across the literature. This whole thing is pointless however, because it's blindingly obvious from what a statistical AI algorithm does, which is empirical function fiting on historical datasets to approximate conditional probability distributions. This process is extremely well-understood, and we know necessarily that it is just approximating a historical distribution.
  - stevenhuang 2y ago
    > This process is extremely well-understood, and we know necessarily that it is just approximating a historical distribution. Prove that this precludes formation of mind. Prove that human minds do not work this way. Until such premises are established, your argument is not sound. This leaves open the possibility what we're seeing is in fact a degree of reasoning ability in LLMs.