7 ms·

Does anyone remember the "Mad Libs" games - you fill out a form with blanks for "verb", "noun", "adjective", etc - then on the next page you fill in the words f

Does anyone remember the "Mad Libs" games - you fill out a form with blanks for "verb", "noun", "adjective", etc - then on the next page you fill in the words from the form to create a silly story. The results are funny because the words you provided initially were without context - they were syntactically correct, but were nonsense in context. LLM's are like Mad Libs with a "contextual predictor" - they produce syntactically correct output, and the "contextual predictor" limits the amount of nonsense because statistical correlations can generate meaningful output most of the time. But there is no "reasoning" occurring here - just syntactic templating and statistical auto-complete.

FeepingCreature 2y ago
I believe reasoning is (sufficiently advanced) syntactic templating and statistical autocomplete. Reminder that syntactic transformations are Turing complete: https://wiki.c2.com/?RewriteRules https://wiki.c2.com/?RewriteRules
- enord 2y ago
  That’s neither here nor there. Everything can be statistically modelled but very few things are reasoning. Same with turing machines.
  - mistermann 2y ago
    Speaking of reasoning, how do you know what is happening inside the mind when it "reasons"? And while you're at it: what is happening inside the mind when it reasons?
    - enord 2y ago
      These are very good questions that deserve unequivocal answers. Alas…
      - mistermann 2y ago
        And here we've encountered the taboo. Best pretend not, kick the can down the road, and hope for the best. That will surely produce good results.
        enord 2y ago
        Ah yes, the taboo of sound reasoning.
        mistermann 2y ago
        The Reasoner analyzes its reasoning and finds it...sound! Case closed, terminate all thought processes.
- EncomLab 2y ago
  In no way does "Turing Completeness" imply the ability to reason - I mean it's like arguing that a nightlight "reasons" about if it is dark out or not.
  - FeepingCreature 2y ago
    However, if reason is computable, then a syntactic transformation can compute it. The point is that stating that something is a "mere" syntactic transformation does not imply computational weakness.
  - thornewolf 2y ago
    that argument is valid, however simple the reasoning may be.
  - naasking 2y ago
    > In no way does "Turing Completeness" imply the ability to reason Unless you believe in magic, then yes, it does. A system that is Turing Complete absolutely can be programmed to reason, aka it has the ability to reason.
    - riku_iki 2y ago
      > A system that is Turing Complete absolutely can be programmed to reason, aka it has the ability to reason. you can write C program which can reason, but C compiler can't reason. So, program part is missing between "Turing Completeness" and reasoning, and it is very non-trivial part.
      - naasking 2y ago
        Given "reasoning" is still undefined, I would not go so far as to claim that a C compiler is not reasoning. What if a C compiler's semantic analysis pass is a limited form of reasoning? Furthermore, the C compiler can do a lot more than you think. The P99/metalang99 macro toolkits give the preprocessor enough state space to encode and run an LLM, in principle.
energy123 2y ago
The network has specific circuits that correspond to concepts and you can see that the network uses and combines those concepts to work through problems. That is reasoning.
- EncomLab 2y ago
  Under this definition an 74LS21 AND gate is reasoning - it has specific circuits that correspond to concepts, and it uses that network to determine an output based on the input. Seems pretty overly broad - we run back into the issue of saying that a nightlight or thermostat is reasoning.
  - energy123 2y ago
    Reasoning is probably better thought of as a spectrum where what you describe is a very little bit of reasoning, and LLMs do a lot more reasoning.
    - EncomLab 2y ago
      For true reasoning you really need to introduce the ability for the circuit to intentionally decide to do something different that is not just a random selection or hallucination - otherwise we are just saying that state machines "reason" for the sake of using an anthropomorphic word.
      - chrisweekly 2y ago
        "intentionally decide" is at least as problematic a term as "reason", no?
      - drdeca 2y ago
        No, I don’t think “reasoning” should require intent. I think a prolog program should be something that can be described as reasoning.
      - ToValueFunfetti 2y ago
        This restriction makes it impossible to determine if something is reasoning. An LLM may well intentionally make decisions; I have as much evidence for that as I have for anybody else doing so, ie. zilch. I'm not even sure that I make intentional decisions, I can only say that it feels like I do. But free will isn't really compliant with my model of physical reality.
nerdponx 2y ago
> statistical auto-complete Yes, but it's a hugely almost unimaginably complicated auto-complete model. And it turns out that a lot of human reasoning is statistically predictable enough in writing that you can actually obtain reasoning-like behavior just by having a good auto-complete model. You shouldn't trvialize how amazingly well it does work, and how surprising it is that it works, just because it doesn't work in all cases. Literally the whole point of TFA is to explore how this phenomenon of something-like-reasoning arises out of a sufficiently huge autocomplete model.
- JangoSteve 2y ago
  > And it turns out that a lot of human reasoning is statistically predictable enough in writing that you can actually obtain reasoning-like behavior just by having a good auto-complete model. I would disagree with this on a technicality that changes the conclusion. It's not that human reasoning is statistically predictable (though it may be), it's that all of the writing that has ever described human reasoning on an unimaginable number of topics is statistically summarizable, and therefore having a good auto-complete model does a good job of describing human reasoning that has been previously described at least combinatorially across various sources. We don't have direct access to anyone else's reasoning. We infer their reasoning by seeing/hearing it described, then we fill in the blanks with our own reasoning-to-description experiences. When we see a model that's great at mimicking descriptions of reasoning, it triggers the same inferences, and we conclude similar reasoning must be going on under the hood. It's like the ELIZA Effect on steroids. It might be the case that neural networks could theoretically, eventually reproduce the same kind of thinking we experience. But I think it's highly unlikely it'd be a single neural network trained on language, especially given the myriad studies showing the logic and reasoning capabilities of humans that are distinct from language. It'd probably be a large number of separate models trained on different domains that come together. At that point though, there are several domains that would be much more efficiently represented with something other than a neural network model, such as the modeling of physics and mathematics with equations (just because we're able to learn them with neurons in our brains doesn't mean that's the most efficient way to learn or remember them). While a "sufficiently huge autocomplete model" is impressive and can do many things related to language, I think it's inaccurate to claim they develop reasoning capabilities. I think of transformer-based neural networks as giant compression algorithms. They're super lossy compression algorithms with super high compression ratios, which allows them to take in more information than any other models we've developed. They work well, because they have the unique ability to determine the least relevant information to lose. The auto-complete part is then using the compressed information in the form of the trained model to decompress prompts with astounding capability. We do similar things in our brains, but again, it's not entirely tied to language; that's just one of many tools we use.
naasking 2y ago
> But there is no "reasoning" occurring here - just syntactic templating and statistical auto-complete. I don't know why people continue to be so sure that "reasoning" is not some form of syntactic templating and statistical auto-complete.
- discreteevent 2y ago
  Well we don't have an understanding of how the brain works so we can't be fully sure but it's clear why they have this intuition: 1) Many people have had to cram for some exam where they didn't have time to fully understand the material. So for those parts they memorized as much as they could and got through the exam by pattern matching. But they knew there was a difference because they knew what it was like to fully understand something where that they could reason about it and play with it in their mind. 2) Crucially, if they understand the key mechanism early then they often don't need to memorize anything (the opposite of LLM's which need millions of examples) 3) LLM's display attributes of someone who has crammed for an exam and when it is probed further [1] it starts to break down in exactly the same way a crammer does. [1] https://arxiv.org/abs/2406.02061 https://arxiv.org/abs/2406.02061
  - naasking 2y ago
    I understand why they intuitively think it isn't. I also think there is probably something more to reasoning. I'm just mystified by why they are so sure it isn't.
- srean 2y ago
  Do you mean human reasoning in their day to day life ? Because there certainly are other kinds of reasoning, for example, logic.
  - naasking 2y ago
    Logic is a syntactic formalism that humans often apply imperfectly. That certainly sounds like we could be employing syntactic templating and statistical auto-complete.
PoignardAzur 2y ago
> But there is no "reasoning" occurring here - just syntactic templating and statistical auto-complete. This is the "stochastic parrot" hypothesis, which people feel obligated to bring up every single time there's a LLM paper on HN. This hypothesis isn't just philosophical, it can lead to falsifiable predictions, and experiments have thoroughly falsified them: LLMs do have a world model. See OthelloGPT for the most famous paper on the subject; see Transformers Represent Belief State Geometry in their Residual Stream for a more recent one.
gsuuon 2y ago
This is (well -- ad-libs) what I based the name of my fill-in-the-blank-with-llm ts library on https://github.com/gsuuon/ad-llama/ https://github.com/gsuuon/ad-llama/

FeepingCreature 2y ago

I believe reasoning is (sufficiently advanced) syntactic templating and statistical autocomplete. Reminder that syntactic transformations are Turing complete: https://wiki.c2.com/?RewriteRules https://wiki.c2.com/?RewriteRules

enord 2y ago
That’s neither here nor there. Everything can be statistically modelled but very few things are reasoning. Same with turing machines.
- mistermann 2y ago
  Speaking of reasoning, how do you know what is happening inside the mind when it "reasons"? And while you're at it: what is happening inside the mind when it reasons?
  - enord 2y ago
    These are very good questions that deserve unequivocal answers. Alas…
    - mistermann 2y ago
      And here we've encountered the taboo. Best pretend not, kick the can down the road, and hope for the best. That will surely produce good results.
      - enord 2y ago
        Ah yes, the taboo of sound reasoning.
        mistermann 2y ago
        The Reasoner analyzes its reasoning and finds it...sound! Case closed, terminate all thought processes.
EncomLab 2y ago
In no way does "Turing Completeness" imply the ability to reason - I mean it's like arguing that a nightlight "reasons" about if it is dark out or not.
- FeepingCreature 2y ago
  However, if reason is computable, then a syntactic transformation can compute it. The point is that stating that something is a "mere" syntactic transformation does not imply computational weakness.
- thornewolf 2y ago
  that argument is valid, however simple the reasoning may be.
- naasking 2y ago
  > In no way does "Turing Completeness" imply the ability to reason Unless you believe in magic, then yes, it does. A system that is Turing Complete absolutely can be programmed to reason, aka it has the ability to reason.
  - riku_iki 2y ago
    > A system that is Turing Complete absolutely can be programmed to reason, aka it has the ability to reason. you can write C program which can reason, but C compiler can't reason. So, program part is missing between "Turing Completeness" and reasoning, and it is very non-trivial part.
    - naasking 2y ago
      Given "reasoning" is still undefined, I would not go so far as to claim that a C compiler is not reasoning. What if a C compiler's semantic analysis pass is a limited form of reasoning? Furthermore, the C compiler can do a lot more than you think. The P99/metalang99 macro toolkits give the preprocessor enough state space to encode and run an LLM, in principle.

energy123 2y ago

The network has specific circuits that correspond to concepts and you can see that the network uses and combines those concepts to work through problems. That is reasoning.

EncomLab 2y ago
Under this definition an 74LS21 AND gate is reasoning - it has specific circuits that correspond to concepts, and it uses that network to determine an output based on the input. Seems pretty overly broad - we run back into the issue of saying that a nightlight or thermostat is reasoning.
- energy123 2y ago
  Reasoning is probably better thought of as a spectrum where what you describe is a very little bit of reasoning, and LLMs do a lot more reasoning.
  - EncomLab 2y ago
    For true reasoning you really need to introduce the ability for the circuit to intentionally decide to do something different that is not just a random selection or hallucination - otherwise we are just saying that state machines "reason" for the sake of using an anthropomorphic word.
    - chrisweekly 2y ago
      "intentionally decide" is at least as problematic a term as "reason", no?
    - drdeca 2y ago
      No, I don’t think “reasoning” should require intent. I think a prolog program should be something that can be described as reasoning.
    - ToValueFunfetti 2y ago
      This restriction makes it impossible to determine if something is reasoning. An LLM may well intentionally make decisions; I have as much evidence for that as I have for anybody else doing so, ie. zilch. I'm not even sure that I make intentional decisions, I can only say that it feels like I do. But free will isn't really compliant with my model of physical reality.

nerdponx 2y ago

> statistical auto-complete Yes, but it's a hugely almost unimaginably complicated auto-complete model. And it turns out that a lot of human reasoning is statistically predictable enough in writing that you can actually obtain reasoning-like behavior just by having a good auto-complete model. You shouldn't trvialize how amazingly well it does work, and how surprising it is that it works, just because it doesn't work in all cases. Literally the whole point of TFA is to explore how this phenomenon of something-like-reasoning arises out of a sufficiently huge autocomplete model.

JangoSteve 2y ago
> And it turns out that a lot of human reasoning is statistically predictable enough in writing that you can actually obtain reasoning-like behavior just by having a good auto-complete model. I would disagree with this on a technicality that changes the conclusion. It's not that human reasoning is statistically predictable (though it may be), it's that all of the writing that has ever described human reasoning on an unimaginable number of topics is statistically summarizable, and therefore having a good auto-complete model does a good job of describing human reasoning that has been previously described at least combinatorially across various sources. We don't have direct access to anyone else's reasoning. We infer their reasoning by seeing/hearing it described, then we fill in the blanks with our own reasoning-to-description experiences. When we see a model that's great at mimicking descriptions of reasoning, it triggers the same inferences, and we conclude similar reasoning must be going on under the hood. It's like the ELIZA Effect on steroids. It might be the case that neural networks could theoretically, eventually reproduce the same kind of thinking we experience. But I think it's highly unlikely it'd be a single neural network trained on language, especially given the myriad studies showing the logic and reasoning capabilities of humans that are distinct from language. It'd probably be a large number of separate models trained on different domains that come together. At that point though, there are several domains that would be much more efficiently represented with something other than a neural network model, such as the modeling of physics and mathematics with equations (just because we're able to learn them with neurons in our brains doesn't mean that's the most efficient way to learn or remember them). While a "sufficiently huge autocomplete model" is impressive and can do many things related to language, I think it's inaccurate to claim they develop reasoning capabilities. I think of transformer-based neural networks as giant compression algorithms. They're super lossy compression algorithms with super high compression ratios, which allows them to take in more information than any other models we've developed. They work well, because they have the unique ability to determine the least relevant information to lose. The auto-complete part is then using the compressed information in the form of the trained model to decompress prompts with astounding capability. We do similar things in our brains, but again, it's not entirely tied to language; that's just one of many tools we use.

naasking 2y ago

> But there is no "reasoning" occurring here - just syntactic templating and statistical auto-complete. I don't know why people continue to be so sure that "reasoning" is not some form of syntactic templating and statistical auto-complete.

discreteevent 2y ago
Well we don't have an understanding of how the brain works so we can't be fully sure but it's clear why they have this intuition: 1) Many people have had to cram for some exam where they didn't have time to fully understand the material. So for those parts they memorized as much as they could and got through the exam by pattern matching. But they knew there was a difference because they knew what it was like to fully understand something where that they could reason about it and play with it in their mind. 2) Crucially, if they understand the key mechanism early then they often don't need to memorize anything (the opposite of LLM's which need millions of examples) 3) LLM's display attributes of someone who has crammed for an exam and when it is probed further [1] it starts to break down in exactly the same way a crammer does. [1] https://arxiv.org/abs/2406.02061 https://arxiv.org/abs/2406.02061
- naasking 2y ago
  I understand why they intuitively think it isn't. I also think there is probably something more to reasoning. I'm just mystified by why they are so sure it isn't.
srean 2y ago
Do you mean human reasoning in their day to day life ? Because there certainly are other kinds of reasoning, for example, logic.
- naasking 2y ago
  Logic is a syntactic formalism that humans often apply imperfectly. That certainly sounds like we could be employing syntactic templating and statistical auto-complete.

PoignardAzur 2y ago

> But there is no "reasoning" occurring here - just syntactic templating and statistical auto-complete. This is the "stochastic parrot" hypothesis, which people feel obligated to bring up every single time there's a LLM paper on HN. This hypothesis isn't just philosophical, it can lead to falsifiable predictions, and experiments have thoroughly falsified them: LLMs do have a world model. See OthelloGPT for the most famous paper on the subject; see Transformers Represent Belief State Geometry in their Residual Stream for a more recent one.

gsuuon 2y ago

This is (well -- ad-libs) what I based the name of my fill-in-the-blank-with-llm ts library on https://github.com/gsuuon/ad-llama/ https://github.com/gsuuon/ad-llama/