3 ms·

> And again, human brains are clearly limited in the number of steps it can compute without writing something down No - there is a loop between the cortex and

> And again, human brains are clearly limited in the number of steps it can compute without writing something down No - there is a loop between the cortex and thalamus, feeding the outputs of the cortex back in as inputs. Our brain can iterate for as long as it likes before initiating any motor output, if any, such as writing something down.

famouswaffles 2y ago
The brain's ability to iterate on information is still constrained by certain cognitive limitations like working memory capacity and attention span. In practice, the cortex-thalamus loop allows for some degree of internal iteration, but the brain cannot endlessly iterate without some form of external aid (e.g., writing something down) to offload information and prevent cognitive overload. I'm not telling you anything here you don't experience in your everyday life. Try indefinitely iterating on any computation you like and see how well that works for you.
- HarHarVeryFunny 2y ago
  What's your point? The discussion is about the architecturally imposed limitations of LLMs, resulting in capabilities that are way less than that of a brain. The fact that the brain has it's own limits doesn't somehow negate this fact!
  - famouswaffles 2y ago
    My point is that for some bizare reason, people have standards of reasoning (for machines) that only exist in fiction or their own imagination. It is beyond silly to dump an architecture for a limitation the human brain has. A reasoning engine that can iterate indefinitely with no external aid does not exist in real life. That the transformer also has this weakness is not any reason for it to have capabilities less than a brain so it's completely moot.
    - HarHarVeryFunny 2y ago
      LLMs are here to stay until something better replaces them, and will be used for those things they are capable of. It shouldn't be surprising they are not great at reasoning, or everything one would hope for from an AGI, since they simply were not built for that. If you look at the development history, the transformer was a successor to LSTM-based seq-2-seq models using Bahdanau attention, whose main goal was to more efficiently utilize parallel hardware by supporting parallel processing. Of course a good language model (word predictor) will look as if it's reasoning because it is trying to model the data it was trained on - a human reasoner. As humans we routinely think for seconds/minutes or even hours before speaking or acting, while an LLM only has that fixed N steps (layers) of computation. I don't know why you claim this difference (among others) should make no difference, but it clearly does, with out-of-training-set reasoning weakness being a notable limitation that people such as Demis Hassabis have recently conceded.
      - famouswaffles 2y ago
        Reasoning is reasoning. "Look as if it is reasoning" is an imaginary distinction you've made up. One that is very clear because everybody touting this "fake reasoning" rhetoric is still somehow unable to define a testable version of reasoning that disqualifies LLMs without also disqualifying some chunk of humans. >As humans we routinely think for seconds/minutes or even hours before speaking or acting No human is iterating on a base thought for hours uninterrupted lol so this is just moot >with out-of-training-set reasoning weakness being a notable limitation that people such as Demis Hassabis have recently conceded. Humans reason weaker out of training. LLMs are simply currently worse
        HarHarVeryFunny 2y ago
        > Reasoning is reasoning. "Look as if it is reasoning" is an imaginary distinction you've made up. No - just because something has the surface appearance of reasoning doesn't mean that the generative process was reasoning, anymore than a cargo cult wooden aircraft reflects any understanding of aerodynamics and would be able to fly. We've already touched on it, but the "farmer crossing river" problems is a great example. When the LLM sometimes degenerates into "cross bank A to B with chicken, cross band B to A with chicken, cross bank A to B with chicken.. that is the fewest trips possible", this is an example of "looks as if it is reasoning" aka cargo-cult surface-level copying of what a solution looks like. Real reasoning would never repeat a crossing without loading/unloading something since that conflicts with the goal of fewest trips possible.
accountnum 2y ago
You seem to repeatedly insist that hidden computation is a distinction of any relevance whatsoever. First of all, your understanding of the architecture itself is mistaken. A transformer can iterate endlessly because each token it produces allows it a forward pass, and each of these tokens is postpended to its input in the next inference. That's the autoregressive in autoregressive transformer, and the entire reason why it was proposed for arbitrary seq2seq transduction. This means you get layers * tokens iterations, where tokens is up to two million, and is in practice unlimited due to the LLM being able to summarize and select from that. Parallelism is irrelevant, since the transformer is sequential in the output of tokens. A transformer can iterate endlessly, it simply has to output enough tokens. And no, the throughput isn't limited either, since each token gets translated into a high-dimensional internal representation, that in turn is influenced by each other token in the model input. Models can encode whatever they want not just by choosing a token, but by choosing an arbitrary pattern of tokens encoding arbitrary latent-space interactions. Secondly, internal thoughts are irrelevant, because something being "internal" is an arbitrary distinction without impact. If I trained an LLM to prepend and postpend <internal_thought> to some part of its output, and then simply didn't show that part, then the LLM wouldn't magically become human. This is something many models do even today, in fact. Similarly, if I were to take a human and modify their brain to only be able to iterate using pen and paper, or by speaking out loud, then I wouldn't magically make them into something non-human. And I would definitely not reduce their capacity for reasoning in any way whatsoever. There are people with aphantasia working in the arts, there are people without an internal monologue working as authors - how "internal" something is can be trivially changed with no influence on either the architecture or the capabilities of that architecture. Reasoning itself isn't some unified process, neither is it infinite iteration. It requires specific understanding about the domain being reasoned over, especially understanding of which transformation rules are applicable to produce desired states, where the judgement about which states are desirable has to be learned itself. LLMs can reason today, they're just not as good at it than humans are in some domains.
- HarHarVeryFunny 2y ago
  Sure - a transformer can iterate endlessly by generating tokens, but this is no substitute for iterating internally and maintaining internal context and goal-based attention. One reason why just blathering on endlessly isn't the same as thinking deeply before answering, is that it's almost impossible to maintain long-term context/attention. Try it. "Think step by step" or other attempts to prompt the model into generating a longer reply that builds upon itself, will only get you so far because keeping a 1-dimensional context is no substitute for the thousands of connections we have in our brain between neurons, and the richness of context we're therefore able to maintain while thinking. The reasoning weakness of LLMs isn't limited to "some domains" that they had less training data for - it's a fundamental architecturally-based limitation. This becomes obvious when you see the failure modes of simple problems like "how few trips does the farmer need to cross the river with his chicken & corn, etc" type problems. You don't need to morph the problem to require out-of-distribution knowledge to get it to fail - small changes to the problem statement can make the model state that crossing the river backwards and forwards multiple times without loading/unloading anything is the optimal way to cross the river. But, hey, no need to believe me, some random internet dude. People like Demis Hassabis (CEO of DeepMind) acknowledge the weakness too.

famouswaffles 2y ago

The brain's ability to iterate on information is still constrained by certain cognitive limitations like working memory capacity and attention span. In practice, the cortex-thalamus loop allows for some degree of internal iteration, but the brain cannot endlessly iterate without some form of external aid (e.g., writing something down) to offload information and prevent cognitive overload. I'm not telling you anything here you don't experience in your everyday life. Try indefinitely iterating on any computation you like and see how well that works for you.

HarHarVeryFunny 2y ago
What's your point? The discussion is about the architecturally imposed limitations of LLMs, resulting in capabilities that are way less than that of a brain. The fact that the brain has it's own limits doesn't somehow negate this fact!
- famouswaffles 2y ago
  My point is that for some bizare reason, people have standards of reasoning (for machines) that only exist in fiction or their own imagination. It is beyond silly to dump an architecture for a limitation the human brain has. A reasoning engine that can iterate indefinitely with no external aid does not exist in real life. That the transformer also has this weakness is not any reason for it to have capabilities less than a brain so it's completely moot.
  - HarHarVeryFunny 2y ago
    LLMs are here to stay until something better replaces them, and will be used for those things they are capable of. It shouldn't be surprising they are not great at reasoning, or everything one would hope for from an AGI, since they simply were not built for that. If you look at the development history, the transformer was a successor to LSTM-based seq-2-seq models using Bahdanau attention, whose main goal was to more efficiently utilize parallel hardware by supporting parallel processing. Of course a good language model (word predictor) will look as if it's reasoning because it is trying to model the data it was trained on - a human reasoner. As humans we routinely think for seconds/minutes or even hours before speaking or acting, while an LLM only has that fixed N steps (layers) of computation. I don't know why you claim this difference (among others) should make no difference, but it clearly does, with out-of-training-set reasoning weakness being a notable limitation that people such as Demis Hassabis have recently conceded.
    - famouswaffles 2y ago
      Reasoning is reasoning. "Look as if it is reasoning" is an imaginary distinction you've made up. One that is very clear because everybody touting this "fake reasoning" rhetoric is still somehow unable to define a testable version of reasoning that disqualifies LLMs without also disqualifying some chunk of humans. >As humans we routinely think for seconds/minutes or even hours before speaking or acting No human is iterating on a base thought for hours uninterrupted lol so this is just moot >with out-of-training-set reasoning weakness being a notable limitation that people such as Demis Hassabis have recently conceded. Humans reason weaker out of training. LLMs are simply currently worse
      - HarHarVeryFunny 2y ago
        > Reasoning is reasoning. "Look as if it is reasoning" is an imaginary distinction you've made up. No - just because something has the surface appearance of reasoning doesn't mean that the generative process was reasoning, anymore than a cargo cult wooden aircraft reflects any understanding of aerodynamics and would be able to fly. We've already touched on it, but the "farmer crossing river" problems is a great example. When the LLM sometimes degenerates into "cross bank A to B with chicken, cross band B to A with chicken, cross bank A to B with chicken.. that is the fewest trips possible", this is an example of "looks as if it is reasoning" aka cargo-cult surface-level copying of what a solution looks like. Real reasoning would never repeat a crossing without loading/unloading something since that conflicts with the goal of fewest trips possible.

accountnum 2y ago

You seem to repeatedly insist that hidden computation is a distinction of any relevance whatsoever. First of all, your understanding of the architecture itself is mistaken. A transformer can iterate endlessly because each token it produces allows it a forward pass, and each of these tokens is postpended to its input in the next inference. That's the autoregressive in autoregressive transformer, and the entire reason why it was proposed for arbitrary seq2seq transduction. This means you get layers * tokens iterations, where tokens is up to two million, and is in practice unlimited due to the LLM being able to summarize and select from that. Parallelism is irrelevant, since the transformer is sequential in the output of tokens. A transformer can iterate endlessly, it simply has to output enough tokens. And no, the throughput isn't limited either, since each token gets translated into a high-dimensional internal representation, that in turn is influenced by each other token in the model input. Models can encode whatever they want not just by choosing a token, but by choosing an arbitrary pattern of tokens encoding arbitrary latent-space interactions. Secondly, internal thoughts are irrelevant, because something being "internal" is an arbitrary distinction without impact. If I trained an LLM to prepend and postpend <internal_thought> to some part of its output, and then simply didn't show that part, then the LLM wouldn't magically become human. This is something many models do even today, in fact. Similarly, if I were to take a human and modify their brain to only be able to iterate using pen and paper, or by speaking out loud, then I wouldn't magically make them into something non-human. And I would definitely not reduce their capacity for reasoning in any way whatsoever. There are people with aphantasia working in the arts, there are people without an internal monologue working as authors - how "internal" something is can be trivially changed with no influence on either the architecture or the capabilities of that architecture. Reasoning itself isn't some unified process, neither is it infinite iteration. It requires specific understanding about the domain being reasoned over, especially understanding of which transformation rules are applicable to produce desired states, where the judgement about which states are desirable has to be learned itself. LLMs can reason today, they're just not as good at it than humans are in some domains.

HarHarVeryFunny 2y ago
Sure - a transformer can iterate endlessly by generating tokens, but this is no substitute for iterating internally and maintaining internal context and goal-based attention. One reason why just blathering on endlessly isn't the same as thinking deeply before answering, is that it's almost impossible to maintain long-term context/attention. Try it. "Think step by step" or other attempts to prompt the model into generating a longer reply that builds upon itself, will only get you so far because keeping a 1-dimensional context is no substitute for the thousands of connections we have in our brain between neurons, and the richness of context we're therefore able to maintain while thinking. The reasoning weakness of LLMs isn't limited to "some domains" that they had less training data for - it's a fundamental architecturally-based limitation. This becomes obvious when you see the failure modes of simple problems like "how few trips does the farmer need to cross the river with his chicken & corn, etc" type problems. You don't need to morph the problem to require out-of-distribution knowledge to get it to fail - small changes to the problem statement can make the model state that crossing the river backwards and forwards multiple times without loading/unloading anything is the optimal way to cross the river. But, hey, no need to believe me, some random internet dude. People like Demis Hassabis (CEO of DeepMind) acknowledge the weakness too.