4 ms·

> I don't understand this view. How I see it the fundamental bottleneck to AGI is continual learning and backpropagation. Models today are static, and human bra

> I don't understand this view. How I see it the fundamental bottleneck to AGI is continual learning and backpropagation. Models today are static, and human brains don't learn or adapt themselves with anything close to backpropagation. Even with continuous backpropagation and "learning", enriching the training data, so called online-learning, the limitations will not disappear. The LLMs will not be able to conclude things about the world based on fact and deduction. They only consider what is likely from their training data. They will not foresee/anticipate events, that are unlikely or non-existent in their training data, but are bound to happen due to real world circumstances. They are not intelligent in that way. Whether humans always apply that much effort to conclude these things is another question. The point is, that humans fundamentally are capable of doing that, while LLMs are structurally not. The problems are structural/architectural. I think it will take another 2-3 major leaps in architectures, before these AI models reach human level general intelligence, if they ever reach it. So far they can "merely" often "fake it" when things are statistically common in their training data.

andy12_ 8mo ago
> Even with continuous backpropagation and "learning" That's what I said. Backpropagation cannot be enough; that's not how neurons work in the slightest. When you put biological neurons in a Pong environment they learn to play not through some kind of loss or reward function; they self-organize to avoid unpredictable stimulation. As far as I know, no architecture learns in such an unsupervised way. https://www.sciencedirect.com/science/article/pii/S0896627322008066 https://www.sciencedirect.com/science/article/pii/S089662732...
- torginus 8mo ago
  Forgive me for being ignorant - but 'loss' in supervised learning ML context encode the difference between how unlikely (high loss) or likely (low loss) was the network in predicting the output based on the input. This sounds very similar to me as to what neurons do (avoid unpredictable stimulation)
  - andy12_ 8mo ago
    So, I have been thinking about this for a little while. Image a model f that takes a world x and makes a prediciton y. At a high-level, a traditional supervised model is trained like this f(x)=y' => loss(y',y) => how good was my prediction? Train f through backprop with that error. While a model trained with reinforcement learning is more similar to this. Where m(y) is the resulting world state of taking an action y the model predicted. f(x)=y' => m(y')=z => reward(z) => how good was the state I was in based on my actions? Train f with an algorithm like REINFORCE with the reward, as the world m is a non-differentiable black-box. While a group of neurons is more like predicting what is the resulting word state of taking my action, g(x,y), and trying to learn by both tuning g and the action taken f(x). f(x)=y' => m(y')=z => g(x,y)=z' => loss(z,z') => how predictable was the results of my actions? Train g normally with backprop, and train f with an algorithm like REINFORCE with negative surprise as a reward. After talking with GPT5.2 for a little while, it seems like Curiosity-driven Exploration by Self-supervised Prediction[1] might be an architecture similar to the one I described for neurons? But with the twist that f is rewarded by making the prediction error bigger (not smaller!) as a proxy of "curiosity". [1] https://arxiv.org/pdf/1705.05363 https://arxiv.org/pdf/1705.05363
- latentsea 8mo ago
  So can't you just use how real neurons learn as training data to to learn how to learn the same way?
wiz21c 8mo ago
I'm sure that if a car appeared from nowhere in the middle of your living room, you would not be prepared at all. So my question is: when is there enough training data that you can handle 99.99% of the world ?
jstummbillig 8mo ago
> They will not foresee/anticipate events, that are unlikely or non-existent in their training data, but are bound to happen due to real world circumstances. They are not intelligent in that way. Can you be a bit more specific at all bounds? Maybe via an example?
steego 8mo ago
I think people MOSTLY foresee and anticipate events in OUR training data, which mostly comprises information collected by our senses. Our training data is a lot more diverse than an LLMs. We also leverage our senses as a carrier for communicating abstract ideas using audio and visual channels that may or may not be grounded in reality. We have TV shows, video games, programming languages and all sorts of rich and interesting things we can engage with that do not reflect our fundamental reality. Like LLMs, we can hallucinate while we sleep or we can delude ourselves with untethered ideas, but UNLIKE LLMs, we can steer our own learning corpus. We can train ourselves with our own untethered “hallucinations” or we can render them in art and share them with others so they can include it in their training corpus. Our hallucinations are often just erroneous models of the world. When we render it into something that has aesthetic appeal, we might call it art. If the hallucination helps us understand some aspect of something, we call it a conjecture or hypothesis. We live in a rich world filled with rich training data. We don’t magically anticipate events not in our training data, but we’re also not void of creativity (“hallucinations”) either. Most of us are stochastic parrots most of the time. We’ve only gotten this far because there are so many of us and we’ve been on this earth for many generations. Most of us are dazzled and instinctively driven to mimic the ideas that a small minority of people “hallucinate”. There is no shame in mimicking or being a stochastic parrot. These are critical features that helped our ancestors survive.
- robwwilliams 8mo ago
  > We can steer our own learning corpus This is critical. We have some degree of attentional autonomy. And we have a complex tapestry of algorithms running in thalamocortical circuits that generate “Nows”. Truncation commands produce sequences of acts (token-like products).
perfmode 8mo ago
Humans are notoriously bad at formal logic. The Wason selection task is the classic example: most people fail a simple conditional reasoning problem unless it’s dressed up in familiar social context, like catching cheaters. That looks a lot more like pattern matching than rule application. Kahneman’s whole framework points the same direction. Most of what people call “reasoning” is fast, associative, pattern-based. The slow, deliberate, step-by-step stuff is effortful and error-prone, and people avoid it when they can. And even when they do engage it, they’re often confabulating a logical-sounding justification for a conclusion they already reached by other means. So maybe the honest answer is: the gap between what LLMs do and what most humans do most of the time might be smaller than people assume. The story that humans have access to some pure deductive engine and LLMs are just faking it with statistics might be flattering to humans more than it’s accurate. Where I’d still flag a possible difference is something like adaptability. A person can learn a totally new formal system and start applying its rules, even if clumsily. Whether LLMs can genuinely do that outside their training distribution or just interpolate convincingly is still an open question. But then again, how often do humans actually reason outside their own “training distribution”? Most human insight happens within well-practiced domains.
- lich_king 8mo ago
  > The Wason selection task is the classic example: most people fail a simple conditional reasoning problem unless it’s dressed up in familiar social context, like catching cheaters. I've never heard about the Wason selection task, looked it up, and could tell the right answer right away. But I can also tell you why: because I have some familiarity with formal logic and can, in your words, pattern-match the gotcha that "if x then y" is distinct from "if not x then not y". In contrast to you, this doesn't make me believe that people are bad at logic or don't really think. It tells me that people are unfamiliar with "gotcha" formalities introduced by logicians that don't match the everyday use of language. If you added a simple additional to the problem, such as "Note that in this context, 'if' only means that...", most people would almost certainly answer it correctly. Mind you, I'm not arguing that human thinking is necessarily more profound from what what LLMs could ever do. However, judging from the output, LLMs have a tenuous grasp on reality, so I don't think that reductionist arguments along the lines of "humans are just as dumb" are fair. There's a difference that we don't really know how to overcome.
conartist6 8mo ago
Models don't care. They aren't alive. This is the source of the chasm between here and AGI. You have to fear death to reason about the world and how to behave in it. I guess I just always thought it was obvious that you can't do better than nature. You can do different things, sure, but if a society of unique individuals wasn't the most effective way of making progress, nature itself would not have chosen it. So in a way I think Yan is smart because he got money, but in a way I think he's a fucking idiot if he can't see just how very, very very far we are from competing with organic intelligence.
- conartist6 8mo ago
  Not only that but people like this aren't actually interested in understanding the physical world. Because we don't understand it yet. If you care about understanding the world I think you become someone more like Jane Goodall than Yan LeCun
- j4k0o 8mo ago
  "You have to fear death to reason about the world and how to behave in it." You're onto something there. If everyone knew they were to die tomorrow, all of a sudden they'd choose to act differently. There is no logical thought process that determines that - it's something else. Something we can't concretely point toward as an object.

andy12_ 8mo ago

> Even with continuous backpropagation and "learning" That's what I said. Backpropagation cannot be enough; that's not how neurons work in the slightest. When you put biological neurons in a Pong environment they learn to play not through some kind of loss or reward function; they self-organize to avoid unpredictable stimulation. As far as I know, no architecture learns in such an unsupervised way. https://www.sciencedirect.com/science/article/pii/S0896627322008066 https://www.sciencedirect.com/science/article/pii/S089662732...

torginus 8mo ago
Forgive me for being ignorant - but 'loss' in supervised learning ML context encode the difference between how unlikely (high loss) or likely (low loss) was the network in predicting the output based on the input. This sounds very similar to me as to what neurons do (avoid unpredictable stimulation)
- andy12_ 8mo ago
  So, I have been thinking about this for a little while. Image a model f that takes a world x and makes a prediciton y. At a high-level, a traditional supervised model is trained like this f(x)=y' => loss(y',y) => how good was my prediction? Train f through backprop with that error. While a model trained with reinforcement learning is more similar to this. Where m(y) is the resulting world state of taking an action y the model predicted. f(x)=y' => m(y')=z => reward(z) => how good was the state I was in based on my actions? Train f with an algorithm like REINFORCE with the reward, as the world m is a non-differentiable black-box. While a group of neurons is more like predicting what is the resulting word state of taking my action, g(x,y), and trying to learn by both tuning g and the action taken f(x). f(x)=y' => m(y')=z => g(x,y)=z' => loss(z,z') => how predictable was the results of my actions? Train g normally with backprop, and train f with an algorithm like REINFORCE with negative surprise as a reward. After talking with GPT5.2 for a little while, it seems like Curiosity-driven Exploration by Self-supervised Prediction[1] might be an architecture similar to the one I described for neurons? But with the twist that f is rewarded by making the prediction error bigger (not smaller!) as a proxy of "curiosity". [1] https://arxiv.org/pdf/1705.05363 https://arxiv.org/pdf/1705.05363
latentsea 8mo ago
So can't you just use how real neurons learn as training data to to learn how to learn the same way?

wiz21c 8mo ago

I'm sure that if a car appeared from nowhere in the middle of your living room, you would not be prepared at all. So my question is: when is there enough training data that you can handle 99.99% of the world ?

jstummbillig 8mo ago

> They will not foresee/anticipate events, that are unlikely or non-existent in their training data, but are bound to happen due to real world circumstances. They are not intelligent in that way. Can you be a bit more specific at all bounds? Maybe via an example?

steego 8mo ago

I think people MOSTLY foresee and anticipate events in OUR training data, which mostly comprises information collected by our senses. Our training data is a lot more diverse than an LLMs. We also leverage our senses as a carrier for communicating abstract ideas using audio and visual channels that may or may not be grounded in reality. We have TV shows, video games, programming languages and all sorts of rich and interesting things we can engage with that do not reflect our fundamental reality. Like LLMs, we can hallucinate while we sleep or we can delude ourselves with untethered ideas, but UNLIKE LLMs, we can steer our own learning corpus. We can train ourselves with our own untethered “hallucinations” or we can render them in art and share them with others so they can include it in their training corpus. Our hallucinations are often just erroneous models of the world. When we render it into something that has aesthetic appeal, we might call it art. If the hallucination helps us understand some aspect of something, we call it a conjecture or hypothesis. We live in a rich world filled with rich training data. We don’t magically anticipate events not in our training data, but we’re also not void of creativity (“hallucinations”) either. Most of us are stochastic parrots most of the time. We’ve only gotten this far because there are so many of us and we’ve been on this earth for many generations. Most of us are dazzled and instinctively driven to mimic the ideas that a small minority of people “hallucinate”. There is no shame in mimicking or being a stochastic parrot. These are critical features that helped our ancestors survive.

robwwilliams 8mo ago
> We can steer our own learning corpus This is critical. We have some degree of attentional autonomy. And we have a complex tapestry of algorithms running in thalamocortical circuits that generate “Nows”. Truncation commands produce sequences of acts (token-like products).

perfmode 8mo ago

Humans are notoriously bad at formal logic. The Wason selection task is the classic example: most people fail a simple conditional reasoning problem unless it’s dressed up in familiar social context, like catching cheaters. That looks a lot more like pattern matching than rule application. Kahneman’s whole framework points the same direction. Most of what people call “reasoning” is fast, associative, pattern-based. The slow, deliberate, step-by-step stuff is effortful and error-prone, and people avoid it when they can. And even when they do engage it, they’re often confabulating a logical-sounding justification for a conclusion they already reached by other means. So maybe the honest answer is: the gap between what LLMs do and what most humans do most of the time might be smaller than people assume. The story that humans have access to some pure deductive engine and LLMs are just faking it with statistics might be flattering to humans more than it’s accurate. Where I’d still flag a possible difference is something like adaptability. A person can learn a totally new formal system and start applying its rules, even if clumsily. Whether LLMs can genuinely do that outside their training distribution or just interpolate convincingly is still an open question. But then again, how often do humans actually reason outside their own “training distribution”? Most human insight happens within well-practiced domains.

lich_king 8mo ago
> The Wason selection task is the classic example: most people fail a simple conditional reasoning problem unless it’s dressed up in familiar social context, like catching cheaters. I've never heard about the Wason selection task, looked it up, and could tell the right answer right away. But I can also tell you why: because I have some familiarity with formal logic and can, in your words, pattern-match the gotcha that "if x then y" is distinct from "if not x then not y". In contrast to you, this doesn't make me believe that people are bad at logic or don't really think. It tells me that people are unfamiliar with "gotcha" formalities introduced by logicians that don't match the everyday use of language. If you added a simple additional to the problem, such as "Note that in this context, 'if' only means that...", most people would almost certainly answer it correctly. Mind you, I'm not arguing that human thinking is necessarily more profound from what what LLMs could ever do. However, judging from the output, LLMs have a tenuous grasp on reality, so I don't think that reductionist arguments along the lines of "humans are just as dumb" are fair. There's a difference that we don't really know how to overcome.

conartist6 8mo ago

Models don't care. They aren't alive. This is the source of the chasm between here and AGI. You have to fear death to reason about the world and how to behave in it. I guess I just always thought it was obvious that you can't do better than nature. You can do different things, sure, but if a society of unique individuals wasn't the most effective way of making progress, nature itself would not have chosen it. So in a way I think Yan is smart because he got money, but in a way I think he's a fucking idiot if he can't see just how very, very very far we are from competing with organic intelligence.

conartist6 8mo ago
Not only that but people like this aren't actually interested in understanding the physical world. Because we don't understand it yet. If you care about understanding the world I think you become someone more like Jane Goodall than Yan LeCun
j4k0o 8mo ago
"You have to fear death to reason about the world and how to behave in it." You're onto something there. If everyone knew they were to die tomorrow, all of a sudden they'd choose to act differently. There is no logical thought process that determines that - it's something else. Something we can't concretely point toward as an object.