16 ms·

The interview which I've watched recently with Rich Sutton left me with the impression that AGI is not just a matter of adding more 9s. The interviewer had an

The interview which I've watched recently with Rich Sutton left me with the impression that AGI is not just a matter of adding more 9s. The interviewer had an idea that he took for granted: that to understand language you have to have a model of the world. LLMs seem to udnerstand language therefore they've trained a model of the world. Sutton rejected the premise immediately. He might be right in being skeptical here.

sysguest 1y ago
yeah that "model of the world" would mean: babies are already born with "the model of the world" but a lot of experiments on babies/young kids tell otherwise
- rwj 1y ago
  Lots of experiments show that babies develop import capabilities at roughly the same times. That speaks to inherited abilities.
- ben_w 1y ago
  > babies are already born with "the model of the world" > but a lot of experiments on babies/young kids tell otherwise I believe they are born with such a model? It's just that model is one where mummy still has fur for the baby to cling on to? And where aged something like 5 to 8 it's somehow useful for us to build small enclosures to hide in, leading to a display of pillow forts in the modern world?
  - sysguest 1y ago
    damn I guess I had to be more specific: "LLM-level world-detail knowledge"
    - ben_w 1y ago
      I think I'm even more confused now about what you mean…
- ekjhgkejhgk 1y ago
  > yeah that "model of the world" would mean: babies are already born with "the model of the world" No, not necessarily. Babies don't interact with the world only by reading what people wrote wikipedia and stackoverflow, like these models are trained. Babies do things to the world and observe what happens. I imagine it's similar to the difference between a person sitting on a bicycle and trying to ride it, vs a person watching videos of people riding bicycles. I think it would actually be a great experiment. If you take a person that never rode a bicycle in their life and feed them videos of people riding bicycles, and literature about bikes, fiction and non-fiction, at some point I'm sure they'll be able to talk about it like they have huge experience in riding bikes, but won't be able to ride one.
  - aerhardt 1y ago
    We’ve been thinking about reaching the singularity from one end, by making computers like humans, but too little thought has been given to approaching the problem from the other end: by making babies build their world model by reading Stack Overflow.
    - pavlov 1y ago
      The “Brave New World meets OpenAI” model where bottle-born babies listen to Stack Overflow 24 hours a day until they one day graduate to Alphas who get to spend Worldcoin on AI-generated feelies.
    - zelphirkalt 1y ago
      That's it. Now you've done it! I will have stackoverflow Q&A, as well as moderator comments and closings of questions playing 24/7 to my first not yet born child! Q&A for the knowledge and the mod comments for good behavior, of course. This will lead to singularity in no time!
- godelski 1y ago
  It's a lot more complicated than that. You have instincts, right? Innate fears? This is definitely something passed down through genetics. The Hawk/Goose Effect isn't just limited to baby chickens. Certainly some mental encoding passes down through genetics as how much the brain controls, down to your breathing and heartbeat. But instinct is basic. It's something humans are even able to override. It's a first order approximation. Inaccurate to do meaningfully complex things, but sufficient to keep you alive. Maybe we don't want to call the instinct a world model (it certainly is naïve) but can't be discounted either. In human development, yeah, the lion's share of it happens post birth. Human babies don't even show typical signs of consciousness, even really till the age of 2. There's many different categories of "awareness" and these certainly grow over time. But the big thing that makes humans so intelligent is that we continue to grow and learn through our whole lifetimes. And we can pass that information along without genetics and have very advanced tools to do this. It is a combination of nature and nurture. But do note that this happens differently in different animals. It's wonderfully complex. LLMs are quite incredible but so too are many other non-thinking machines. I don't think we should throw them out, but we never needed to make the jump to intelligence. Certainly not so quickly. I mean what did Carl Sagan say?
  - imtringued 1y ago
    One of the biggest mysteries of humans Vs LLMs is that LLMs need an absurd amount of data during pre training, then a little bit of data during fine tuning to make them behave more human. Meanwhile humans don't need any data at all, but have the blind spot that they can only know and learn about what they have observed. This raises two questions. What is the loss function of the supervised learning algorithm equivalent? Supposedly neurons do predictive coding. They predict what their neighbours are doing. That includes input only neurons like touch, pain, vision, sound, taste, etc. The observations never contain actions. E.g. you can look at another human, but that will never teach you how to walk because your legs are different from other people's legs. How do humans avoid starving to death? How do they avoid leaving no children? How do they avoid eating food that will kill them? These things require a complicated chain of actions. You need to find food, a partner and you need to spit out poison. This means you need a reinforcement learning analogue, but what is going to be the reward function equivalent? The reward function can't be created by the brain, because it would be circular. It would be like giving yourself a high, without even needing drugs. Hence, the reward signal must remain inside the body but outside the brain, where the brain can't hack it. The first and most important reward is to perform reproduction. If food and partners are abundant, the ones that don't reproduce simply die out. This means that reward functions that don't reward reproduction disappear. Reproduction is costly in terms of energy. Do it too many times and you need to recover and eat. Hunger evolved as a result of the brain needing to know about the energy state of the body. It overrides reproductive instincts. Now let's say you have a poisonous plant that gives you diarrhea, but you are hungry. What stops you from eating it? Pain evolves as a response to a damaged body. Harmful activities signal themselves in the form of pain to the brain. Pain overrides hunger. However, what if the plant is so deadly that it will kill you? The pain sensors wouldn't be fast enough. You need to sense the poison before it enters your body. So the tongue evolves taste and cyanide starts tasting bitter. Notice something? The feelings only exist internally inside the human body, but they are all coupled with continued survival in one way or another. There is no such thing for robots or LLMs. They won't accidentally evolve a complex reward function like that.
exe34 1y ago
To me, it's a matter of a very big checklist - you can keep adding tasks to the list, but if it keeps marching onwards checking things off your list, some day you will get there. whether it's a linear or asymptotic march, only time will tell.
- imadierich 1y ago
  [dead]
- cactusplant7374 1y ago
  That's like saying that if we image every neuron in the brain we will understand thinking. We can build these huge databases and they tell us nothing about the process of thinking.
  - exe34 1y ago
    What if we copy the functionality of every neuron? what if we simply copy all the skills that those neurons compute?
    - rootusrootus 1y ago
      Do we even know the functionality of every neuron?
      - exe34 1y ago
        Not yet.
- ekjhgkejhgk 1y ago
  I don't know if you will get there, that's far from clear at this stage. Did you see the recent video by Nick Beato [1] where he asks various models about a specific number? The models that get it right are the models that consume youtube videos, because there was a youtube video about that specific number. It's like, these models are capable of telling you about very similar things that they've seen, but they don't seem like they understand it. It's totally unclear whether this is a quantitative or qualitative gap. [1] https://www.youtube.com/watch?v=TiwADS600Jc https://www.youtube.com/watch?v=TiwADS600Jc
godelski 1y ago
> that to understand knowledge you have to have a model of the world. You have a small but important mistake. It's to recite (or even apply) knowledge. To understand does actually require a world model. Think of it this way: can you pass a test without understanding the test material? Certainly we all saw people we thought were idiots do well in class while we've also seen people we thought were geniuses fail. The test and understanding usually correlates but it's not perfect, right? The reason I say understanding requires a world model (and I would not say LLMs understand) is because to understand you have to be able to detail things. Look at physics, or the far more detail oriented math. Physicists don't conclude things just off of experimental results. It's an important part, but not the whole story. They also write equations, ones which are counterfactual. You can call this compression if you want (I would and do), but it's only that because of the generalization. But it also only has that power because of the details and nuance. With AI many of these people have been screaming for years (check my history) that what we're doing won't get us all the way there. Not because we want to stop the progress, but because we wanted to ensure continued and accelerate progress. We knew the limits and were saying "let's try to get ahead of this problem" but were told "that'll never be a problem. And if it is, we'll deal with it when we deal with it." It's why Chollet made the claim that LLMs have actually held AI progress back. Because the story that was sold was "AGI is solved, we just need to scale" (i.e. more money). I do still wonder how different things would be if those of us pushing back were able to continue and scale our works (research isn't free, so yes, people did stop us). We always had the math to show that scale wasn't enough, but it's easy to say "you don't need math" when you can see progress. The math never said no progress nor no acceleration, the math said there's a wall and it's easier to adjust now than when we're closer and moving faster. Sadly I don't think we'll ever shift the money over. We still evaluate success weirdly. Successful predictions don't matter. You're still heralded if you made a lot of money in VR and Bitcoin, right?
- robotresearcher 1y ago
  In my view 'understand' is a folk psychology term that does not have a technical meaning. Like 'intelligent', 'beautiful', and 'interesting'. It usefully labels a basket of behaviors we see in others, and that is all it does. In this view, if a machine performs a task as well as a human, it understands it exactly as much as a human. There's no problem of how to do understanding, only how to do tasks. The 'problem' melts away when you take this stance. Just my opinion, but my professional opinion from thirty-plus years in AI.
tyre 1y ago
There is some evidence from Anthropic that LLMs do model the world. This paper[0] tracing their "thought" is fascinating. Basically an LLM translating across languages will "light up" (to use a rough fMRI equivalent) for the same concepts (e.g. bigness) across languages. It does have clusters of parameters that correlate with concepts, not just randomly "after X word tends to have Y word." Otherwise you would expect all of Chinese to be grouped in one place, all of French in another, all of English in another. This is empirically not the case. I don't know whether to understand knowledge you have to have a model of the world, but at least as far as language, LLMs very much do seem to have modeling. [0]: https://www.anthropic.com/research/tracing-thoughts-language-model https://www.anthropic.com/research/tracing-thoughts-language...
- manmal 1y ago
  > Basically an LLM translating across languages will "light up" (to use a rough fMRI equivalent) for the same concepts (e.g. bigness) across languages I thought that’s the basic premise of how transformers work - they encode concepts into high dimensional space, and similar concepts will be clustered together. I don’t think it models the world, but just the texts it ingested. It’s observation and regurgitation, not understanding. I do use agents a lot (soon on my second codex subscription), so I don’t think that’s a bad thing. But I’m firmly in the “they are useful tools” camp.
  - bryanlarsen 1y ago
    That's a model. Not a higher-order model like most humans use, but it's still a model.
    - manmal 1y ago
      Yes, not of the world, but of the ingested text. Almost verbatim what I wrote.
      - timschmidt 1y ago
        The ingested text itself contains a model of the world which we have encoded in it. That's what language is. Therefore by the transitive property...
deleted 1y ago
[deleted]
bentt 1y ago
I think this a useful challenge to our normal way of thinking. At the same time, "the world" exists only in our imagination (per our brain). Therefore, if LLMs need a model of a world, and they're trained on the corpus of human knowledge (which passed through our brains), then what's the difference, especially when LLMs are going back into our brains anyway?
- qlm 1y ago
  Language isn't thought. It's a representation of thought.
  - naasking 1y ago
    Are the particles that make up thoughts in our brain not also a representation of a thought? Isn't "thought" really some kind of Platonic ideal that only has approximate material representations? If so, why couldn't some language sentences be thoughts?
    - qlm 1y ago
      The sentence is the result of a thought. The sentence in itself does not capture every process that went into producing the sentence.
      - naasking 1y ago
        > The sentence in itself does not capture every process that went into producing the sentence. A thought does not capture every process that went into producing the thought either.
        qlm 1y ago
        I guess? I would argue that a thought doesn't "capture" anything though, because unlike language it's not a representational tool. A painting of a landscape can capture details of the landscape it's representing. The landscape itself doesn't capture anything, it just is.
imtringued 1y ago
Model based reinforcement learning is a thing and it is kind of a crazy idea. Look up temporal difference model predictive control. The fundamental idea behind temporal difference is that you can record any observable data stream over time and predict the difference between past and present based on your decision variables (e.g. camera movement, actuator movement, and so on). Think of it like the Minecraft clone called Oasis AI. The AI predicts the response to a user provided action. Now imagine if it worked as presented. The data problem would be solved, because you are receiving a constant stream of data every single second. If anything, the RL algorithms are nowhere near where they need to be and continual learning has not been solved yet, but the best known way is through automatic continual learning ala Schmidhuber (co-inventor of LSTMs along with Hochreiter). So, model based control is solved right? Everything that can be observed can be controlled once you have a model! Wrong. Unfortunately. You still need the rest of reinforcement learning: an objective and a way to integrate the model. It turns out that reconstructing the observations is too computationally challenging and the standard computational tricks like U-Nets learn a latent representation that is optimized for reconstruction rather than for your RL objectives. There is a data exchange problem that can only realistically be solved by throwing an even bigger model at it, but here is why that won't work either: Model predictive control tries to find the best trajectory over a receding horizon. It is inherently future oriented. This means that you need to optimize through your big model and that is expensive to do. So you're going to have to take shortcuts by optimizing for a specific task. You reduce the dimension of the latent space and stop reconstructing the observations. The price? You are now learning a latent space for your particular task, which is less demanding. The dream of continual learning with infinite data shatters and you are brought down to earth: it's better than what came before, but not that much better.
LarsDu88 1y ago
This world model talk is interesting, and Yann Lecunn has broached on the same topic, but the fact is there are video diffusion models that are quite good at representing the "video world" and even counterfactually and temporally coherently generating a representation of that "world" under different perturbations. In fact you can go to a SOTA LLM today, and it will do quite well at predicting the outcomes of basic counterfactual scenarios. Animal brains such as our own have evolved to compress information about our world to aide in survival. LLMs and recent diffusion/conditional flow matching models have been quite successful in compressing the "text world" and the "pixel world" to score good loss metrics on training data. It's incredibly difficult to compress information without have at least some internal model of that information. Whether that model is a "world model" that fits the definition of folks like Sutton and LeCunn is semantic.
- timschmidt 1y ago
  1000% this. I would only add this has been demonstrated explicitly with chess: https://adamkarvonen.github.io/machine_learning/2024/01/03/chess-world-models.html https://adamkarvonen.github.io/machine_learning/2024/01/03/c...
- dreambuffer 1y ago
  Photons hit a human eye and then the human came up with language to describe that and then encoded the language into the LLM. The LLM can capture some of this relationship, but the LLM is not sensing actual photons, nor experiencing actual light cone stimulation, nor generating thoughts. Its "world model" is several degrees removed from the real world. So whatever fragment of a model it gains through learning to compress that causal chain of events does not mean much when it cannot generate the actual causal chain.
  - bckr 1y ago
    > Its "world model" is several degrees removed from the real world. Like insects that weave tokens
  - ziofill 1y ago
    I agree with this. A metaphor I like is that the reason why humans say the night sky is beautiful is because they see that it is, whereas an LLM says it because it’s been said enough times in its training data.
DrewADesign 1y ago
I think current AI is a human language/behavior mirror. A cat might believe they see another cat looking in a mirror, but you can’t create a new cat by creating a perfect mirror.
xmichael909 1y ago
love the intentional use of udnerstand, brilliant!
zaphos 1y ago
"just a matter of adding more 9s" is a wild place to use a "just" ...
theptip 1y ago
> LLMs seem to udnerstand language therefore they've trained a model of the world. This isn’t the claim, obviously. LLMs seem to understand a lot more than just language. If you’ve worked with one for hundreds of hours actually exercising frontier capabilities I don’t see how you could think otherwise.
- ekjhgkejhgk 1y ago
  > This isn’t the claim, obviously. This is precisely the claim that leads a of lot people to believe that all you need to reach AGI is more compute.
  - deleted 1y ago
    [deleted]
  - theptip 1y ago
    What I mean here is that this is certainly not what Dwarkesh would claim. It’s a ludicrous strawman position. Dwarkesh is AGI-pilled and would base his assumption of a world model on much more impressive feats than mere language understanding.
    - baobun 1y ago
      Watching the video it seems that Dwarkesh doesn't really have a clue what he's confidently talking about yet running fast with his personal half-baked ideas, to the points where it gets both confusing and cringe when Karpathy apparently manages to make sense of it and yes-anding the word salad AK. Karpathy is supposedly there to clear up misunderstanding yet lets all the nonsense Dwarkesh is putting before him slide. "ludicrous" sure but I wouldn't be so certain about "strawman" or that Dwarkesh has a consistent view.
harrall 1y ago
I don’t have a deep understand of LLMs but don’t they fundamentally work on tokens and generate a multi-dimensional statistical relationship map between tokens? So it doesn’t have to be LLM. You could theoretically have image tokens (though I don’t know in practice, but the important part is the statistical map). And it’s not like my brain doesn’t work like that either. When I say a funny joke in response to people in a group, I can clearly observe my brain pull together related “tokens” (Mary just talked about X, X is related to Y, Y is relevant to Bob), filter them, sort them and then spit out a joke. And that happens in like less than a second.
- tacitusarc 1y ago
  Yes! Absolutely. And this is likely what would be necessary for anything approaching actual AGI. And not just visual input, but all kinds of sensory input. The problem is that we have no ability, not even close, to process that even near the level of a human yet, much less some super genius being.
zwnow 1y ago
A world model can not exist, the context windows aren't even near big enough for that. Weird that every serious scientist agrees on AGI not being a thing in the next decades. LLMs are good if you train them for a specific thing. Not so much if you expect them to explain the whole world to you. This is not possible yet.
Animats 1y ago
> The interviewer had an idea that he took for granted: that to understand language you have to have a model of the world. LLMs seem to understand language therefore they've trained a model of the world. Sutton rejected the premise immediately. He might be right in being skeptical here. That's the basic success of LLMs. They don't have much of a model of the world, and they still work. "Attention is all you need". Good Old Fashioned AI was all about developing models, yet that was a dead end. There's been some progress on representation in an unexpected area. Try Perchance's AI character chat. It seems to be an ordinary chatbot. But at any point in the conversation, you can ask it to generate a picture, which it does using a Stable Diffusion type system. You can generate several pictures, and pick the one you like best. Then let the LLM continue the conversation continue from there. It works from a character sheet, which it will create if asked. It's possible to start from an image and get to a character sheet and a story. The back and forth between the visual and textural domains seems to help. For storytelling, such system may need to generate the collateral materials needed for a stage or screen production - storyboards, scripts with stage directions, character summaries, artwork of sets, blocking (where everybody is positioned on stage), character sheets (poses and costumes) etc. Those are the modeling tools real productions use to keep a work created by many people on track. Those are a form of world model for storytelling. I've been amazed at how good the results I can get from this thing are. You have to coax it a bit. It tends to stay stuck in a scene unless you push the plot forward. But give it a hint of what happens next and it will run with it. [1]https://perchance.org/ai-character-chat https://perchance.org/ai-character-chat
- deleted 1y ago
  [deleted]
senko 1y ago
The thing is, achieving say, 99.99999% reliable AI would be spectacularly useful even if it's a dead end from the AGI perspective. People routinely conflate the "useful LLMs" and "AGI", likely because AGI has been so hyped up, but you don't need AGI to have useful AI. It's like saying the Internet is dead end because it didn't lead to telepathy. It didn't, but it sure as hell is useful. It's beneficial to have both discussions: whether and how to achieve AGI and how to grapple with it, and how to improve a reliability, performance and cost of LLMs for more prosaic use cases. It's just that they are separate discussions.
DanielHB 1y ago
Problem is that these models feels like they are 8 and getting more 8's (maybe 7)
skopje 1y ago
What "9" do you add to AGI? I don't think we even have the axes defined, let alone a way to measure them. "Mistakes per query?" It's like Cantor's diagonal test, where do we even start?
wkat4242 1y ago
Absolutely. AGI isn't a matter of adding more 9s. It's a matter of solving more "???"s. And those require not just work but also a healthy serving of luck. As I understand it, to the breadth of LLMs was also something that was stumbled on kinda by accident, I understand they got developed as translators and were just 'smarter' than expected. Also, to understand the world you don't need language. People don't think in language. Thought is understanding. Language is knowledge transfer and expression.

sysguest 1y ago

yeah that "model of the world" would mean: babies are already born with "the model of the world" but a lot of experiments on babies/young kids tell otherwise

rwj 1y ago
Lots of experiments show that babies develop import capabilities at roughly the same times. That speaks to inherited abilities.
ben_w 1y ago
> babies are already born with "the model of the world" > but a lot of experiments on babies/young kids tell otherwise I believe they are born with such a model? It's just that model is one where mummy still has fur for the baby to cling on to? And where aged something like 5 to 8 it's somehow useful for us to build small enclosures to hide in, leading to a display of pillow forts in the modern world?
- sysguest 1y ago
  damn I guess I had to be more specific: "LLM-level world-detail knowledge"
  - ben_w 1y ago
    I think I'm even more confused now about what you mean…
ekjhgkejhgk 1y ago
> yeah that "model of the world" would mean: babies are already born with "the model of the world" No, not necessarily. Babies don't interact with the world only by reading what people wrote wikipedia and stackoverflow, like these models are trained. Babies do things to the world and observe what happens. I imagine it's similar to the difference between a person sitting on a bicycle and trying to ride it, vs a person watching videos of people riding bicycles. I think it would actually be a great experiment. If you take a person that never rode a bicycle in their life and feed them videos of people riding bicycles, and literature about bikes, fiction and non-fiction, at some point I'm sure they'll be able to talk about it like they have huge experience in riding bikes, but won't be able to ride one.
- aerhardt 1y ago
  We’ve been thinking about reaching the singularity from one end, by making computers like humans, but too little thought has been given to approaching the problem from the other end: by making babies build their world model by reading Stack Overflow.
  - pavlov 1y ago
    The “Brave New World meets OpenAI” model where bottle-born babies listen to Stack Overflow 24 hours a day until they one day graduate to Alphas who get to spend Worldcoin on AI-generated feelies.
  - zelphirkalt 1y ago
    That's it. Now you've done it! I will have stackoverflow Q&A, as well as moderator comments and closings of questions playing 24/7 to my first not yet born child! Q&A for the knowledge and the mod comments for good behavior, of course. This will lead to singularity in no time!
godelski 1y ago
It's a lot more complicated than that. You have instincts, right? Innate fears? This is definitely something passed down through genetics. The Hawk/Goose Effect isn't just limited to baby chickens. Certainly some mental encoding passes down through genetics as how much the brain controls, down to your breathing and heartbeat. But instinct is basic. It's something humans are even able to override. It's a first order approximation. Inaccurate to do meaningfully complex things, but sufficient to keep you alive. Maybe we don't want to call the instinct a world model (it certainly is naïve) but can't be discounted either. In human development, yeah, the lion's share of it happens post birth. Human babies don't even show typical signs of consciousness, even really till the age of 2. There's many different categories of "awareness" and these certainly grow over time. But the big thing that makes humans so intelligent is that we continue to grow and learn through our whole lifetimes. And we can pass that information along without genetics and have very advanced tools to do this. It is a combination of nature and nurture. But do note that this happens differently in different animals. It's wonderfully complex. LLMs are quite incredible but so too are many other non-thinking machines. I don't think we should throw them out, but we never needed to make the jump to intelligence. Certainly not so quickly. I mean what did Carl Sagan say?
- imtringued 1y ago
  One of the biggest mysteries of humans Vs LLMs is that LLMs need an absurd amount of data during pre training, then a little bit of data during fine tuning to make them behave more human. Meanwhile humans don't need any data at all, but have the blind spot that they can only know and learn about what they have observed. This raises two questions. What is the loss function of the supervised learning algorithm equivalent? Supposedly neurons do predictive coding. They predict what their neighbours are doing. That includes input only neurons like touch, pain, vision, sound, taste, etc. The observations never contain actions. E.g. you can look at another human, but that will never teach you how to walk because your legs are different from other people's legs. How do humans avoid starving to death? How do they avoid leaving no children? How do they avoid eating food that will kill them? These things require a complicated chain of actions. You need to find food, a partner and you need to spit out poison. This means you need a reinforcement learning analogue, but what is going to be the reward function equivalent? The reward function can't be created by the brain, because it would be circular. It would be like giving yourself a high, without even needing drugs. Hence, the reward signal must remain inside the body but outside the brain, where the brain can't hack it. The first and most important reward is to perform reproduction. If food and partners are abundant, the ones that don't reproduce simply die out. This means that reward functions that don't reward reproduction disappear. Reproduction is costly in terms of energy. Do it too many times and you need to recover and eat. Hunger evolved as a result of the brain needing to know about the energy state of the body. It overrides reproductive instincts. Now let's say you have a poisonous plant that gives you diarrhea, but you are hungry. What stops you from eating it? Pain evolves as a response to a damaged body. Harmful activities signal themselves in the form of pain to the brain. Pain overrides hunger. However, what if the plant is so deadly that it will kill you? The pain sensors wouldn't be fast enough. You need to sense the poison before it enters your body. So the tongue evolves taste and cyanide starts tasting bitter. Notice something? The feelings only exist internally inside the human body, but they are all coupled with continued survival in one way or another. There is no such thing for robots or LLMs. They won't accidentally evolve a complex reward function like that.

exe34 1y ago

To me, it's a matter of a very big checklist - you can keep adding tasks to the list, but if it keeps marching onwards checking things off your list, some day you will get there. whether it's a linear or asymptotic march, only time will tell.

imadierich 1y ago
[dead]
cactusplant7374 1y ago
That's like saying that if we image every neuron in the brain we will understand thinking. We can build these huge databases and they tell us nothing about the process of thinking.
- exe34 1y ago
  What if we copy the functionality of every neuron? what if we simply copy all the skills that those neurons compute?
  - rootusrootus 1y ago
    Do we even know the functionality of every neuron?
    - exe34 1y ago
      Not yet.
ekjhgkejhgk 1y ago
I don't know if you will get there, that's far from clear at this stage. Did you see the recent video by Nick Beato [1] where he asks various models about a specific number? The models that get it right are the models that consume youtube videos, because there was a youtube video about that specific number. It's like, these models are capable of telling you about very similar things that they've seen, but they don't seem like they understand it. It's totally unclear whether this is a quantitative or qualitative gap. [1] https://www.youtube.com/watch?v=TiwADS600Jc https://www.youtube.com/watch?v=TiwADS600Jc

godelski 1y ago

> that to understand knowledge you have to have a model of the world. You have a small but important mistake. It's to recite (or even apply) knowledge. To understand does actually require a world model. Think of it this way: can you pass a test without understanding the test material? Certainly we all saw people we thought were idiots do well in class while we've also seen people we thought were geniuses fail. The test and understanding usually correlates but it's not perfect, right? The reason I say understanding requires a world model (and I would not say LLMs understand) is because to understand you have to be able to detail things. Look at physics, or the far more detail oriented math. Physicists don't conclude things just off of experimental results. It's an important part, but not the whole story. They also write equations, ones which are counterfactual. You can call this compression if you want (I would and do), but it's only that because of the generalization. But it also only has that power because of the details and nuance. With AI many of these people have been screaming for years (check my history) that what we're doing won't get us all the way there. Not because we want to stop the progress, but because we wanted to ensure continued and accelerate progress. We knew the limits and were saying "let's try to get ahead of this problem" but were told "that'll never be a problem. And if it is, we'll deal with it when we deal with it." It's why Chollet made the claim that LLMs have actually held AI progress back. Because the story that was sold was "AGI is solved, we just need to scale" (i.e. more money). I do still wonder how different things would be if those of us pushing back were able to continue and scale our works (research isn't free, so yes, people did stop us). We always had the math to show that scale wasn't enough, but it's easy to say "you don't need math" when you can see progress. The math never said no progress nor no acceleration, the math said there's a wall and it's easier to adjust now than when we're closer and moving faster. Sadly I don't think we'll ever shift the money over. We still evaluate success weirdly. Successful predictions don't matter. You're still heralded if you made a lot of money in VR and Bitcoin, right?

robotresearcher 1y ago
In my view 'understand' is a folk psychology term that does not have a technical meaning. Like 'intelligent', 'beautiful', and 'interesting'. It usefully labels a basket of behaviors we see in others, and that is all it does. In this view, if a machine performs a task as well as a human, it understands it exactly as much as a human. There's no problem of how to do understanding, only how to do tasks. The 'problem' melts away when you take this stance. Just my opinion, but my professional opinion from thirty-plus years in AI.

tyre 1y ago

There is some evidence from Anthropic that LLMs do model the world. This paper[0] tracing their "thought" is fascinating. Basically an LLM translating across languages will "light up" (to use a rough fMRI equivalent) for the same concepts (e.g. bigness) across languages. It does have clusters of parameters that correlate with concepts, not just randomly "after X word tends to have Y word." Otherwise you would expect all of Chinese to be grouped in one place, all of French in another, all of English in another. This is empirically not the case. I don't know whether to understand knowledge you have to have a model of the world, but at least as far as language, LLMs very much do seem to have modeling. [0]: https://www.anthropic.com/research/tracing-thoughts-language-model https://www.anthropic.com/research/tracing-thoughts-language...

manmal 1y ago
> Basically an LLM translating across languages will "light up" (to use a rough fMRI equivalent) for the same concepts (e.g. bigness) across languages I thought that’s the basic premise of how transformers work - they encode concepts into high dimensional space, and similar concepts will be clustered together. I don’t think it models the world, but just the texts it ingested. It’s observation and regurgitation, not understanding. I do use agents a lot (soon on my second codex subscription), so I don’t think that’s a bad thing. But I’m firmly in the “they are useful tools” camp.
- bryanlarsen 1y ago
  That's a model. Not a higher-order model like most humans use, but it's still a model.
  - manmal 1y ago
    Yes, not of the world, but of the ingested text. Almost verbatim what I wrote.
    - timschmidt 1y ago
      The ingested text itself contains a model of the world which we have encoded in it. That's what language is. Therefore by the transitive property...

deleted 1y ago

[deleted]

bentt 1y ago

I think this a useful challenge to our normal way of thinking. At the same time, "the world" exists only in our imagination (per our brain). Therefore, if LLMs need a model of a world, and they're trained on the corpus of human knowledge (which passed through our brains), then what's the difference, especially when LLMs are going back into our brains anyway?

qlm 1y ago
Language isn't thought. It's a representation of thought.
- naasking 1y ago
  Are the particles that make up thoughts in our brain not also a representation of a thought? Isn't "thought" really some kind of Platonic ideal that only has approximate material representations? If so, why couldn't some language sentences be thoughts?
  - qlm 1y ago
    The sentence is the result of a thought. The sentence in itself does not capture every process that went into producing the sentence.
    - naasking 1y ago
      > The sentence in itself does not capture every process that went into producing the sentence. A thought does not capture every process that went into producing the thought either.
      - qlm 1y ago
        I guess? I would argue that a thought doesn't "capture" anything though, because unlike language it's not a representational tool. A painting of a landscape can capture details of the landscape it's representing. The landscape itself doesn't capture anything, it just is.

imtringued 1y ago

Model based reinforcement learning is a thing and it is kind of a crazy idea. Look up temporal difference model predictive control. The fundamental idea behind temporal difference is that you can record any observable data stream over time and predict the difference between past and present based on your decision variables (e.g. camera movement, actuator movement, and so on). Think of it like the Minecraft clone called Oasis AI. The AI predicts the response to a user provided action. Now imagine if it worked as presented. The data problem would be solved, because you are receiving a constant stream of data every single second. If anything, the RL algorithms are nowhere near where they need to be and continual learning has not been solved yet, but the best known way is through automatic continual learning ala Schmidhuber (co-inventor of LSTMs along with Hochreiter). So, model based control is solved right? Everything that can be observed can be controlled once you have a model! Wrong. Unfortunately. You still need the rest of reinforcement learning: an objective and a way to integrate the model. It turns out that reconstructing the observations is too computationally challenging and the standard computational tricks like U-Nets learn a latent representation that is optimized for reconstruction rather than for your RL objectives. There is a data exchange problem that can only realistically be solved by throwing an even bigger model at it, but here is why that won't work either: Model predictive control tries to find the best trajectory over a receding horizon. It is inherently future oriented. This means that you need to optimize through your big model and that is expensive to do. So you're going to have to take shortcuts by optimizing for a specific task. You reduce the dimension of the latent space and stop reconstructing the observations. The price? You are now learning a latent space for your particular task, which is less demanding. The dream of continual learning with infinite data shatters and you are brought down to earth: it's better than what came before, but not that much better.

LarsDu88 1y ago

This world model talk is interesting, and Yann Lecunn has broached on the same topic, but the fact is there are video diffusion models that are quite good at representing the "video world" and even counterfactually and temporally coherently generating a representation of that "world" under different perturbations. In fact you can go to a SOTA LLM today, and it will do quite well at predicting the outcomes of basic counterfactual scenarios. Animal brains such as our own have evolved to compress information about our world to aide in survival. LLMs and recent diffusion/conditional flow matching models have been quite successful in compressing the "text world" and the "pixel world" to score good loss metrics on training data. It's incredibly difficult to compress information without have at least some internal model of that information. Whether that model is a "world model" that fits the definition of folks like Sutton and LeCunn is semantic.

timschmidt 1y ago
1000% this. I would only add this has been demonstrated explicitly with chess: https://adamkarvonen.github.io/machine_learning/2024/01/03/chess-world-models.html https://adamkarvonen.github.io/machine_learning/2024/01/03/c...
dreambuffer 1y ago
Photons hit a human eye and then the human came up with language to describe that and then encoded the language into the LLM. The LLM can capture some of this relationship, but the LLM is not sensing actual photons, nor experiencing actual light cone stimulation, nor generating thoughts. Its "world model" is several degrees removed from the real world. So whatever fragment of a model it gains through learning to compress that causal chain of events does not mean much when it cannot generate the actual causal chain.
- bckr 1y ago
  > Its "world model" is several degrees removed from the real world. Like insects that weave tokens
- ziofill 1y ago
  I agree with this. A metaphor I like is that the reason why humans say the night sky is beautiful is because they see that it is, whereas an LLM says it because it’s been said enough times in its training data.

DrewADesign 1y ago

I think current AI is a human language/behavior mirror. A cat might believe they see another cat looking in a mirror, but you can’t create a new cat by creating a perfect mirror.

xmichael909 1y ago

love the intentional use of udnerstand, brilliant!

zaphos 1y ago

"just a matter of adding more 9s" is a wild place to use a "just" ...

theptip 1y ago

> LLMs seem to udnerstand language therefore they've trained a model of the world. This isn’t the claim, obviously. LLMs seem to understand a lot more than just language. If you’ve worked with one for hundreds of hours actually exercising frontier capabilities I don’t see how you could think otherwise.

ekjhgkejhgk 1y ago
> This isn’t the claim, obviously. This is precisely the claim that leads a of lot people to believe that all you need to reach AGI is more compute.
- deleted 1y ago
  [deleted]
- theptip 1y ago
  What I mean here is that this is certainly not what Dwarkesh would claim. It’s a ludicrous strawman position. Dwarkesh is AGI-pilled and would base his assumption of a world model on much more impressive feats than mere language understanding.
  - baobun 1y ago
    Watching the video it seems that Dwarkesh doesn't really have a clue what he's confidently talking about yet running fast with his personal half-baked ideas, to the points where it gets both confusing and cringe when Karpathy apparently manages to make sense of it and yes-anding the word salad AK. Karpathy is supposedly there to clear up misunderstanding yet lets all the nonsense Dwarkesh is putting before him slide. "ludicrous" sure but I wouldn't be so certain about "strawman" or that Dwarkesh has a consistent view.

harrall 1y ago

I don’t have a deep understand of LLMs but don’t they fundamentally work on tokens and generate a multi-dimensional statistical relationship map between tokens? So it doesn’t have to be LLM. You could theoretically have image tokens (though I don’t know in practice, but the important part is the statistical map). And it’s not like my brain doesn’t work like that either. When I say a funny joke in response to people in a group, I can clearly observe my brain pull together related “tokens” (Mary just talked about X, X is related to Y, Y is relevant to Bob), filter them, sort them and then spit out a joke. And that happens in like less than a second.

tacitusarc 1y ago
Yes! Absolutely. And this is likely what would be necessary for anything approaching actual AGI. And not just visual input, but all kinds of sensory input. The problem is that we have no ability, not even close, to process that even near the level of a human yet, much less some super genius being.

zwnow 1y ago

A world model can not exist, the context windows aren't even near big enough for that. Weird that every serious scientist agrees on AGI not being a thing in the next decades. LLMs are good if you train them for a specific thing. Not so much if you expect them to explain the whole world to you. This is not possible yet.

Animats 1y ago

> The interviewer had an idea that he took for granted: that to understand language you have to have a model of the world. LLMs seem to understand language therefore they've trained a model of the world. Sutton rejected the premise immediately. He might be right in being skeptical here. That's the basic success of LLMs. They don't have much of a model of the world, and they still work. "Attention is all you need". Good Old Fashioned AI was all about developing models, yet that was a dead end. There's been some progress on representation in an unexpected area. Try Perchance's AI character chat. It seems to be an ordinary chatbot. But at any point in the conversation, you can ask it to generate a picture, which it does using a Stable Diffusion type system. You can generate several pictures, and pick the one you like best. Then let the LLM continue the conversation continue from there. It works from a character sheet, which it will create if asked. It's possible to start from an image and get to a character sheet and a story. The back and forth between the visual and textural domains seems to help. For storytelling, such system may need to generate the collateral materials needed for a stage or screen production - storyboards, scripts with stage directions, character summaries, artwork of sets, blocking (where everybody is positioned on stage), character sheets (poses and costumes) etc. Those are the modeling tools real productions use to keep a work created by many people on track. Those are a form of world model for storytelling. I've been amazed at how good the results I can get from this thing are. You have to coax it a bit. It tends to stay stuck in a scene unless you push the plot forward. But give it a hint of what happens next and it will run with it. [1]https://perchance.org/ai-character-chat https://perchance.org/ai-character-chat

deleted 1y ago
[deleted]

senko 1y ago

The thing is, achieving say, 99.99999% reliable AI would be spectacularly useful even if it's a dead end from the AGI perspective. People routinely conflate the "useful LLMs" and "AGI", likely because AGI has been so hyped up, but you don't need AGI to have useful AI. It's like saying the Internet is dead end because it didn't lead to telepathy. It didn't, but it sure as hell is useful. It's beneficial to have both discussions: whether and how to achieve AGI and how to grapple with it, and how to improve a reliability, performance and cost of LLMs for more prosaic use cases. It's just that they are separate discussions.

DanielHB 1y ago

Problem is that these models feels like they are 8 and getting more 8's (maybe 7)

skopje 1y ago

What "9" do you add to AGI? I don't think we even have the axes defined, let alone a way to measure them. "Mistakes per query?" It's like Cantor's diagonal test, where do we even start?

wkat4242 1y ago

Absolutely. AGI isn't a matter of adding more 9s. It's a matter of solving more "???"s. And those require not just work but also a healthy serving of luck. As I understand it, to the breadth of LLMs was also something that was stumbled on kinda by accident, I understand they got developed as translators and were just 'smarter' than expected. Also, to understand the world you don't need language. People don't think in language. Thought is understanding. Language is knowledge transfer and expression.