11 ms·

> that to understand knowledge you have to have a model of the world. You have a small but important mistake. It's to recite (or even apply) knowledge. To unde

> that to understand knowledge you have to have a model of the world. You have a small but important mistake. It's to recite (or even apply) knowledge. To understand does actually require a world model. Think of it this way: can you pass a test without understanding the test material? Certainly we all saw people we thought were idiots do well in class while we've also seen people we thought were geniuses fail. The test and understanding usually correlates but it's not perfect, right? The reason I say understanding requires a world model (and I would not say LLMs understand) is because to understand you have to be able to detail things. Look at physics, or the far more detail oriented math. Physicists don't conclude things just off of experimental results. It's an important part, but not the whole story. They also write equations, ones which are counterfactual. You can call this compression if you want (I would and do), but it's only that because of the generalization. But it also only has that power because of the details and nuance. With AI many of these people have been screaming for years (check my history) that what we're doing won't get us all the way there. Not because we want to stop the progress, but because we wanted to ensure continued and accelerate progress. We knew the limits and were saying "let's try to get ahead of this problem" but were told "that'll never be a problem. And if it is, we'll deal with it when we deal with it." It's why Chollet made the claim that LLMs have actually held AI progress back. Because the story that was sold was "AGI is solved, we just need to scale" (i.e. more money). I do still wonder how different things would be if those of us pushing back were able to continue and scale our works (research isn't free, so yes, people did stop us). We always had the math to show that scale wasn't enough, but it's easy to say "you don't need math" when you can see progress. The math never said no progress nor no acceleration, the math said there's a wall and it's easier to adjust now than when we're closer and moving faster. Sadly I don't think we'll ever shift the money over. We still evaluate success weirdly. Successful predictions don't matter. You're still heralded if you made a lot of money in VR and Bitcoin, right?

robotresearcher 1y ago
In my view 'understand' is a folk psychology term that does not have a technical meaning. Like 'intelligent', 'beautiful', and 'interesting'. It usefully labels a basket of behaviors we see in others, and that is all it does. In this view, if a machine performs a task as well as a human, it understands it exactly as much as a human. There's no problem of how to do understanding, only how to do tasks. The 'problem' melts away when you take this stance. Just my opinion, but my professional opinion from thirty-plus years in AI.
- compass_copium 1y ago
  Nonsense. A QC operator may be able to carry out a test with as much accuracy (or perhaps better accuracy, with enough practice) than the PhD quality chemist who developed it. They could plausibly do so with a high school education and not be able to explain the test in any detail. They do not understand the test in the same way as the chemist. If 'understand' is a meaningless term to someone who's spent 30 years in AI research, I understand why LLMs are being sold and hyped in the way they are.
  - robotresearcher 1y ago
    > They do not understand the test in the same way as the chemist. Can you explain precisely what 'understand' means here, without using the word 'understand'? I don't think anyone can.
    - throw4847285 1y ago
      There are a number of competing models. The SEP page is probably a good place to start. https://plato.stanford.edu/entries/understanding https://plato.stanford.edu/entries/understanding
    - bandrami 1y ago
      Not to be flippant but have you considered that that question is an entire branch of philosophy with a several-millennias long history which people in some cases spend their entire life studying?
      - robotresearcher 1y ago
        I have. It robustly has the folk-psychological meaning I mentioned in my first sentence. Call it ‘philosophical’ instead of ‘folk-psychological’ if you like. It’s a useful concept. But the concept doesn’t require AI engineers to do anything. It certainly doesn’t give any hints about AI engineers what they should actually do. “Make it understand.” “How? What does that look like?” “… But it needs to understand…” “It answers your questions.” “But it doesn’t understand.” “Ok. Get back to me when that entails anything.”
naasking 1y ago
> It's to recite (or even apply) knowledge. To understand does actually require a world model. This is a shell game, or a god of the gaps. All you're saying is that the models "understand" how to recite or apply knowledge or language, but somehow don't understand knowledge or language. Well what else is there really?
- godelski 1y ago
  > Well what else is there really? Differentiate from memorization. I'd say there's a difference between a database and understanding. If they're the same, well I think Google created AGI a long time ago.
  - naasking 1y ago
    A database doesn't recite or apply knowledge, it stores knowledge.
    - godelski 1y ago
      It sure recites it when I query it
      - naasking 1y ago
        It makes perfect sense to say that the database understands your query. It also makes sense to say that the database's factorization of domain knowledge + domain queries exhibit at least a static domain understanding (which still isn't general ala AGI). This is the standard systems response to the Chinese Room. The "general" part comes from whether that static aspect can be made dynamic and extensible. In what sense is a system that can be arbitrarily extended to "recite" or "apply" knowledge not AGI?
        godelski 1y ago
        I would say that the database processes my query, not understands it. You reference the Chinese Room, but that's the thing. The Chinese Room processes the inputs without understanding them.
munksbeer 1y ago
> We always had the math to show that scale wasn't enough Math, to show that scale (presumably of LLMs) wasn't enough for AGI? This sounds like it would be quite a big deal, what math is that?
- hodgehog11 1y ago
  As someone who is invested in researching said math, I can say with some confidence that it does not exist, or at least not in the form claimed here. That's the whole problem. I would be ecstatic if it did though, so if anyone has any examples or rebuttal, I would very much appreciate it.
  - nsagent 1y ago
    You're right that there is no purely mathematical argument; it's almost non-sensical to claim such. Instead you can simply make the rather reasonable observation that LLMs are a product of their training distribution, which only contains partial coverage of all possible observable states of the world. Some highly regular observable states are thus likely missing, but an embodied agent (like a human) would be able to correctly reason about these states due to other sensory modes that observe these states. It's very clear that the number of possible states far exceeds any text describing those states (the number of states transcribed to text is vanishingly small in comparison, but they are locally coherent and quite regular when observed through other sensory modes like vision). That's the closest you can make to a mathematical argument. As a result one common failure case for an LLM might involve describing a specific physical environment and associated set of actions that are in an area of low (or no) probability mass in the training data and asking the LLM to reason about it. Here's an example where I asked ChatGPT the following: imagine i walked into a room through a doorway, then turned around and walked right back out of the room. from the perspective of a person in the room facing the door, which side would my right arm be? please explain. It responded by incorrectly stating the following: Let’s break this down step by step. 1. *You walk into the room through the doorway*: * As you're entering, the person in the room is facing the door. So, you're entering from the **outside**. * Your **right arm** will be on the **right side** of your body, just as it is normally when you're standing. 2. *You turn around and walk right back out*: * When you turn around to walk out, you’ll be facing **away** from the room, looking back toward the doorway. * Since you're walking **back out**, the person inside the room is still facing the door, while you are walking away from them, so your **right arm** is still on the right side of your body. So, from the perspective of the person in the room, as you’re exiting, your *right arm* will still be on their *left side*. That’s because they’re facing the door, and your right side is to their left when you’re turning around and leaving. To visualize: * Imagine a door in front of them. * You walk through and turn around, so you're now facing away from them. * Your right arm is on your right side, but to them, it's on their left side since you’re now moving in the opposite direction.
nebula8804 1y ago
Only problem is this time enough money is being burned that if AGI does not come, it will probably be extremely painful/fatal for a lot of people that had nothing to do with this field or the decisions being made. What will be the consequences if that comes to pass? So many lives were permanently ruined due to the GFC.
JKCalhoun 1y ago
I'm not sure. There's a view that, as I understand it, suggests that language is intelligence. That language is a requirement for understanding. An example might be kind of the contrary—that you might not be able to hold an idea in your head until it has been named. For myself, until I heard the word gestalt (maybe a fitting example?) I am not sure I could have understood the concept. But when it is described it starts to coalesce—and then when named, it became real. (If that makes sense.) FWIW, Zeitgeist is another one of those concepts/words for me. I guess I have to thank the German language. Perhaps it is why other animals on this planet seem to us lacking intelligence. Perhaps it is their lack of complex language holding their minds back.
- godelski 1y ago
  > There's a view that suggests that language is intelligence. I think you find the limits when you dig in. What are you calling language? Can you really say that Eliza doesn't meet your criteria? What about a more advanced version? I mean we've been passing the Turing Test for decades now. > That language is a requirement for understanding. But this contradicts your earlier statement. If language is a requirement then it must precede intelligence, right? I think you must then revisit your definition of language and ensure that it matches to all the creatures that you consider intelligent. At least by doing this you'll make some falsifiable claims and can make progress. I think an ant is intelligent, but I also think ants do things far more sophisticated than the average person thinks. It's an easy trap, not knowing what you don't know. But if we do the above we get some path to aid in discovery, right? > that you might not be able to hold an idea in your head until it has been named Are you familiar with Anendophasia? It is the condition where a person does not have an internal monologue. They think without words. The definition of language is still flexible enough that you can probably still call that language, just like in your example, but it shows a lack of precision in the definition, even if it is accurate. > Perhaps it is why other animals on this planet seem to us lacking intelligence One thing to also consider is if language is necessary for societies or intelligence. Can we decouple the two? I'm not aware of any great examples, although octopi and many other cephalopods are fairly asocial creatures. Yet they are considered highly intelligent due to their adaptive and creative nature. Perhaps language is a necessary condition for advanced intelligence, but not intelligence alone. Perhaps it is communication and societies, differentiating from an internalized language. Certainly the social group can play an influence here, as coalitions can do more than the sum of the individuals (by definition). But the big question is if these things are necessary. Getting the correct causal graph, removing the confounding variables, is no easy task. But I think we should still try and explore differing ideas. While I don't think you're right, I'll encourage you to pursue your path if you encourage me to pursue mine. We can compete, but it should be friendly, as our competition forces us to help see flaws in our models. Maybe the social element isn't a necessary condition, but I have no doubt that it is a beneficial tool. I'm more frustrated by those wanting to call the problem solved. It obviously isn't, as it's been so difficult to get generalization and consensus among experts (across fields).
camillomiller 1y ago
Fantastic comment!

robotresearcher 1y ago

In my view 'understand' is a folk psychology term that does not have a technical meaning. Like 'intelligent', 'beautiful', and 'interesting'. It usefully labels a basket of behaviors we see in others, and that is all it does. In this view, if a machine performs a task as well as a human, it understands it exactly as much as a human. There's no problem of how to do understanding, only how to do tasks. The 'problem' melts away when you take this stance. Just my opinion, but my professional opinion from thirty-plus years in AI.

compass_copium 1y ago
Nonsense. A QC operator may be able to carry out a test with as much accuracy (or perhaps better accuracy, with enough practice) than the PhD quality chemist who developed it. They could plausibly do so with a high school education and not be able to explain the test in any detail. They do not understand the test in the same way as the chemist. If 'understand' is a meaningless term to someone who's spent 30 years in AI research, I understand why LLMs are being sold and hyped in the way they are.
- robotresearcher 1y ago
  > They do not understand the test in the same way as the chemist. Can you explain precisely what 'understand' means here, without using the word 'understand'? I don't think anyone can.
  - throw4847285 1y ago
    There are a number of competing models. The SEP page is probably a good place to start. https://plato.stanford.edu/entries/understanding https://plato.stanford.edu/entries/understanding
  - bandrami 1y ago
    Not to be flippant but have you considered that that question is an entire branch of philosophy with a several-millennias long history which people in some cases spend their entire life studying?
    - robotresearcher 1y ago
      I have. It robustly has the folk-psychological meaning I mentioned in my first sentence. Call it ‘philosophical’ instead of ‘folk-psychological’ if you like. It’s a useful concept. But the concept doesn’t require AI engineers to do anything. It certainly doesn’t give any hints about AI engineers what they should actually do. “Make it understand.” “How? What does that look like?” “… But it needs to understand…” “It answers your questions.” “But it doesn’t understand.” “Ok. Get back to me when that entails anything.”

naasking 1y ago

> It's to recite (or even apply) knowledge. To understand does actually require a world model. This is a shell game, or a god of the gaps. All you're saying is that the models "understand" how to recite or apply knowledge or language, but somehow don't understand knowledge or language. Well what else is there really?

godelski 1y ago
> Well what else is there really? Differentiate from memorization. I'd say there's a difference between a database and understanding. If they're the same, well I think Google created AGI a long time ago.
- naasking 1y ago
  A database doesn't recite or apply knowledge, it stores knowledge.
  - godelski 1y ago
    It sure recites it when I query it
    - naasking 1y ago
      It makes perfect sense to say that the database understands your query. It also makes sense to say that the database's factorization of domain knowledge + domain queries exhibit at least a static domain understanding (which still isn't general ala AGI). This is the standard systems response to the Chinese Room. The "general" part comes from whether that static aspect can be made dynamic and extensible. In what sense is a system that can be arbitrarily extended to "recite" or "apply" knowledge not AGI?
      - godelski 1y ago
        I would say that the database processes my query, not understands it. You reference the Chinese Room, but that's the thing. The Chinese Room processes the inputs without understanding them.

munksbeer 1y ago

> We always had the math to show that scale wasn't enough Math, to show that scale (presumably of LLMs) wasn't enough for AGI? This sounds like it would be quite a big deal, what math is that?

hodgehog11 1y ago
As someone who is invested in researching said math, I can say with some confidence that it does not exist, or at least not in the form claimed here. That's the whole problem. I would be ecstatic if it did though, so if anyone has any examples or rebuttal, I would very much appreciate it.
- nsagent 1y ago
  You're right that there is no purely mathematical argument; it's almost non-sensical to claim such. Instead you can simply make the rather reasonable observation that LLMs are a product of their training distribution, which only contains partial coverage of all possible observable states of the world. Some highly regular observable states are thus likely missing, but an embodied agent (like a human) would be able to correctly reason about these states due to other sensory modes that observe these states. It's very clear that the number of possible states far exceeds any text describing those states (the number of states transcribed to text is vanishingly small in comparison, but they are locally coherent and quite regular when observed through other sensory modes like vision). That's the closest you can make to a mathematical argument. As a result one common failure case for an LLM might involve describing a specific physical environment and associated set of actions that are in an area of low (or no) probability mass in the training data and asking the LLM to reason about it. Here's an example where I asked ChatGPT the following: imagine i walked into a room through a doorway, then turned around and walked right back out of the room. from the perspective of a person in the room facing the door, which side would my right arm be? please explain. It responded by incorrectly stating the following: Let’s break this down step by step. 1. *You walk into the room through the doorway*: * As you're entering, the person in the room is facing the door. So, you're entering from the **outside**. * Your **right arm** will be on the **right side** of your body, just as it is normally when you're standing. 2. *You turn around and walk right back out*: * When you turn around to walk out, you’ll be facing **away** from the room, looking back toward the doorway. * Since you're walking **back out**, the person inside the room is still facing the door, while you are walking away from them, so your **right arm** is still on the right side of your body. So, from the perspective of the person in the room, as you’re exiting, your *right arm* will still be on their *left side*. That’s because they’re facing the door, and your right side is to their left when you’re turning around and leaving. To visualize: * Imagine a door in front of them. * You walk through and turn around, so you're now facing away from them. * Your right arm is on your right side, but to them, it's on their left side since you’re now moving in the opposite direction.

nebula8804 1y ago

Only problem is this time enough money is being burned that if AGI does not come, it will probably be extremely painful/fatal for a lot of people that had nothing to do with this field or the decisions being made. What will be the consequences if that comes to pass? So many lives were permanently ruined due to the GFC.

JKCalhoun 1y ago

I'm not sure. There's a view that, as I understand it, suggests that language is intelligence. That language is a requirement for understanding. An example might be kind of the contrary—that you might not be able to hold an idea in your head until it has been named. For myself, until I heard the word gestalt (maybe a fitting example?) I am not sure I could have understood the concept. But when it is described it starts to coalesce—and then when named, it became real. (If that makes sense.) FWIW, Zeitgeist is another one of those concepts/words for me. I guess I have to thank the German language. Perhaps it is why other animals on this planet seem to us lacking intelligence. Perhaps it is their lack of complex language holding their minds back.

godelski 1y ago
> There's a view that suggests that language is intelligence. I think you find the limits when you dig in. What are you calling language? Can you really say that Eliza doesn't meet your criteria? What about a more advanced version? I mean we've been passing the Turing Test for decades now. > That language is a requirement for understanding. But this contradicts your earlier statement. If language is a requirement then it must precede intelligence, right? I think you must then revisit your definition of language and ensure that it matches to all the creatures that you consider intelligent. At least by doing this you'll make some falsifiable claims and can make progress. I think an ant is intelligent, but I also think ants do things far more sophisticated than the average person thinks. It's an easy trap, not knowing what you don't know. But if we do the above we get some path to aid in discovery, right? > that you might not be able to hold an idea in your head until it has been named Are you familiar with Anendophasia? It is the condition where a person does not have an internal monologue. They think without words. The definition of language is still flexible enough that you can probably still call that language, just like in your example, but it shows a lack of precision in the definition, even if it is accurate. > Perhaps it is why other animals on this planet seem to us lacking intelligence One thing to also consider is if language is necessary for societies or intelligence. Can we decouple the two? I'm not aware of any great examples, although octopi and many other cephalopods are fairly asocial creatures. Yet they are considered highly intelligent due to their adaptive and creative nature. Perhaps language is a necessary condition for advanced intelligence, but not intelligence alone. Perhaps it is communication and societies, differentiating from an internalized language. Certainly the social group can play an influence here, as coalitions can do more than the sum of the individuals (by definition). But the big question is if these things are necessary. Getting the correct causal graph, removing the confounding variables, is no easy task. But I think we should still try and explore differing ideas. While I don't think you're right, I'll encourage you to pursue your path if you encourage me to pursue mine. We can compete, but it should be friendly, as our competition forces us to help see flaws in our models. Maybe the social element isn't a necessary condition, but I have no doubt that it is a beneficial tool. I'm more frustrated by those wanting to call the problem solved. It obviously isn't, as it's been so difficult to get generalization and consensus among experts (across fields).

camillomiller 1y ago

Fantastic comment!