5 ms·

Generative AI doesn't have a coherent understanding of the world

58 points by gmays 2y ago

h_tbob 2y ago
To be honest… it’s amazing it can have any understanding given the only “senses” it has are the 1024 inputs of d_model! Imagine trying to understand the world if you were simply given books and books in a language you had never read… and you didn’t even know how to read or even talk! So it’s pretty incredible it’s got this far!
- pjerem 2y ago
  I mean, I’m amazed by LLMs. But what you describe is basically done by any human on Earth : you are born not knowing how to read or how to talk and after years of learning, reading all the books may give you some understanding of the world. Contrary to LLM though, human brains don’t have a virtually infinite energy supply, cannot be parallelized and have to dedicate their already scarce energy to do a lot of other things than reading books including moving in a 3D world, living in a society, feeding themselves, doing their own hardware maintenance (sleep …), pay attention not to die every single day etc etc. So, for sure, LLMs _algorithms_ are really incredible, but they are also useful only if you throw a lot of hardware and energy into them. I’d be curious to see how long you’d need to train (not use) a useful LLM with only 20W of power (which is more or less the power we estimate the brain is using to function). We can still be impressed by the results, but not really by the speed. And when you have to learn the entire written corpus in some weeks/months, speed is pretty useful.
  - corimaith 2y ago
    Pretty sure the human brain is far more parallelized than a regular CPU or GPU. Our image recognition for example probably dosen't take shortcuts like convulution because of the cost of processing each "pixel", we directly do it all with those millions of eye neurons. Well, to be fair there is alot of post-processing and "magic" involved in getting the final image.
    - danielbln 2y ago
      There is soooo much post processing happening in the visual cortex. The image from the eye is crude, only in focus in a tiny area, warped and upside down. It's quite incredible how our brains create this seamless virtual reality out of that input soup.
  - jjk166 2y ago
    > But what you describe is basically done by any human on Earth You don't simply learn on training data, you are taught by other humans who were in turn taught by more humans still. When you do learn genuinely new information, you know exactly how it makes sense to you, and you can share it with others in a way optimized for them to incorporate it as well. LLMs aren't just learning to speak and read, they are developing from scratch a framework for comprehending the training data thrown at them. An LLM has no concept of phonetics because it has no ears with which to hear nor a mouth with which to speak. It derived english from pulses of electricity. Leave a child in a box with no human contact and a library of books and perhaps it will ascribe meaning to the symbols written on the pages, but it will never know what they actually mean. Even if they lived a million lifetimes, and were able to recognize that the symbols appear with certain regularity and could infer some rules of the language, they would never know what concept the string "comprehension" refers to.
jmole 2y ago
None of us has a coherent view of the world, the map is not the territory..
- partomniscient 2y ago
  Yeah I was going to call out MIT for pointing out the obvious, but there's enough noise/misunderstanding out there, that this kind of article can lead to the 'I get it' moment for someone.
- krapp 2y ago
  A map, if it is useful (which the subjective human experience of reality tends to be, for most people most of the time,) is by definition a coherent view of the territory. Coherent doesn't imply perfect objective accuracy.
  - drdeca 2y ago
    By definition, you say? By what definition? I don’t see why something would need to be entirely coherent to be useful. If you have a program which, when given a statement, assigns a value between 0 and 1, to be interpreted as if it were a probability of the statement being true, it needn’t define an entirely coherent probability distribution.
    - krapp 2y ago
      The coherence comes from the relationship between the virtual (mapped) data and the real data it represents (territory), and the ability of the map to correctly predict the attributes of the territory. Human perception is coherent because it tends to accurately model the physical world from which it's constructed, by means of sensory input and memory, and if this were not the case, we would have gone extinct as a species ages ago. While it is true that the model of human perception isn't perfect, it would be incorrect to call it incoherent to the same or similar degree as the incoherence demostrated by LLMs, as jmole's earlier comment implies. Even if one must consider the model of human perception incoherent, one must also consider it vastly more coherent than whatever LLMs model reality on, if anything.
mediumsmart 2y ago
AI generates a world that is language mapped by word pointing to another word like we did?
- Pet_Ant 2y ago
  Actually that is a real question and it has been studied in philosophy of the mind: https://en.wikipedia.org/wiki/Intentional_object https://en.wikipedia.org/wiki/Intentional_object
southernplaces7 2y ago
>Generative AI doesn't have a coherent understanding of the world The very headline is based on a completely faulty assumption, that AI has any capacity for understanding at all, which it doesn't. That would require self-directed reasoning and self-awareness, both of which it lacks based on any available evidence. (though there's no shortage of irrational defenders here who somehow leap to say that there's no difference between consciousness in humans and the pattern matching of AI technology today, because they happened to have a "conversation" with ChatGPT and etc).
- _heimdall 2y ago
  > The very headline is based on a completely faulty assumption, that AI has any capacity for understanding at all, which it doesn't. And this right here is why its so frustrating to me that the term "AI" is used for LLMs. They are impressive for certain tasks, but they are nothing close to artificial intelligence and were never designed to be.
  - furyofantares 2y ago
    I love that "AI" is suddenly a high bar that the best AI technologies we have don't pass, when the field is 70 years old and includes, you know, straightforward search algorithms. I mean, I get it. These things are mainstream and close enough in some ways to what we see in science fiction that the term evokes the sci-fi meaning of the term rather than the computer science meaning.
    - _heimdall 2y ago
      I can't speak to 70 years ago, but in my 30ish years I don't think I have raised the bar of what I'd consider AI. I never thought of text prediction or auto complete as AI for example, though out of context I could see many 70 years ago considering it AI. I've always considered AI as a much more complex concept than simply appearing to be human in speech or text. Extremely complex prediction algorithms that appear on the surface to be creative wouldn't have met my bar either. An LLM can predict how a human may write a white paper, for example. It will sound impressive, but the model and related algorithms seem to have absolutely no way of logically working through a problem or modelling the world to know if the ideas proposed in the paper might work. Its a really impressive sounding impersonator, but it is nothing like intelligence.
upghost 2y ago
"MIT Researchers have yet to discover self-aware sentient dataset. In unrelated news, Sam Altman promises with another 7 trillion dollars he can turn lead into gold. 'For real this time,' claims Altman."
- mdp2021 2y ago
  Joke aside, the piece is not about «self-aware sentient dataset[s]».
  - upghost 2y ago
    Isn't it, though? Interesting -- I'm surprised this joke didn't land harder. Thought experiment-- let's say you have the PERFECT transformer architecture and perfectly tuned weights with infinite free compute and instantaneous compute speed. What does that give? Something that allows you to perfectly generalize over your dataset. You can perfectly extract the signal from the noise. Having a "world model" implies a model of the world and thus implies a model of the self distinct from the world. Either that or the world model incorporates the self, right? Because the thing doing the modeling must be part of the world in order for it to have a model of the world. So, given a perfect transformer and perfect language model, you will at best get a perfect generalization of the dataset. You cannot take a dataset that has no "world model" and run it through a perfect transformer and somehow produce a world model, aka, cannot turn lead into gold-- you have created information from thin air and now we are in the realm of magic and not science. It does not matter how much compute you throw at the problem. Looking at the model is a red herring. The answers are in the dataset.
    - NLPaep 2y ago
      Is this argument not also applicable to us and our lives?
      - upghost 2y ago
        No, not even close. We are not "tabula rasa"[1] or blank slate*. If you would actually like to understand why, some good books about this are "The Self Assembling Brain"[2] and "The Archaeology of Mind"[3]. [1] https://en.wikipedia.org/wiki/Tabula_rasa https://en.wikipedia.org/wiki/Tabula_rasa [2] https://press.princeton.edu/books/hardcover/9780691181226/the-self-assembling-brain https://press.princeton.edu/books/hardcover/9780691181226/th... [3] https://www.amazon.com/Archaeology-Mind-Neuroevolutionary-Interpersonal-Neurobiology/dp/0393705315 https://www.amazon.com/Archaeology-Mind-Neuroevolutionary-In... [*] One of the things that frustrates me the most in the discourse on LLMs is that people who should know better deliberately mislead others into believing that there is something similar to "intelligence" going on with LLMs -- because they are heavily financially incentivized to do so. Comparisons with humans are categorical errors in everything but metaphor. They call them "neural networks" instead of "systems of nonlinear equations", because "neural network" sounds way sexier than vectorized y=f(mx+b).
penjelly 2y ago
LLMs in their current form seem to me like intuitive thought, like what we use when in conversation with friends over a beer. It seems like only one part of a future AI brain, there still needs to be another system for maintaining a world model, another for background planning, etc. These + vision and hearing models to take in new info about the world and we'll have something resembling human intelligence where all that data meets

58 points by gmays 2y ago

To be honest… it’s amazing it can have any understanding given the only “senses” it has are the 1024 inputs of d_model! Imagine trying to understand the world if you were simply given books and books in a language you had never read… and you didn’t even know how to read or even talk! So it’s pretty incredible it’s got this far!

pjerem 2y ago
I mean, I’m amazed by LLMs. But what you describe is basically done by any human on Earth : you are born not knowing how to read or how to talk and after years of learning, reading all the books may give you some understanding of the world. Contrary to LLM though, human brains don’t have a virtually infinite energy supply, cannot be parallelized and have to dedicate their already scarce energy to do a lot of other things than reading books including moving in a 3D world, living in a society, feeding themselves, doing their own hardware maintenance (sleep …), pay attention not to die every single day etc etc. So, for sure, LLMs _algorithms_ are really incredible, but they are also useful only if you throw a lot of hardware and energy into them. I’d be curious to see how long you’d need to train (not use) a useful LLM with only 20W of power (which is more or less the power we estimate the brain is using to function). We can still be impressed by the results, but not really by the speed. And when you have to learn the entire written corpus in some weeks/months, speed is pretty useful.
- corimaith 2y ago
  Pretty sure the human brain is far more parallelized than a regular CPU or GPU. Our image recognition for example probably dosen't take shortcuts like convulution because of the cost of processing each "pixel", we directly do it all with those millions of eye neurons. Well, to be fair there is alot of post-processing and "magic" involved in getting the final image.
  - danielbln 2y ago
    There is soooo much post processing happening in the visual cortex. The image from the eye is crude, only in focus in a tiny area, warped and upside down. It's quite incredible how our brains create this seamless virtual reality out of that input soup.
- jjk166 2y ago
  > But what you describe is basically done by any human on Earth You don't simply learn on training data, you are taught by other humans who were in turn taught by more humans still. When you do learn genuinely new information, you know exactly how it makes sense to you, and you can share it with others in a way optimized for them to incorporate it as well. LLMs aren't just learning to speak and read, they are developing from scratch a framework for comprehending the training data thrown at them. An LLM has no concept of phonetics because it has no ears with which to hear nor a mouth with which to speak. It derived english from pulses of electricity. Leave a child in a box with no human contact and a library of books and perhaps it will ascribe meaning to the symbols written on the pages, but it will never know what they actually mean. Even if they lived a million lifetimes, and were able to recognize that the symbols appear with certain regularity and could infer some rules of the language, they would never know what concept the string "comprehension" refers to.

jmole 2y ago

None of us has a coherent view of the world, the map is not the territory..

partomniscient 2y ago
Yeah I was going to call out MIT for pointing out the obvious, but there's enough noise/misunderstanding out there, that this kind of article can lead to the 'I get it' moment for someone.
krapp 2y ago
A map, if it is useful (which the subjective human experience of reality tends to be, for most people most of the time,) is by definition a coherent view of the territory. Coherent doesn't imply perfect objective accuracy.
- drdeca 2y ago
  By definition, you say? By what definition? I don’t see why something would need to be entirely coherent to be useful. If you have a program which, when given a statement, assigns a value between 0 and 1, to be interpreted as if it were a probability of the statement being true, it needn’t define an entirely coherent probability distribution.
  - krapp 2y ago
    The coherence comes from the relationship between the virtual (mapped) data and the real data it represents (territory), and the ability of the map to correctly predict the attributes of the territory. Human perception is coherent because it tends to accurately model the physical world from which it's constructed, by means of sensory input and memory, and if this were not the case, we would have gone extinct as a species ages ago. While it is true that the model of human perception isn't perfect, it would be incorrect to call it incoherent to the same or similar degree as the incoherence demostrated by LLMs, as jmole's earlier comment implies. Even if one must consider the model of human perception incoherent, one must also consider it vastly more coherent than whatever LLMs model reality on, if anything.

mediumsmart 2y ago

AI generates a world that is language mapped by word pointing to another word like we did?

Pet_Ant 2y ago
Actually that is a real question and it has been studied in philosophy of the mind: https://en.wikipedia.org/wiki/Intentional_object https://en.wikipedia.org/wiki/Intentional_object

southernplaces7 2y ago

>Generative AI doesn't have a coherent understanding of the world The very headline is based on a completely faulty assumption, that AI has any capacity for understanding at all, which it doesn't. That would require self-directed reasoning and self-awareness, both of which it lacks based on any available evidence. (though there's no shortage of irrational defenders here who somehow leap to say that there's no difference between consciousness in humans and the pattern matching of AI technology today, because they happened to have a "conversation" with ChatGPT and etc).

_heimdall 2y ago
> The very headline is based on a completely faulty assumption, that AI has any capacity for understanding at all, which it doesn't. And this right here is why its so frustrating to me that the term "AI" is used for LLMs. They are impressive for certain tasks, but they are nothing close to artificial intelligence and were never designed to be.
- furyofantares 2y ago
  I love that "AI" is suddenly a high bar that the best AI technologies we have don't pass, when the field is 70 years old and includes, you know, straightforward search algorithms. I mean, I get it. These things are mainstream and close enough in some ways to what we see in science fiction that the term evokes the sci-fi meaning of the term rather than the computer science meaning.
  - _heimdall 2y ago
    I can't speak to 70 years ago, but in my 30ish years I don't think I have raised the bar of what I'd consider AI. I never thought of text prediction or auto complete as AI for example, though out of context I could see many 70 years ago considering it AI. I've always considered AI as a much more complex concept than simply appearing to be human in speech or text. Extremely complex prediction algorithms that appear on the surface to be creative wouldn't have met my bar either. An LLM can predict how a human may write a white paper, for example. It will sound impressive, but the model and related algorithms seem to have absolutely no way of logically working through a problem or modelling the world to know if the ideas proposed in the paper might work. Its a really impressive sounding impersonator, but it is nothing like intelligence.

upghost 2y ago

"MIT Researchers have yet to discover self-aware sentient dataset. In unrelated news, Sam Altman promises with another 7 trillion dollars he can turn lead into gold. 'For real this time,' claims Altman."

mdp2021 2y ago
Joke aside, the piece is not about «self-aware sentient dataset[s]».
- upghost 2y ago
  Isn't it, though? Interesting -- I'm surprised this joke didn't land harder. Thought experiment-- let's say you have the PERFECT transformer architecture and perfectly tuned weights with infinite free compute and instantaneous compute speed. What does that give? Something that allows you to perfectly generalize over your dataset. You can perfectly extract the signal from the noise. Having a "world model" implies a model of the world and thus implies a model of the self distinct from the world. Either that or the world model incorporates the self, right? Because the thing doing the modeling must be part of the world in order for it to have a model of the world. So, given a perfect transformer and perfect language model, you will at best get a perfect generalization of the dataset. You cannot take a dataset that has no "world model" and run it through a perfect transformer and somehow produce a world model, aka, cannot turn lead into gold-- you have created information from thin air and now we are in the realm of magic and not science. It does not matter how much compute you throw at the problem. Looking at the model is a red herring. The answers are in the dataset.
  - NLPaep 2y ago
    Is this argument not also applicable to us and our lives?
    - upghost 2y ago
      No, not even close. We are not "tabula rasa"[1] or blank slate*. If you would actually like to understand why, some good books about this are "The Self Assembling Brain"[2] and "The Archaeology of Mind"[3]. [1] https://en.wikipedia.org/wiki/Tabula_rasa https://en.wikipedia.org/wiki/Tabula_rasa [2] https://press.princeton.edu/books/hardcover/9780691181226/the-self-assembling-brain https://press.princeton.edu/books/hardcover/9780691181226/th... [3] https://www.amazon.com/Archaeology-Mind-Neuroevolutionary-Interpersonal-Neurobiology/dp/0393705315 https://www.amazon.com/Archaeology-Mind-Neuroevolutionary-In... [*] One of the things that frustrates me the most in the discourse on LLMs is that people who should know better deliberately mislead others into believing that there is something similar to "intelligence" going on with LLMs -- because they are heavily financially incentivized to do so. Comparisons with humans are categorical errors in everything but metaphor. They call them "neural networks" instead of "systems of nonlinear equations", because "neural network" sounds way sexier than vectorized y=f(mx+b).

penjelly 2y ago

LLMs in their current form seem to me like intuitive thought, like what we use when in conversation with friends over a beer. It seems like only one part of a future AI brain, there still needs to be another system for maintaining a world model, another for background planning, etc. These + vision and hearing models to take in new info about the world and we'll have something resembling human intelligence where all that data meets