5 ms·

Gen AI doesn't have a coherent understanding of the world, researchers suggest

gnabgib 2y ago
Related: Despite its impressive output, generative AI doesn’t have a coherent understanding of the world https://news.ycombinator.com/item?id=42049482 https://news.ycombinator.com/item?id=42049482
davydm 2y ago
Gen ai doesn't "understand" _anything_. It simply spits out the next "likely" occurrence of something based on preceding patterns. I'm not sure why people keep thinking that it "understands" anything.
- Arnt 2y ago
  Because it isn't a Markov chain. Gaining a good understanding of what it is and of its limits makes sense. Too many people think of the recent AI advances as a sort of Markov chain with lipstick. That's deeply wrong.
- indigoabstract 2y ago
  It's pretty easy to make that mistake. Tbh, can you really say you understand something until you've built it yourself? I think LLMs are still pretty much "magic" black boxes for most people (programmers included).
- Ukv 2y ago
  > It simply spits out the next "likely" occurrence I'd claim that the "likely" continuation is a bit of a nebulous concept, or at least not always trivial to work out. Completing sequences like "Q: Solve `7x - 4 = 2x` for x. \nA: " may require some internal model of basic algebraic rules, for instance. Where present, I don't think it's unreasonable to call that an understanding.
  - jqpabc123 2y ago
    Q: Count the letters in 'Hippopotamus'. A: 9 letters Wrong! The "Large Language Model" obviously lacks a coherent understanding of language. "Language" is supposedly it's bread and butter but the "language" it produces is "correct" only from a statistical perspective. In other words, it simply can't be trusted --- because it lacks any coherent understanding of what you or it are saying/asking. LLMs fundamentally turn computing on it's head. Instead of accurate results at low cost, they produce inaccurate results at high cost.
    - Ukv 2y ago
      > Wrong! The "Large Language Model" obviously lacks a coherent understanding of language. It's not all-or-nothing for the entirety of language, but yeah it's known that LLMs are imprecise at mapping from tokens to characters (as a human who only saw tokens may be, to be fair). For an unseen question, even just to follow the Q/A format and give a right-category answer already demonstrates some model of language, I'd argue. > "Language" is supposedly it's bread and butter but the "language" it produces is "correct" only from a statistical perspective. In other words, it simply can't be trusted. Nothing can be trusted 100%, and current LLMs are far below humans, but both the structure and the meaning of their language is correct significantly above a random-guessing baseline. A pretrained foundation model will have representations of fiction and non-fiction content, and can be guided towards either one with RL or a system prompt. > LLMs fundamentally turn computing on it's head. Instead of accurate results at low cost, they produce inaccurate results at high cost. Compared to what? LLMs often do things that traditional algorithms couldn't, faster and cheaper than hiring a human.
      - jqpabc123 2y ago
        ... demonstrates some model of language, I'd argue. Yes, it's model is all based on statistics. It lacks any real "coherent understanding". It produces nice prose --- that simply can't be trusted for even rudimentary results.
        Ukv 2y ago
- krapp 2y ago
  >I'm not sure why people keep thinking that it "understands" anything. Because we humans are hard wired to interpret language as a signal of intelligence and sapience, of a "thing like us." It's extremely counterintuitive to most people to be able to hold a coherent conversation with an entity that has no actual understanding of anything it seems to be talking about. Because there are billions of dollars being poured into advertising LLMs as essentially intelligent agents that can and do understand things, often better and more reliably than human beings. I've seen numerous people refer to them having access to "the sum of all human knowledge," implying that not only are they intelligent, they should be considered nigh omniscient. There's a lot of money at state in establishing cultural (and then legal) precedent for LLMs being intelligent agents equivalent to human beings, where issues like fair use and copyright are concerned. Because people want to believe. For various reasons, many to do with decades of pop culture and sci fi informing people's idea of what "AI" is, there's a transhumanist faith in the inevitability of AGI and singularity within tech that has just been waiting for some emergent phenomenon to fill that void, and LLMs seem good enough. I think there's a similar phenomenon at work with people's enthusiasm for Elon Musk as well, these things appear to point towards the kind of future that nerds read about in books.

Related: Despite its impressive output, generative AI doesn’t have a coherent understanding of the world https://news.ycombinator.com/item?id=42049482 https://news.ycombinator.com/item?id=42049482

davydm 2y ago

Gen ai doesn't "understand" _anything_. It simply spits out the next "likely" occurrence of something based on preceding patterns. I'm not sure why people keep thinking that it "understands" anything.

Arnt 2y ago
Because it isn't a Markov chain. Gaining a good understanding of what it is and of its limits makes sense. Too many people think of the recent AI advances as a sort of Markov chain with lipstick. That's deeply wrong.
indigoabstract 2y ago
It's pretty easy to make that mistake. Tbh, can you really say you understand something until you've built it yourself? I think LLMs are still pretty much "magic" black boxes for most people (programmers included).
Ukv 2y ago
> It simply spits out the next "likely" occurrence I'd claim that the "likely" continuation is a bit of a nebulous concept, or at least not always trivial to work out. Completing sequences like "Q: Solve `7x - 4 = 2x` for x. \nA: " may require some internal model of basic algebraic rules, for instance. Where present, I don't think it's unreasonable to call that an understanding.
- jqpabc123 2y ago
  Q: Count the letters in 'Hippopotamus'. A: 9 letters Wrong! The "Large Language Model" obviously lacks a coherent understanding of language. "Language" is supposedly it's bread and butter but the "language" it produces is "correct" only from a statistical perspective. In other words, it simply can't be trusted --- because it lacks any coherent understanding of what you or it are saying/asking. LLMs fundamentally turn computing on it's head. Instead of accurate results at low cost, they produce inaccurate results at high cost.
  - Ukv 2y ago
    > Wrong! The "Large Language Model" obviously lacks a coherent understanding of language. It's not all-or-nothing for the entirety of language, but yeah it's known that LLMs are imprecise at mapping from tokens to characters (as a human who only saw tokens may be, to be fair). For an unseen question, even just to follow the Q/A format and give a right-category answer already demonstrates some model of language, I'd argue. > "Language" is supposedly it's bread and butter but the "language" it produces is "correct" only from a statistical perspective. In other words, it simply can't be trusted. Nothing can be trusted 100%, and current LLMs are far below humans, but both the structure and the meaning of their language is correct significantly above a random-guessing baseline. A pretrained foundation model will have representations of fiction and non-fiction content, and can be guided towards either one with RL or a system prompt. > LLMs fundamentally turn computing on it's head. Instead of accurate results at low cost, they produce inaccurate results at high cost. Compared to what? LLMs often do things that traditional algorithms couldn't, faster and cheaper than hiring a human.
    - jqpabc123 2y ago
      ... demonstrates some model of language, I'd argue. Yes, it's model is all based on statistics. It lacks any real "coherent understanding". It produces nice prose --- that simply can't be trusted for even rudimentary results.
      - Ukv 2y ago
krapp 2y ago
>I'm not sure why people keep thinking that it "understands" anything. Because we humans are hard wired to interpret language as a signal of intelligence and sapience, of a "thing like us." It's extremely counterintuitive to most people to be able to hold a coherent conversation with an entity that has no actual understanding of anything it seems to be talking about. Because there are billions of dollars being poured into advertising LLMs as essentially intelligent agents that can and do understand things, often better and more reliably than human beings. I've seen numerous people refer to them having access to "the sum of all human knowledge," implying that not only are they intelligent, they should be considered nigh omniscient. There's a lot of money at state in establishing cultural (and then legal) precedent for LLMs being intelligent agents equivalent to human beings, where issues like fair use and copyright are concerned. Because people want to believe. For various reasons, many to do with decades of pop culture and sci fi informing people's idea of what "AI" is, there's a transhumanist faith in the inevitability of AGI and singularity within tech that has just been waiting for some emergent phenomenon to fill that void, and LLMs seem good enough. I think there's a similar phenomenon at work with people's enthusiasm for Elon Musk as well, these things appear to point towards the kind of future that nerds read about in books.