4 ms·

> To make this point stick, you would also have to show why it's not an illusion when humans "appear" to have a theory. This idea has already been explored by

• CamperBob2 1y ago
  You're seriously still going to invoke the Chinese Room argument after what we've seen lately? Wow. The computer understands Chinese better than Searle (or anyone else) understood the nature and functionality of language.
  • ryandv 1y ago
    You're seriously going to invoke this braindead reddit-tier of "argumentation," or rather lack thereof, by claiming bewilderment and offering zero substantive points? Wow.
    • CamperBob2 1y ago
      Yes, because the Chinese Room was a weak test the day it was proposed, and it's a heap of smoldering rhetorical wreckage now. It's Searle who failed to offer any substantive points. How do you know you're not arguing with an LLM at the moment? You don't... any more than I do.
      • ryandv 1y ago
        > How do you know you're not arguing with an LLM at the moment? You don't. I wish I was right now. It would probably provide at least the semblance of greater insight into these topics. > the Chinese Room was a weak test the day it was proposed Why?
        • CamperBob2 1y ago
          It would probably provide at least the semblance of greater insight into these topics. That's very safe to say. You should try it. Then ask yourself how a real Chinese Room would have responded. Why? My beef with the argument is that simulating intelligence well enough to get a given job done is indistinguishable from intelligence itself, with respect to the job in question. More specific arguments along the lines of "Humans can do job X but computers cannot" have not held up well lately, but they were never on solid logical ground. Searle set out to construct such a logical ground, but he obviously failed. If you took today's LLMs back to the 1960s when he proposed that argument, either Searle would be laughed out of town, or you would be burned as a witch. Arguments along the lines of "Machines can never do X, only humans can do that" never belonged in the scientific literature in the first place, and I think the Chinese Room falls into that class. I believe that any such argument needs to begin by explaining what's special about human thought. Right now, the only thing you can say about human thought that you can't say about AI is that humans have real-time sensory input and can perform long-term memory consolidation. Those advantages impose real limitations on what current-generation LLM-based technology can do compared to humans, but they sound like temporary ones to me.
          • Jensson 1y ago
            > Arguments along the lines of "Machines can never do X, only humans can do that" That isn't the argument though. > If you took today's LLMs back to the 1960s when he proposed that argument, either Searle would be laughed out of town, or you would be burned as a witch. Do you think humans were different in the 1960s? No they would see the same limitations as people point out today. 1960s was when AI optimism was still very high.
        • nullstyle 1y ago
• TeMPOraL 1y ago
  Wait, isn't the conclusion to take from the "Chinese room" literally the opposite of what you suggest? I.e. it's the most basic, go-to example of a larger system showing capability (here, understanding Chinese) that is not present in any of its constituent parts individually. > Their words aren't "about" anything, they don't represent concepts or ideas or physical phenomena the same way the words and thoughts of a human do. The computer doesn't actually "understand Chinese" the way a human can. That's very much unclear at this point. We don't fully understand how we relate words to concepts and meaning ourselves, but to the extent we do, LLMs are by far the closest implementation of those same ideas in a computer.
  • vacuity 1y ago
    The Chinese room experiment was originally intended by Searle to (IIUC) do as you claim and justify computers as being capable of understanding like humans do. Since then, it has been used both in this pro-computer, "black box" sense and in the anti-computer, "white box" sense. Personally, I think both are relevant, and the issue with LLMs currently is not a theoretical failing but rather that they aren't convincing when viewed as black boxes (e.g. the Turing test fails).
    • MarkusQ 1y ago
      No, it was used to argue that computers could pass the Turing test and still _not_ understand anything. It was a reducto intended to dispute exactly the claim you are ascribing to it, and argue _against_ "computers as being capable of understanding like humans do".
      • vacuity 1y ago
        Thanks. I stand corrected. I guess I should also add that, aside from the black box view, there are pro-computer stances that claim there is mentality and intentionality.
  • ryandv 1y ago
    > the conclusion to take from the "Chinese room" We can hem and haw about whether or not there are others, but the particular conclusion I am drawing from is that computers lack "intentionality" regarding language, and indeed about anything at all. Symbol shunting, pencil pushing, and the mechanics of syntax are insufficient for the production of meaning and understanding. That is, to oversimplify, the broad distinction drawn in Naur's article regarding the "programming as text manipulation" view vis-a-vis "programming as theory building." > That's very much unclear at this point. It's certainly a central point of contention.
• looofooo0 1y ago
  But the LLM interacts with the program and the world through debugger, run-time feedback, linter, fuzzer etc., we can collect all the user feedback, user pattern ... Moreover, it can also get visual feedback. Reason through other programs like physic simulation etc. Use a robot to interact with the device running the code physically. Can use proof verifier like lean, to ensure its logical model of the program is sound. Do some back and forth between the logical model and the actual program through experiments. Maybe not now, but I don't see why the LLM needs to be kept in the Chinese Room.
  • jimbokun 1y ago
    That's true in general but not true of any current LLM, to my knowledge. Different subsets of those inputs and modalities, yes. But no current LLM has access to all of them.
• smithkl42 1y ago
  The Chinese Room argument is a great thought experiment for understanding why the computational model is an inadequate explanation of consciousness and qualia. But it proves nothing about reason, which LLMs have clearly shown needs to be distinguished from consciousness. And theories fall into the category of reason, not of consciousness. Or another way of putting it that you might find more acceptable: maybe a computer will never, internally, know that it has developed a theory - but it sure seems like it will be able to act and talk as if it had, much like a philosophical zombie.
  • dingnuts 1y ago
    > it proves nothing about reason, which LLMs have clearly shown needs to be distinguished from consciousness. Uh, they have? Are you saying they know how to reason? Because if so, why is it that when I give a state of the art model documentation lacking examples for a new library and ask it to write something, it cannot even begin to do that, even if the documentation is in the training data? A model that can reason should be able to understand the documentation and create novel examples. It cannot. This happened to me just the other day. If the model can reason, examples of the language, which it has, and the expository documentation should have been sufficient. Instead, the model repeatedly inserted bullshitted code in the style of the language I wanted, but with library calls and names based on a version of the library for another language. This is evidence of reasoning ability? Claude Sonnet 3.7 and Gemini Pro both exhibited this behavior last week. I think this technology is fundamentally the same as it has been since GPT2
    • smithkl42 1y ago
      Absolutely LLMs can reason. There are limitations on their ability to reason, as you and everyone else has discovered. But they can absolutely reason about both concepts and the physical world in ways that, say, animals can't - even though presumably animals have at least some sort of self-consciousness and LLM's do not.
    • slippybit 1y ago
      > A model that can reason should be able to understand the documentation and create novel examples. It cannot. That's due to limitations imposed for "security". "Here's a new X, do Y with it" can result in holes bigger and more complex than anyone can currently handle "in time". It's not about "abilities" with LLMs for now, but about functions that work within the range of edge cases, sometimes including them, some other times not. You could still guide it to fulfill the task, though. It just cannot be allowed to do it on it's own but since just "forbidding" an LLM to do something is about as effective as doing that to a child with mischievous older brothers, the only ways to actually do it result in "bullshitted" code and "hallucinations". If I understood the problem correctly, that is.
• jimbokun 1y ago
  The flaw of the Chinese Room argument is the need to explain why it does not apply to humans as well. Does a single neuron "understand" Chinese? 10 neurons? 100? 1 million? If no individual neuron or small group of neurons understand Chinese, how can you say any brain made of neurons understands Chinese?
  • ryandv 1y ago
    > The flaw of the Chinese Room argument is the need to explain why it does not apply to humans as well. But it does - the thought experiment continues by supposing that I gave a human those lookup tables and instructions on how to use them, instead of having the computer run the procedure. The human doesn't understand the foreign language either, not in the same way a native speaker does. The point is that no formal procedure or algorithm is sufficient for such a system to have understanding. Even if you memorized all the lookup tables and instructions and executed this procedure entirely in your head, you would still lack understanding. > Does a single neuron "understand" Chinese? 10 neurons? 100? 1 million? This sounds like a sorites paradox [0]. I don't know how to resolve this, other than to observe that our notions of "understanding" and "thought" and "intelligence" are ill-defined and more heuristic approximations than terms with a precise meaning; hence the tendency of the field of computer science to use thought experiments like Turing's imitation game or Searle's Chinese room as proxies for assessing intelligence, in lieu of being able to treat these terms and ideas more rigorously. [0] https://plato.stanford.edu/entries/sorites-paradox/ https://plato.stanford.edu/entries/sorites-paradox/
• im3w1l 1y ago
  You can state the argument formally as A has property B. Property B' implies property C. Hence A has property C. The fallacy is the sleight of hand where two almost but not quite identical properties B and B' are used, in this case two different defitions of theory, only one of which requires some ineffable mind consciousness. It's important not to get caught up in a discussion about whether B or B' is the proper definition, but instead see that it's the inconsistency that is the issue. LLM's build an internal representation that let's them efficiently and mostly successfully manipulate source code. Whether that internal representation is satisfies your criteria for a theory doesn't change that fact. What does matter to the highest degree however is where they succeed and where they fail, and how the representations and computing can improve the success rate and capabilities.
  • ryandv 1y ago
    No, I don't agree with this formalization. It's more that (some) humans have a "theory" of the program (in the same sense used by Ryle and Naur); let's take for granted that if one has a theory, then they have understanding; thus (some) humans have an understanding of the program. It's not equivocating between B and B', but rather observing that B implies B'. Thus, if an LLM lacks understanding (Searle), then they don't have a theory either. > LLM's build an internal representation that let's them efficiently and mostly successfully manipulate source code. Whether that internal representation is satisfies your criteria for a theory doesn't change that fact. The entire point of Naur's paper is that the activity of programming, of software engineering, is not just "manipulating source code." It is, rather, building a theory of the software system (which implies an understanding of it), in a way that an LLM or an AI cannot, as posited by Searle.
    • n4r9 1y ago
      > let's take for granted that if one has a theory, then they have understanding Leaving aside what is actually meant by "theory" and "understanding". Could it not be argued that eventually LLMs will simulate understanding well enough that - for all intents and purposes - they might as well be said to have a theory? The parallel I've got in my head is the travelling salesman problem. Yes, it's NP-Hard, which means we are unlikely to ever get a polynomial-time algorithm to solve it. But that doesn't stop us solving TSP problems near-optimally at an industrial scales. Similarly, although LLMs may not literally have a theory, they could become powerful enough that the edge cases in which a theory is really needed are infinitesimally unlikely.