5 ms·

It’s a crappy thought experiment free from the constraints of any reality, and given that these fancy lookup tables understand most languages better than I do,

It’s a crappy thought experiment free from the constraints of any reality, and given that these fancy lookup tables understand most languages better than I do, it doesnt hold water. Thought experiments arent science.

ryandv 1y ago
> these fancy lookup tables understand most languages better than I do I see. So if I gave you a full set of those lookup tables, a whole library full, and a set of instructions for their usage... you would now understand the world's languages?
Jensson 1y ago
If the Chinese rooms tells you "I just left the train, see you in 5 minutes", what do you think the Chinese room try to convey? Do you think it knows what it just said? LLMs say such things all the time if you don't RLHF them to stop, why do you think they wouldn't be just as clueless about other things?
- CamperBob2 1y ago
  If you ask an LLM to do some math, what happens is interesting. Simple arithmetic ("What is 2+2") is obviously going to be well-represented in the training data, so the model will simply regurgitate "4."' For more advanced questions like "What are the roots of 14.338x^5 + 4.005x^4 + 3.332x^3 - 99.7x^2 + 120x = 0?", the model will either yield random nonsense as GPT-4o did, or write a Python script and execute it to return the correct answer(s) as o4-mini-high did: https://chatgpt.com/share/680fb812-76b8-800b-a19e-7469cbcc4350 https://chatgpt.com/share/680fb812-76b8-800b-a19e-7469cbcc43... Now, give the model an intermediate arithmetic problem, one that isn't especially hard but also isn't going to be in-distribution ("If a is 3 and b is 11.4, what is the fourth root of a*b?"). How would YOU expect the operator of a Chinese Room to respond to that? Here's how GPT-4o responded: https://chatgpt.com/share/680fb616-45e0-800b-b592-789f3f8c58ca https://chatgpt.com/share/680fb616-45e0-800b-b592-789f3f8c58... Now, that's not a great answer, it's clearly an imprecise estimate. But it's more or less right, and the fact that it isn't a perfect answer suggests that the model didn't cheat somehow. A similar but easier problem would almost certainly have been answered correctly. Where did that answer come from, if the model doesn't "understand" the math to a nontrivial extent? If it can "understand" basic high-school math, what else can it "understand?" What exactly are the limits of what a transformer can "understand" without resorting to web search or tool use? An adherent of Searle's argument is going to have a terrible time explaining phenomena like this... and it's only going to get worse for them over time.
  - Jensson 1y ago
    > If it can "understand" basic high-school math, what else can it "understand?" What exactly are the limits of what a transformer can "understand" without resorting to web search or tool use? It is basically a grammar machine, it mostly understands stuff that can be encoded as a grammar. That is extremely inefficient for math but it can do it, that gives you a really simple way to figure out what it can do and can't do. Knowing this LLM never really surprised me, you can encode a ton of stuff as grammars, but that is still never going to be enough given how inefficient grammars are at lots of things. But when you have a grammar the size of many billions of bytes then you can do quite a lot with it.
    - CamperBob2 1y ago
      Let's stick with the Chinese Room specifically for a moment. 1) The operator doesn't know math, but the Chinese books in the room presumably include math lessons. 2) The operator's instruction manual does not include anything about math, only instructions for translation using English and Chinese vocabulary and grammar. 3) Someone walks up and hands the operator the word problem in question, written in Chinese. Does the operator succeed in returning the Chinese characters corresponding to the equation's roots? Remember, he doesn't even know he's working on a math problem, much less how to solve it himself. As humans, you and I were capable of reading high-school math textbooks by the time we reached the third or fourth grade. Just being able to read the books, though, would not have taught us how to attack math problems that were well beyond our skill level at the time. So much for grammar. How can a math problem be solved by someone who not only doesn't understand math, but the language the question is written in? Searle's proposal only addresses the latter: language can indeed be translated symbolically. Wow, yeah, thanks for that insight. Meanwhile, to arrive at the right answers, an understanding of the math must exist somewhere... but where? My position is that no, the operator of the Room could not have arrived at the answer to the question that the LLM succeeded (more or less) at solving.
      - Jensson 1y ago
        > Meanwhile, to arrive at the right answers, an understanding of the math must exist somewhere... but where? In the grammar, you can have grammar rules like "1 + 1 = " must be followed by 2 etc. Then add a lot of dependency rules like "He did X" the He depends on some previous sentence to stuff like that, in same way "1 plus 1" translates to "1 + 1" or "add 1 to 1" is also "1 + 1", and now you have a machine that can do very complex things. Then you take such a grammar machine and train it on all text human has ever written, and it learns a lot of such grammar structures, and can thus parse and solve some basic math problems since the solution to them is a part of the grammar it learned. Such a machine is still unable to solve anything outside of the grammar it has learned. But it is still very useful, pose a question in a way that makes it easy to parse, and that has a lot of such grammar dependencies you know it can handle, and it will almost always output the right response.
psychoslave 1y ago
>Thought experiments arent science. By that standard we should have drop many of the cutting edge theory that was ever produced in science. It took like a century between some of Einstein’s thought experiments and any possibility to challenge them experimentally. And while Lucretius’ idea of atom was very different than the one we kept with standard model, it actually has put the concept on the table several thousand years before they could be falsified experimentally. It looks like you should seriously consider to expand your epistemological knowledge if you want to contribute more relevantly on the topic. https://bigthink.com/surprising-science/einstein-is-right-again-his-elevator-thought-experiment-is-proven-correct/ https://bigthink.com/surprising-science/einstein-is-right-ag...
emorning3 1y ago
>>Thought experiments arent science.<< Thought experiments provide conclusions based on deductive or inductive reasoning from their starting assumptions. Thought experiments are proofs. That's science.
- nullstyle 1y ago
  [dead]

ryandv 1y ago

> these fancy lookup tables understand most languages better than I do I see. So if I gave you a full set of those lookup tables, a whole library full, and a set of instructions for their usage... you would now understand the world's languages?

Jensson 1y ago

If the Chinese rooms tells you "I just left the train, see you in 5 minutes", what do you think the Chinese room try to convey? Do you think it knows what it just said? LLMs say such things all the time if you don't RLHF them to stop, why do you think they wouldn't be just as clueless about other things?

CamperBob2 1y ago
If you ask an LLM to do some math, what happens is interesting. Simple arithmetic ("What is 2+2") is obviously going to be well-represented in the training data, so the model will simply regurgitate "4."' For more advanced questions like "What are the roots of 14.338x^5 + 4.005x^4 + 3.332x^3 - 99.7x^2 + 120x = 0?", the model will either yield random nonsense as GPT-4o did, or write a Python script and execute it to return the correct answer(s) as o4-mini-high did: https://chatgpt.com/share/680fb812-76b8-800b-a19e-7469cbcc4350 https://chatgpt.com/share/680fb812-76b8-800b-a19e-7469cbcc43... Now, give the model an intermediate arithmetic problem, one that isn't especially hard but also isn't going to be in-distribution ("If a is 3 and b is 11.4, what is the fourth root of a*b?"). How would YOU expect the operator of a Chinese Room to respond to that? Here's how GPT-4o responded: https://chatgpt.com/share/680fb616-45e0-800b-b592-789f3f8c58ca https://chatgpt.com/share/680fb616-45e0-800b-b592-789f3f8c58... Now, that's not a great answer, it's clearly an imprecise estimate. But it's more or less right, and the fact that it isn't a perfect answer suggests that the model didn't cheat somehow. A similar but easier problem would almost certainly have been answered correctly. Where did that answer come from, if the model doesn't "understand" the math to a nontrivial extent? If it can "understand" basic high-school math, what else can it "understand?" What exactly are the limits of what a transformer can "understand" without resorting to web search or tool use? An adherent of Searle's argument is going to have a terrible time explaining phenomena like this... and it's only going to get worse for them over time.
- Jensson 1y ago
  > If it can "understand" basic high-school math, what else can it "understand?" What exactly are the limits of what a transformer can "understand" without resorting to web search or tool use? It is basically a grammar machine, it mostly understands stuff that can be encoded as a grammar. That is extremely inefficient for math but it can do it, that gives you a really simple way to figure out what it can do and can't do. Knowing this LLM never really surprised me, you can encode a ton of stuff as grammars, but that is still never going to be enough given how inefficient grammars are at lots of things. But when you have a grammar the size of many billions of bytes then you can do quite a lot with it.
  - CamperBob2 1y ago
    Let's stick with the Chinese Room specifically for a moment. 1) The operator doesn't know math, but the Chinese books in the room presumably include math lessons. 2) The operator's instruction manual does not include anything about math, only instructions for translation using English and Chinese vocabulary and grammar. 3) Someone walks up and hands the operator the word problem in question, written in Chinese. Does the operator succeed in returning the Chinese characters corresponding to the equation's roots? Remember, he doesn't even know he's working on a math problem, much less how to solve it himself. As humans, you and I were capable of reading high-school math textbooks by the time we reached the third or fourth grade. Just being able to read the books, though, would not have taught us how to attack math problems that were well beyond our skill level at the time. So much for grammar. How can a math problem be solved by someone who not only doesn't understand math, but the language the question is written in? Searle's proposal only addresses the latter: language can indeed be translated symbolically. Wow, yeah, thanks for that insight. Meanwhile, to arrive at the right answers, an understanding of the math must exist somewhere... but where? My position is that no, the operator of the Room could not have arrived at the answer to the question that the LLM succeeded (more or less) at solving.
    - Jensson 1y ago
      > Meanwhile, to arrive at the right answers, an understanding of the math must exist somewhere... but where? In the grammar, you can have grammar rules like "1 + 1 = " must be followed by 2 etc. Then add a lot of dependency rules like "He did X" the He depends on some previous sentence to stuff like that, in same way "1 plus 1" translates to "1 + 1" or "add 1 to 1" is also "1 + 1", and now you have a machine that can do very complex things. Then you take such a grammar machine and train it on all text human has ever written, and it learns a lot of such grammar structures, and can thus parse and solve some basic math problems since the solution to them is a part of the grammar it learned. Such a machine is still unable to solve anything outside of the grammar it has learned. But it is still very useful, pose a question in a way that makes it easy to parse, and that has a lot of such grammar dependencies you know it can handle, and it will almost always output the right response.

psychoslave 1y ago

>Thought experiments arent science. By that standard we should have drop many of the cutting edge theory that was ever produced in science. It took like a century between some of Einstein’s thought experiments and any possibility to challenge them experimentally. And while Lucretius’ idea of atom was very different than the one we kept with standard model, it actually has put the concept on the table several thousand years before they could be falsified experimentally. It looks like you should seriously consider to expand your epistemological knowledge if you want to contribute more relevantly on the topic. https://bigthink.com/surprising-science/einstein-is-right-again-his-elevator-thought-experiment-is-proven-correct/ https://bigthink.com/surprising-science/einstein-is-right-ag...

emorning3 1y ago

>>Thought experiments arent science.<< Thought experiments provide conclusions based on deductive or inductive reasoning from their starting assumptions. Thought experiments are proofs. That's science.

nullstyle 1y ago
[dead]