4 ms·

> the current algorithms are clearly a dead end for thinking machines. These discussions often get derailed into debates about what "thinking" means. If we def

> the current algorithms are clearly a dead end for thinking machines. These discussions often get derailed into debates about what "thinking" means. If we define thinking as the capacity to produce and evaluate arguments, as the cognitive scientists Sperber and Mercier do, then we can see LLMs are certainly producing arguments, but they're weak at the evaluation. In some cases, arguments can be formalised, and then evaluating them is a solved problem, as in the examples of using the Lean proofchecker in combination with LLMs to write mathematical proofs. That suggests a way forward will come from formalising natural language arguments. So LLMs by themselves might be a dead end but in combination with formalisation they could be very powerful. That might not be "thinking" in the sense of the full suite of human abilities that we group with that word but it seems an important component of it.

kjellsbells 9mo ago
> suggests a way forward will come from formalising natural language arguments If by this you mean "reliably convert expressions made in human natural language to unambiguous, formally parseable expressions that a machine can evaluate the same way every time"... isn't that essentially an unreachable holy grail? I mean, everyone from Plato to Russell and Wittgenstein struggled with the meaning of human statements. And the best solution we have today is to ask the human to restrict the set of statement primitives and combinations that they can use to a small subset of words like "const", "let foo = bar", and so on.
- griffzhowl 9mo ago
  Whether the Holy Grail is unreachable or not is the question. Of course, the problem in full generality is hard, but that doesn't mean it can't be approached in various partial ways, either by restricting the inputs as you suggest or by coming up with some kind of evaluation procedures that are less strict than formal verifiability. I don't have any detailed proposals tbh
benrutter 9mo ago
> That suggests a way forward will come from formalising natural language arguments. Hot take (and continue with the derailment), but I'd argue that analytic philosophy from the last 100 years suggests this is a dead end. The idea that belief systems could be formalized was huge in the early 20th century (movements like Logical Positivism, or Russell's principia mathematica being good examples of this). Those approaches haven't really yielded many results, and by far the more fruitful form of analysis has been to conceptually "reframe" different problems (folks like Hillary Putnam, Wittgenstein, Quine being good examples). We've stacked up a lot of evidence that human language is much too loose and mushy to be formalised in a meaningful way.
- master_crab 9mo ago
  We've stacked up a lot of evidence that human language is much too loose and mushy to be formalized in a meaningful way. Lossy might also be a way of putting it, like a bad compression algorithm. Written language carries far less information than spoken and nonverbal cues.
- griffzhowl 9mo ago
  True, maybe full formalisation is too strict and the evaluation should be fuzzier
- sdwr 9mo ago
  Wittgenstein obliterates any hope of formalizing language, yeah. I think modeling language usefully looks a lot more like psychoanalysis than first order logic.
staticman2 9mo ago
I think you may mean Sperber and Mercier define "reasoning" as the capacity to produce and evaluate arguments?
- griffzhowl 9mo ago
  True, they use the word "reasoning". Part of my point was just to focus on the more concrete concept: the capacity to produce and evaluate arguments.
coldtea 9mo ago
>If we define thinking as the capacity to produce and evaluate arguments That bar is so low that even a political pundit on TV can clear it.
AstroBen 9mo ago
Yesterday I got AI (a sota model) to write some tests for a backend I'm working on. One set of tests was for a function that does a somewhat complex SQL query that should return multiple rows In the test setup, the AI added a single database row, ran the query and then asserted the single added row was returned. Clearly this doesn't show that the query works as intended. Is this what people are referring to when they say AI writes their tests? I don't know what to call this kind of thinking. Any intelligent, reasoning human would immediately see that it's not even close to enough. You barely even need a coding background to see the issues. AI just doesn't have it, and it hasn't improved in this area for years This kind of thing happens over and over again. I look at the stuff it outputs and it's clear to me that no reasoning thing would act this way
- elcritch 9mo ago
  As a counter I’ve had OpenAI Codex and Claude Code both catch logic cases I’d missed in both tests and codes. The tooling in the Code tools is key to useable LLM coding. Those tools prompt the models to “reason” whether they’ve caught edge cases or met the logic. Without that external support they’re just fancy autocompletes. In some ways it’s no different than working with some interns. You have to prompt them to “did you consider if your code matched all of the requirements?”. LLMs are different in that they’re sorta lobotomized. They won’t learn from tutoring “did you consider” which needs to essentially be encoded manually still.
  - AstroBen 9mo ago
    > As a counter I’ve had OpenAI Codex and Claude Code both catch logic cases I’d missed in both tests and codes That has other explanations than that it reasoned its way to the correct answers. Maybe it had very similar code in its training data This specific example was with Codex. I didn't mention it because I didn't want it to sound like I think codex is worse than claude code I do realize my prompt wasn't optimal to get the best out of AI here, and I improved it on the second pass, mainly to give it more explicit instruction on what to do My point though is that I feel these situations are heavily indicative of it not having true reasoning and understanding of the goals presented to it Why can it sometimes catch the logic cases you miss, such as in your case, and then utterly fail at something that a simple understanding of the problem and thinking it through would solve? The only explanation I have is that it's not using actual reasoning to solve the problems
hulitu 9mo ago
> These discussions often get derailed into debates about what "thinking" means. "SAL-9000: Will I dream? Dr. Chandra: Of course you will. All intelligent beings dream. Nobody knows why. Perhaps you will dream of HAL... just as I often do." From 2010

kjellsbells 9mo ago

> suggests a way forward will come from formalising natural language arguments If by this you mean "reliably convert expressions made in human natural language to unambiguous, formally parseable expressions that a machine can evaluate the same way every time"... isn't that essentially an unreachable holy grail? I mean, everyone from Plato to Russell and Wittgenstein struggled with the meaning of human statements. And the best solution we have today is to ask the human to restrict the set of statement primitives and combinations that they can use to a small subset of words like "const", "let foo = bar", and so on.

griffzhowl 9mo ago
Whether the Holy Grail is unreachable or not is the question. Of course, the problem in full generality is hard, but that doesn't mean it can't be approached in various partial ways, either by restricting the inputs as you suggest or by coming up with some kind of evaluation procedures that are less strict than formal verifiability. I don't have any detailed proposals tbh

benrutter 9mo ago

> That suggests a way forward will come from formalising natural language arguments. Hot take (and continue with the derailment), but I'd argue that analytic philosophy from the last 100 years suggests this is a dead end. The idea that belief systems could be formalized was huge in the early 20th century (movements like Logical Positivism, or Russell's principia mathematica being good examples of this). Those approaches haven't really yielded many results, and by far the more fruitful form of analysis has been to conceptually "reframe" different problems (folks like Hillary Putnam, Wittgenstein, Quine being good examples). We've stacked up a lot of evidence that human language is much too loose and mushy to be formalised in a meaningful way.

master_crab 9mo ago
We've stacked up a lot of evidence that human language is much too loose and mushy to be formalized in a meaningful way. Lossy might also be a way of putting it, like a bad compression algorithm. Written language carries far less information than spoken and nonverbal cues.
griffzhowl 9mo ago
True, maybe full formalisation is too strict and the evaluation should be fuzzier
sdwr 9mo ago
Wittgenstein obliterates any hope of formalizing language, yeah. I think modeling language usefully looks a lot more like psychoanalysis than first order logic.

staticman2 9mo ago

I think you may mean Sperber and Mercier define "reasoning" as the capacity to produce and evaluate arguments?

griffzhowl 9mo ago
True, they use the word "reasoning". Part of my point was just to focus on the more concrete concept: the capacity to produce and evaluate arguments.

coldtea 9mo ago

>If we define thinking as the capacity to produce and evaluate arguments That bar is so low that even a political pundit on TV can clear it.

AstroBen 9mo ago

Yesterday I got AI (a sota model) to write some tests for a backend I'm working on. One set of tests was for a function that does a somewhat complex SQL query that should return multiple rows In the test setup, the AI added a single database row, ran the query and then asserted the single added row was returned. Clearly this doesn't show that the query works as intended. Is this what people are referring to when they say AI writes their tests? I don't know what to call this kind of thinking. Any intelligent, reasoning human would immediately see that it's not even close to enough. You barely even need a coding background to see the issues. AI just doesn't have it, and it hasn't improved in this area for years This kind of thing happens over and over again. I look at the stuff it outputs and it's clear to me that no reasoning thing would act this way

elcritch 9mo ago
As a counter I’ve had OpenAI Codex and Claude Code both catch logic cases I’d missed in both tests and codes. The tooling in the Code tools is key to useable LLM coding. Those tools prompt the models to “reason” whether they’ve caught edge cases or met the logic. Without that external support they’re just fancy autocompletes. In some ways it’s no different than working with some interns. You have to prompt them to “did you consider if your code matched all of the requirements?”. LLMs are different in that they’re sorta lobotomized. They won’t learn from tutoring “did you consider” which needs to essentially be encoded manually still.
- AstroBen 9mo ago
  > As a counter I’ve had OpenAI Codex and Claude Code both catch logic cases I’d missed in both tests and codes That has other explanations than that it reasoned its way to the correct answers. Maybe it had very similar code in its training data This specific example was with Codex. I didn't mention it because I didn't want it to sound like I think codex is worse than claude code I do realize my prompt wasn't optimal to get the best out of AI here, and I improved it on the second pass, mainly to give it more explicit instruction on what to do My point though is that I feel these situations are heavily indicative of it not having true reasoning and understanding of the goals presented to it Why can it sometimes catch the logic cases you miss, such as in your case, and then utterly fail at something that a simple understanding of the problem and thinking it through would solve? The only explanation I have is that it's not using actual reasoning to solve the problems

hulitu 9mo ago

> These discussions often get derailed into debates about what "thinking" means. "SAL-9000: Will I dream? Dr. Chandra: Of course you will. All intelligent beings dream. Nobody knows why. Perhaps you will dream of HAL... just as I often do." From 2010