We know that formal logic can be based on a small number of fundamental rules. Same with computation (lamda calculus, Turing machines, ...), mathematics (axioms

We know that formal logic can be based on a small number of fundamental rules. Same with computation (lamda calculus, Turing machines, ...), mathematics (axioms), etc. We know that theorems can be proven using a combination of search, rule application and heuristics. E.g. back in 1950s, Logic Theorist (https://en.wikipedia.org/wiki/Logic_Theorist https://en.wikipedia.org/wiki/Logic_Theorist) proved 38 of the first 52 theorems in chapter two of Whitehead and Russell's Principia Mathematica, and found new and shorter proofs for some of them. We know that language models are good at transforming text, e.g. they can convert a sentence in English to Python code. We know that language models have only a fixed computing budget per token, i.e. they cannot stop to think. We know that logic puzzles and proofs might require a considerable amount of computations, e.g. to search through a tree of possibilities, backtrack and so on. If we believe that reasoning is kinda like logic, we'll be better off using LLM to translate reasoning tasks into computing task to be solved by specialized computing tools (such as Python interpreter or theorem prover or SAT solver) instead of asking LLM to reason directly. Of course, GPT-4 is trained to be over-confident in its reasoning capability, and it will try to reply immediately, essentially just guessing the answer, and quite often it would fail. But the question "Can GPT-4 reason with the default assistant prompt?" is different from "Can GPT-4 reason?". Even without external tool, we can ask GPT-4 to translate the problem into primitive fragments in excruciating detail, and to consider all possibilities, and it might work much better than the default prompt. Given that GPT-4 is essentially just weights, I'd consider "Can GPT-4 reason?" question be more like "Is there a prompt X such that being prepended to reasoning tasks it produces correct answers?", not "If I enter my question into a box, does it give the right answer?". So this paper's author does a bit of a category mistake, it's more like "Can ChatGPT (the product) reason?".