4 ms·

>And in any case, I think OpenAI’s o1 models are crushing it in math right now. My man, it cannot solve even the simplest problems which it hasn't seen the sol

>And in any case, I think OpenAI’s o1 models are crushing it in math right now. My man, it cannot solve even the simplest problems which it hasn't seen the solution to yet, and routinely makes elementary errors in simple algebraic manipulations or arithmetic! All of this points to the fact that it cannot actually perform mathematical or logical reason, only mimic it superficially if trained in enough examples. I challenge you to give it even a simple, but original, problem to solve.

ukuina 2y ago
Do you have some categories of such original problems? It seems markedly better at reasoning/logic puzzles, and programmatically-solvable problems are often offloaded to the Python interpreter.
Workaccount2 2y ago
>I challenge you to give it even a simple, but original, problem to solve. (34903173/x)+(238 * 2650) - 323326 = 45323434, solve for x Statistically, no one has ever done this calculation ever before. It's entirely unique. O1 answered "x = 34,903,173 divided by 45,016,060", which is correct.[1][2] Now I guess you can pick up the goal post and move it. [1]https://chatgpt.com/share/6709481a-3144-8004-a7fd-0ccd9e3bc58d https://chatgpt.com/share/6709481a-3144-8004-a7fd-0ccd9e3bc5... [2]https://www.wolframalpha.com/input?i=%2834903173%2Fx%29%2B%28238+*+2650%29+-++323326+%3D+45323434 https://www.wolframalpha.com/input?i=%2834903173%2Fx%29%2B%2...
- bob1029 2y ago
  > Now I guess you can pick up the goal post and move it. The central problem with math is that you have an infinite amount of space within which to move these goalposts. How many variants on this trial before we find a mistake? What is an acceptable error rate?
  - jimhefferon 2y ago
    At this moment, the error rate seems to be that of a beginning graduate student. Or at least, that's what Terry Tao thinks. That's pretty good.
    - lupire 2y ago
      That is not at all what Tao said. https://mathstodon.xyz/@tao/113132502735585408 https://mathstodon.xyz/@tao/113132502735585408 "Here the results were better than previous models, but still slightly disappointing: the new model could work its way to a correct (and well-written) solution if provided a lot of hints and prodding, but did not generate the key conceptual ideas on its own, and did make some non-trivial mistakes. The experience seemed roughly on par with trying to advise a mediocre, but not completely incompetent, (static simulation of a) graduate student. "
      - buneskamin 2y ago
        Yea it does seem pretty clear that it's mainly Terrance's contributions to the context window that is bringing the model to the right answer
  - naasking 2y ago
    > How many variants on this trial before we find a mistake? How many variants would it take for a human to make a mistake? It's certainly not "infinity", so is this an indication that humans don't reason?
- andrepd 2y ago
  My brother in christ, how is A/B + C*D - E = F, solve for B an original problem? How many tens of thousands of examples of this exact form do you think it came across? It's the same as with coding by the way: it can reshuffle things it has already seen while changing variable names and so on. Ask it something which is not in stackoverflow or geeks4geeks and it goes tits up. PS: Tested it on GPT 3.5: same answer.
WhitneyLand 2y ago
Please provide your precise definitions of “reasoning” and “original”. There’s no consensus in the literature on what these mean even if you make it more specific by talking about “mathematical reasoning”, so I don’t really understand what opinions like these are based on. I see a lot of no true Scottsman fallacy going around, even the paper resorts to this as it actually uses phrases like “true reasoning” several times. I don’t think the paper is very convincing btw, the abstract is kind of click-baity and talks about 65% variation when that was a cherry picked example from a tiny phi model and the SOTA models showed way less variation which was arguably not that interesting.
- YeGoblynQueenne 2y ago
  >> There’s no consensus in the literature on what these mean even if you make it more specific by talking about “mathematical reasoning”, so I don’t really understand what opinions like these are based on. What literature is that? You can find plenty of very clear consensus on what reasoning is if you read e.g. the literature on automated reasoning. A brief taste: Automated Reasoning Reasoning is the ability to make inferences, and automated reasoning is concerned with the building of computing systems that automate this process. Although the overall goal is to mechanize different forms of reasoning, the term has largely been identified with valid deductive reasoning as practiced in mathematics and formal logic. In this respect, automated reasoning is akin to mechanical theorem proving. Building an automated reasoning program means providing an algorithmic description to a formal calculus so that it can be implemented on a computer to prove theorems of the calculus in an efficient manner. Important aspects of this exercise involve defining the class of problems the program will be required to solve, deciding what language will be used by the program to represent the information given to it as well as new information inferred by the program, specifying the mechanism that the program will use to conduct deductive inferences, and figuring out how to perform all these computations efficiently. While basic research work continues in order to provide the necessary theoretical framework, the field has reached a point where automated reasoning programs are being used by researchers to attack open questions in mathematics and logic, provide important applications in computing science, solve problems in engineering, and find novel approaches to questions in exact philosophy. https://plato.stanford.edu/entries/reasoning-automated/ https://plato.stanford.edu/entries/reasoning-automated/ After that you may want to look at the SEP articles on Analogical reasoning and Defeasible Reasoning: https://plato.stanford.edu/entries/reasoning-analogy/ https://plato.stanford.edu/entries/reasoning-analogy/ https://seop.illc.uva.nl/entries/reasoning-defeasible/ https://seop.illc.uva.nl/entries/reasoning-defeasible/

ukuina 2y ago

Do you have some categories of such original problems? It seems markedly better at reasoning/logic puzzles, and programmatically-solvable problems are often offloaded to the Python interpreter.

Workaccount2 2y ago

>I challenge you to give it even a simple, but original, problem to solve. (34903173/x)+(238 * 2650) - 323326 = 45323434, solve for x Statistically, no one has ever done this calculation ever before. It's entirely unique. O1 answered "x = 34,903,173 divided by 45,016,060", which is correct.[1][2] Now I guess you can pick up the goal post and move it. [1]https://chatgpt.com/share/6709481a-3144-8004-a7fd-0ccd9e3bc58d https://chatgpt.com/share/6709481a-3144-8004-a7fd-0ccd9e3bc5... [2]https://www.wolframalpha.com/input?i=%2834903173%2Fx%29%2B%28238+*+2650%29+-++323326+%3D+45323434 https://www.wolframalpha.com/input?i=%2834903173%2Fx%29%2B%2...

bob1029 2y ago
> Now I guess you can pick up the goal post and move it. The central problem with math is that you have an infinite amount of space within which to move these goalposts. How many variants on this trial before we find a mistake? What is an acceptable error rate?
- jimhefferon 2y ago
  At this moment, the error rate seems to be that of a beginning graduate student. Or at least, that's what Terry Tao thinks. That's pretty good.
  - lupire 2y ago
    That is not at all what Tao said. https://mathstodon.xyz/@tao/113132502735585408 https://mathstodon.xyz/@tao/113132502735585408 "Here the results were better than previous models, but still slightly disappointing: the new model could work its way to a correct (and well-written) solution if provided a lot of hints and prodding, but did not generate the key conceptual ideas on its own, and did make some non-trivial mistakes. The experience seemed roughly on par with trying to advise a mediocre, but not completely incompetent, (static simulation of a) graduate student. "
    - buneskamin 2y ago
      Yea it does seem pretty clear that it's mainly Terrance's contributions to the context window that is bringing the model to the right answer
- naasking 2y ago
  > How many variants on this trial before we find a mistake? How many variants would it take for a human to make a mistake? It's certainly not "infinity", so is this an indication that humans don't reason?
andrepd 2y ago
My brother in christ, how is A/B + C*D - E = F, solve for B an original problem? How many tens of thousands of examples of this exact form do you think it came across? It's the same as with coding by the way: it can reshuffle things it has already seen while changing variable names and so on. Ask it something which is not in stackoverflow or geeks4geeks and it goes tits up. PS: Tested it on GPT 3.5: same answer.

WhitneyLand 2y ago

Please provide your precise definitions of “reasoning” and “original”. There’s no consensus in the literature on what these mean even if you make it more specific by talking about “mathematical reasoning”, so I don’t really understand what opinions like these are based on. I see a lot of no true Scottsman fallacy going around, even the paper resorts to this as it actually uses phrases like “true reasoning” several times. I don’t think the paper is very convincing btw, the abstract is kind of click-baity and talks about 65% variation when that was a cherry picked example from a tiny phi model and the SOTA models showed way less variation which was arguably not that interesting.

YeGoblynQueenne 2y ago
>> There’s no consensus in the literature on what these mean even if you make it more specific by talking about “mathematical reasoning”, so I don’t really understand what opinions like these are based on. What literature is that? You can find plenty of very clear consensus on what reasoning is if you read e.g. the literature on automated reasoning. A brief taste: Automated Reasoning Reasoning is the ability to make inferences, and automated reasoning is concerned with the building of computing systems that automate this process. Although the overall goal is to mechanize different forms of reasoning, the term has largely been identified with valid deductive reasoning as practiced in mathematics and formal logic. In this respect, automated reasoning is akin to mechanical theorem proving. Building an automated reasoning program means providing an algorithmic description to a formal calculus so that it can be implemented on a computer to prove theorems of the calculus in an efficient manner. Important aspects of this exercise involve defining the class of problems the program will be required to solve, deciding what language will be used by the program to represent the information given to it as well as new information inferred by the program, specifying the mechanism that the program will use to conduct deductive inferences, and figuring out how to perform all these computations efficiently. While basic research work continues in order to provide the necessary theoretical framework, the field has reached a point where automated reasoning programs are being used by researchers to attack open questions in mathematics and logic, provide important applications in computing science, solve problems in engineering, and find novel approaches to questions in exact philosophy. https://plato.stanford.edu/entries/reasoning-automated/ https://plato.stanford.edu/entries/reasoning-automated/ After that you may want to look at the SEP articles on Analogical reasoning and Defeasible Reasoning: https://plato.stanford.edu/entries/reasoning-analogy/ https://plato.stanford.edu/entries/reasoning-analogy/ https://seop.illc.uva.nl/entries/reasoning-defeasible/ https://seop.illc.uva.nl/entries/reasoning-defeasible/