3 ms·

The paper includes several examples of their modified questions. There has been a substantial jump from o1-preview to o1, so I gave several samples to o1 and o1

by e1g 2y ago

The paper includes several examples of their modified questions. There has been a substantial jump from o1-preview to o1, so I gave several samples to o1 and o1-pro (not o1-preview), and current o1s gave the correct answer to those modified problems. SOTA changes fast.

suddenlybananas 2y ago
LLM boosters are so tiresome. You hardly did a rigorous evaluation, the set has been public since October and could have easily been added to the training data.
- gdhkgdhkvff 2y ago
  Your points would be more convincing if you didn’t preface them with arrogant cynicism.
- deleted 2y ago
  [deleted]
- e1g 2y ago
  I'm not skilled enough in math to do a rigorous evaluation, so it was a quick check. Terence Tao is skilled enough, and he describes O1's math ability is "...roughly on par with a mediocre, but not completely incompetent graduate student" (good discussion at https://news.ycombinator.com/item?id=41540902 https://news.ycombinator.com/item?id=41540902), and the next iteration O3 just got 25% on his brand new Frontier Math test. Seeing LLMs as useless is banal, but downplaying their rate of improvement is self-sabotage.
  - fumeux_fume 2y ago
    > "...roughly on par with a mediocre, but not completely incompetent graduate student" Let it sink in how vague and almost meaningless that statement is.
    - pizza 2y ago
      What types of questions are you hoping to answer for that to be considered a vague statement?
jtefera 2y ago
The paper mentions that on several occasions the LLM will provide a correct answer but will either take big jumps without justifying them or will take illogical steps but end up with the right solution at the end. Did you check for that?
- e1g 2y ago
  No, I don't know enough math to test the logic, only the check questions against their expected answers in https://anonymous.4open.science/r/putnam-axiom-B57C/data/Putnam-AXIOM_Variations_v3/test.json https://anonymous.4open.science/r/putnam-axiom-B57C/data/Put...
  - zeroonetwothree 2y ago
    Putnam problems need to actually be graded, often the answer itself is trivial.

by e1g 2y ago

suddenlybananas 2y ago

LLM boosters are so tiresome. You hardly did a rigorous evaluation, the set has been public since October and could have easily been added to the training data.

gdhkgdhkvff 2y ago
Your points would be more convincing if you didn’t preface them with arrogant cynicism.
deleted 2y ago
[deleted]
e1g 2y ago
I'm not skilled enough in math to do a rigorous evaluation, so it was a quick check. Terence Tao is skilled enough, and he describes O1's math ability is "...roughly on par with a mediocre, but not completely incompetent graduate student" (good discussion at https://news.ycombinator.com/item?id=41540902 https://news.ycombinator.com/item?id=41540902), and the next iteration O3 just got 25% on his brand new Frontier Math test. Seeing LLMs as useless is banal, but downplaying their rate of improvement is self-sabotage.
- fumeux_fume 2y ago
  > "...roughly on par with a mediocre, but not completely incompetent graduate student" Let it sink in how vague and almost meaningless that statement is.
  - pizza 2y ago
    What types of questions are you hoping to answer for that to be considered a vague statement?

jtefera 2y ago

The paper mentions that on several occasions the LLM will provide a correct answer but will either take big jumps without justifying them or will take illogical steps but end up with the right solution at the end. Did you check for that?

e1g 2y ago
No, I don't know enough math to test the logic, only the check questions against their expected answers in https://anonymous.4open.science/r/putnam-axiom-B57C/data/Putnam-AXIOM_Variations_v3/test.json https://anonymous.4open.science/r/putnam-axiom-B57C/data/Put...
- zeroonetwothree 2y ago
  Putnam problems need to actually be graded, often the answer itself is trivial.