3 ms·

>I challenge you to give it even a simple, but original, problem to solve. (34903173/x)+(238 * 2650) - 323326 = 45323434, solve for x Statistically, no one h

>I challenge you to give it even a simple, but original, problem to solve. (34903173/x)+(238 * 2650) - 323326 = 45323434, solve for x Statistically, no one has ever done this calculation ever before. It's entirely unique. O1 answered "x = 34,903,173 divided by 45,016,060", which is correct.[1][2] Now I guess you can pick up the goal post and move it. [1]https://chatgpt.com/share/6709481a-3144-8004-a7fd-0ccd9e3bc58d https://chatgpt.com/share/6709481a-3144-8004-a7fd-0ccd9e3bc5... [2]https://www.wolframalpha.com/input?i=%2834903173%2Fx%29%2B%28238+*+2650%29+-++323326+%3D+45323434 https://www.wolframalpha.com/input?i=%2834903173%2Fx%29%2B%2...

bob1029 2y ago
> Now I guess you can pick up the goal post and move it. The central problem with math is that you have an infinite amount of space within which to move these goalposts. How many variants on this trial before we find a mistake? What is an acceptable error rate?
- jimhefferon 2y ago
  At this moment, the error rate seems to be that of a beginning graduate student. Or at least, that's what Terry Tao thinks. That's pretty good.
  - lupire 2y ago
    That is not at all what Tao said. https://mathstodon.xyz/@tao/113132502735585408 https://mathstodon.xyz/@tao/113132502735585408 "Here the results were better than previous models, but still slightly disappointing: the new model could work its way to a correct (and well-written) solution if provided a lot of hints and prodding, but did not generate the key conceptual ideas on its own, and did make some non-trivial mistakes. The experience seemed roughly on par with trying to advise a mediocre, but not completely incompetent, (static simulation of a) graduate student. "
    - buneskamin 2y ago
      Yea it does seem pretty clear that it's mainly Terrance's contributions to the context window that is bringing the model to the right answer
- naasking 2y ago
  > How many variants on this trial before we find a mistake? How many variants would it take for a human to make a mistake? It's certainly not "infinity", so is this an indication that humans don't reason?
andrepd 2y ago
My brother in christ, how is A/B + C*D - E = F, solve for B an original problem? How many tens of thousands of examples of this exact form do you think it came across? It's the same as with coding by the way: it can reshuffle things it has already seen while changing variable names and so on. Ask it something which is not in stackoverflow or geeks4geeks and it goes tits up. PS: Tested it on GPT 3.5: same answer.

bob1029 2y ago

> Now I guess you can pick up the goal post and move it. The central problem with math is that you have an infinite amount of space within which to move these goalposts. How many variants on this trial before we find a mistake? What is an acceptable error rate?

jimhefferon 2y ago
At this moment, the error rate seems to be that of a beginning graduate student. Or at least, that's what Terry Tao thinks. That's pretty good.
- lupire 2y ago
  That is not at all what Tao said. https://mathstodon.xyz/@tao/113132502735585408 https://mathstodon.xyz/@tao/113132502735585408 "Here the results were better than previous models, but still slightly disappointing: the new model could work its way to a correct (and well-written) solution if provided a lot of hints and prodding, but did not generate the key conceptual ideas on its own, and did make some non-trivial mistakes. The experience seemed roughly on par with trying to advise a mediocre, but not completely incompetent, (static simulation of a) graduate student. "
  - buneskamin 2y ago
    Yea it does seem pretty clear that it's mainly Terrance's contributions to the context window that is bringing the model to the right answer
naasking 2y ago
> How many variants on this trial before we find a mistake? How many variants would it take for a human to make a mistake? It's certainly not "infinity", so is this an indication that humans don't reason?

andrepd 2y ago

My brother in christ, how is A/B + C*D - E = F, solve for B an original problem? How many tens of thousands of examples of this exact form do you think it came across? It's the same as with coding by the way: it can reshuffle things it has already seen while changing variable names and so on. Ask it something which is not in stackoverflow or geeks4geeks and it goes tits up. PS: Tested it on GPT 3.5: same answer.