3 ms·
I respectfully disagree. While tokenization certainly plays a role in how language models process input, it's simplistic to attribute the challenges in mathema
by ttul 2y ago
I respectfully disagree.
While tokenization certainly plays a role in how language models process input, it's simplistic to attribute the challenges in mathematical reasoning solely to tokenization.
SOTA language models don't just rely on individual token predictions, but build up contextual representations across multiple layers. This allows them to capture higher-level meaning beyond simple token-to-token relationships. If this weren’t the case, it would be inconceivable that models would work at all in all but the most utterly simplistic scenarios.
The decline in performance as complexity increases might be due to other factors, such as:
- Limitations in working memory or attention span
- Difficulty in maintaining coherence over longer sequences
- Challenges in managing multiple interdependent logical constraints simultaneously (simply due to the KQV matrices being too small)
And in any case, I think OpenAI’s o1 models are crushing it in math right now. The iterative, model-guided CoT approach seems to be able to handle very complex problems.
- m3kw9 2y agoI would say the more variable you give it the more the probability drifts for each of the facts they have to hold, maybe LLMs still doesn’t have the ability to ignore useless stuff you add to the prompt
- l33t7332273 2y agoI thought attention was all you need
- altruios 2y agoHow much attention do you need? ...is probably an important question too.
- andrepd 2y ago>And in any case, I think OpenAI’s o1 models are crushing it in math right now. My man, it cannot solve even the simplest problems which it hasn't seen the solution to yet, and routinely makes elementary errors in simple algebraic manipulations or arithmetic! All of this points to the fact that it cannot actually perform mathematical or logical reason, only mimic it superficially if trained in enough examples. I challenge you to give it even a simple, but original, problem to solve.
- ukuina 2y agoDo you have some categories of such original problems? It seems markedly better at reasoning/logic puzzles, and programmatically-solvable problems are often offloaded to the Python interpreter.
- Workaccount2 2y ago>I challenge you to give it even a simple, but original, problem to solve. (34903173/x)+(238 * 2650) - 323326 = 45323434, solve for x Statistically, no one has ever done this calculation ever before. It's entirely unique. O1 answered "x = 34,903,173 divided by 45,016,060", which is correct.[1][2] Now I guess you can pick up the goal post and move it. [1]https://chatgpt.com/share/6709481a-3144-8004-a7fd-0ccd9e3bc58d https://chatgpt.com/share/6709481a-3144-8004-a7fd-0ccd9e3bc5... [2]https://www.wolframalpha.com/input?i=%2834903173%2Fx%29%2B%28238+*+2650%29+-++323326+%3D+45323434 https://www.wolframalpha.com/input?i=%2834903173%2Fx%29%2B%2...
- bob1029 2y ago> Now I guess you can pick up the goal post and move it. The central problem with math is that you have an infinite amount of space within which to move these goalposts. How many variants on this trial before we find a mistake? What is an acceptable error rate?
- jimhefferon 2y agoAt this moment, the error rate seems to be that of a beginning graduate student. Or at least, that's what Terry Tao thinks. That's pretty good.
- lupire 2y agoThat is not at all what Tao said. https://mathstodon.xyz/@tao/113132502735585408 https://mathstodon.xyz/@tao/113132502735585408 "Here the results were better than previous models, but still slightly disappointing: the new model could work its way to a correct (and well-written) solution if provided a lot of hints and prodding, but did not generate the key conceptual ideas on its own, and did make some non-trivial mistakes. The experience seemed roughly on par with trying to advise a mediocre, but not completely incompetent, (static simulation of a) graduate student. "