25 ms·

Understanding the Limitations of Mathematical Reasoning in LLMs

282 points by hnhn34 2y ago

thenoblesunfish 2y ago
Very interesting, and aligns with what I would expect in terms of the type of "thinking" LLMs do. I think that it's also the type of "thinking" that will let a student pass most school courses, except of course for the ones where the teacher has taken the time to pose test questions that aren't as amenable to pattern matching. (Hard, but I assume most readers here are familiar with leetcode style interviews and what makes questions of that kind higher or lower quality for assessing candidates) (And yes, I know people are hard at work adding other types of thinking to work along with the pure language models)
dr_dshiv 2y ago
It seems incredibly easy to generate an enormous amount of synthetic data for math. Is that happening? Does it work?
- ninetyninenine 2y ago
  I don’t think so. The data is biased towards being very general.
- Davidzheng 2y ago
  In which distribution? Like school math or competition or unsolved problems? FWIW I think one and three and probably easier to generated as synethetically. It's harder to bound the difficulty but I think the recent David silver talk implies it doesn't matter much. Anyway there's some work on this you can find online--they claim to improve gsm8k and MATH a bit but not saturate it. Idk in practice how useful it is
- MacsHeadroom 2y ago
  Yes, this is how o1 was trained. Math and programming, because they are verifiable. This is also why o1 is not better at English. Math skills transfer to general reasoning but not so much to creative writing.
- bentice 2y ago
  Data is the wrong approach to develop reasoning. You we don't want LLM's to simply memorize 3x3 = 9 we want them to understand that 3 + 3 + 3 = 9 therefore 3x3 = 9 (obviously a trivial example). If they have developed reasoning very few examples should be needed. The way I see it reasoning is actually the ability of the model to design and train smaller models that can learn with very few examples.
  - hackinthebochs 2y ago
    > If they have developed reasoning very few examples should be needed. Yes, once the modules for reasoning have converged, it will take very few examples for it to update to new types of reasoning. But to develop those modules from scratch requires large amounts of examples that overtax its ability to memorize. We see this pattern in the "grokking" papers. Memorization happens first, then "grokking" (god I hate that word). It's not like humans bootstrap reasoning out of nothing. We have a billion years of evolution that encoded the right inductive biases in our developmental pathways to quickly converge on the structures for reasoning. Training an LLM from scratch is like recapitulating the entire history of evolution in a few months.
  - dr_dshiv 2y ago
    My understanding is that, if you train these enough, it becomes likely to develop efficient compressions— which “reasoning” would be.
- ilaksh 2y ago
  They did that for o1 and o1-preview. Which if you read the paper or do your own testing with that SOTA model you will see that the paper is nonsense. With the best models the problems they point out are mostly marginal like one or two percentage points when changing numbers etc. They are taking poor performance of undersized models and claiming that proves some fundamental limitation of large models, even though their own tests show that isn't true.
  - foobarqux 2y ago
    You choose to ignore Figure 8 which shows a 18% drop when simply adding an irrelevant detail. In the other test the perturbations aren’t particularly sophisticated and modify the problem according to a template. As the parent comment said this is pretty easy to generate test data for (and for the model to pattern match against) so maybe that is what they did. A better test of “reasoning” would be to isolate the concept/algorithm and generate novel instances that are completely textually different from existing problems to see if the model really isn’t just pattern matching. But we already know the answer to this because it can’t do things like arbitrary length multiplication.
    - ilaksh 2y ago
      This shows there are limitations but it doesn't prove they can't be overcome by changing training data. I don't think that LLMs are the end of AGI research at all, but the extreme skepticism of their current utility is mostly based on failures of small models. It's like 65% for most of the small models they tested and that is what they are really basing their conclusions on
- aithrowawaycomm 2y ago
  It's easy enough to generate an enormous amount of formal math problems, but utterly quixotic to generate an enormous amount of quantitative reasoning problems, which is the thing LLMs are lacking.
s-macke 2y ago
These results are very similar to the "Alice in Wonderland" problem [1, 2], which was already discussed a few months ago. However the authors of the other paper are much more critical and call it a "Complete Reasoning Breakdown". You could argue that the issue lies in the models being in an intermediate state between pattern matching and reasoning. To me, such results indicate that you can't trust any LLM benchmark results related to math and reasoning when you see, that changing the characters, numbers or the sentence structure in a problem alter the outcome by more than 20 percentage points. [1] https://arxiv.org/html/2406.02061v1 https://arxiv.org/html/2406.02061v1 [2] https://news.ycombinator.com/item?id=40811329 https://news.ycombinator.com/item?id=40811329
- s-macke 2y ago
  Here is the larger discussion about the Alice in Wonderland Paper on Hacker News. https://news.ycombinator.com/item?id=40585039 https://news.ycombinator.com/item?id=40585039
- apsec112 2y ago
  Both Claude-3.5 and o1-preview nail this problem "Let's think through this step-by-step: 1. Alice has 3 brothers 2. Alice has 2 sisters 3. We need to find out how many sisters Alice's brother has The key here is to realize that Alice's brothers would have the same sisters as Alice, except they would also count Alice as their sister. So, Alice's brothers would have: - The 2 sisters Alice has - Plus Alice herself as a sister Therefore, Alice's brothers have 3 sisters in total."
  - s-macke 2y ago
    And here lies the exact issue. Single tests don’t provide any meaningful insights. You need to perform this test at least twenty times in separate chat windows or via the API to obtain meaningful statistics. For the "Alice in Wonderland" paper, neither Claude-3.5 nor o1-preview was available at that time. But I have tested them as well a few weeks ago with the issue translated into German, achieving also a 100% success rate with both models. However, when I add irrelevant information (My mother ...), Claude's success rate drops to 85%: "My mother has a sister called Alice. Alice has 2 sisters and 1 brother. How many sisters does Alice's brother have?"
yk 2y ago
I test llms actually similar. For example there is a well known logic puzzle were a farmer tries to cross a river with a cabbage a goat and a wolf. Llms can solve that since at least GPT-2, however if we replace the wolf with a cow, gpt-o does correctly infer the rules of the puzzle but can't solve it.
- voidUpdate 2y ago
  I'm scared of the cows around you if they eat goats
  - Manabu-eo 2y ago
    I think their point is that cows don't eat goats, unlike wolves, and that causes the LLMs to answer it wrong.
- getoffmyyawn 2y ago
  I've found that the River Crossing puzzle is a great way to show how LLMs break down. For example, I tested Gemini with several versions of the puzzle that are easy to solve because they don't have the restrictions such as the farmer's boat only being able to carry one passenger/item at a time. Ask this version, "A farmer has a spouse, chicken, cabbage, and baby with them. The farmer needs to get them all across the river in their boat. What is the best way to do it?" In my tests the LLMs nearly always assume that the boat has a carry-restriction and they come up with wild solutions involving multiple trips.
- chasd00 2y ago
  What happens if you sit down and invent a logic game that is brand new and has never been documented before anywhere then ask an LLM to solve it? That, to a layman like me, seems like a good way to measure reasoning in AI.
  - jprete 2y ago
    I think the problem is inventing new structures for logic games. The shape of the problem ideally would be different than any existing puzzle, and that's hard. If a person can look at it and say "oh, that's just the sheep-wolf-cabbage/liar-and-truthteller/etc. problem with extra features" then it's not an ideal test because it can be pattern-matched.
criddell 2y ago
It would be interesting if this kind of work could ever be extended to show the limitations of mathematical reasoning in animals and humans. For example, just as a dog will never understand a fourier transform, there are likely ideas that humans cannot understand. If we know what our limits are, I wonder if we could build machines that can reason in ways we aren't capable of?
- myrmidon 2y ago
  I think it is a naive assumption that such a limitation even exists ("exists" in a sense that it is actually useful, by being consistent and somewhat simple to describe). We investigated similar ideas for language (=> Noam Chomsky), where we tried to draw clear, formalized limits for understanding (to show e.g. how human capabilities contrast with animals). The whole approach failed completely and irredeemably (personal opinion), but researching it was far from useless to be fair.
  - r2_pilot 2y ago
    As the human brain is finitely bounded in space and time, any idea that can't be compressed or represented by condensing notation, which is "larger" than the 100B cells+100T synapses can represent, or whose integration into said human's brain would take longer than 150 years, would be considered unable to be contemplated by a normal human.
    - klabb3 2y ago
      Yes but we overcome. We can do absolutely insane things like just large prime number testing, because of reasoning + tool use. Humans invent tools and wield them. Whether it's pen & paper to extend our memory, a horse to become stronger, a calculator to speed up our thinking or an airplane to literally fly, the tools we wield become extensions of our agency and control. A lonely human without knowledge sharing or tools isn’t that much more capable in their lifetime than the smartest animals. When we talk about human ability colloquially, we’re generally talking about what we can do with access to our human heritage, civilization, safety and access to materials and tools. Pattern matching against something others have already done is great but this is shared with at the very least all mammals to some extent. Pushing the boundaries of our species forward over time is a different game. Or at least, it seems to be… It certainly seems like we’ve found the holy grail of pattern matching (system 1 thinking), which is an insane leap! But what about system 2? The million dollar question is what the hell is the topology of that pre-frontal cortex thinking machine? Is it just more pattern matching but against different patterns? Or is it completely and qualitatively different? And if so, is it more or less hard? To me, following the debate is just watching one bad prediction after another, (including my own of course). We just don't know how it works. Not you or me, not Sam Altman in full though-leading leather jacket uniform, or even our top neuro-scientists.
beardyw 2y ago
I honestly can't see why LLMs should be good at this sort of thing. I am convinced you need a completely different approach. At the very least you mostly only want one completely correct result. Good luck getting current models to do that.
- golol 2y ago
  I'm a math phd student at the moment and I regularly use o1 to try some quick calculations I don't feel like doing. While I feel like GPT-4o is so distilled that it just tries to know the answer from memory, o1 actually works with what you gave it and tries to calculate. It's can be quite useful.
  - banditelol 2y ago
    I'm curious what kind of quick calculation do you usually use llm for? Edited for clarity
    - golol 2y ago
      Just earlier today I wanted to check if exp(inx) is an orthonormal basis on L^2((0, 1)) or if it needs normalization. This is an extremely trivial one though. Less trivially I had an issue where a paper claimed that a certain white noise, a random series which diverges in a certain Hilbert space, is actually convergent in some L^infinity type space. I had tried to use a Sobolev embedding but that was too crude so it didn't work. o1 correctly realized that you have to use the decay of the L^infinity norm of the eigenbasis, a technique which I had used before but just didn't think of in the moment. It also gave me the eigenbasis and checked that everything works (again, standard but takes a while to find in YOUR setting). I wasn't sure about the normalization so again I asked it to calculate the integral. This kind of adaptation to your specific setting instead of just spitting out memorized answers in commonn settings is what makes o1 useful for me. Now again, it is often wrong, but if I am completely clueless I like to watch it attempt things and I can get inspiration from that. That's much more useful than seeing a confident wrong answer like 4o would give it.
- deleted 2y ago
apsec112 2y ago
()
- ilaksh 2y ago
  That makes the whole conclusion obviously false. I don't really understand why, but I think we are going to see total denial from a significant percentage of the population all the way up to and past the point where many average mathematicians and software engineers cannot in any way compete with AI. We already are reportedly getting pretty close with o1 (not o1-preview). There are also new paradigms for machine learning and hardware in the pipeline that will continue to provide orders of magnitude performance gains and new capabilities in the next 5-10 years. Many people still claim that "self driving cars don't exist", in so many words, even though they are deployed in multiple cities.
  - sottol 2y ago
    > Many people still claim that "self driving cars don't exist", in so many words, even though they are deployed in multiple cities. But just look at the predictions of that time - cities will change, ... and so on. Sure, we have self-driving cars but the reality looks very different (and a lot more like the past!) than the pundits and futurists imagined! I'm not sure anyone will make their billions of dollars investmented back within even 20 years. Just two random examples from ~10 years ago (2013-2016), you can google many more of that time. * "Ford Targets Fully Autonomous Vehicle for Ride Sharing in 2021; Invests in New Tech Companies, Doubles Silicon Valley Team" [1] * "Disruptions: How Driverless Cars Could Reshape Cities" [2] [1] https://media.ford.com/content/fordmedia/fna/us/en/news/2016/08/16/ford-targets-fully-autonomous-vehicle-for-ride-sharing-in-2021.html https://media.ford.com/content/fordmedia/fna/us/en/news/2016... [2] https://archive.nytimes.com/bits.blogs.nytimes.com/2013/07/07/disruptions-how-driverless-cars-could-reshape-cities/?smid=tw-share https://archive.nytimes.com/bits.blogs.nytimes.com/2013/07/0... [3] https://www.gensler.com/dialogue/30/the-game-changer-for-cities-driverless-cars https://www.gensler.com/dialogue/30/the-game-changer-for-cit...
dev1ycan 2y ago
I don't understand the idiocracy we live in, it is beyond obvious not just that the stock market is a bubble but ESPECIALLY the AI related stocks are a massive bubble, when it pops, and it will, it is going to be very very ugly, yet people keep pouring in, as Sabine said it, it's starting to look like particle physics where they keep asking for bigger colliders, just because you have a bigger collider, if your methodology is flawed you aren't gonna get any more significant returns. Eventually they will run out of exponential cash to pour in, and investors will start asking questions, stocks are already valued at 60x+ their earnings, whenever it pops you don't want to be the one who bought the top. Guess it's still gonna take a while more for the layman to realize the issues with LLMs, but it'll happen.
- Workaccount2 2y ago
  >if your methodology is flawed you aren't gonna get any more significant returns. The problem with this statement is that predictions made about scaling 5 years ago have held true[1]. We keep adding parameters, adding compute, and the models keep getting more capable. The flaws of LLM's from 2024 are not what is relevant. Just like the flaws of LLMs from 2021 were not relevant. What is relevant is the rate of change, and the lack of evidence that things won't continue on this steep incline. Especially if you consider that GPT4 was sort of a preview model that motivated big money to make ungodly investments to see how far we can push this. Those models will start to show up over the next 2 years. If they break the trend and the scaling flops, then I think a lot of air is gonna blow out of the bubble. [1]https://arxiv.org/pdf/2001.08361 https://arxiv.org/pdf/2001.08361
  - dev1ycan 2y ago
    They are very literally asking for trillions and even nuclear powered data centers, pretty sure we've gotten to the point where it's not sustainable.
    - Workaccount2 2y ago
      Those are roadmap items being asked for, but the next gen models are already in training. If they keep moving along the same trend line, like all the previous models have, then they probably will be able to find the investors for the next next gen. Even if it's a few trillion dollars and a few nuclear power plants. This doesn't even factor in the tech inertia. We could stop making new models today, and it would probably be 4-5 years before integration slowed down. Google still hasn't even put Gemini in their home speakers.
bob1029 2y ago
> we investigate the fragility of mathematical reasoning in these models and demonstrate that their performance significantly deteriorates as the number of clauses in a question increases. We hypothesize that this decline is due to the fact that current LLMs are not capable of genuine logical reasoning I'd offer a simpler explanation: Tokenization. If you tokenize "12345 * 27271" you will get the following: "123", "45", " *", " ", "272", "71" The statistical likelihood that any of these tokens predicts any of the others is completely meaningless in the context of simple arithmetic. You can argue that this is where tool use comes in (and I would be inclined to agree), but I don't think this bodes well for "genuine logical reasoning".
- m3kw9 2y ago
  The llm will know 123 and 45 is a contiguious number just like how humans can tell if you say 123 and then a slight pause 45 as a single number
  - soulofmischief 2y ago
    I think that as long as the attention mechanism has been trained on each possible numerical token enough, this is true. But if a particular token is underrepresented, it could potentially cause inaccuracies.
  - TZubiri 2y ago
    It's just so dissonant to me that the tokens in mathematics are the digits, and not bundles of digits. The idea of tokenization makes sense for taking the power off letters, it provides language agnosticism. But for maths, it doesn't seem appropriate. I wonder what the effect of forcing tokenization for each separate digit be.
    - taeric 2y ago
      This reminds me of the riddle of someone buying the numerals to put their address on their house. When you are looking at text, the point is all you have are the characters/symbols/tokens/whatever you want to call them. You can't really shepherd some over to their numeric value while leaving some at their token value. Unless you want to cause other issues when it comes time to reason about them later. I'd hazard that the majority of numbers in most text are not such that they should be converted to a number, per se. Consider addresses, postal codes, phone numbers, ... ok, I may have run out of things to consider. :D
singularity2001 2y ago
If the argument is that LLMs are bad at reasoning because they are easily distractible and the results vary with modifications in the question, one should be reminded of the consistency and distractability of humans.
- zeroonetwothree 2y ago
  Why? LLMs are supposedly better than humans (as many comments claim in this thread).
- deleted 2y ago
  [deleted]
- riku_iki 2y ago
  Trained human can tell if distracted: "I am distracted and can't figure out answer", while LLM will confidently gives you wrong answer, which makes whole results not reliable.
woopwoop 2y ago
I'm curious about what happens with the no-op dataset if you include in the prompt that the questions may contain irrelevant information.
parsimo2010 2y ago
I won't take a strong stance on whether or not LLMs actually do reasoning, but I will say that this decrease in performance is similar to what I see in college freshmen (I'm currently teaching a calculus course in which almost half of the students took AP calc in high school). They perform well on simple questions. Requiring students to chain multiple steps together, even simple steps, results in decreased accuracy and higher variance (I have no data on whether this decrease is linear or not, as the paper assumes that the decrease should be linear with the number of steps). We see similar results with adding unrelated statements into a problem- many students are trained to make sure to use all given information in solving a problem- if you leave out something that the instructor gives you, then you probably forgot to do something important. So while I don't take a stance on what an LLM does should be considered reasoning, I do think that SOTA LLMs like GPT-4o perform about as good as high school graduates in America with average intelligence. In other words, average Americans exhibit similar limitations on their reasoning as good LLMs. Which on the one hand is a little disappointing to me in terms of the human performance but is kind of good news for LLMs- they aren't doing graduate-level research but they are already capable of helping a large portion of the population.
- skydhash 2y ago
  Not to disparage American school system (my country’s is worse) but it’s very much easy mode. I know that not everyone is suited to academic excellence, but it’s definitely easier to learn when young. I do believe too much hand holding actively harm learning.
  - BriggyDwiggs42 2y ago
    I don’t think the issue with American schools is that there’s too much hand holding. If anything, it’s the opposite; teachers at drastically underfunded schools don’t have any time to help the students of their 50 person class through the confused curriculum.
    - skydhash 2y ago
      Here, we have to go through 4 state exams just to get to university. The first when you’re 11, the second at 14, then two consecutive ones at 17 and 18. There’s a national curriculum that the exams will be about, although the schools are free to add to it. So however you feel about the school or the teacher, you have to master the subjects enough to go through. And that means paying attention in class, cram before it, or hoping you can cheat. We have our own problem too, but the consensus among all the people I know that have moved to the US is that classes are easy there. Not a bad thing per se (better explanation, better understanding instead of rote memorizing).
woopwoop 2y ago
This paper, among other things, shows that LLMs have dramatically worse performance on basic algebra questions when you add in irrelevant information. The examples are things like "John picked 43 kiwis on Monday, 24 kiwis on Tuesday. On Wednesday, 5 of the kiwis he picked were smaller than usual. Altogether, on Monday, Tuesday, and Wednesday, John picked 87 kiwis. How many kiwis did John pick on Wednesday?" In this question, the remark about some of the kiwis on Wednesday being small is irrelevant, but adding things like this reduces performance on a popular benchmark from 95% to 77% for GPT-4o, for example. I don't find this very impressive. Forget LLMs for a second. Let's say _you_ read a question of that kind with some bit of irrelevant information. There are two possibilities you have to consider: the question may as well have excluded the irrelevant information, or the question was miswritten and the irrelevant information was meant to be relevant. The latter is a perfectly live possibility, and I don't think it's a dramatic failure to assume that this is correct. I have to confess that when I read some people's LLM gotcha questions, where they take some popular logic puzzle and invert things, I think I would get them "wrong" too. And not wrong because I don't understand the question, but wrong because with no context I'd just assume the inversion was a typo.
- jfrbfbreudh 2y ago
  I think it’s an important result because filtering signal from noise is just as, if not more, important than forming conclusions from signal.
- swatcoder 2y ago
  Real discourse has tons of irrelevant information for all sorts of reasons. There are some contexts, academic or professional, where questions are posed carefully and specifically, but these are narrow contexts. A useful general purpose assistant needs to be able to find what's relevant among what's irrelevant. Excellence at just solving math problems that are especially well specified can be a useful domain assistant (no small win!), but is not the same thing. That said, if you've got a hundred billion dollars betting on your AI project achieving AGI, you benefit a lot by conflating those contexts. In that case, grinding on formal SAT, LSAT, GRE, etc problems amounts to tuning for microbenchmarks rather than real world use cases.
resters 2y ago
I think it's obvious that LLMs will be able to do "reasoning" far better than humans. We must separate our notion of what is remarkably human. Rarely is it the reasoning, it's the intuition that a logical path exists -- for example a mathematical proof that draws from separate sub-disciplines of mathematics, etc. Consider that in a LLM, language inputs are tokenized and fed as inputs into the neural network, and connections in the network create output sequences that are not just syntactically correct (trivial) or form semantically plausible sentences (early transformers did this). LLM output sequences follow the deep patterns of language which include sometjhing that resembles reasoning as the model has learnt from its training data. LLMs seem to fall short because they often fail at truly abstract reasoning tasks that humans find easy. If trained properly, LLMs can develop advanced representations of logical systems that will surely outpace what humans can do in terms of raw reasoning. However, human mathematicians have not even unified around constructive mathematics as a must for the study of mathematics. This reveals that even highly evolved mathematical disciplines rely on objects whose characteristics do not lend themselves to full logical scrutiny and are in a way socially constructed and effectively hard to audit. While notation in mathematics is incredible technology it is also a highly limiting factor that suffers major tradeoffs. Humans struggle to invent new notation fast enough and to discard outdated notation fast enough. If we do see an AI-powered boom in mathematics, I suspect our notion of notation and the fluidity we demand from it will change dramatically.
- sottol 2y ago
  > If trained properly, LLMs can develop advanced representations of logical systems that will surely outpace what humans can do in terms of raw reasoning. We have already trained the LLMs on most of the human knowledge base (so like 4-5000 years?) - imo training data will become a problem and will soon be more expensive than compute. Sure, you can work around some of this using synthetic training data but I personally would not count on general-purpose LLMs (especially LLMs aka transformer models) developing super-human representations of logical systems anytime soon.
  - resters 2y ago
    I don't disagree, however I'm optimistic because most of the current reasoning "ability" of LLMs comes from the accidental reasoning embedded in language patterns. For example, the prompt completion: "The mouse has a unique digestive system compared to other rodents, however the sparrow" on GPT-4o is "exhibits a highly specialized digestive system adapted for rapid processing of food, particularly seeds and insects, through structures like the crop and gizzard, which are not found in rodents." Claude 3.5 completes it as "has a completely different digestive anatomy as a bird. Birds like sparrows have adaptations for flight, including a lightweight skeletal system and a specialized digestive tract. Unlike mice, sparrows have a crop for storing food, a gizzard for grinding it, and generally shorter intestines to reduce weight. They also lack teeth, instead using their beak to manipulate food." What appears to be a thoughtful contrast is merely a language pattern. Similarly, a prompt like "Assume -B, A->B. Under what circumstances is B true?" will simply follow the gradient to return output that is likely correct. Prompts like "what is 2+2" fail only because nobody bothers to write about it so simple arithmetic was not in the training data. However the way that multi-modal LLMs handle images is inspiring as it effectively converts from the visual domain into the sequential token domain. The same could be done for symbolic systems, etc.
trehalose 2y ago
I see a lot of discussion about irrelevant clauses tripping up the LLMs and why that does or doesn't matter. To me, what's far more damning is this: > Specifically, the performance of all models declines when only the numerical values in the question are altered in the GSM-Symbolic benchmark. This seems like irrefutable evidence of overfitting, that in the best case scenario is epidemic among current LLMs (and in the worst case interpretation, is covering up fundamental inabilities to learn mathematical reasoning from the training data).
gradientsrneat 2y ago
Could this be Goodhart's Law in action? AI tools like to showcase benchmarks in bar graphs to show how well they perform compared to other models. Maybe the benchmark Qs/As snuck into training sets accidentally. Is it still Goodhart's Law if it's unintentional? Daniel Lemire has blogged about being impressed with how well the LLM answers his CS problem questions. I was impressed too. Not sure where the line of competence lies.
eigenform 2y ago
The difference is that, if we are solving a math problem together, you and I [explicitly or implicitly] can come to an agreement over the context and decide to restrict our use of language with certain rules. The utility behind our conversation [generally] rests on those rules! An LLM is very good at recovering rules, but being good at pattern recognition is not the same thing as being good at unambiguously following rules in the appropriate context. edit: Natural language is far from an efficient/sufficient/necessary intermediate representation for doing math, just ask any general-purpose computer. Sometimes, it's worth "putting rules in stone," and it seems unreasonable to believe that there is always an unambiguous rule for this that you can mechanically recover from a corpus of language use.
deleted 2y ago
[deleted]
dang 2y ago
Related ongoing thread: LLMs don't do formal reasoning - https://news.ycombinator.com/item?id=41812523 https://news.ycombinator.com/item?id=41812523 - Oct 2024 (70 comments)
jumploops 2y ago
> Overall, while o1-preview and o1-mini exhibit significantly stronger results compared to current open models—potentially due to improved training data and post-training procedures—they still share similar limitations with the open models. tl;dr - the best open model dropped from 89.7% on GSM8K(full) to 30% on Symbolic-NoOp, while o1-preview dropped from 94.9% to 77.4%, respectively. I think all this paper shows is that LLMs need space to "think" outside of their inference layer, (for the current architectures at least). It's similar to the "draw a room, but DO NOT put an elephant in the corner" prompts that people were using with image models. This is something that practitioners have been doing for awhile (via CoT, ToT, etc.) and the whole rationale behind OpenAI's newly launched o1-series "model." There's another post that says this paper proves LLMs can't be used to build "reliable agents" -- which doesn't appear to be true when you look at o1's stellar performance here.
- data_maan 2y ago
  Can you send a paper regarding that LLMs can build "reliable agents"?
qwerty456127 2y ago
Can't al LLM just detect a mathematical reasoning task then produce a formula (not even display it in the production mode) to invoke on an external service engineered for formal logical and mathematical computations?
- aithrowawaycomm 2y ago
  In many of these examples it produces the wrong formula because it misunderstands the word problem, so a computer algebra system wouldn't help - garbage in, garbage out. The problem here is more serious than mathematics: the quantitative reasoning itself is highly unreliable.
ak_111 2y ago
As an outsider can anyone enlighten me how this squares with the news that models that adapt similar LLM architecture can obtain silver medal in mathematical olympiad?
- lionkor 2y ago
  careful statistical massaging, maybe. would you pick only winning results and only present favorable, massaged results if it got you 150+B USD of worth?
throwaway918299 2y ago
limitations of mathematical reasoning? They have none. Literally zero. That’s the limit. Thank you for reading my paper.
codelion 2y ago
This is surprising to only those that have not worked in formal reasoning. Yes, LLMs cannot do true logical reasoning in a formal sense, you can do better with an SMT solver. But it is also true that you can solve a lot of logical problems by just applying “reasoning steps” from the training data, specially when your training data is the entirety of written content ever produced. Both of these can be true at the same time it is not a contradiction just an interesting dichotomy.
jgord 2y ago
I propose 'gords rule' : "any sufficiently advanced LLM will learn the laws of logic, the principles of scientific method, and Reinforcement Learning" until that happens .. I think RL startups focused on real problems are much undervalued : https://quantblog.wordpress.com/2024/10/11/llm-hype-means-there-are-bargains-in-ml-deep-tech-startups-doing-real-things/ https://quantblog.wordpress.com/2024/10/11/llm-hype-means-th...
uptownfunk 2y ago
The very fundamental problem with LLM is there is no guarantee on any of the reasoning it gives you without a human there to give a thumbs up. They are working on solving this (alpha proof, lean agent etc) but getting this to run at inference time in an optimized way is what I would call one of the millenial prize problems of AI which will lead to a quantum leap in the path towards the singularity.
Animats 2y ago
It's an expected result. Whatever happened with that result which found some representation of the state of a game inside an LLM? That indicated some degree of model-building. Haven't heard about that again/
i007 2y ago
LLMs are designed to carry out "associative reasoning" which captures logic based on recognition and recall of compositional patterns learned during training. Having said that, we can still get semantically and logically idempotent output that makes sense but with lots of work outside of the LLM, which contrasts with the current hyper focus on the LLM itself as the be all and end all. It is just one component in what ought to be a larger and more involved system for reasoning. Look at what we were able to accomplish here for Legal AI, not so mathematical logic per se but mimicking (capturing) axiomatic logic in the legal domain: https://www.youtube.com/watch?v=_9Galw9-Z3Q https://www.youtube.com/watch?v=_9Galw9-Z3Q marc at sunami dot ai
teleforce 2y ago
In terms of usefulness and realistic implementation mathematical reasoning is the next frontier of LLM not autonomous level 5 driving or AGI. More research fund and investment are much better spent on the former rather than the latter but apparently it seems that the reverse situation is the case.
gtsop 2y ago
LLMs are inherently emulators of digitaly imprinted artifacts of human consciousness. When people trully grasp what this means they will stop being buffled by the fact that LLMs performance deteriorate when novelty of the task increases. EDIT: Had there been an ounce of actual true reasoning emerging in LLMs, openai would have been running this thing privatly 24/7 to produce new science and capture pattents that would give them economic dominance. Not trying to sell tokens to us all.
K0balt 2y ago
Trying to solve (much less explore) mathematics using probabilistic next-token prediction seems like the really long way around, especially when we have pretty good deterministic tools available for our use. I don’t know why anyone would bother doing anything besides working on the correct manipulation of tools. Brains have various structures that have distinct architectures. I don’t see any indication that the best way forward is to try to shoehorn everything into a single computational paradigm. It’s like trying to make a flying submarine car. It might technically be possible, but it might not be worth the trouble, and it’s unlikely to result in a vehicle that works excellently in any of its environments.
bubble12345 2y ago
Can LLMs even do addition, with say 20+ digit numbers? Multiplication?

282 points by hnhn34 2y ago

Very interesting, and aligns with what I would expect in terms of the type of "thinking" LLMs do. I think that it's also the type of "thinking" that will let a student pass most school courses, except of course for the ones where the teacher has taken the time to pose test questions that aren't as amenable to pattern matching. (Hard, but I assume most readers here are familiar with leetcode style interviews and what makes questions of that kind higher or lower quality for assessing candidates) (And yes, I know people are hard at work adding other types of thinking to work along with the pure language models)

dr_dshiv 2y ago

It seems incredibly easy to generate an enormous amount of synthetic data for math. Is that happening? Does it work?

ninetyninenine 2y ago
I don’t think so. The data is biased towards being very general.
Davidzheng 2y ago
In which distribution? Like school math or competition or unsolved problems? FWIW I think one and three and probably easier to generated as synethetically. It's harder to bound the difficulty but I think the recent David silver talk implies it doesn't matter much. Anyway there's some work on this you can find online--they claim to improve gsm8k and MATH a bit but not saturate it. Idk in practice how useful it is
MacsHeadroom 2y ago
Yes, this is how o1 was trained. Math and programming, because they are verifiable. This is also why o1 is not better at English. Math skills transfer to general reasoning but not so much to creative writing.
bentice 2y ago
Data is the wrong approach to develop reasoning. You we don't want LLM's to simply memorize 3x3 = 9 we want them to understand that 3 + 3 + 3 = 9 therefore 3x3 = 9 (obviously a trivial example). If they have developed reasoning very few examples should be needed. The way I see it reasoning is actually the ability of the model to design and train smaller models that can learn with very few examples.
- hackinthebochs 2y ago
  > If they have developed reasoning very few examples should be needed. Yes, once the modules for reasoning have converged, it will take very few examples for it to update to new types of reasoning. But to develop those modules from scratch requires large amounts of examples that overtax its ability to memorize. We see this pattern in the "grokking" papers. Memorization happens first, then "grokking" (god I hate that word). It's not like humans bootstrap reasoning out of nothing. We have a billion years of evolution that encoded the right inductive biases in our developmental pathways to quickly converge on the structures for reasoning. Training an LLM from scratch is like recapitulating the entire history of evolution in a few months.
- dr_dshiv 2y ago
  My understanding is that, if you train these enough, it becomes likely to develop efficient compressions— which “reasoning” would be.
ilaksh 2y ago
They did that for o1 and o1-preview. Which if you read the paper or do your own testing with that SOTA model you will see that the paper is nonsense. With the best models the problems they point out are mostly marginal like one or two percentage points when changing numbers etc. They are taking poor performance of undersized models and claiming that proves some fundamental limitation of large models, even though their own tests show that isn't true.
- foobarqux 2y ago
  You choose to ignore Figure 8 which shows a 18% drop when simply adding an irrelevant detail. In the other test the perturbations aren’t particularly sophisticated and modify the problem according to a template. As the parent comment said this is pretty easy to generate test data for (and for the model to pattern match against) so maybe that is what they did. A better test of “reasoning” would be to isolate the concept/algorithm and generate novel instances that are completely textually different from existing problems to see if the model really isn’t just pattern matching. But we already know the answer to this because it can’t do things like arbitrary length multiplication.
  - ilaksh 2y ago
    This shows there are limitations but it doesn't prove they can't be overcome by changing training data. I don't think that LLMs are the end of AGI research at all, but the extreme skepticism of their current utility is mostly based on failures of small models. It's like 65% for most of the small models they tested and that is what they are really basing their conclusions on
aithrowawaycomm 2y ago
It's easy enough to generate an enormous amount of formal math problems, but utterly quixotic to generate an enormous amount of quantitative reasoning problems, which is the thing LLMs are lacking.

s-macke 2y ago

These results are very similar to the "Alice in Wonderland" problem [1, 2], which was already discussed a few months ago. However the authors of the other paper are much more critical and call it a "Complete Reasoning Breakdown". You could argue that the issue lies in the models being in an intermediate state between pattern matching and reasoning. To me, such results indicate that you can't trust any LLM benchmark results related to math and reasoning when you see, that changing the characters, numbers or the sentence structure in a problem alter the outcome by more than 20 percentage points. [1] https://arxiv.org/html/2406.02061v1 https://arxiv.org/html/2406.02061v1 [2] https://news.ycombinator.com/item?id=40811329 https://news.ycombinator.com/item?id=40811329

s-macke 2y ago
Here is the larger discussion about the Alice in Wonderland Paper on Hacker News. https://news.ycombinator.com/item?id=40585039 https://news.ycombinator.com/item?id=40585039
apsec112 2y ago
Both Claude-3.5 and o1-preview nail this problem "Let's think through this step-by-step: 1. Alice has 3 brothers 2. Alice has 2 sisters 3. We need to find out how many sisters Alice's brother has The key here is to realize that Alice's brothers would have the same sisters as Alice, except they would also count Alice as their sister. So, Alice's brothers would have: - The 2 sisters Alice has - Plus Alice herself as a sister Therefore, Alice's brothers have 3 sisters in total."
- s-macke 2y ago
  And here lies the exact issue. Single tests don’t provide any meaningful insights. You need to perform this test at least twenty times in separate chat windows or via the API to obtain meaningful statistics. For the "Alice in Wonderland" paper, neither Claude-3.5 nor o1-preview was available at that time. But I have tested them as well a few weeks ago with the issue translated into German, achieving also a 100% success rate with both models. However, when I add irrelevant information (My mother ...), Claude's success rate drops to 85%: "My mother has a sister called Alice. Alice has 2 sisters and 1 brother. How many sisters does Alice's brother have?"

yk 2y ago

I test llms actually similar. For example there is a well known logic puzzle were a farmer tries to cross a river with a cabbage a goat and a wolf. Llms can solve that since at least GPT-2, however if we replace the wolf with a cow, gpt-o does correctly infer the rules of the puzzle but can't solve it.

voidUpdate 2y ago
I'm scared of the cows around you if they eat goats
- Manabu-eo 2y ago
  I think their point is that cows don't eat goats, unlike wolves, and that causes the LLMs to answer it wrong.
getoffmyyawn 2y ago
I've found that the River Crossing puzzle is a great way to show how LLMs break down. For example, I tested Gemini with several versions of the puzzle that are easy to solve because they don't have the restrictions such as the farmer's boat only being able to carry one passenger/item at a time. Ask this version, "A farmer has a spouse, chicken, cabbage, and baby with them. The farmer needs to get them all across the river in their boat. What is the best way to do it?" In my tests the LLMs nearly always assume that the boat has a carry-restriction and they come up with wild solutions involving multiple trips.
chasd00 2y ago
What happens if you sit down and invent a logic game that is brand new and has never been documented before anywhere then ask an LLM to solve it? That, to a layman like me, seems like a good way to measure reasoning in AI.
- jprete 2y ago
  I think the problem is inventing new structures for logic games. The shape of the problem ideally would be different than any existing puzzle, and that's hard. If a person can look at it and say "oh, that's just the sheep-wolf-cabbage/liar-and-truthteller/etc. problem with extra features" then it's not an ideal test because it can be pattern-matched.

criddell 2y ago

It would be interesting if this kind of work could ever be extended to show the limitations of mathematical reasoning in animals and humans. For example, just as a dog will never understand a fourier transform, there are likely ideas that humans cannot understand. If we know what our limits are, I wonder if we could build machines that can reason in ways we aren't capable of?

myrmidon 2y ago
I think it is a naive assumption that such a limitation even exists ("exists" in a sense that it is actually useful, by being consistent and somewhat simple to describe). We investigated similar ideas for language (=> Noam Chomsky), where we tried to draw clear, formalized limits for understanding (to show e.g. how human capabilities contrast with animals). The whole approach failed completely and irredeemably (personal opinion), but researching it was far from useless to be fair.
- r2_pilot 2y ago
  As the human brain is finitely bounded in space and time, any idea that can't be compressed or represented by condensing notation, which is "larger" than the 100B cells+100T synapses can represent, or whose integration into said human's brain would take longer than 150 years, would be considered unable to be contemplated by a normal human.
  - klabb3 2y ago
    Yes but we overcome. We can do absolutely insane things like just large prime number testing, because of reasoning + tool use. Humans invent tools and wield them. Whether it's pen & paper to extend our memory, a horse to become stronger, a calculator to speed up our thinking or an airplane to literally fly, the tools we wield become extensions of our agency and control. A lonely human without knowledge sharing or tools isn’t that much more capable in their lifetime than the smartest animals. When we talk about human ability colloquially, we’re generally talking about what we can do with access to our human heritage, civilization, safety and access to materials and tools. Pattern matching against something others have already done is great but this is shared with at the very least all mammals to some extent. Pushing the boundaries of our species forward over time is a different game. Or at least, it seems to be… It certainly seems like we’ve found the holy grail of pattern matching (system 1 thinking), which is an insane leap! But what about system 2? The million dollar question is what the hell is the topology of that pre-frontal cortex thinking machine? Is it just more pattern matching but against different patterns? Or is it completely and qualitatively different? And if so, is it more or less hard? To me, following the debate is just watching one bad prediction after another, (including my own of course). We just don't know how it works. Not you or me, not Sam Altman in full though-leading leather jacket uniform, or even our top neuro-scientists.

beardyw 2y ago

I honestly can't see why LLMs should be good at this sort of thing. I am convinced you need a completely different approach. At the very least you mostly only want one completely correct result. Good luck getting current models to do that.

golol 2y ago
I'm a math phd student at the moment and I regularly use o1 to try some quick calculations I don't feel like doing. While I feel like GPT-4o is so distilled that it just tries to know the answer from memory, o1 actually works with what you gave it and tries to calculate. It's can be quite useful.
- banditelol 2y ago
  I'm curious what kind of quick calculation do you usually use llm for? Edited for clarity
  - golol 2y ago
    Just earlier today I wanted to check if exp(inx) is an orthonormal basis on L^2((0, 1)) or if it needs normalization. This is an extremely trivial one though. Less trivially I had an issue where a paper claimed that a certain white noise, a random series which diverges in a certain Hilbert space, is actually convergent in some L^infinity type space. I had tried to use a Sobolev embedding but that was too crude so it didn't work. o1 correctly realized that you have to use the decay of the L^infinity norm of the eigenbasis, a technique which I had used before but just didn't think of in the moment. It also gave me the eigenbasis and checked that everything works (again, standard but takes a while to find in YOUR setting). I wasn't sure about the normalization so again I asked it to calculate the integral. This kind of adaptation to your specific setting instead of just spitting out memorized answers in commonn settings is what makes o1 useful for me. Now again, it is often wrong, but if I am completely clueless I like to watch it attempt things and I can get inspiration from that. That's much more useful than seeing a confident wrong answer like 4o would give it.
deleted 2y ago

apsec112 2y ago

()

ilaksh 2y ago
That makes the whole conclusion obviously false. I don't really understand why, but I think we are going to see total denial from a significant percentage of the population all the way up to and past the point where many average mathematicians and software engineers cannot in any way compete with AI. We already are reportedly getting pretty close with o1 (not o1-preview). There are also new paradigms for machine learning and hardware in the pipeline that will continue to provide orders of magnitude performance gains and new capabilities in the next 5-10 years. Many people still claim that "self driving cars don't exist", in so many words, even though they are deployed in multiple cities.
- sottol 2y ago
  > Many people still claim that "self driving cars don't exist", in so many words, even though they are deployed in multiple cities. But just look at the predictions of that time - cities will change, ... and so on. Sure, we have self-driving cars but the reality looks very different (and a lot more like the past!) than the pundits and futurists imagined! I'm not sure anyone will make their billions of dollars investmented back within even 20 years. Just two random examples from ~10 years ago (2013-2016), you can google many more of that time. * "Ford Targets Fully Autonomous Vehicle for Ride Sharing in 2021; Invests in New Tech Companies, Doubles Silicon Valley Team" [1] * "Disruptions: How Driverless Cars Could Reshape Cities" [2] [1] https://media.ford.com/content/fordmedia/fna/us/en/news/2016/08/16/ford-targets-fully-autonomous-vehicle-for-ride-sharing-in-2021.html https://media.ford.com/content/fordmedia/fna/us/en/news/2016... [2] https://archive.nytimes.com/bits.blogs.nytimes.com/2013/07/07/disruptions-how-driverless-cars-could-reshape-cities/?smid=tw-share https://archive.nytimes.com/bits.blogs.nytimes.com/2013/07/0... [3] https://www.gensler.com/dialogue/30/the-game-changer-for-cities-driverless-cars https://www.gensler.com/dialogue/30/the-game-changer-for-cit...

dev1ycan 2y ago

I don't understand the idiocracy we live in, it is beyond obvious not just that the stock market is a bubble but ESPECIALLY the AI related stocks are a massive bubble, when it pops, and it will, it is going to be very very ugly, yet people keep pouring in, as Sabine said it, it's starting to look like particle physics where they keep asking for bigger colliders, just because you have a bigger collider, if your methodology is flawed you aren't gonna get any more significant returns. Eventually they will run out of exponential cash to pour in, and investors will start asking questions, stocks are already valued at 60x+ their earnings, whenever it pops you don't want to be the one who bought the top. Guess it's still gonna take a while more for the layman to realize the issues with LLMs, but it'll happen.

Workaccount2 2y ago
>if your methodology is flawed you aren't gonna get any more significant returns. The problem with this statement is that predictions made about scaling 5 years ago have held true[1]. We keep adding parameters, adding compute, and the models keep getting more capable. The flaws of LLM's from 2024 are not what is relevant. Just like the flaws of LLMs from 2021 were not relevant. What is relevant is the rate of change, and the lack of evidence that things won't continue on this steep incline. Especially if you consider that GPT4 was sort of a preview model that motivated big money to make ungodly investments to see how far we can push this. Those models will start to show up over the next 2 years. If they break the trend and the scaling flops, then I think a lot of air is gonna blow out of the bubble. [1]https://arxiv.org/pdf/2001.08361 https://arxiv.org/pdf/2001.08361
- dev1ycan 2y ago
  They are very literally asking for trillions and even nuclear powered data centers, pretty sure we've gotten to the point where it's not sustainable.
  - Workaccount2 2y ago
    Those are roadmap items being asked for, but the next gen models are already in training. If they keep moving along the same trend line, like all the previous models have, then they probably will be able to find the investors for the next next gen. Even if it's a few trillion dollars and a few nuclear power plants. This doesn't even factor in the tech inertia. We could stop making new models today, and it would probably be 4-5 years before integration slowed down. Google still hasn't even put Gemini in their home speakers.

bob1029 2y ago

> we investigate the fragility of mathematical reasoning in these models and demonstrate that their performance significantly deteriorates as the number of clauses in a question increases. We hypothesize that this decline is due to the fact that current LLMs are not capable of genuine logical reasoning I'd offer a simpler explanation: Tokenization. If you tokenize "12345 * 27271" you will get the following: "123", "45", " *", " ", "272", "71" The statistical likelihood that any of these tokens predicts any of the others is completely meaningless in the context of simple arithmetic. You can argue that this is where tool use comes in (and I would be inclined to agree), but I don't think this bodes well for "genuine logical reasoning".

m3kw9 2y ago
The llm will know 123 and 45 is a contiguious number just like how humans can tell if you say 123 and then a slight pause 45 as a single number
- soulofmischief 2y ago
  I think that as long as the attention mechanism has been trained on each possible numerical token enough, this is true. But if a particular token is underrepresented, it could potentially cause inaccuracies.
- TZubiri 2y ago
  It's just so dissonant to me that the tokens in mathematics are the digits, and not bundles of digits. The idea of tokenization makes sense for taking the power off letters, it provides language agnosticism. But for maths, it doesn't seem appropriate. I wonder what the effect of forcing tokenization for each separate digit be.
  - taeric 2y ago
    This reminds me of the riddle of someone buying the numerals to put their address on their house. When you are looking at text, the point is all you have are the characters/symbols/tokens/whatever you want to call them. You can't really shepherd some over to their numeric value while leaving some at their token value. Unless you want to cause other issues when it comes time to reason about them later. I'd hazard that the majority of numbers in most text are not such that they should be converted to a number, per se. Consider addresses, postal codes, phone numbers, ... ok, I may have run out of things to consider. :D

singularity2001 2y ago

If the argument is that LLMs are bad at reasoning because they are easily distractible and the results vary with modifications in the question, one should be reminded of the consistency and distractability of humans.

zeroonetwothree 2y ago
Why? LLMs are supposedly better than humans (as many comments claim in this thread).
deleted 2y ago
[deleted]
riku_iki 2y ago
Trained human can tell if distracted: "I am distracted and can't figure out answer", while LLM will confidently gives you wrong answer, which makes whole results not reliable.

woopwoop 2y ago

I'm curious about what happens with the no-op dataset if you include in the prompt that the questions may contain irrelevant information.

parsimo2010 2y ago

I won't take a strong stance on whether or not LLMs actually do reasoning, but I will say that this decrease in performance is similar to what I see in college freshmen (I'm currently teaching a calculus course in which almost half of the students took AP calc in high school). They perform well on simple questions. Requiring students to chain multiple steps together, even simple steps, results in decreased accuracy and higher variance (I have no data on whether this decrease is linear or not, as the paper assumes that the decrease should be linear with the number of steps). We see similar results with adding unrelated statements into a problem- many students are trained to make sure to use all given information in solving a problem- if you leave out something that the instructor gives you, then you probably forgot to do something important. So while I don't take a stance on what an LLM does should be considered reasoning, I do think that SOTA LLMs like GPT-4o perform about as good as high school graduates in America with average intelligence. In other words, average Americans exhibit similar limitations on their reasoning as good LLMs. Which on the one hand is a little disappointing to me in terms of the human performance but is kind of good news for LLMs- they aren't doing graduate-level research but they are already capable of helping a large portion of the population.

skydhash 2y ago
Not to disparage American school system (my country’s is worse) but it’s very much easy mode. I know that not everyone is suited to academic excellence, but it’s definitely easier to learn when young. I do believe too much hand holding actively harm learning.
- BriggyDwiggs42 2y ago
  I don’t think the issue with American schools is that there’s too much hand holding. If anything, it’s the opposite; teachers at drastically underfunded schools don’t have any time to help the students of their 50 person class through the confused curriculum.
  - skydhash 2y ago
    Here, we have to go through 4 state exams just to get to university. The first when you’re 11, the second at 14, then two consecutive ones at 17 and 18. There’s a national curriculum that the exams will be about, although the schools are free to add to it. So however you feel about the school or the teacher, you have to master the subjects enough to go through. And that means paying attention in class, cram before it, or hoping you can cheat. We have our own problem too, but the consensus among all the people I know that have moved to the US is that classes are easy there. Not a bad thing per se (better explanation, better understanding instead of rote memorizing).

woopwoop 2y ago

This paper, among other things, shows that LLMs have dramatically worse performance on basic algebra questions when you add in irrelevant information. The examples are things like "John picked 43 kiwis on Monday, 24 kiwis on Tuesday. On Wednesday, 5 of the kiwis he picked were smaller than usual. Altogether, on Monday, Tuesday, and Wednesday, John picked 87 kiwis. How many kiwis did John pick on Wednesday?" In this question, the remark about some of the kiwis on Wednesday being small is irrelevant, but adding things like this reduces performance on a popular benchmark from 95% to 77% for GPT-4o, for example. I don't find this very impressive. Forget LLMs for a second. Let's say _you_ read a question of that kind with some bit of irrelevant information. There are two possibilities you have to consider: the question may as well have excluded the irrelevant information, or the question was miswritten and the irrelevant information was meant to be relevant. The latter is a perfectly live possibility, and I don't think it's a dramatic failure to assume that this is correct. I have to confess that when I read some people's LLM gotcha questions, where they take some popular logic puzzle and invert things, I think I would get them "wrong" too. And not wrong because I don't understand the question, but wrong because with no context I'd just assume the inversion was a typo.

jfrbfbreudh 2y ago
I think it’s an important result because filtering signal from noise is just as, if not more, important than forming conclusions from signal.
swatcoder 2y ago
Real discourse has tons of irrelevant information for all sorts of reasons. There are some contexts, academic or professional, where questions are posed carefully and specifically, but these are narrow contexts. A useful general purpose assistant needs to be able to find what's relevant among what's irrelevant. Excellence at just solving math problems that are especially well specified can be a useful domain assistant (no small win!), but is not the same thing. That said, if you've got a hundred billion dollars betting on your AI project achieving AGI, you benefit a lot by conflating those contexts. In that case, grinding on formal SAT, LSAT, GRE, etc problems amounts to tuning for microbenchmarks rather than real world use cases.

resters 2y ago

I think it's obvious that LLMs will be able to do "reasoning" far better than humans. We must separate our notion of what is remarkably human. Rarely is it the reasoning, it's the intuition that a logical path exists -- for example a mathematical proof that draws from separate sub-disciplines of mathematics, etc. Consider that in a LLM, language inputs are tokenized and fed as inputs into the neural network, and connections in the network create output sequences that are not just syntactically correct (trivial) or form semantically plausible sentences (early transformers did this). LLM output sequences follow the deep patterns of language which include sometjhing that resembles reasoning as the model has learnt from its training data. LLMs seem to fall short because they often fail at truly abstract reasoning tasks that humans find easy. If trained properly, LLMs can develop advanced representations of logical systems that will surely outpace what humans can do in terms of raw reasoning. However, human mathematicians have not even unified around constructive mathematics as a must for the study of mathematics. This reveals that even highly evolved mathematical disciplines rely on objects whose characteristics do not lend themselves to full logical scrutiny and are in a way socially constructed and effectively hard to audit. While notation in mathematics is incredible technology it is also a highly limiting factor that suffers major tradeoffs. Humans struggle to invent new notation fast enough and to discard outdated notation fast enough. If we do see an AI-powered boom in mathematics, I suspect our notion of notation and the fluidity we demand from it will change dramatically.

sottol 2y ago
> If trained properly, LLMs can develop advanced representations of logical systems that will surely outpace what humans can do in terms of raw reasoning. We have already trained the LLMs on most of the human knowledge base (so like 4-5000 years?) - imo training data will become a problem and will soon be more expensive than compute. Sure, you can work around some of this using synthetic training data but I personally would not count on general-purpose LLMs (especially LLMs aka transformer models) developing super-human representations of logical systems anytime soon.
- resters 2y ago
  I don't disagree, however I'm optimistic because most of the current reasoning "ability" of LLMs comes from the accidental reasoning embedded in language patterns. For example, the prompt completion: "The mouse has a unique digestive system compared to other rodents, however the sparrow" on GPT-4o is "exhibits a highly specialized digestive system adapted for rapid processing of food, particularly seeds and insects, through structures like the crop and gizzard, which are not found in rodents." Claude 3.5 completes it as "has a completely different digestive anatomy as a bird. Birds like sparrows have adaptations for flight, including a lightweight skeletal system and a specialized digestive tract. Unlike mice, sparrows have a crop for storing food, a gizzard for grinding it, and generally shorter intestines to reduce weight. They also lack teeth, instead using their beak to manipulate food." What appears to be a thoughtful contrast is merely a language pattern. Similarly, a prompt like "Assume -B, A->B. Under what circumstances is B true?" will simply follow the gradient to return output that is likely correct. Prompts like "what is 2+2" fail only because nobody bothers to write about it so simple arithmetic was not in the training data. However the way that multi-modal LLMs handle images is inspiring as it effectively converts from the visual domain into the sequential token domain. The same could be done for symbolic systems, etc.

trehalose 2y ago

I see a lot of discussion about irrelevant clauses tripping up the LLMs and why that does or doesn't matter. To me, what's far more damning is this: > Specifically, the performance of all models declines when only the numerical values in the question are altered in the GSM-Symbolic benchmark. This seems like irrefutable evidence of overfitting, that in the best case scenario is epidemic among current LLMs (and in the worst case interpretation, is covering up fundamental inabilities to learn mathematical reasoning from the training data).

gradientsrneat 2y ago

Could this be Goodhart's Law in action? AI tools like to showcase benchmarks in bar graphs to show how well they perform compared to other models. Maybe the benchmark Qs/As snuck into training sets accidentally. Is it still Goodhart's Law if it's unintentional? Daniel Lemire has blogged about being impressed with how well the LLM answers his CS problem questions. I was impressed too. Not sure where the line of competence lies.

eigenform 2y ago

The difference is that, if we are solving a math problem together, you and I [explicitly or implicitly] can come to an agreement over the context and decide to restrict our use of language with certain rules. The utility behind our conversation [generally] rests on those rules! An LLM is very good at recovering rules, but being good at pattern recognition is not the same thing as being good at unambiguously following rules in the appropriate context. edit: Natural language is far from an efficient/sufficient/necessary intermediate representation for doing math, just ask any general-purpose computer. Sometimes, it's worth "putting rules in stone," and it seems unreasonable to believe that there is always an unambiguous rule for this that you can mechanically recover from a corpus of language use.

deleted 2y ago

[deleted]

dang 2y ago

Related ongoing thread: LLMs don't do formal reasoning - https://news.ycombinator.com/item?id=41812523 https://news.ycombinator.com/item?id=41812523 - Oct 2024 (70 comments)

jumploops 2y ago

> Overall, while o1-preview and o1-mini exhibit significantly stronger results compared to current open models—potentially due to improved training data and post-training procedures—they still share similar limitations with the open models. tl;dr - the best open model dropped from 89.7% on GSM8K(full) to 30% on Symbolic-NoOp, while o1-preview dropped from 94.9% to 77.4%, respectively. I think all this paper shows is that LLMs need space to "think" outside of their inference layer, (for the current architectures at least). It's similar to the "draw a room, but DO NOT put an elephant in the corner" prompts that people were using with image models. This is something that practitioners have been doing for awhile (via CoT, ToT, etc.) and the whole rationale behind OpenAI's newly launched o1-series "model." There's another post that says this paper proves LLMs can't be used to build "reliable agents" -- which doesn't appear to be true when you look at o1's stellar performance here.

data_maan 2y ago
Can you send a paper regarding that LLMs can build "reliable agents"?

qwerty456127 2y ago

Can't al LLM just detect a mathematical reasoning task then produce a formula (not even display it in the production mode) to invoke on an external service engineered for formal logical and mathematical computations?

aithrowawaycomm 2y ago
In many of these examples it produces the wrong formula because it misunderstands the word problem, so a computer algebra system wouldn't help - garbage in, garbage out. The problem here is more serious than mathematics: the quantitative reasoning itself is highly unreliable.

ak_111 2y ago

As an outsider can anyone enlighten me how this squares with the news that models that adapt similar LLM architecture can obtain silver medal in mathematical olympiad?

lionkor 2y ago
careful statistical massaging, maybe. would you pick only winning results and only present favorable, massaged results if it got you 150+B USD of worth?

throwaway918299 2y ago

limitations of mathematical reasoning? They have none. Literally zero. That’s the limit. Thank you for reading my paper.

codelion 2y ago

This is surprising to only those that have not worked in formal reasoning. Yes, LLMs cannot do true logical reasoning in a formal sense, you can do better with an SMT solver. But it is also true that you can solve a lot of logical problems by just applying “reasoning steps” from the training data, specially when your training data is the entirety of written content ever produced. Both of these can be true at the same time it is not a contradiction just an interesting dichotomy.

jgord 2y ago

I propose 'gords rule' : "any sufficiently advanced LLM will learn the laws of logic, the principles of scientific method, and Reinforcement Learning" until that happens .. I think RL startups focused on real problems are much undervalued : https://quantblog.wordpress.com/2024/10/11/llm-hype-means-there-are-bargains-in-ml-deep-tech-startups-doing-real-things/ https://quantblog.wordpress.com/2024/10/11/llm-hype-means-th...

uptownfunk 2y ago

The very fundamental problem with LLM is there is no guarantee on any of the reasoning it gives you without a human there to give a thumbs up. They are working on solving this (alpha proof, lean agent etc) but getting this to run at inference time in an optimized way is what I would call one of the millenial prize problems of AI which will lead to a quantum leap in the path towards the singularity.

Animats 2y ago

It's an expected result. Whatever happened with that result which found some representation of the state of a game inside an LLM? That indicated some degree of model-building. Haven't heard about that again/

i007 2y ago

LLMs are designed to carry out "associative reasoning" which captures logic based on recognition and recall of compositional patterns learned during training. Having said that, we can still get semantically and logically idempotent output that makes sense but with lots of work outside of the LLM, which contrasts with the current hyper focus on the LLM itself as the be all and end all. It is just one component in what ought to be a larger and more involved system for reasoning. Look at what we were able to accomplish here for Legal AI, not so mathematical logic per se but mimicking (capturing) axiomatic logic in the legal domain: https://www.youtube.com/watch?v=_9Galw9-Z3Q https://www.youtube.com/watch?v=_9Galw9-Z3Q marc at sunami dot ai

teleforce 2y ago

In terms of usefulness and realistic implementation mathematical reasoning is the next frontier of LLM not autonomous level 5 driving or AGI. More research fund and investment are much better spent on the former rather than the latter but apparently it seems that the reverse situation is the case.

gtsop 2y ago

LLMs are inherently emulators of digitaly imprinted artifacts of human consciousness. When people trully grasp what this means they will stop being buffled by the fact that LLMs performance deteriorate when novelty of the task increases. EDIT: Had there been an ounce of actual true reasoning emerging in LLMs, openai would have been running this thing privatly 24/7 to produce new science and capture pattents that would give them economic dominance. Not trying to sell tokens to us all.

K0balt 2y ago

Trying to solve (much less explore) mathematics using probabilistic next-token prediction seems like the really long way around, especially when we have pretty good deterministic tools available for our use. I don’t know why anyone would bother doing anything besides working on the correct manipulation of tools. Brains have various structures that have distinct architectures. I don’t see any indication that the best way forward is to try to shoehorn everything into a single computational paradigm. It’s like trying to make a flying submarine car. It might technically be possible, but it might not be worth the trouble, and it’s unlikely to result in a vehicle that works excellently in any of its environments.

bubble12345 2y ago

Can LLMs even do addition, with say 20+ digit numbers? Multiplication?