12 ms·
I am kind of amazed at how many commenters respond to this result by confidently asserting that LLMs will never generate 'truly novel' ideas or problem solution
by qnleigh 7mo ago
I am kind of amazed at how many commenters respond to this result by confidently asserting that LLMs will never generate 'truly novel' ideas or problem solutions.
> AI is a remixer; it remixes all known ideas together. It won't come up with new ideas
> it's not because the model is figuring out something new
> LLMs will NEVER be able to do that, because it doesn't exist
It's not enough to say 'it will never be able to do X because it's not in the training data,' because we have countless counterexamples to this statement (e.g. 167,383 * 426,397 = 71,371,609,051, or the above announcement). You need to say why it can do some novel tasks but could never do others. And it should be clear why this post or others like it don't contradict your argument.
If you have been making these kinds of arguments against LLMs and acknowledge that novelty lies on a continuum, I am really curious why you draw the line where you do. And most importantly, what evidence would change your mind?
- tornikeo 7mo agoBeliefs are not rooted in facts. Beliefs are a part of you, and people aren't all that happy to say "this LLM is better than me"
- ChrisGreenHeur 7mo agoIt's not possible to know something without believing it to be true. https://en.wikipedia.org/wiki/Belief#/media/File:Classical_definition_of_Kno.svg https://en.wikipedia.org/wiki/Belief#/media/File:Classical_d...
- bilekas 7mo agoThis is objectively wrong. If that was the case every scientist performing a test would have always had their expectations and beliefs proven true. If you're trying to disprove something also because you believe it to be wrong you would never be proven wrong.
- ChrisGreenHeur 7mo agore-read your post - it's just a bunch of nonsense, no actual reasoning in there
- benterix 7mo agoI'm very happy to say calculators are far better than me in calculations (to a given precision). I'm happy to admit computers are so much better than me in so many aspects. And I have problem saying LLMs are very helpful tools able to generate output so much better than mine in almost every field of knowledge. Yet, whenever I ask it to do something novel or creative, it falls very short. But humans are ingenious beasts and I'm sure or later they will design an architecture able to be creative - I just doubt it will be Transformer-based, given the results so far.
- stavros 7mo agoBut the question isn't whether you can get LLMs to do something novel, it's whether anyone can get them to do something novel. Apparently someone can, and the fact that you can't doesn't mean LLMs aren't good for that.
- tovej 7mo agoNovel is a tricky word. In this case, the LLM produced a python program that was similar to other programs in its corpus, and this oython program generated examples of hypergraphs that hadn't been seen before. That's a new result, but I don't know about novel. The technique was the same as earlier work in this vein. And it seems like not much computational power was needed at all. (The article mentions that an undergrad left a laptop running overnight to produce one of the previous results, that's absolute peanuts when compared to most computational research).
- deleted 7mo ago[deleted]
- jacquesm 7mo ago> e.g. 167,383 * 426,397 = 71,371,609,051 They may be wrong, but so are you.
- tourist2d 7mo ago[dead]
- KellyCriterion 7mo agoNo, its correct: https://www.google.com/search?q=167383+*+426397 https://www.google.com/search?q=167383+*+426397
- jacquesm 7mo agoYou missed the point.
- karmakurtisaani 7mo agoI missed it too, care to explain?
- KellyCriterion 7mo agoNot sure what you mean? Can you elaborate?
- swingboy 7mo agoYou could have just checked the math yourself, you know.
- bdbdbdb 7mo agoI guess when it can't be tripped up by simple things like multiplying numbers, counting to 100 sequentially or counting letters in a string without writing a python program, then I might believe it. Also no matter how many math problems it solves it still gets lost in a codebase
- tourist2d 7mo ago[dead]
- anal_reactor 7mo agoArguments like "but AI cannot reliably multiply numbers" fundamentally misunderstand how AI works. AI cannot do basic math not because AI is stupid, but because basic math is an inherently difficult task for otherwise smart AI. Lots of human adults can do complex abstract thinking but when you ask them to count it's "one... two... three... five... wait I got lost".
- datsci_est_2015 7mo ago> fundamentally misunderstand how AI works Who does fundamentally understand how LLMs work? Many claims flying around these days, all backed by some of the largest investments ever collectively made by humans. Lots of money to be lost because of fundamental misunderstandings. Personally, I find that AI influencers conveniently brush away any evidence (like inability to perform basic arithmetic) about how LLMs fundamentally work as something that should be ignored in favor of results like TFA. Do LLMs have utility? Undoubtedly. But it’s a giant red flag for me that their fundamental limitations, of which there are many, are verboten to be spoken about.
- stavros 7mo agoYou're not doing yourself a favor when you point out "but they can't do arithmetic!" as if anyone says otherwise. Yes, we all know they can't do arithmetic, and that's just how they work. I feel like I'm saying "this hammer is so cool, it's made driving nails a breeze" and people go "but it can't screw screws in! Why won't anyone talk about that! Hammers really aren't all they're cracked up to be".
- PUSH_AX 7mo agoThe hardest part about any creativity is hiding your influences
- busyant 7mo agoThis is poetry.
- deleted 7mo ago[deleted]
- ekjhgkejhgk 7mo agoYes! I call these the "it's just a stochastic parrot" crowd. Ironically, they are the stochastic parrots, because they're confidently repeating something that they read somehwere and haven't examined critically.
- louiereederson 7mo agoThat would not be stochastic, just parroting
- ekjhgkejhgk 7mo agoIt's not deterministic therefore it's stochastic.
- bluecalm 7mo ago>>AI is a remixer; it remixes all known ideas together. It won't come up with new ideas I always found this argument very weak. There isn't that much truly new anyway. Creativity is often about mixing old ideas. Computers can do that faster than humans if they have a good framework. Especially with something as simple as math - limited set of formal rules and easy to verify results - I find a belief computers won't beat humans at it to be very naive.
- qsera 7mo agoIt is like not trusting someone who attained highest score in some exam by by-hearting the whole text book, to do the corresponding job. Not very hard to understand.
- LatencyKills 7mo agoI've been working on a utility that lets me "see through" app windows on macOS [1] (I was a dev on Apple's Xcode team and have a strong understanding of how to do this efficiently using private APIs). I wondered how Claude Code would approach the problem. I fully expected it to do something most human engineers would do: brute-force with ScreenCaptureKit. It almost instantly figured out that it didn't have to "see through" anything and (correctly) dismissed ScreenCaptureKit due to the performance overhead. This obviously isn't a "frontier" type problem, but I was impressed that it came up with a novel solution. [1]: https://imgur.com/a/gWTGGYa https://imgur.com/a/gWTGGYa
- stavros 7mo agoWhat was the solution?
- LatencyKills 7mo agoWell, I'm not going to share either solution as this is actually a pretty useful utility that I plan on releasing, but the short answer is: 1) don't use ScreenCaptureKit, and 2) take advantage of what CGWindowListCreateImage() offers through the content server. This is a simple IPC mechanism that does not trigger all the SKC limitations (i.e., no multi-space or multi-desktop support). In fact, when using SKC, the user cannot even minimize the "watched" window. Claude realized those issues right from the start. One of the trickiest parts is tracking the window content while the window is moving - the content server doesn't, natively, provide that information.
- stavros 7mo agoHuh, Claude one-shotted it out of a single message from me. Man, LLMs have gotten good.
- LatencyKills 7mo agoNo it didn't. Like I said... it may have gotten something that worked but there is no way Claude got it to work while supporting multi-spaces, multi-desktops, and using under 2% cpu utilization. My solution can display app window content even when those windows are minimized, which is not something the content server supports. My point was that Claude realized all the SKC problems and came up with a solution that 99% of macOS devs wouldn't even know existed.
- SequoiaHope 7mo agoMost inventions are an interpolation of three existing ideas. These systems are very good at that.
- mikkupikku 7mo agoMy take as well. Furthermore, most innovations come relatively shortly after their technological prerequisites have been met, so that suggests the "novelty space" that humans generally explore is a relatively narrow band around the current frontier. Just as humans can search through this space, so too should machines be capable of it. It's not an infinitely unbounded search which humans are guided through by some manner of mystic soul or other supernatural forces.
- fsflover 7mo agoI can't even find a good example of an invention that is not an interpolation.
- franktankbank 7mo agoThe inclined plane, the wheel, shall I keep going?
- Terretta 7mo agoStand on a fallen log on a hillside and you'll interpolate pretty hard.
- ryandvm 7mo agoIndeed. Every time someone complains that LLMs can't come up with anything new, I'm assaulted with the depressing remembrance that neither do I.
- veltas 7mo agoDo we know for a fact that LLMs aren't now configured to pass simple arithmetic like this in a simpler calculator, to add illusion of actual insight?
- GaggiX 7mo agoYou can train a LLM on just multiplication and test it on ones it has never seen before, it's nothing particularly magical.
- veltas 7mo agoIt's not 'magic' though but previously LLMs have performed very badly on longer multiplication, 'insight' is the wrong word but I'm saying maybe they're not wildly better at this calculation... maybe they are just optimising these well known jagged edges.
- bananamogul 7mo agoThe major AIs have access to all sorts of tools, including a math library. I thought this was well-known. There's no "illusion of actual insight" - they're just "using a calculator" (in the sense that they call a math library when needed). AIs are not just LLMs.
- Yizahi 7mo agoLLMs can generate anything by design. LLMs can't understand what they are generating so it may be true, it may be wrong, it may be novel or it may be known thing. It doesn't discern between them, just looks for the best statistical fit. The core of the issue lies in our human language and our human assumptions. We humans have implicitly assigned phrases "truly novel" and "solving unsolved math problem" a certain meaning in our heads. Some of us at least, think that truly novel means something truly novel and important, something significant. Like, I don't know, finding a high temperature superconductor formula or creating a new drug etc. Something which involver real intelligent thinking and not randomizing possible solutions until one lands. But formally there can be a truly novel way to pack the most computer cables in a drawer, or truly novel way to tie shoelaces, or indeed a truly novel way to solve some arbitrary math equation with an enormous numbers. Which a formally novel things, but we really never needed any of that and so relegated these "issues" to a deepest backlog possible. Utilizing LLMs we can scour for the solutions to many such problems, but they are not that impressive in the first place.
- logicprog 7mo agoIf LLMs can come up with formerly truly novel solutions to things, and you have a verification loop to ensure that they are actual proper solutions, I don't understand why you think they could never come up with solutions to impressive problems, especially considering the thread we are literally on right now? That seems like a pure assertion at this point that they will always be limited to coming up with truly novel solutions to uninteresting problems.
- eru 7mo ago"Truly novel" is fast becoming a True Scotsman.
- getnormality 7mo agoNo True Novelty, No True Understanding, etc. The problem with these bromides is not that they're wrong, it's that they're not even wrong. They're predictive nulls. What observable differences can we expect between an entity with True Understanding and an entity without True Understanding? It's a theological question, not a scientific one. I'm not an AI booster by any means, but I do strongly prefer we address the question of AI agent intelligence scientifically rather than theologically.
- cyanydeez 7mo agoWhen I read through what they're doing? It sure doesn't sound like it's generating something new as people typically think of it. The link, they provide a very well defined problem and they just loop through it. I think you're arguing with semantics.
- energy123 7mo ago> 67,383 * 426,397 = 71,371,609,051 ... You need to say why it can do some novel tasks but could never do others. Model interpretability gives us the answers. The reason LLMs can (almost) do new multiplication tasks is because it saw many multiplication problems in its training data, and it was cheaper to learn the compressed/abstract multiplication strategies and encode them as circuits in the network, rather than memorize the times tables up to some large N. This gives it the ability to approximate multiplication problems it hasn't seen before.
- adam_arthur 7mo agoWhich is exactly how humans learn many things too. E.g. observing a game being played to form an understanding of the rules, rather than reading the rulebook Or: Observing language as a baby. Suddenly you can speak grammatically correctly even if you can't explain the grammar rules.
- qnleigh 7mo agoYup, I agree with this. So based on this, where do you draw the line between what will be possible and what will not be possible?
- bananamogul 7mo agoWhy are we reducing AIs to LLMs? Claude, OpenAI, etc.'s AIs are not just LLMs. If you ask it to multiply something, it's going to call a math library. Go feed it a thousand arithmetic problems and it'll get them 100% right. The major AIs are a lot more than just LLMs. They have access to all sorts of systems they can call on. They can write code and execute it to get answers. Etc.
- nearbuy 7mo ago> This gives it the ability to approximate multiplication problems it hasn't seen before. More than approximate. It straight up knows the algorithms and will do arbitrarily long multiplications correctly. (Within reason. Obviously it couldn't do a multiplication so large the reasoning tokens would exceed its context window.) Having ChatGPT 5.4 do 1566168165163321561 * 115616131811365737 without tools, after multiplying out a lot of coefficients, it eventually answered 181074305022287409585376614708755457, which is correct. At this point, it's less misleading to say it knows the algorithm.
- staticassertion 7mo agoI think "novel" is ill defined here, perhaps. LLMs do appear to be poor general reasoners[0], and it's unclear if they'll improve here. It would be unintuitive for them to be good at this, given that we know exactly how they're implemented - by looking at text and then building a statistical model to predict the next token. From this, if we wanted to commit to LLMs having generalizable knowledge, we'd have to assume something like "general reasoning is an emergent property of statistical token generation", which I'm not totally against but I think that's something that warrants a good deal of evidence. A single math problem being solved just isn't rising to that level of evidence for me. I think it is more on you to: 1. Provide a theory for how LLMs can do things that seemingly go beyond expectations based on their implementation (for example, saying that certain properties of reasoning are emergent or reduce to statistical constructs). 2. Provide evidence that supports your theory and ideally can not be just as well accounted for another theory. I'm not sure if an LLM will never generate "novel" content because I'm not sure that "novel" is well defined. If novel means "new", of course they generate new content. If novel means "impressive", well I'm certainly impressed. If "novel" means "does not follow directly from what they were trained on", well I'm still skeptical of that. Even in this case, are we sure that the LLM wasn't trained on previous published works, potentially informal comments on some forum, etc, that could have steered it towards this? Are we sure that the gap was so large? Do we truly have countless counterexamples? Obviously this math problem being solved is not a rigorous study - the authors of this don't even have access to the training data, we'd need quite a bit more than this to form assumptions. I'm willing to take a position here if you make a good case for it. I'm absolutely not opposed to the idea that other forms of reasoning can't reduce to statistical token generation, it just strikes me as unintuitive and so I'm going to need to hear something to compel me. [0] https://jamesfodor.com/2025/06/22/line-goes-up-large-language-models-are-poor-general-reasoners/ https://jamesfodor.com/2025/06/22/line-goes-up-large-languag...
- aspenmartin 7mo agoThe “good deal of evidence” is everywhere. The proof is in the pudding. Of course you can find failure modes, the blog article (not an actual paper?) rightfully derides benchmarks and then…creates a benchmark? Designed to elicit failure modes, ok so what? As if this is surprising to anyone and somehow negates everything else? Anyone who says that “statistical models for next token generation” are unlikely to provide emergent intelligence I think is really not understanding what a statistical model for next token generation really means. That is a proxy task DESIGNED to elicit intelligence because in order to excel at that task beyond a certain point you need to develop the right abstractions and decide how to manipulate them to predict the next token (which, by the way, is only one of many many stages of training). That’s indistinguishable from intelligence. It’s like saying “I think it’s surprising that a jumble of trillions of little cells zapping each other would produce emergent intelligence” while ignoring the fact that brains are clearly intelligent.
- storus 7mo agoOk, I'll bite. Show me an LLM that comes up with a new math operator. Or which will come up with theory of relativity if only Newton physics is in its training dataset. That it could remix existing ideas which leads to novel insights is expected, however the current LLMs can't come up with paradigm shifts that require novel insights. Even humans have a rather limited time they can come up with novel insights (when they are young, capable of latent thinking, not yet ossified from the existing formalization of science and their brain is still energetically capable without vascular and mitochondrial dysfunction common as we age).
- tigershark 7mo agoHow many humans have been born until now and how many Einsteins have been born? And in how many hundreds of thousands of years?
- storus 7mo agoThe point is that humans do have some edge compared to current LLMs which are essentially next token predictors. If we all start relying on current AI and stop thinking, we would only be able to "exhaust the remix space" of existing ideas but won't be able to do any paradigm jumps. Moreover, it's quite likely that current training sets are self-contradictory, containing Dutch books, carrying some innate error in them.
- int_19h 7mo agoIt takes a lot of intelligence to "essentially predict" next token when you're doing a math proof. Or writing code.
- qnleigh 7mo agoI might as well answer my own question, because I do think there are some coherent arguments for fundamental LLM limitations: 1. LLMs are trained on human-quality data, so they will naturally learn to mimic our limitations. Their capabilities should saturate at human or maybe above-average human performance. 2. LLMs do not learn from experience. They might perform as well as most humans on certain tasks, but a human who works in a certain field/code base etc. for long enough will internalize the relevant information more deeply than an LLM. However I'm increasingly doubtful that these arguments are actually correct. Here are some counterarguments: 1. It may be more efficient to just learn correct logical reasoning, rather than to mimic every human foible. I stopped believing this argument when LLMs got a gold metal at the Math Olympiad. 2. LLMs alone may suffer from this limitation, but RL could change the story. People may find ways to add memory. Finally, it can't be ruled out that a very large, well-trained LLM could internalize new information as deeply as a human can. Maybe this is what's happening here: https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/ https://metr.org/blog/2025-03-19-measuring-ai-ability-to-com...
- scoofy 7mo agoI studied philosophy focusing on the analytic school and proto-computer science. LLMs are going to force many people start getting a better understanding about what "Knowledge" and "Truth" are, especially the distinction between deductive and inductive knowledge. Math is a perfect field for machine learning to thrive because theoretically, all the information ever needed is tied up in the axioms. In the empirical world, however, knowledge only moves at the speed of experimentation, which is an entirely different framework and much, much slower, even if there are some areas to catch up in previous experimental outcomes. Having a focus in philosophy of language is something I genuinely never thought would be useful. It’s really been helpful with LLMs, but probably not in the way most people think. I’d say that folks curious should all be reading Quine, Wittgenstein’s investigations, and probably Austin.
- qnleigh 7mo agoWhere can I read about how LLMs have changed epistemology? Is there a field of philosophy that tries to define and understand 'intelligence'? That sounds very interesting.
- muyuu 7mo agoIt's fear.
- aaroninsf 7mo agoXimm's Law applies ITT: every critique of AI assumes to some degree that contemporary implementations will not, or cannot, be improved upon. Especially the lemmas: - any statement about AI which uses the word "never" to preclude some feature from future realization is false. - contemporary implementations have almost always already been improved upon, but are unevenly distributed.
- zeroonetwothree 7mo agoAnti-Ximm's Law: every response to a critique of AI assumes as much arbitrary level of future improvement as is necessary to make the case.
- deleted 7mo ago[deleted]
- zamalek 7mo agoLLMs are notoriously terrible at multiplying large numbers: https://claude.ai/share/538f7dca-1c4e-4b51-b887-8eaaf7e6c7d3 https://claude.ai/share/538f7dca-1c4e-4b51-b887-8eaaf7e6c7d3 > Let me calculate that. 729,278,429 × 2,969,842,939 = 2,165,878,555,365,498,631 Real answer is: https://www.wolframalpha.com/input?i=729278429*2969842939 https://www.wolframalpha.com/input?i=729278429*2969842939 > 2 165 842 392 930 662 831 Your example seems short enough to not pose a problem.
- peacebeard 7mo agoThis doesn’t address the author’s point about novelty at all. You don’t need 100% accuracy to have the capability to solve novel problems.
- zamalek 7mo agoIt does address the GP comment about math.
- luma 7mo agoModern LLMs, just like everyone reading this, will instead reach for a calculator to perform such tasks. I can't do that in my head either, but a python script can so that's what any tool-using LLM will (and should) do.
- zamalek 7mo agoThis is special pleading. Long multiplication is a trivial form of reasoning that is taught at elementary level. Furthermore, the LLM isn't doing things "in its head" - the headline feature of GPT LLMs is attention across all previous tokens, all of its "thoughts" are on paper. That was Opus with extended reasoning, it had all the opportunity to get it right, but didn't. There are people who can quickly multiply such numbers in their head (I am not one of them). LLMs don't reason.
- markatto 7mo agoI tried this with Claude - it has to be explicitly instructed to not make an external tool call, and it can get the right answer if asked to show its work long-form.
- drfloyd51 7mo agoPeople rarely create things that are wholly new. Most created things are remixes of existing things. Hallucinations are “something new”. And like most new things, useless. But the truth is the entire conversation is a hallucination. We just happen to agree that most of it is useful.
- acchow 7mo ago> asserting that LLMs will never generate 'truly novel' ideas or problem solutions I don't think I've had one of these my entire life. Truly novel ideas are exceptionally rare: - Darwin's origin of the species - Godel's Incompleteness - Buddhist detachment Can't think of many.
- _doctor_love 7mo ago> You need to say why it can do some novel tasks but could never do others. This is actually quite a tall order. Reasoning about AI and making sense of what the LLMs are doing, and learning to think about it as technology, is a very difficult and very tricky problem. You get into all kinds of weird things about a person’s outlook on life: personal philosophy, understanding of ontology and cosmology, and then whatever other headcanon they happen to be carrying around about how they think life works. I know that might sound kind of poetic, but I really believe it’s true. I am a great fan of Dr Richard Hamming and he gave a wonderful series of lectures on the topic. The book Learning to Learn has the full set of his lectures transcribed (highly recommend this book!). But don't take my word for it, listen to Dr Hamming say it himself: https://www.youtube.com/watch?v=aq_PLEQ9YzI https://www.youtube.com/watch?v=aq_PLEQ9YzI "The biggest problem is your ego. The second biggest problem is your religion."