11 ms·

Recent results show that LLMs struggle with compositional tasks

345 points by thm 2y ago

d0mine 2y ago
> multilayer transformers indeed cannot solve certain complicated compositional tasks > chain-of-thought prompting essentially turns a large problem into a sequence of smaller problems, making it possible for transformers to tackle more complex compositional tasks --- [out of order] > the model could be trained on 20-digit numbers and still reliably (with 98% accuracy) add 100-digit numbers, whereas a model trained without the extra positional embedding was only about 3% accurate
- thfuran 2y ago
  98% seems pretty damn unreliable to me given that a computer ought to be able to do that addition with probably about 12 nines instead of 1 without really putting much effort into reliability.
  - szvsw 2y ago
    Can you add 100 digit numbers reliably in a short amount of time over a large sample size? How about without a piece of paper and a pen? Can you even remember a 100 digit number? It’s likely quite difficult… This is not about comparing to reliability “of a computer” but rather to a process which responds to a wide array of natural language queries (both human processes an algorithmic processes)… and obviously the reliability goes up once tool use is included.
    - achierius 2y ago
      With pen and paper? Yes, definitely. Without? LLMs have scratch space, so they're not operating without either.
      - szvsw 2y ago
        At pace though? What would your throughput be to maintain 98% reliability? How long do you think one execution would take? my guess would be between 30s-1m for a single summation of two 100 digit numbers? So let’s say you want a sample size of 200 operations to estimate reliability… maybe something like 2-4hours to get through the computations working non stop? That’s actually a nontrivial amount of focus on a highly repetitive task, which isn’t easy for a lot of people. I’m now genuinely curious what the average HN user’s reliability and throughput would be for adding together two random hundred digit numbers working as fast as they can.
        thfuran 2y ago
        Computers have been faster than humans at arithmetic since the 1940s. I don't really understand your point.
        szvsw 2y ago
        The point made by a commenter a few levels up questioned whether 98% reliability for adding two 100-digit numbers together should be considered noteworthy/acceptable etc. My point is that assessing 98% reliability qualitatively depends on whatever kind of baseline system you are comparing against. Obviously this reliability (and speed) of an LLM with 98% accuracy is atrocious compared to any basic automated arithmetic computation system. If you need to reliably add together 100 digit numbers, of course an LLM is a bad choice. However, comparing it to the general class of systems which an LLM belongs to - ie “things that can add 100-digit numbers together quickly and also tell you about this history of the Ottoman Empire and a bunch of other stuff too, all within a relatively short amount of time” - this reliability is (potentially) impressive. We don’t have many such systems - a moderately educated human; a generic bespoke piece of software which process natural language requests and attempts to decide whether or not to retrieve information from a database/the internet or write and execute code, but could conceivably have been written in a crude manner 5 years ago; and LLMs (which might also take advantage of tool use as suggested in the previous example). In this context - comparing to other systems with similar capabilities - the 98% reliability might be considered impressive, especially when you consider that it likely is much higher for more common simpler arithmetic problems. My intention in bringing up “an average HN user” was to bring up an example of a reasonable benchmark to compare to, in my opinion. A HN user is a stand-in for a moderately educated person, which could reasonably respond to a variety of generic natural language requests in a short amount of time. My point was simply that adding together 100 digit numbers reliably and quickly while also being able to chat about Shakespeare or Kalman filters or planning a wedding is likely a more difficult task than we tend to give credence to, due to our familiarity with specialized systems which do arithmetic extremely well and extremely quickly.
  - bruce511 2y ago
    This speaks to the psychology of it all. Over my life I've been "trained" that humans are unreliable and subject-to-change while computers are deterministic and will give the same output every time (given the same input.) LLMs though are very human. Ask it the same question and you'll get different answers. Sometimes grammatically different, sometimes factually different. This is partly why hallucinations freaked people out. I expect a correct, and deterministic answer. And it just made stuff up. Then it made up URLs to validate the answer. In other words you have computers behaving like humans. Which was indeed the whole point of them to begin with. But psychologically this is disturbing.
    - suddenlybananas 2y ago
      If you genuinely believe that LLM hallucinations are like human behaviour, you must not interact a lot with humans.
      - anshumankmr 2y ago
        LLMs are not human intelligence, but are similarly fallible, though we don't have ways to jailbreak humans yet. And perhaps even worse off.
    - habinero 2y ago
      The reason you get different answers is there's randomness built into generating text from the statistical model. It's not any more complex than that.
  - imtringued 2y ago
    Try entering a 20 digit number into your calculator.
klodolph 2y ago
When one of these limitations gets spelled out in an article, it feels like six months later, somebody has a demo of a chatbot without that particular limitation. These limitations don’t seem in any way “fundamental” to me. I’m sure there are a ton of people gluing LLMs to SAT solvers as we speak.
- wslh 2y ago
  SMT solvers really.
- xigency 2y ago
  I've been slacking but yeah it's on my list.
- chefandy 2y ago
  Could you give an example of something we recently solved that was considered an unsolvable problem six months beforehand? I don’t have any specific examples, but it seems like most of the huge breakthrough discoveries I’ve seen announced end up being overstated and for practical usage, our choice of LLM-driven tools is only marginally better than they were a couple of years ago. It seems like the preponderance of practical advancement in recent times has come from the tooling/interface improvements rather than generating miracles from the models themselves. But it could be that I just don’t have the right use cases.
  - intelkishan 2y ago
    Performance of OpenAI o3 in the ARC-AGI challenge fits the bill, however the model is not released publicly.
  - gallerdude 2y ago
    Completely disagree… there are a crazy amount of cases that didn’t work, until the models scaled to a point they magically did. Best example I can think of is the ARC AGI benchmark. It was seen to measure human-like intelligence through special symmetries and abstract patterns. From GPT-2 to GPT-4 there was basically had no progress, then o1 got about 20%. Now o3 has basically solved the benchmark.
changoplatanero 2y ago
By the time these academic studies get published they are usually already several months out of date. o3-mini was released yesterday and if one wants to know about the limitations of current technology they are much better to check twitter than some research paper
- Xmd5a 2y ago
  The paper is recent and being discussed here: https://news.ycombinator.com/item?id=42889786 https://news.ycombinator.com/item?id=42889786
- FuckButtons 2y ago
  I think the breathless hype train of twitter is probably the worst place to get an actually grounded take on what the real world implications of the technology is. Seeing the 100th example of an llm generating some toy code for which there are a vast number of examples of approximately similar things in the training corpus doesn’t give you a clearer view of what is or isn’t possible.
  - unification_fan 2y ago
    I think that most of the developers who advocate for AI coding have never worked all by themselves on projects with over 500/1000 files. Because if they had they would not advocate for AI coding.
    - AtlasBarfed 2y ago
      I posted this earlier, but I wanted a java port of sed for ... Reasons, and despite the existence of man pages and source code it couldn't do anything but the most basic flags. Imo this should be low hanging fruit. Porting non-trivial but yet 3-4 core code files that are already debugged and interface specified should be what an LLM excels at.
      - Kye 2y ago
        I tried this with Microsoft's Copilot + the Think Deeper button. That allegedly uses the new o1 model. It goes into a lot of fancy talk about...pretty much what you said older models did. Then it said "here's some other stuff you could extend this with!" and a list of all the important sed functionality. It's possible it could do it if prompted to finish the code with those things, but I don't know the secret number of fancy o1 uses I get and I don't want to burn them on something that's not for me. You should be able to access it here if you have a Microsoft account and want to try the button: https://copilot.microsoft.com/ https://copilot.microsoft.com/
novemp 2y ago
It feels like every few weeks someone puts out a new article breathlessly talking about how bad the fancy autocorrect bot is at math and logic like it's brand new information. How have we as a society not gotten this through our skulls yet?
simonw 2y ago
Here's "Einstein’s puzzle" from this paper: https://www.researchgate.net/publication/341189675_Is_Einstein's_Puzzle_Over-Specified https://www.researchgate.net/publication/341189675_Is_Einste... H01 There are five houses. H02 The Englishman lives in the red house. H03 The Spaniard owns the dog. H04 Coffee is drunk in the green house. H05 The Ukrainian drinks tea. H06 The green house is immediately to the right of the ivory house. H07 The Old Gold smoker owns snails. H08 Kools are smoked in the yellow house. H09 Milk is drunk in the middle house. H10 The Norwegian lives in the first house. H11 The man who smokes Chesterfields lives in the house next to the man with the fox. H12 Kools are smoked in a house next to the house where the horse is kept. H13 The Lucky Strike smoker drinks orange juice. H14 The Japanese smokes Parliaments. H15 The Norwegian lives next to the blue house. Now, Q1 Who drinks water? Q2 Who owns the zebra?
- gorgoiler 2y ago
  I am reminded of this Reddit post from November last year, except this is fifth grade math homework: https://www.reddit.com/r/mildlyinfuriating/comments/1gippqm/fifth_grade_math_homework_has_kid_in_tears_and/ https://www.reddit.com/r/mildlyinfuriating/comments/1gippqm/... The puzzle is a grid of simultaneous equations with 9 unknowns: 9 + ? - 6 = ? - 3 - = - = x 4 + ? x 5 - ? = 6 / + + - - 8 - 5 = ? - ? x 2 x - - - x ? + 1 + 7 - 4 = 8 = - = + = 7 = ? x 3 - ? - 2
  - UncleEntity 2y ago
    Deepthink thought for 644 seconds and says the answer is 4. Probably didn't help that I just pasted it with no explanation, the spaces between the columns really threw it off.
    - gorgoiler 2y ago
      Here’s the same puzzle minus the grid parsing step: A1: 9+a-6=b-3 A2: 4+(c*5)-d=6 A3: 8-5=e-(f*2) A4: g+1+7-4=8 A5: 7=(h*3)-i-2 D1: 9-((4/8)*g)=7 D2: a=c+5-1-h D3: 6-5+e-7=3 D4: b=d-f-4+i D5: (3*6)-(2*8)=2
anon291 2y ago
This is ultimately a basic adaptation of the pigeonhole principle and is not surprising. A finite system of matrix multiplications cannot be turing complete. You cannot expect one trip through a series of matrix multiplications and bias additions and a final sampling at the end which commits it to a certain answer to ever produce a correct answer. It's a mathematical impossibility. No talk of quantum woo, emergent phenomenen, or whatever other pseudo-science has arisen to explain AI intelligence can get around this simple truth of mathematics. However, chain of thought reasoning where token streams can continue ad infinitum could potentially solve large swaths of problems whose ordinary solutions require turing machines. It could also solve problems that cannot generally be solved by turing machines, but where you only need solutions for a few classes of problems. Either way, even with chain of thought, you would expect that... in some instances, the model output diverges and does not complete. And unsurprisingly... this is exactly what you see with the DeepSeek models (and other CoT models) when you pose it difficult questions. It will never emit the </think> tag.
geoffhill 2y ago
Idk, `o3-mini-high` was able to pop this Prolog code out in about 20 seconds: solve(WaterDrinker, ZebraOwner) :- % H01: Five houses with positions 1..5. Houses = [ house(1, _, norwegian, _, _, _), % H10: Norwegian lives in the first house. house(2, blue, _, _, _, _), % H15: Since the Norwegian lives next to the blue house, house(3, _, _, milk, _, _), % and house1 is Norwegian, house2 must be blue. house(4, _, _, _, _, _), house(5, _, _, _, _, _) ], % H02: The Englishman lives in the red house. member(house(_, red, englishman, _, _, _), Houses), % H03: The Spaniard owns the dog. member(house(_, _, spaniard, _, dog, _), Houses), % H04: Coffee is drunk in the green house. member(house(_, green, _, coffee, _, _), Houses), % H05: The Ukrainian drinks tea. member(house(_, _, ukrainian, tea, _, _), Houses), % H06: The green house is immediately to the right of the ivory house. right_of(house(_, green, _, _, _, _), house(_, ivory, _, _, _, _), Houses), % H07: The Old Gold smoker owns snails. member(house(_, _, _, _, snails, old_gold), Houses), % H08: Kools are smoked in the yellow house. member(house(_, yellow, _, _, _, kools), Houses), % H11: The man who smokes Chesterfields lives in the house next to the man with the fox. next_to(house(_, _, _, _, _, chesterfields), house(_, _, _, _, fox, _), Houses), % H12: Kools are smoked in a house next to the house where the horse is kept. next_to(house(_, _, _, _, horse, _), house(_, _, _, _, _, kools), Houses), % H13: The Lucky Strike smoker drinks orange juice. member(house(_, _, _, orange_juice, _, lucky_strike), Houses), % H14: The Japanese smokes Parliaments. member(house(_, _, japanese, _, _, parliaments), Houses), % (H09 is built in: Milk is drunk in the middle house, i.e. house3.) % Finally, find out: % Q1: Who drinks water? member(house(_, _, WaterDrinker, water, _, _), Houses), % Q2: Who owns the zebra? member(house(_, _, ZebraOwner, _, zebra, _), Houses). right_of(Right, Left, Houses) :- nextto(Left, Right, Houses). next_to(X, Y, Houses) :- nextto(X, Y, Houses); nextto(Y, X, Houses). Seems ok to me. ?- solve(WaterDrinker, ZebraOwner). WaterDrinker = norwegian, ZebraOwner = japanese .
- simonw 2y ago
  Solving it with Prolog is neat, and a very realistic way of how LLMs with tools should be expected to handle this kind of thing.
WillAdams 2y ago
Perhaps applying: https://www.doc.ic.ac.uk/~re14/Evans-R-2020-PhD-Thesis.pdf https://www.doc.ic.ac.uk/~re14/Evans-R-2020-PhD-Thesis.pdf "Kant’s Cognitive Architecture" will help? > ...we provide a precise formalization of what it means > to “make sense” of a sensory sequence. According to our definition, making sense means constructing > a symbolic causal theory that explains the sensory sequence and satisfies a set of unity conditions > that were inspired by Kant’s discussion in the first half of the Critique of Pure Reason. According to > our interpretation, making sense of sensory input is a type of program synthesis, but it is unsupervised > program synthesis.
apsec112 2y ago
"showing that multilayer transformers indeed cannot solve certain complicated compositional tasks. Basically, some compositional problems will always be beyond the ability of transformer-based LLMs." Pretty sure this is just false and the paper doesn't show this. I could be misunderstanding, but it looks like the result is only about a single token/forward pass, not a reasoning model with many thousands of tokens like o1/o3
- simonw 2y ago
  I'm not sure that the statement "some compositional problems will always be beyond the ability of transformer-based LLMs" is even controversial to be honest. There's a reason all of the AI labs have been leaning hard into tool use and (more recently) inference-scaling compute (o1/o3/Gemini Thinking/R1 etc) recently - those are just some of the techniques you can apply to move beyond the unsurprising limitations of purely guessing-the-next-token.
  - apsec112 2y ago
    o3 is still a transformer-based LLM, just one with a different loss function
    - simonw 2y ago
      Huh, yeah that's a good point. The various distilled R1 models are definitely regular transformer-based LLMs because the GGUF file versions of them work without any upgrades to the underlying llama.cpp library.
mohsen1 2y ago
https://chatgpt.com/share/679f0353-12bc-8007-91cf-dd63d52044a0 https://chatgpt.com/share/679f0353-12bc-8007-91cf-dd63d52044... O1 Pro gets the answer with proper reasoning Can’t tell if this is dude to data contamination or it really figured it out? How can we form the question in another way to avoid data contamination?
- mohsen1 2y ago
  With modified prompt to make less look like the original prompt it thought 6x more: https://chatgpt.com/share/679f086d-a758-8007-b240-38e684303786 https://chatgpt.com/share/679f086d-a758-8007-b240-38e6843037...
  - pollinations 2y ago
    I don't think individual examples make sense to solve these kinds of discussions as for me it can vary easily by 6x thinking with exactly the same input and parameters.
- deleted 2y ago
  [deleted]
EGreg 2y ago
Maybe they’re not training them on the right thing, maybe math can be a different “mode” just lime sound or images
moolimon 2y ago
The main thesis here seems to be that LLMs behave like almost all other machine learning models, in that they are doing pattern matching on their input data, and short circuiting to a statistically likely result. Chain of thought reasoning is still bound by this basic property of reflexive pattern matching, except the LLM is forced to go through a process of iteratively refining the domain it does matching on. Chain of thought is interesting, because you can combine it with reinforcement learning to get models to solve (seemingly) arbitrarily hard problems. This comes with the caveat that you need some reward model for all RL. This means you need a clear definition of success, and some way of rewarding being closer to success, to actually solve those problems. Framing transformer based models as pattern matchers makes all the sense in the world. Pattern matching is obviously vital to human problem solving skills too. Interesting to think about what structures human intelligence has that these models don't. For one, humans can integrate absolutely gargantuan amounts of information extremely efficiently.
- drakenot 2y ago
  With DeepSeek-R1-Zero, their usage of RL didn't have reward functions really that indicated progress towards the goal afaik. It was "correct structure, wrong answer", "correct answer", "wrong answer". This was for Math & Coding, where they could verify answers deterministically.
  - deleted 2y ago
    [deleted]
  - mountainriver 2y ago
    It is a reward function it’s just a deterministic one. Reward models are often hacked preventing real reasoning from being discovered
- huijzer 2y ago
  > Framing transformer based models as pattern matchers makes all the sense in the world. Pattern matching is obviously vital to human problem solving skills too. Interesting to think about what structures human intelligence has that these models don't. For one, humans can integrate absolutely gargantuan amounts of information extremely efficiently. What is also a benefit for humans, I think, is that people are typically much more selective. LLMs train to predict anything on the internet, so for example for finance that includes clickbait articles which have a lifetime of about 2 hours. Experts would probably reject any information in these articles and instead try to focus on high quality sources only. Similarly, a math researcher will probably have read a completely set of sources throughout the life than, say, a lawyer. I’m not sure it’s a fundamental difference, but current models do seem to not specialize from the start unlike humans. And that might be in the way of learning the best representations. I know from ice hockey for example, that you can see within 3 seconds whether someone played ice hockey from young age or not. Same with language. People can usually hear an accent within seconds. Relatedly, I've used OpenAI's text to speech a while back and the Dutch voice had an American accent. What this means is that even if you ask LLMs about Buffett's strategy, maybe they have a "clickbait accent" too. So with the current approach to training, the models might never reach absolute expert performance.
mikeknoop 2y ago
One must now ask whether research results are analyzing pure LLMs (eg. gpt-series) or LLM synthesis engines (eg. o-series, r-series). In this case, the headline is summarizing a paper originally published in 2023 and does not necessarily have bearing on new synthesis engines. In fact, evidence strongly suggests the opposite given o3's significant performance on ARC-AGI-1 which requires on-the-fly composition capability.
- orbital-decay 2y ago
  It's Quanta being misleading. They mention several papers but end up with this [1] which talks about decoder-only transformers, not LLMs in general, chatbots, or LLM synthesis engines, whatever that means. The paper also proves that CoT-like planning lets you squeeze more computation from a transformer, which is... obvious? but formally proven this time. Models trained to do CoT don't have some magical on-the-fly compositional ability, they just invest more computation (could be dozens millions of tokens in case of o3 solving the tasks from that benchmark). [1] https://arxiv.org/abs/2412.02975 https://arxiv.org/abs/2412.02975
antirez 2y ago
LLMs keep showing more and more they are the wonder of AI that we awaited for decades: talking machines that every two months do progresses that two months before were anticipated impossible because of <put here some limit that was actually in the prejudice of the skeptical AI community> (just stochastic parrots, no reasoning possible without symbolic representations, there are no longer tokens, ...) At the same time, part of the scientific community continues to diminish what was accomplished and the steps that are being made. A few months ago LeCun arrived to tell new researchers to move away from LLMs since they are a dead end: imagine the disservice he made to the surely non-zero folks that followed the advice, putting themselves out of the AI research that matters. (Incidentally, this skepticism of the Meta AI head must have something to do with the fact that Meta, despite the huge efforts allocated, produced the worst LLM among Anthropic, OpenAI, DeepSeek -- I bet Zuckerberg is asking questions lately). It's very hard to explain this behavior if not by psychological denial. [EDIT: you can't see the score of this comment, but I can: it's incredible how it goes from 3, to -2, to 1, and so forth. The community is split in two, and it is pretty sad since this is not a matter of taste or political inclination: there must be a single truth]
- ramblerman 2y ago
  I get the sentiment, but I actually think some skepticism in the system is healthy. Billions are flowing towards LLMS, and Sam Altman will overpromise AGI is just around the corner and the days of jobs are gone to fill his coffers to anyone that will listen. Additionally if we begin to use these things in real production environments where mistakes matters, knowing the exact limitations is key. None of this takes away from the fact that these are exciting times.
  - antirez 2y ago
    Yes there is another part of the community that overhypes everything. But I can expect that from a CEO of an AI company (especially if he is Altman), but from researches? Also the fact that LLMs may reach superhuman expertise in certain fields in a short timeframe (a few years), since reinforcement learning is starting to be applied to LLMs may no longer be a totally crazy position. If it is possible to extend considerably the same approach seen in R1-Zero there could be low hanging fruits around the corner.
comeonbro 2y ago
> Chatbot Software Begins to Face Fundamental Limitations > Recent results show that large language models struggle with compositional tasks, suggesting a hard limit to their abilities. Your first question with anything like this should always be WHICH MODELS: > For our experiments, we evaluate the performance of 6 LLMs: GPT4 (gpt-4) [58], ChatGPT (GPT3.5-turbo) [57], GPT3 (text-davinci-003) [11], FlanT5 [17] and LLaMa [75]. This is ancient. This research was done centuries ago. This is research about the possibility of isotopes, written about radium in 1903, published in 1946. It is a criminal level of journalistic malpractice to leave uninformed readers with the impression that this is where AI stands yesterday.
deleted 2y ago
[deleted]
deleted 2y ago
[deleted]
MagicMoonlight 2y ago
Do you really need a study to work out that a Markov chain can’t solve problem questions? It feels pretty intuitive. LLMs are not intelligent. Human intelligence is stored in text, so mimicking text mimicks that intelligence. The sheer quantity of text means they’ve probably seen something similar to the prompt. If LLMs were an intelligent system they wouldn’t need to steal 1000TB of text and media content. Have you ever seen a person require a million books? It shows that the learning is brute force rather than intelligent. It’s close, but it’s just not there. The methodology is wrong.
- red75prime 2y ago
  The Markov chain wouldn't fit in a googol of observable universes. The analogy is absolutely misleading. There's no reason to think that the chain of that humongous size cannot capture at least something of what humans with their limited brain size can do. It cannot capture non in-context learning, true. But we have tools to modify the "Markov chain".
- cjblomqvist 2y ago
  I guess your point is that it's obvious, but I'd argue it's not. A lot of people are hyping up LLMs as the solution to all our problems, and there have been a lot of discussion around what's reasonable use cases (and a lot of anecdotal evidence trying to argue and support certain stances). My point is that we need (scientific) proof to move forward, and create confidence in "facts" we base our decisions on. Obviously that doesn't seem to be very fashionable nowadays - on both sides of the LLM discussion there seems to be a lot of people arguing hard for there stances. I'm very glad some are working on providing some facts into this discussion. (one can argue about science vs truth, but regardless it's a step in the right direction/a good motive) (this is btw why a solid educational foundation, not only skill focused, is very critical for society. cf democracy)
seydor 2y ago
"Fundamental" results cannot come from empirical findings, they have to be proven rigorously. Until then, they are not fundamental
mmcnl 2y ago
There's so much talk about the advancements in AI/LLMs, yet for me ChatGPT as of this date is basically just a faster search engine without cookie banners, clickbait and ads. It hallucinates a lot and it can keep very limited context. Why is there so much promise about future progress but so little actual progress?
- EA-3167 2y ago
  It's the same cycle we saw with Crypto, there's so much money flying around that the motivation to "believe" is overwhelming. The hype is coming from all directions, and people are social animals that put greater weight on words that come from multiple sources. It's also a platform for people excited about the future to fantasize, and for people terrified of the future to catastrophize.
- knowaveragejoe 2y ago
  I have to wonder how you are using ChatGPT to get a lot of hallucinations or run into issues with limited context.
kebsup 2y ago
I've managed to get llms fail on simple questions, that require thinking graphically - 2D or 3D. An example would be: you have a NxM grid. How many shapes of XYZ shape can you fit on it? However, thinking of the transformer video games, AI can be trained to have a good representation of 2D/3D worlds. I wonder how it can be combined so that this graphical representation is used to compute text output.
anshumankmr 2y ago
Isn't the math calculation thing solvable through the integration with a simple Python script for basic math and more complex stuff something like an open source version of Wolfram?
Ndv_MC 2y ago
Quite deep and informative. One thing to point out here is how they measure accuracy when testing their LLMs. Like most neural networks, the n-dimensional space of LLMs is largely sparse. In simple terms, this means that it’s very easy to “exercise” the model in areas where it’s weak. That doesn’t mean that the model doesn’t have the potential of doing a (much) better job if you “exercise” its space properly—which is exactly what techniques like CoT basically do. LLMs are inherently limited, and they don’t have “true” reasoning capabilities—beyond marketing hype, this much should be obvious to most serious ML researchers today. The only question is how well we can get them to “mimic” reasoning for practical applications, and this is where “prompt engineering”, strictly speaking, is a true form of engineering, which has to take into account the mathematical foundations of the models and figure out how to extract out of them the best performance they can deliver.
starchild3001 2y ago
What a poorly informed article. It's very shallow and out of touch with LLM research. As it stands 6-12 months old models are system 1 thinkers, everybody knows this and knew this even at the time. You need system 2 thinking (test time compute) for more complex logical, algorithmic and reasoning tasks. We knew this when Daniel kahneman wrote thinking fast, thinking slow (over a decade ago) and we still know it today. So LLMs can think but they have to be programmed to think (a la system 2, reasoning, thinking models). There's nothing inherently wrong or limited with LLMs themselves as far as we can tell.
- tshadley 2y ago
  > To understand the capabilities of LLMs, we evaluate GPT3 (text-davinci-003) [11], ChatGPT (GPT-3.5-turbo) [57] and GPT4 (gpt-4) Oh dear, this is embarrassing. Anil Anathaswamy, are you aware a year in AI research now is like 10 years in every other field?
- astrange 2y ago
  This is an example of "metaphor-driven development" in AI, which Phil Agre criticized a few decades ago. System 1/System 2 isn't a real thing. It's just a metaphor Kahneman invented for a book. AI developers continually find metaphors about the brain, decide they are real, implement something which they give the same name, decide it's both real and the same thing because they have given it the same name, and then find it doesn't work. (Another common example is "world model", something which has never had a clear meaning, and if you did define it you'd find that people don't have one and don't need one.)
strangescript 2y ago
Why do articles testing ancient LLMs keep getting posted?
Workaccount2 2y ago
"Recent results" = two year old study looking at GPT-3, 3.5, and first gen 4.
fyygfdrryhcss 2y ago
[dead]
have-a-break 2y ago
Maybe it's just me but when i code/program i typically think of how different variations of implementations would affect roi of the associated company i would be working for. That's something that AI in the current state probably wouldn't be able to take into account.
emorning3 2y ago
I'm a software developer with just a basic understanding of LLMs. But its not surprising to me that LLMs wouldnt be good at composition and planning. To me, LLMs seem like half a brain, the right half, that does pattern matching and such. To me, the current crop of AI seems to be missing the other half of the brain, the part that's good at planning and composition and such. LLMs can tell me all I want to know about logic and deduction, so its funny that LLMs cant understand their own words.
malaigo 2y ago
I'd like to propose a modified Zebra puzzle. Still has unique solutions. Not on the internet, so not part of the training set. Similar problem description complexity. Just remove the last constraint (Norwegian lives next to the blue house) and asked negated questions: Who cannot be the water drinker? Who cannot own the zebra? Text of problem here: sandiway.arizona.edu/mzebra.pdf (I tested it on free ChatGPT and DeepSeek-R1 https://youtu.be/3gotauWUcew https://youtu.be/3gotauWUcew)
bpositiveteam1 2y ago
[dead]

345 points by thm 2y ago

> multilayer transformers indeed cannot solve certain complicated compositional tasks > chain-of-thought prompting essentially turns a large problem into a sequence of smaller problems, making it possible for transformers to tackle more complex compositional tasks --- [out of order] > the model could be trained on 20-digit numbers and still reliably (with 98% accuracy) add 100-digit numbers, whereas a model trained without the extra positional embedding was only about 3% accurate

thfuran 2y ago
98% seems pretty damn unreliable to me given that a computer ought to be able to do that addition with probably about 12 nines instead of 1 without really putting much effort into reliability.
- szvsw 2y ago
  Can you add 100 digit numbers reliably in a short amount of time over a large sample size? How about without a piece of paper and a pen? Can you even remember a 100 digit number? It’s likely quite difficult… This is not about comparing to reliability “of a computer” but rather to a process which responds to a wide array of natural language queries (both human processes an algorithmic processes)… and obviously the reliability goes up once tool use is included.
  - achierius 2y ago
    With pen and paper? Yes, definitely. Without? LLMs have scratch space, so they're not operating without either.
    - szvsw 2y ago
      At pace though? What would your throughput be to maintain 98% reliability? How long do you think one execution would take? my guess would be between 30s-1m for a single summation of two 100 digit numbers? So let’s say you want a sample size of 200 operations to estimate reliability… maybe something like 2-4hours to get through the computations working non stop? That’s actually a nontrivial amount of focus on a highly repetitive task, which isn’t easy for a lot of people. I’m now genuinely curious what the average HN user’s reliability and throughput would be for adding together two random hundred digit numbers working as fast as they can.
      - thfuran 2y ago
        Computers have been faster than humans at arithmetic since the 1940s. I don't really understand your point.
        szvsw 2y ago
        The point made by a commenter a few levels up questioned whether 98% reliability for adding two 100-digit numbers together should be considered noteworthy/acceptable etc. My point is that assessing 98% reliability qualitatively depends on whatever kind of baseline system you are comparing against. Obviously this reliability (and speed) of an LLM with 98% accuracy is atrocious compared to any basic automated arithmetic computation system. If you need to reliably add together 100 digit numbers, of course an LLM is a bad choice. However, comparing it to the general class of systems which an LLM belongs to - ie “things that can add 100-digit numbers together quickly and also tell you about this history of the Ottoman Empire and a bunch of other stuff too, all within a relatively short amount of time” - this reliability is (potentially) impressive. We don’t have many such systems - a moderately educated human; a generic bespoke piece of software which process natural language requests and attempts to decide whether or not to retrieve information from a database/the internet or write and execute code, but could conceivably have been written in a crude manner 5 years ago; and LLMs (which might also take advantage of tool use as suggested in the previous example). In this context - comparing to other systems with similar capabilities - the 98% reliability might be considered impressive, especially when you consider that it likely is much higher for more common simpler arithmetic problems. My intention in bringing up “an average HN user” was to bring up an example of a reasonable benchmark to compare to, in my opinion. A HN user is a stand-in for a moderately educated person, which could reasonably respond to a variety of generic natural language requests in a short amount of time. My point was simply that adding together 100 digit numbers reliably and quickly while also being able to chat about Shakespeare or Kalman filters or planning a wedding is likely a more difficult task than we tend to give credence to, due to our familiarity with specialized systems which do arithmetic extremely well and extremely quickly.
- bruce511 2y ago
  This speaks to the psychology of it all. Over my life I've been "trained" that humans are unreliable and subject-to-change while computers are deterministic and will give the same output every time (given the same input.) LLMs though are very human. Ask it the same question and you'll get different answers. Sometimes grammatically different, sometimes factually different. This is partly why hallucinations freaked people out. I expect a correct, and deterministic answer. And it just made stuff up. Then it made up URLs to validate the answer. In other words you have computers behaving like humans. Which was indeed the whole point of them to begin with. But psychologically this is disturbing.
  - suddenlybananas 2y ago
    If you genuinely believe that LLM hallucinations are like human behaviour, you must not interact a lot with humans.
    - anshumankmr 2y ago
      LLMs are not human intelligence, but are similarly fallible, though we don't have ways to jailbreak humans yet. And perhaps even worse off.
  - habinero 2y ago
    The reason you get different answers is there's randomness built into generating text from the statistical model. It's not any more complex than that.
- imtringued 2y ago
  Try entering a 20 digit number into your calculator.

klodolph 2y ago

When one of these limitations gets spelled out in an article, it feels like six months later, somebody has a demo of a chatbot without that particular limitation. These limitations don’t seem in any way “fundamental” to me. I’m sure there are a ton of people gluing LLMs to SAT solvers as we speak.

wslh 2y ago
SMT solvers really.
xigency 2y ago
I've been slacking but yeah it's on my list.
chefandy 2y ago
Could you give an example of something we recently solved that was considered an unsolvable problem six months beforehand? I don’t have any specific examples, but it seems like most of the huge breakthrough discoveries I’ve seen announced end up being overstated and for practical usage, our choice of LLM-driven tools is only marginally better than they were a couple of years ago. It seems like the preponderance of practical advancement in recent times has come from the tooling/interface improvements rather than generating miracles from the models themselves. But it could be that I just don’t have the right use cases.
- intelkishan 2y ago
  Performance of OpenAI o3 in the ARC-AGI challenge fits the bill, however the model is not released publicly.
- gallerdude 2y ago
  Completely disagree… there are a crazy amount of cases that didn’t work, until the models scaled to a point they magically did. Best example I can think of is the ARC AGI benchmark. It was seen to measure human-like intelligence through special symmetries and abstract patterns. From GPT-2 to GPT-4 there was basically had no progress, then o1 got about 20%. Now o3 has basically solved the benchmark.

changoplatanero 2y ago

By the time these academic studies get published they are usually already several months out of date. o3-mini was released yesterday and if one wants to know about the limitations of current technology they are much better to check twitter than some research paper

Xmd5a 2y ago
The paper is recent and being discussed here: https://news.ycombinator.com/item?id=42889786 https://news.ycombinator.com/item?id=42889786
FuckButtons 2y ago
I think the breathless hype train of twitter is probably the worst place to get an actually grounded take on what the real world implications of the technology is. Seeing the 100th example of an llm generating some toy code for which there are a vast number of examples of approximately similar things in the training corpus doesn’t give you a clearer view of what is or isn’t possible.
- unification_fan 2y ago
  I think that most of the developers who advocate for AI coding have never worked all by themselves on projects with over 500/1000 files. Because if they had they would not advocate for AI coding.
  - AtlasBarfed 2y ago
    I posted this earlier, but I wanted a java port of sed for ... Reasons, and despite the existence of man pages and source code it couldn't do anything but the most basic flags. Imo this should be low hanging fruit. Porting non-trivial but yet 3-4 core code files that are already debugged and interface specified should be what an LLM excels at.
    - Kye 2y ago
      I tried this with Microsoft's Copilot + the Think Deeper button. That allegedly uses the new o1 model. It goes into a lot of fancy talk about...pretty much what you said older models did. Then it said "here's some other stuff you could extend this with!" and a list of all the important sed functionality. It's possible it could do it if prompted to finish the code with those things, but I don't know the secret number of fancy o1 uses I get and I don't want to burn them on something that's not for me. You should be able to access it here if you have a Microsoft account and want to try the button: https://copilot.microsoft.com/ https://copilot.microsoft.com/

novemp 2y ago

It feels like every few weeks someone puts out a new article breathlessly talking about how bad the fancy autocorrect bot is at math and logic like it's brand new information. How have we as a society not gotten this through our skulls yet?

simonw 2y ago

Here's "Einstein’s puzzle" from this paper: https://www.researchgate.net/publication/341189675_Is_Einstein's_Puzzle_Over-Specified https://www.researchgate.net/publication/341189675_Is_Einste... H01 There are five houses. H02 The Englishman lives in the red house. H03 The Spaniard owns the dog. H04 Coffee is drunk in the green house. H05 The Ukrainian drinks tea. H06 The green house is immediately to the right of the ivory house. H07 The Old Gold smoker owns snails. H08 Kools are smoked in the yellow house. H09 Milk is drunk in the middle house. H10 The Norwegian lives in the first house. H11 The man who smokes Chesterfields lives in the house next to the man with the fox. H12 Kools are smoked in a house next to the house where the horse is kept. H13 The Lucky Strike smoker drinks orange juice. H14 The Japanese smokes Parliaments. H15 The Norwegian lives next to the blue house. Now, Q1 Who drinks water? Q2 Who owns the zebra?

gorgoiler 2y ago
I am reminded of this Reddit post from November last year, except this is fifth grade math homework: https://www.reddit.com/r/mildlyinfuriating/comments/1gippqm/fifth_grade_math_homework_has_kid_in_tears_and/ https://www.reddit.com/r/mildlyinfuriating/comments/1gippqm/... The puzzle is a grid of simultaneous equations with 9 unknowns: 9 + ? - 6 = ? - 3 - = - = x 4 + ? x 5 - ? = 6 / + + - - 8 - 5 = ? - ? x 2 x - - - x ? + 1 + 7 - 4 = 8 = - = + = 7 = ? x 3 - ? - 2
- UncleEntity 2y ago
  Deepthink thought for 644 seconds and says the answer is 4. Probably didn't help that I just pasted it with no explanation, the spaces between the columns really threw it off.
  - gorgoiler 2y ago
    Here’s the same puzzle minus the grid parsing step: A1: 9+a-6=b-3 A2: 4+(c*5)-d=6 A3: 8-5=e-(f*2) A4: g+1+7-4=8 A5: 7=(h*3)-i-2 D1: 9-((4/8)*g)=7 D2: a=c+5-1-h D3: 6-5+e-7=3 D4: b=d-f-4+i D5: (3*6)-(2*8)=2

anon291 2y ago

This is ultimately a basic adaptation of the pigeonhole principle and is not surprising. A finite system of matrix multiplications cannot be turing complete. You cannot expect one trip through a series of matrix multiplications and bias additions and a final sampling at the end which commits it to a certain answer to ever produce a correct answer. It's a mathematical impossibility. No talk of quantum woo, emergent phenomenen, or whatever other pseudo-science has arisen to explain AI intelligence can get around this simple truth of mathematics. However, chain of thought reasoning where token streams can continue ad infinitum could potentially solve large swaths of problems whose ordinary solutions require turing machines. It could also solve problems that cannot generally be solved by turing machines, but where you only need solutions for a few classes of problems. Either way, even with chain of thought, you would expect that... in some instances, the model output diverges and does not complete. And unsurprisingly... this is exactly what you see with the DeepSeek models (and other CoT models) when you pose it difficult questions. It will never emit the </think> tag.

geoffhill 2y ago

Idk, `o3-mini-high` was able to pop this Prolog code out in about 20 seconds: solve(WaterDrinker, ZebraOwner) :- % H01: Five houses with positions 1..5. Houses = [ house(1, _, norwegian, _, _, _), % H10: Norwegian lives in the first house. house(2, blue, _, _, _, _), % H15: Since the Norwegian lives next to the blue house, house(3, _, _, milk, _, _), % and house1 is Norwegian, house2 must be blue. house(4, _, _, _, _, _), house(5, _, _, _, _, _) ], % H02: The Englishman lives in the red house. member(house(_, red, englishman, _, _, _), Houses), % H03: The Spaniard owns the dog. member(house(_, _, spaniard, _, dog, _), Houses), % H04: Coffee is drunk in the green house. member(house(_, green, _, coffee, _, _), Houses), % H05: The Ukrainian drinks tea. member(house(_, _, ukrainian, tea, _, _), Houses), % H06: The green house is immediately to the right of the ivory house. right_of(house(_, green, _, _, _, _), house(_, ivory, _, _, _, _), Houses), % H07: The Old Gold smoker owns snails. member(house(_, _, _, _, snails, old_gold), Houses), % H08: Kools are smoked in the yellow house. member(house(_, yellow, _, _, _, kools), Houses), % H11: The man who smokes Chesterfields lives in the house next to the man with the fox. next_to(house(_, _, _, _, _, chesterfields), house(_, _, _, _, fox, _), Houses), % H12: Kools are smoked in a house next to the house where the horse is kept. next_to(house(_, _, _, _, horse, _), house(_, _, _, _, _, kools), Houses), % H13: The Lucky Strike smoker drinks orange juice. member(house(_, _, _, orange_juice, _, lucky_strike), Houses), % H14: The Japanese smokes Parliaments. member(house(_, _, japanese, _, _, parliaments), Houses), % (H09 is built in: Milk is drunk in the middle house, i.e. house3.) % Finally, find out: % Q1: Who drinks water? member(house(_, _, WaterDrinker, water, _, _), Houses), % Q2: Who owns the zebra? member(house(_, _, ZebraOwner, _, zebra, _), Houses). right_of(Right, Left, Houses) :- nextto(Left, Right, Houses). next_to(X, Y, Houses) :- nextto(X, Y, Houses); nextto(Y, X, Houses). Seems ok to me. ?- solve(WaterDrinker, ZebraOwner). WaterDrinker = norwegian, ZebraOwner = japanese .

simonw 2y ago
Solving it with Prolog is neat, and a very realistic way of how LLMs with tools should be expected to handle this kind of thing.

WillAdams 2y ago

Perhaps applying: https://www.doc.ic.ac.uk/~re14/Evans-R-2020-PhD-Thesis.pdf https://www.doc.ic.ac.uk/~re14/Evans-R-2020-PhD-Thesis.pdf "Kant’s Cognitive Architecture" will help? > ...we provide a precise formalization of what it means > to “make sense” of a sensory sequence. According to our definition, making sense means constructing > a symbolic causal theory that explains the sensory sequence and satisfies a set of unity conditions > that were inspired by Kant’s discussion in the first half of the Critique of Pure Reason. According to > our interpretation, making sense of sensory input is a type of program synthesis, but it is unsupervised > program synthesis.

apsec112 2y ago

"showing that multilayer transformers indeed cannot solve certain complicated compositional tasks. Basically, some compositional problems will always be beyond the ability of transformer-based LLMs." Pretty sure this is just false and the paper doesn't show this. I could be misunderstanding, but it looks like the result is only about a single token/forward pass, not a reasoning model with many thousands of tokens like o1/o3

simonw 2y ago
I'm not sure that the statement "some compositional problems will always be beyond the ability of transformer-based LLMs" is even controversial to be honest. There's a reason all of the AI labs have been leaning hard into tool use and (more recently) inference-scaling compute (o1/o3/Gemini Thinking/R1 etc) recently - those are just some of the techniques you can apply to move beyond the unsurprising limitations of purely guessing-the-next-token.
- apsec112 2y ago
  o3 is still a transformer-based LLM, just one with a different loss function
  - simonw 2y ago
    Huh, yeah that's a good point. The various distilled R1 models are definitely regular transformer-based LLMs because the GGUF file versions of them work without any upgrades to the underlying llama.cpp library.

mohsen1 2y ago

https://chatgpt.com/share/679f0353-12bc-8007-91cf-dd63d52044a0 https://chatgpt.com/share/679f0353-12bc-8007-91cf-dd63d52044... O1 Pro gets the answer with proper reasoning Can’t tell if this is dude to data contamination or it really figured it out? How can we form the question in another way to avoid data contamination?

mohsen1 2y ago
With modified prompt to make less look like the original prompt it thought 6x more: https://chatgpt.com/share/679f086d-a758-8007-b240-38e684303786 https://chatgpt.com/share/679f086d-a758-8007-b240-38e6843037...
- pollinations 2y ago
  I don't think individual examples make sense to solve these kinds of discussions as for me it can vary easily by 6x thinking with exactly the same input and parameters.
deleted 2y ago
[deleted]

EGreg 2y ago

Maybe they’re not training them on the right thing, maybe math can be a different “mode” just lime sound or images

moolimon 2y ago

The main thesis here seems to be that LLMs behave like almost all other machine learning models, in that they are doing pattern matching on their input data, and short circuiting to a statistically likely result. Chain of thought reasoning is still bound by this basic property of reflexive pattern matching, except the LLM is forced to go through a process of iteratively refining the domain it does matching on. Chain of thought is interesting, because you can combine it with reinforcement learning to get models to solve (seemingly) arbitrarily hard problems. This comes with the caveat that you need some reward model for all RL. This means you need a clear definition of success, and some way of rewarding being closer to success, to actually solve those problems. Framing transformer based models as pattern matchers makes all the sense in the world. Pattern matching is obviously vital to human problem solving skills too. Interesting to think about what structures human intelligence has that these models don't. For one, humans can integrate absolutely gargantuan amounts of information extremely efficiently.

drakenot 2y ago
With DeepSeek-R1-Zero, their usage of RL didn't have reward functions really that indicated progress towards the goal afaik. It was "correct structure, wrong answer", "correct answer", "wrong answer". This was for Math & Coding, where they could verify answers deterministically.
- deleted 2y ago
  [deleted]
- mountainriver 2y ago
  It is a reward function it’s just a deterministic one. Reward models are often hacked preventing real reasoning from being discovered
huijzer 2y ago
> Framing transformer based models as pattern matchers makes all the sense in the world. Pattern matching is obviously vital to human problem solving skills too. Interesting to think about what structures human intelligence has that these models don't. For one, humans can integrate absolutely gargantuan amounts of information extremely efficiently. What is also a benefit for humans, I think, is that people are typically much more selective. LLMs train to predict anything on the internet, so for example for finance that includes clickbait articles which have a lifetime of about 2 hours. Experts would probably reject any information in these articles and instead try to focus on high quality sources only. Similarly, a math researcher will probably have read a completely set of sources throughout the life than, say, a lawyer. I’m not sure it’s a fundamental difference, but current models do seem to not specialize from the start unlike humans. And that might be in the way of learning the best representations. I know from ice hockey for example, that you can see within 3 seconds whether someone played ice hockey from young age or not. Same with language. People can usually hear an accent within seconds. Relatedly, I've used OpenAI's text to speech a while back and the Dutch voice had an American accent. What this means is that even if you ask LLMs about Buffett's strategy, maybe they have a "clickbait accent" too. So with the current approach to training, the models might never reach absolute expert performance.

mikeknoop 2y ago

One must now ask whether research results are analyzing pure LLMs (eg. gpt-series) or LLM synthesis engines (eg. o-series, r-series). In this case, the headline is summarizing a paper originally published in 2023 and does not necessarily have bearing on new synthesis engines. In fact, evidence strongly suggests the opposite given o3's significant performance on ARC-AGI-1 which requires on-the-fly composition capability.

orbital-decay 2y ago
It's Quanta being misleading. They mention several papers but end up with this [1] which talks about decoder-only transformers, not LLMs in general, chatbots, or LLM synthesis engines, whatever that means. The paper also proves that CoT-like planning lets you squeeze more computation from a transformer, which is... obvious? but formally proven this time. Models trained to do CoT don't have some magical on-the-fly compositional ability, they just invest more computation (could be dozens millions of tokens in case of o3 solving the tasks from that benchmark). [1] https://arxiv.org/abs/2412.02975 https://arxiv.org/abs/2412.02975

antirez 2y ago

LLMs keep showing more and more they are the wonder of AI that we awaited for decades: talking machines that every two months do progresses that two months before were anticipated impossible because of <put here some limit that was actually in the prejudice of the skeptical AI community> (just stochastic parrots, no reasoning possible without symbolic representations, there are no longer tokens, ...) At the same time, part of the scientific community continues to diminish what was accomplished and the steps that are being made. A few months ago LeCun arrived to tell new researchers to move away from LLMs since they are a dead end: imagine the disservice he made to the surely non-zero folks that followed the advice, putting themselves out of the AI research that matters. (Incidentally, this skepticism of the Meta AI head must have something to do with the fact that Meta, despite the huge efforts allocated, produced the worst LLM among Anthropic, OpenAI, DeepSeek -- I bet Zuckerberg is asking questions lately). It's very hard to explain this behavior if not by psychological denial. [EDIT: you can't see the score of this comment, but I can: it's incredible how it goes from 3, to -2, to 1, and so forth. The community is split in two, and it is pretty sad since this is not a matter of taste or political inclination: there must be a single truth]

ramblerman 2y ago
I get the sentiment, but I actually think some skepticism in the system is healthy. Billions are flowing towards LLMS, and Sam Altman will overpromise AGI is just around the corner and the days of jobs are gone to fill his coffers to anyone that will listen. Additionally if we begin to use these things in real production environments where mistakes matters, knowing the exact limitations is key. None of this takes away from the fact that these are exciting times.
- antirez 2y ago
  Yes there is another part of the community that overhypes everything. But I can expect that from a CEO of an AI company (especially if he is Altman), but from researches? Also the fact that LLMs may reach superhuman expertise in certain fields in a short timeframe (a few years), since reinforcement learning is starting to be applied to LLMs may no longer be a totally crazy position. If it is possible to extend considerably the same approach seen in R1-Zero there could be low hanging fruits around the corner.

comeonbro 2y ago

> Chatbot Software Begins to Face Fundamental Limitations > Recent results show that large language models struggle with compositional tasks, suggesting a hard limit to their abilities. Your first question with anything like this should always be WHICH MODELS: > For our experiments, we evaluate the performance of 6 LLMs: GPT4 (gpt-4) [58], ChatGPT (GPT3.5-turbo) [57], GPT3 (text-davinci-003) [11], FlanT5 [17] and LLaMa [75]. This is ancient. This research was done centuries ago. This is research about the possibility of isotopes, written about radium in 1903, published in 1946. It is a criminal level of journalistic malpractice to leave uninformed readers with the impression that this is where AI stands yesterday.

deleted 2y ago

[deleted]

deleted 2y ago

[deleted]

MagicMoonlight 2y ago

Do you really need a study to work out that a Markov chain can’t solve problem questions? It feels pretty intuitive. LLMs are not intelligent. Human intelligence is stored in text, so mimicking text mimicks that intelligence. The sheer quantity of text means they’ve probably seen something similar to the prompt. If LLMs were an intelligent system they wouldn’t need to steal 1000TB of text and media content. Have you ever seen a person require a million books? It shows that the learning is brute force rather than intelligent. It’s close, but it’s just not there. The methodology is wrong.

red75prime 2y ago
The Markov chain wouldn't fit in a googol of observable universes. The analogy is absolutely misleading. There's no reason to think that the chain of that humongous size cannot capture at least something of what humans with their limited brain size can do. It cannot capture non in-context learning, true. But we have tools to modify the "Markov chain".
cjblomqvist 2y ago
I guess your point is that it's obvious, but I'd argue it's not. A lot of people are hyping up LLMs as the solution to all our problems, and there have been a lot of discussion around what's reasonable use cases (and a lot of anecdotal evidence trying to argue and support certain stances). My point is that we need (scientific) proof to move forward, and create confidence in "facts" we base our decisions on. Obviously that doesn't seem to be very fashionable nowadays - on both sides of the LLM discussion there seems to be a lot of people arguing hard for there stances. I'm very glad some are working on providing some facts into this discussion. (one can argue about science vs truth, but regardless it's a step in the right direction/a good motive) (this is btw why a solid educational foundation, not only skill focused, is very critical for society. cf democracy)

seydor 2y ago

"Fundamental" results cannot come from empirical findings, they have to be proven rigorously. Until then, they are not fundamental

mmcnl 2y ago

There's so much talk about the advancements in AI/LLMs, yet for me ChatGPT as of this date is basically just a faster search engine without cookie banners, clickbait and ads. It hallucinates a lot and it can keep very limited context. Why is there so much promise about future progress but so little actual progress?

EA-3167 2y ago
It's the same cycle we saw with Crypto, there's so much money flying around that the motivation to "believe" is overwhelming. The hype is coming from all directions, and people are social animals that put greater weight on words that come from multiple sources. It's also a platform for people excited about the future to fantasize, and for people terrified of the future to catastrophize.
knowaveragejoe 2y ago
I have to wonder how you are using ChatGPT to get a lot of hallucinations or run into issues with limited context.

kebsup 2y ago

I've managed to get llms fail on simple questions, that require thinking graphically - 2D or 3D. An example would be: you have a NxM grid. How many shapes of XYZ shape can you fit on it? However, thinking of the transformer video games, AI can be trained to have a good representation of 2D/3D worlds. I wonder how it can be combined so that this graphical representation is used to compute text output.

anshumankmr 2y ago

Isn't the math calculation thing solvable through the integration with a simple Python script for basic math and more complex stuff something like an open source version of Wolfram?

Ndv_MC 2y ago

Quite deep and informative. One thing to point out here is how they measure accuracy when testing their LLMs. Like most neural networks, the n-dimensional space of LLMs is largely sparse. In simple terms, this means that it’s very easy to “exercise” the model in areas where it’s weak. That doesn’t mean that the model doesn’t have the potential of doing a (much) better job if you “exercise” its space properly—which is exactly what techniques like CoT basically do. LLMs are inherently limited, and they don’t have “true” reasoning capabilities—beyond marketing hype, this much should be obvious to most serious ML researchers today. The only question is how well we can get them to “mimic” reasoning for practical applications, and this is where “prompt engineering”, strictly speaking, is a true form of engineering, which has to take into account the mathematical foundations of the models and figure out how to extract out of them the best performance they can deliver.

starchild3001 2y ago

What a poorly informed article. It's very shallow and out of touch with LLM research. As it stands 6-12 months old models are system 1 thinkers, everybody knows this and knew this even at the time. You need system 2 thinking (test time compute) for more complex logical, algorithmic and reasoning tasks. We knew this when Daniel kahneman wrote thinking fast, thinking slow (over a decade ago) and we still know it today. So LLMs can think but they have to be programmed to think (a la system 2, reasoning, thinking models). There's nothing inherently wrong or limited with LLMs themselves as far as we can tell.

tshadley 2y ago
> To understand the capabilities of LLMs, we evaluate GPT3 (text-davinci-003) [11], ChatGPT (GPT-3.5-turbo) [57] and GPT4 (gpt-4) Oh dear, this is embarrassing. Anil Anathaswamy, are you aware a year in AI research now is like 10 years in every other field?
astrange 2y ago
This is an example of "metaphor-driven development" in AI, which Phil Agre criticized a few decades ago. System 1/System 2 isn't a real thing. It's just a metaphor Kahneman invented for a book. AI developers continually find metaphors about the brain, decide they are real, implement something which they give the same name, decide it's both real and the same thing because they have given it the same name, and then find it doesn't work. (Another common example is "world model", something which has never had a clear meaning, and if you did define it you'd find that people don't have one and don't need one.)

strangescript 2y ago

Why do articles testing ancient LLMs keep getting posted?

Workaccount2 2y ago

"Recent results" = two year old study looking at GPT-3, 3.5, and first gen 4.

fyygfdrryhcss 2y ago

[dead]

have-a-break 2y ago

Maybe it's just me but when i code/program i typically think of how different variations of implementations would affect roi of the associated company i would be working for. That's something that AI in the current state probably wouldn't be able to take into account.

emorning3 2y ago

I'm a software developer with just a basic understanding of LLMs. But its not surprising to me that LLMs wouldnt be good at composition and planning. To me, LLMs seem like half a brain, the right half, that does pattern matching and such. To me, the current crop of AI seems to be missing the other half of the brain, the part that's good at planning and composition and such. LLMs can tell me all I want to know about logic and deduction, so its funny that LLMs cant understand their own words.

malaigo 2y ago

I'd like to propose a modified Zebra puzzle. Still has unique solutions. Not on the internet, so not part of the training set. Similar problem description complexity. Just remove the last constraint (Norwegian lives next to the blue house) and asked negated questions: Who cannot be the water drinker? Who cannot own the zebra? Text of problem here: sandiway.arizona.edu/mzebra.pdf (I tested it on free ChatGPT and DeepSeek-R1 https://youtu.be/3gotauWUcew https://youtu.be/3gotauWUcew)

bpositiveteam1 2y ago

[dead]