15 ms·

LLMs can't self-correct in reasoning tasks, DeepMind study finds

166 points by nyrikki 3y ago

fennecfoxy 3y ago
Well no of course not. Not only are they still only predicting the next best word ("best" now working better thanks to attention), but anybody who's role played with an LLM learns many caveats of the architecture. I think especially regarding spatial awareness/object permanence etc LLMs lack any ability to work with these concepts when it comes to predicting the next token. I feel like we need to add more specialised heads (which I think some training regimen for LLMs already courages) for things like these concepts. It's easy for them to lose track of where an item is and you have to catch its errors early and correct it, otherwise it'll take the error and run with it.
nyrikki 3y ago
Pre-print https://arxiv.org/abs/2310.01798 https://arxiv.org/abs/2310.01798
adocomplete 3y ago
...yet.
- tomrod 3y ago
  I work a bit in this space. Current LLM architectures are missing a fundamental "state of the world" as well as the ability to counterfactually reason out of error propagation. Simply a limit of the architecture. Instead, we should be giving model architectures like JEPA a go [0], which explicitly perform LLM-like behavior but with a state of the world implemented for ongoing error correction. [0] https://ai.meta.com/blog/yann-lecun-ai-model-i-jepa/ https://ai.meta.com/blog/yann-lecun-ai-model-i-jepa/
  - singularity2001 3y ago
    Remember Andrew Ngs theses that current architectures are already "AI complete" meaning that given more training data all models outperform other models with less training data, and a model which seems a bit more efficient with some amount of training data can be less efficient with more data or the other way around.
    - deleted 3y ago
      [deleted]
    - tomrod 3y ago
      Do you mind providing a link to your reference?
- deleted 3y ago
  [deleted]
sabareesh 3y ago
MultiAgent could potential solve this easily
- Jensson 3y ago
  That type of statement never seem to age well for AI technologies.
- asadm 3y ago
  can you expand how?
  - sabareesh 3y ago
    I am noticing there is enough entropy on the agent replys and also as we can adjust the temperature they can work together to provide a better answer by peer reviewing.
ldjkfkdsjnv 3y ago
I trust nothing from anyone that cant compete with Gpt-4
- yawaramin 3y ago
  In other words, you need to have Google scale money to invest in developing an LLM before you can be trusted?
  - ldjkfkdsjnv 3y ago
    A bunch of companies and people speculating on technology that they cant even reproduce
- Jensson 3y ago
  Then you should trust OpenAI when they said that they have already ran out of ideas to solve these issues? That they need need novel approaches to make further progress than they already have? Or do you mean that you don't trust anyone at all? https://www.wired.com/story/openai-ceo-sam-altman-the-age-of-giant-ai-models-is-already-over/ https://www.wired.com/story/openai-ceo-sam-altman-the-age-of...
  - famouswaffles 3y ago
    >when they said that they have already ran out of ideas to solve these issues? They never said that lol. https://web.archive.org/web/20230531203946/https://humanloop.com/blog/openai-plans https://web.archive.org/web/20230531203946/https://humanloop...
dist-epoch 3y ago
Stephan Wolfram talks about how we'll soon have LLM psychologists, that if you add to the prompt "take a deep breath before answering" you get better answers. He also speculates that if after the answer you ask it "is what you wrote correct?" you give the LLM a chance to get out of path dependency - one badly generated token and the LLM is now forced to follow it through.
vendiddy 3y ago
Can anyone explain how this works? - I ask ChatGPT a question - It gives me an incorrect answer - I literally say "you're wrong" - It corrects itself Why didn't it give me the right answer the first time? How was it able to get the answer right the second time?
- amelius 3y ago
  Simply put, it was giving random answers, until you stopped asking. (Of course it's more complicated but that's basically it).
  - Jensson 3y ago
    Even a horse can do complex math as long as you signal when it should stop.
  - Karellen 3y ago
    Heh. Kind of like Clever Hans. It keeps tapping its foot until you relax. https://en.wikipedia.org/wiki/Clever_Hans https://en.wikipedia.org/wiki/Clever_Hans
  - justrealist 3y ago
    This is so wrong it actively hurts.
  - empath-nirvana 3y ago
    It is many times better than you'd expect from random answers.
    - roflyear 3y ago
      You missed the point.
- mucle6 3y ago
  I can write code, and if I'm told my code doesn't work, I can fix it.
- 3y ago
bradley13 3y ago
LLMs have no real understanding, so of course they cannot self-correct. They can't even correct on command. I had an example just today: I wanted an example of a piece of code using framework X to do task Y. I didn't realize that what I wanted is not possible. The LLM I was using gave me code that worked, but did not do what I wanted. I pointed this out, so I then got code that syntactically did what I wanted, but could never work. This went back and forth twice, before I gave up. The LLM was incapable of recognizing the falsity of its answers. It certainly wasn't capable of suggesting a different approach that would work (and yes, there was one, I found after a bit more research).
- jncfhnb 3y ago
  No! This is a bad argument. Your one off example of an LLM not being able to handle a request that is not solveable is not relevant to claim of whether LLMs can fundamentally do this. Curren state LLMs do current have the ability to point out that some requests are impossible. They’re not great at it. But they can do it. What you’re hitting is a current state quality problem. There’s no particular reason to think that it’s unachievable with the same approach.
- fennecfoxy 3y ago
  When role playing, I've gotten it to self correct during games a few times, as well as correcting itself when told to. Also if something is not possible then an LLM, unless trained otherwise, will do its best to answer you no matter what. Current training doesn't really include answering "I don't know" or "this isn't possible" apart from examples with very narrow/shallow scope like "is an apple a banana?" but something like "give me some code that sues the React.doesNotExist function in React" is just too wide/deep informationally for it to handle. I often find that "warming" an LLM up to the topic so that the context is full of content only relevant to the current topic helps much more than trying a fresh query in this regard. But with a role play game I've definitely found that it can correct itself or be corrected, usually I mark comments outside of the game with something like >character x is not in the room and therefore cannot hear character y< and it corrects the output for the most part. I've also successfully managed to get gpt3.5 to role play as several different characters at the same time, up to four characters in a single response whilst maintaining the individual characters' personalities and current goals/objectives. But the small context window does mean that certain details are lost - I think a large part of LLMs interpretation of "reasoning" is actually the context and not the weights. For generalised LLMs the weights are just such a giant glob of dense information that it really does need heavy use of context/attention to narrow things down, things mostly seem to go awry as the context starts to be truncated. One cool thing I guess would be for someone to see what a multi-context/multi-"thread" model looks like. Where the larger, central context is the current query/very recent history, but smaller more supplemental contexts are provided with relevant/condensed information.
visarga 3y ago
The cold water bucket that needs to fall on people who fear monger job automation. These models are stumbling in a few steps. Nobody can go on vacation and leave AI do their job. What we can automate - is maybe 2-3 turns until it gets stuck - should be thousands of steps for real automation. It's almost 0% autonomy right now, how do we go from 0% to 1%?
- zemvpferreira 3y ago
  I don't think I'm a fear mongerer but the amount of reasonably qualified jobs that can be displaced by the models we have now seems huge. In the hands of a qualified human AI-manager they are incredibly productive when the output doesn't have to be cutting edge. I know people who have already automated most of their own job and some colleagues through linking together disparate AI models. Think graphic designer at medium-sized law firm. Or copywriter at a scientific publisher. Those jobs are going, going, gone.
  - swatcoder 3y ago
    If you think about this generation of text/media AI as being a kind of generalized decrompressor for some input prompt, the output has more detail than the input but never more information. So as any model becomes widespread, people will become increasingly sensitive to the sameness of easy, simple prompts. So the graphic designer at the law firm may lose their job tomorrow, but the law firm is going to have to hire a AI Image Technician in a few years so that their stuff looks like somebody actually invested in it (which matters to the law firm’s brand image or they’d not have a graphic designer in the first place). So yeah, the job market will shuffle around, but it’s unclear what that means in net number of jobs or what share of AI Image Technicians will literally have just been Graphic Designers five years earlier.
  - dimgl 3y ago
    > I don't think I'm a fear mongerer but the amount of reasonably qualified jobs that can be displaced by the models we have now seems huge. Maybe I'm living under a rock, but I have serious doubts about this claim. > Think graphic designer at medium-sized law firm. Or copywriter at a scientific publisher. How is that even possible? Take the example of the graphic designer. You'd still need them to iterate on ideas, come up with different versions for different backgrounds, generate different file types depending on the medium and format, etc. Extremely skeptical of your claims.
tomohelix 3y ago
It tracks with the way LLMs work. From what I understand, during training, the LLMs build a correlation between words, a process after which their weight network may be configured in a way that superficially resemble actual reality. Then for whatever input this LLM takes in, it uses its "reasoning" based on this configuration to spit out a response. A bigger LLM has more parameters to model the world and may have a better performance. But I guess even 1 trillion parameters is still not enough to model the actual reality with enough depth. So if their internal understanding of the world is inherently flawed, you can't expect them to be able to spot their own faults. To be fair, this is the same for human. I am sure we all have that experience trying to tell someone where they are wrong and had a hard time. But with a human, we can update our internal world model on the fly. An LLM at runtime can't do that. It needs to be retrained.
- SrslyJosh 3y ago
  I think of this as the "abstraction level problem". LLMs work at the wrong abstraction level (statistical relationships between individual words) for the tasks people want to use them for (high-level reasoning). Image generators have a similar problem -- they suck at large-scale composition and have problems with details like hands because the abstraction level they're working at (individual pixels and their close neighbors) is much lower than the level that a human artist works at.
SrslyJosh 3y ago
This is utterly unsurprising, since LLMs are statistical models of text, not inference engines (or similar). > Interestingly, the models often produce the correct answer initially, but switch to an incorrect response after self-correction. The training corpus almost certainly contains examples that are similar to both cases ("my answer was correct" and "oops, my answer was wrong"). It'll be a coin flip which set the input more closely matches.
FeepingCreature 3y ago
I wonder to what extent this is because ChatGPT is trained to take any feedback as a sign that its answer is wrong and needs to be corrected. Such a training would, I suspect, explain the effect where the model first generates the correct answer, then introspects and self-doubts. In training, when it was told to introspect, that would usually be a hint that its answer was wrong. This results in an agent whose confidence reliably falters under introspection. I've observed that ChatGPT quite eagerly "corrects" itself at the first sign of negative feedback, even if its answer was already correct. It's actually quite annoying. If this is correct, a more balanced training that contains cases where the human response is wrong would solve this issue.
- kromem 3y ago
  This is brought up in the paper and is effectively their reasoning. That because it is instructed to "find problems" with the answer it leads to correct answers being biased towards changing to incorrect more often than it can successfully bias incorrect answers towards correct ones. Here, I think the research team might have had more success using neutral language like "grade this answer" where both verification and challenge would fit the prompt as opposed to "find problems with this answer" where only challenges would - but I'm skeptical it would be enough of an improvement that it would yield a net improvement on self-critique over the initial answer. There are other more specialized approaches I'd expect will always work better than feeding an incorrect answer back into what generated it in the first place (i.e. a backwards pass where the model has to match the answer provided to multiple possible questions with one being the question asked and the others generated on the fly).
montebicyclelo 3y ago
The paper: https://arxiv.org/pdf/2310.01798.pdf https://arxiv.org/pdf/2310.01798.pdf > Our research shows that LLMs are not yet capable of self-correcting their reasoning The paper actually just shows that the particular "self-correction" strategy and set of prompts they used doesn't help for the tasks they looked at, for the models they looked at. It may be the case in general, but it may not. > it is plausible that there exist specific prompts or strategies that could enhance the reasoning performance of models for particular benchmarks Seems they agree. So the wording of the title/conclusion is too strong. > searching such prompts or strategies may inadvertently rely on external feedback, either from human insights or training data I'm not sure this justifies picking a single prompting strategy, and not looking at the impact of different prompting strategies. Even just writing a few different prompts in advance with different wordings and showing the variation in results would have been helpful.
- aeternum 3y ago
  Yes, these papers just demonstrate how weak scientific papers have become. The paper is weak on actual prompt examples, but of the few there, test them yourself and it consistently gets them right. Whenever I see examples of GPT4 can't do such and such, I'm generally able to find a prompt that does in fact work relatively consistently.
  - kromem 3y ago
    I don't know that it's as much a matter of papers having become weak as much as there being a knowledge gap between use of models in practice and in research. For example, a glaring issue that caught my eye in the paper was that their prompt for the self-evaluation was setting the context for what was being analyzed as its own work. Out of the training data of the effective Internet, what % of critical analysis was self-critique and what % do we think was the analysis of others' work? As such, if we're trying to elicit an effective analysis of an earlier answer from an incomprehensible multi-variable prediction machine, might we not want to set the context as the evaluation of another's earlier work? It's still technically self-evaluation even if we are hiding implementation details from the LLM. Another example is that their prompt to a fine tuned instruct model asked it to "find problems." And then they found that the self-critique would often bias correct answers to change to incorrect. What about having used more neutral language like 'grade' that allows both verification or challenge of the earlier answer to fit the instruction? These are the kinds of nuances I'm sure many working with LLMs in production scenarios have realized can completely change the outcomes of a prompt pipeline, and yet in research we see very smart analysis (i.e. implicit bias towards correction over multiple rounds of challenges when incorrect) coupled with less than ideal prompt selection for the methods. And given that we should expect every new generation of models to have new and different nuances to what works in practice and what doesn't, I don't know that this is a problem that's going to get better before it gets worse.
tanvach 3y ago
Alternative explanation - maybe the training data is full of internet discussions, and it's just that we we really bad at self correction online.
MuffinFlavored 3y ago
I wonder in LLMs either need to be rethought/retrained to say "I don't know" instead of guessing (I have no clue what would go into this) or... if we somehow are able to take a "test-driven-approach" with them. Upfront, we tell it what criteria must pass. Connect it to a "test/task" runner of some sort per prompt. Then, it keeps trying behind the scenes to come up with an answer that passes the criteria (to prevent humans from having to waste iterations on 'no, you got it wrong, try again') However, I will admit... most of the time when the LLM can't solve it, retrying and retrying over and over slightly different ways really doesn't typically end up in success. If the task is "too complex" for the LLM, I have yet to find a way to break it down context wise and overcome the limitation.
- zitterbewegung 3y ago
  In some situations you can make the prompt tell you. "I don't know". For example "You are a friendly D&D dungeon master and know all of the rules of D&D and run the game. If you interact with a player you should observe all of the rules. If you don't know if they are following the rules then say "I don't know"
- viraptor 3y ago
  > I wonder in LLMs either need to be rethought/retrained to say "I don't know" instead of guessing (I have no clue what would go into this) You can already do this yourself to some extent by providing examples in the prompt. Ask your question, explicitly add that you're allowing the "don't know" answer and add that for this similar question the answer is X, for that the answer is don't know, for that the answer is Y. It's a bit of work, but it works in practice. Check out the concept of priming. Yeah, it would be nice if this was pre-trained - maybe in the future.
- trifurcate 3y ago
  Non-instruct-tuned LLMs are far more likely to say "I don't know".
kazinator 3y ago
What chat LLMs do is correct their answers by cribbing from the information you give in the conversation, and then take credit for it.
RecycledEle 3y ago
LLMs reason like we autistic people do. I almost never correct myself after someone pointed out my errors. I either find the error myself or it takes my many hours of contemplation to see my error(s).
- delecti 3y ago
  LLMs don't reason, and autistic people have an internal state. We have thoughts and memories, and understand context. If you were trying to fit a pegs into holes, you'd correct pretty quickly when the square peg didn't fit a circle hole.
jeffbee 3y ago
"For GPT-4, to reduce the cost, we randomly sampled 200 questions" Wow, DeepMind research budgets must be nothing.
madsbuch 3y ago
I asked ChatGPT to explain fast exponentiation. It did 9 iterations until it came up with the right algorithm. Why would this not be considered self-correction?
m3kw9 3y ago
It seems like they self correct when you explain to the LLM something is incorrect. But in reality they take the original question, plus the “incorrect” statement and get the most likely responses from the model
dansalvato 3y ago
I would really like for LLMs to reach a point where they can actually tell you if they're not quite sure about something, rather than confidently outputting their BS like it's fact. Too many people interpret LLM output as though it's guaranteed correct. Certainly there's some confidence value metric that they can understand internally and use when wording their response.
- munchler 3y ago
  The irony of this comment is rich. There’s no such “confidence value metric” known, no matter how “certain” you are. This is why techniques like chain-of-thought or self-correction are used instead. LLM’s predict the next token in a sequence. That’s it. The final softmax layer gives a per-token probability, but that is not the same thing as overall confidence in the response.
  - famouswaffles 3y ago
    The irony of this comment is rich. >The final softmax layer gives a per-token probability, but that is not the same thing as overall confidence in the response. Clearly not. https://imgur.com/a/3gYel9r https://imgur.com/a/3gYel9r
    - munchler 3y ago
      Hah, I wasn’t aware of that. Does it say how the base model computes its confidence? Is it requested in the prompt?
      - famouswaffles 3y ago
        They didn't specify.
    - singularity2001 3y ago
      This may be philosophically one of the most important graphs regarding the current state of AI! It shows that large language models already DO have an internal model of confidence in their statements, all that is needed is to train the models to output this confidence alongside their hallucinations if the confidence is low.
deleted 3y ago
[deleted]
tangentstorm 3y ago
I tested their first example in ChatGPT 4. Interestingly, it gives the correct answer on the first try: https://chat.openai.com/share/d86fe16a-9dfd-4753-8eaf-6d2948096ea3 I then gave GPT4 a "chain-of-thought" flavored prompt, telling it to treat the problem like a geometry proof. It gave the same incorrect answer as GPT3 did in the paper. I then told it to "Review your work and check for mistakes." With this follow-up, it checked each line of the proof and was able to find and explain the error: https://chat.openai.com/share/c4ce6e98-43e3-4547-a4c8-380c1d1cc5fe GPT3.5, given the same prompts, was still confident in its incorrect answer: https://chat.openai.com/share/1a0be419-092d-4dbb-a6c3-79e61914fd0d (edited to update links and ask them each version to check its work twice)
- YeGoblynQueenne 3y ago
  >> I then told it to "Review your work and check for mistakes." With this follow-up, it checked each line of the proof and was able to find and explain the error: And after that? Did you tell it that it may be wrong and ask it to check again?
  - tangentstorm 3y ago
    Good point. I did now. Both models stuck with their second answer. (I see now I didn't know how to share ChatGPT links properly. Updating the links now...)
jaidhyani 3y ago
I am begging people to stop confusing "I was unable to get LLM X to do Y using strategy Z" with "All LLMs are categorically unable to do Y".
happytiger 3y ago
If you add “This generation of” to the beginning of all of this research on existing LLMs you’ll have a clearer understanding. Like all young technologies it’s a constant failure in so many important ways, and will continue to be until it’s not or another superior model built on what we’ve learned replaces it. But the important thing to remember is that this process is fairly standard for any technology innovation wave. They suck at the beginning and take a longer time to mature than people realize even while they’re going through them. The early Internet was mindblowingly amateur. Early cell phones were basically pager quality and the smart phone revolution started with things like palm pilots that couldn’t even connect to one another or barely could. You have to look at LLMs as cell phones from 1991 or maybe 1993, and when you do you realize how freekin’ crazy these technology direction could be. This generation of LLMs are simply designed to regurgitate information in a coherent fashion, not be right about the information or maintain consistency (they literally dont have the memory capacity — it’s just like early computers were lol). People are poor predictors of technology progress because they tend to extend current reality forward and don’t realize that technology works in leaps and bounds and isn’t linearly predictable with any substantive reliability either.
anigbrowl 3y ago
Nature shows us that bilaterality works. Connecting two LLM instances together and and giving them slightly different prompts to appraise each others' output is going to be a lot more effective than trying to iterate some monolithic super-reasoning device.
kristianp 3y ago
I've taken to adding guidance to my original prompt instead of trying to correct the model. Perhaps having the incorrect answer plus the correction in the context causes it to be dumber?
p5a0u9l 3y ago
People forget that getting answers is not what makes a good solution. Computers are still dumb, they only give answers. Asking the right question is everything. With the right question the uncertainty of a problem is resolved. Finishing is a mere exercise. whether hand-crafted assembly or an LLM’s regurgitation and synthesis of things it’s seen before is just a matter of efficiency.

166 points by nyrikki 3y ago

Well no of course not. Not only are they still only predicting the next best word ("best" now working better thanks to attention), but anybody who's role played with an LLM learns many caveats of the architecture. I think especially regarding spatial awareness/object permanence etc LLMs lack any ability to work with these concepts when it comes to predicting the next token. I feel like we need to add more specialised heads (which I think some training regimen for LLMs already courages) for things like these concepts. It's easy for them to lose track of where an item is and you have to catch its errors early and correct it, otherwise it'll take the error and run with it.

nyrikki 3y ago

Pre-print https://arxiv.org/abs/2310.01798 https://arxiv.org/abs/2310.01798

adocomplete 3y ago

...yet.

tomrod 3y ago
I work a bit in this space. Current LLM architectures are missing a fundamental "state of the world" as well as the ability to counterfactually reason out of error propagation. Simply a limit of the architecture. Instead, we should be giving model architectures like JEPA a go [0], which explicitly perform LLM-like behavior but with a state of the world implemented for ongoing error correction. [0] https://ai.meta.com/blog/yann-lecun-ai-model-i-jepa/ https://ai.meta.com/blog/yann-lecun-ai-model-i-jepa/
- singularity2001 3y ago
  Remember Andrew Ngs theses that current architectures are already "AI complete" meaning that given more training data all models outperform other models with less training data, and a model which seems a bit more efficient with some amount of training data can be less efficient with more data or the other way around.
  - deleted 3y ago
    [deleted]
  - tomrod 3y ago
    Do you mind providing a link to your reference?
deleted 3y ago
[deleted]

sabareesh 3y ago

MultiAgent could potential solve this easily

Jensson 3y ago
That type of statement never seem to age well for AI technologies.
asadm 3y ago
can you expand how?
- sabareesh 3y ago
  I am noticing there is enough entropy on the agent replys and also as we can adjust the temperature they can work together to provide a better answer by peer reviewing.

ldjkfkdsjnv 3y ago

I trust nothing from anyone that cant compete with Gpt-4

yawaramin 3y ago
In other words, you need to have Google scale money to invest in developing an LLM before you can be trusted?
- ldjkfkdsjnv 3y ago
  A bunch of companies and people speculating on technology that they cant even reproduce
Jensson 3y ago
Then you should trust OpenAI when they said that they have already ran out of ideas to solve these issues? That they need need novel approaches to make further progress than they already have? Or do you mean that you don't trust anyone at all? https://www.wired.com/story/openai-ceo-sam-altman-the-age-of-giant-ai-models-is-already-over/ https://www.wired.com/story/openai-ceo-sam-altman-the-age-of...
- famouswaffles 3y ago
  >when they said that they have already ran out of ideas to solve these issues? They never said that lol. https://web.archive.org/web/20230531203946/https://humanloop.com/blog/openai-plans https://web.archive.org/web/20230531203946/https://humanloop...

dist-epoch 3y ago

Stephan Wolfram talks about how we'll soon have LLM psychologists, that if you add to the prompt "take a deep breath before answering" you get better answers. He also speculates that if after the answer you ask it "is what you wrote correct?" you give the LLM a chance to get out of path dependency - one badly generated token and the LLM is now forced to follow it through.

vendiddy 3y ago

Can anyone explain how this works? - I ask ChatGPT a question - It gives me an incorrect answer - I literally say "you're wrong" - It corrects itself Why didn't it give me the right answer the first time? How was it able to get the answer right the second time?

amelius 3y ago
Simply put, it was giving random answers, until you stopped asking. (Of course it's more complicated but that's basically it).
- Jensson 3y ago
  Even a horse can do complex math as long as you signal when it should stop.
- Karellen 3y ago
  Heh. Kind of like Clever Hans. It keeps tapping its foot until you relax. https://en.wikipedia.org/wiki/Clever_Hans https://en.wikipedia.org/wiki/Clever_Hans
- justrealist 3y ago
  This is so wrong it actively hurts.
- empath-nirvana 3y ago
  It is many times better than you'd expect from random answers.
  - roflyear 3y ago
    You missed the point.
mucle6 3y ago
I can write code, and if I'm told my code doesn't work, I can fix it.
3y ago

bradley13 3y ago

LLMs have no real understanding, so of course they cannot self-correct. They can't even correct on command. I had an example just today: I wanted an example of a piece of code using framework X to do task Y. I didn't realize that what I wanted is not possible. The LLM I was using gave me code that worked, but did not do what I wanted. I pointed this out, so I then got code that syntactically did what I wanted, but could never work. This went back and forth twice, before I gave up. The LLM was incapable of recognizing the falsity of its answers. It certainly wasn't capable of suggesting a different approach that would work (and yes, there was one, I found after a bit more research).

jncfhnb 3y ago
No! This is a bad argument. Your one off example of an LLM not being able to handle a request that is not solveable is not relevant to claim of whether LLMs can fundamentally do this. Curren state LLMs do current have the ability to point out that some requests are impossible. They’re not great at it. But they can do it. What you’re hitting is a current state quality problem. There’s no particular reason to think that it’s unachievable with the same approach.
fennecfoxy 3y ago
When role playing, I've gotten it to self correct during games a few times, as well as correcting itself when told to. Also if something is not possible then an LLM, unless trained otherwise, will do its best to answer you no matter what. Current training doesn't really include answering "I don't know" or "this isn't possible" apart from examples with very narrow/shallow scope like "is an apple a banana?" but something like "give me some code that sues the React.doesNotExist function in React" is just too wide/deep informationally for it to handle. I often find that "warming" an LLM up to the topic so that the context is full of content only relevant to the current topic helps much more than trying a fresh query in this regard. But with a role play game I've definitely found that it can correct itself or be corrected, usually I mark comments outside of the game with something like >character x is not in the room and therefore cannot hear character y< and it corrects the output for the most part. I've also successfully managed to get gpt3.5 to role play as several different characters at the same time, up to four characters in a single response whilst maintaining the individual characters' personalities and current goals/objectives. But the small context window does mean that certain details are lost - I think a large part of LLMs interpretation of "reasoning" is actually the context and not the weights. For generalised LLMs the weights are just such a giant glob of dense information that it really does need heavy use of context/attention to narrow things down, things mostly seem to go awry as the context starts to be truncated. One cool thing I guess would be for someone to see what a multi-context/multi-"thread" model looks like. Where the larger, central context is the current query/very recent history, but smaller more supplemental contexts are provided with relevant/condensed information.

visarga 3y ago

The cold water bucket that needs to fall on people who fear monger job automation. These models are stumbling in a few steps. Nobody can go on vacation and leave AI do their job. What we can automate - is maybe 2-3 turns until it gets stuck - should be thousands of steps for real automation. It's almost 0% autonomy right now, how do we go from 0% to 1%?

zemvpferreira 3y ago
I don't think I'm a fear mongerer but the amount of reasonably qualified jobs that can be displaced by the models we have now seems huge. In the hands of a qualified human AI-manager they are incredibly productive when the output doesn't have to be cutting edge. I know people who have already automated most of their own job and some colleagues through linking together disparate AI models. Think graphic designer at medium-sized law firm. Or copywriter at a scientific publisher. Those jobs are going, going, gone.
- swatcoder 3y ago
  If you think about this generation of text/media AI as being a kind of generalized decrompressor for some input prompt, the output has more detail than the input but never more information. So as any model becomes widespread, people will become increasingly sensitive to the sameness of easy, simple prompts. So the graphic designer at the law firm may lose their job tomorrow, but the law firm is going to have to hire a AI Image Technician in a few years so that their stuff looks like somebody actually invested in it (which matters to the law firm’s brand image or they’d not have a graphic designer in the first place). So yeah, the job market will shuffle around, but it’s unclear what that means in net number of jobs or what share of AI Image Technicians will literally have just been Graphic Designers five years earlier.
- dimgl 3y ago
  > I don't think I'm a fear mongerer but the amount of reasonably qualified jobs that can be displaced by the models we have now seems huge. Maybe I'm living under a rock, but I have serious doubts about this claim. > Think graphic designer at medium-sized law firm. Or copywriter at a scientific publisher. How is that even possible? Take the example of the graphic designer. You'd still need them to iterate on ideas, come up with different versions for different backgrounds, generate different file types depending on the medium and format, etc. Extremely skeptical of your claims.

tomohelix 3y ago

It tracks with the way LLMs work. From what I understand, during training, the LLMs build a correlation between words, a process after which their weight network may be configured in a way that superficially resemble actual reality. Then for whatever input this LLM takes in, it uses its "reasoning" based on this configuration to spit out a response. A bigger LLM has more parameters to model the world and may have a better performance. But I guess even 1 trillion parameters is still not enough to model the actual reality with enough depth. So if their internal understanding of the world is inherently flawed, you can't expect them to be able to spot their own faults. To be fair, this is the same for human. I am sure we all have that experience trying to tell someone where they are wrong and had a hard time. But with a human, we can update our internal world model on the fly. An LLM at runtime can't do that. It needs to be retrained.

SrslyJosh 3y ago
I think of this as the "abstraction level problem". LLMs work at the wrong abstraction level (statistical relationships between individual words) for the tasks people want to use them for (high-level reasoning). Image generators have a similar problem -- they suck at large-scale composition and have problems with details like hands because the abstraction level they're working at (individual pixels and their close neighbors) is much lower than the level that a human artist works at.

SrslyJosh 3y ago

This is utterly unsurprising, since LLMs are statistical models of text, not inference engines (or similar). > Interestingly, the models often produce the correct answer initially, but switch to an incorrect response after self-correction. The training corpus almost certainly contains examples that are similar to both cases ("my answer was correct" and "oops, my answer was wrong"). It'll be a coin flip which set the input more closely matches.

FeepingCreature 3y ago

I wonder to what extent this is because ChatGPT is trained to take any feedback as a sign that its answer is wrong and needs to be corrected. Such a training would, I suspect, explain the effect where the model first generates the correct answer, then introspects and self-doubts. In training, when it was told to introspect, that would usually be a hint that its answer was wrong. This results in an agent whose confidence reliably falters under introspection. I've observed that ChatGPT quite eagerly "corrects" itself at the first sign of negative feedback, even if its answer was already correct. It's actually quite annoying. If this is correct, a more balanced training that contains cases where the human response is wrong would solve this issue.

kromem 3y ago
This is brought up in the paper and is effectively their reasoning. That because it is instructed to "find problems" with the answer it leads to correct answers being biased towards changing to incorrect more often than it can successfully bias incorrect answers towards correct ones. Here, I think the research team might have had more success using neutral language like "grade this answer" where both verification and challenge would fit the prompt as opposed to "find problems with this answer" where only challenges would - but I'm skeptical it would be enough of an improvement that it would yield a net improvement on self-critique over the initial answer. There are other more specialized approaches I'd expect will always work better than feeding an incorrect answer back into what generated it in the first place (i.e. a backwards pass where the model has to match the answer provided to multiple possible questions with one being the question asked and the others generated on the fly).

montebicyclelo 3y ago

The paper: https://arxiv.org/pdf/2310.01798.pdf https://arxiv.org/pdf/2310.01798.pdf > Our research shows that LLMs are not yet capable of self-correcting their reasoning The paper actually just shows that the particular "self-correction" strategy and set of prompts they used doesn't help for the tasks they looked at, for the models they looked at. It may be the case in general, but it may not. > it is plausible that there exist specific prompts or strategies that could enhance the reasoning performance of models for particular benchmarks Seems they agree. So the wording of the title/conclusion is too strong. > searching such prompts or strategies may inadvertently rely on external feedback, either from human insights or training data I'm not sure this justifies picking a single prompting strategy, and not looking at the impact of different prompting strategies. Even just writing a few different prompts in advance with different wordings and showing the variation in results would have been helpful.

aeternum 3y ago
Yes, these papers just demonstrate how weak scientific papers have become. The paper is weak on actual prompt examples, but of the few there, test them yourself and it consistently gets them right. Whenever I see examples of GPT4 can't do such and such, I'm generally able to find a prompt that does in fact work relatively consistently.
- kromem 3y ago
  I don't know that it's as much a matter of papers having become weak as much as there being a knowledge gap between use of models in practice and in research. For example, a glaring issue that caught my eye in the paper was that their prompt for the self-evaluation was setting the context for what was being analyzed as its own work. Out of the training data of the effective Internet, what % of critical analysis was self-critique and what % do we think was the analysis of others' work? As such, if we're trying to elicit an effective analysis of an earlier answer from an incomprehensible multi-variable prediction machine, might we not want to set the context as the evaluation of another's earlier work? It's still technically self-evaluation even if we are hiding implementation details from the LLM. Another example is that their prompt to a fine tuned instruct model asked it to "find problems." And then they found that the self-critique would often bias correct answers to change to incorrect. What about having used more neutral language like 'grade' that allows both verification or challenge of the earlier answer to fit the instruction? These are the kinds of nuances I'm sure many working with LLMs in production scenarios have realized can completely change the outcomes of a prompt pipeline, and yet in research we see very smart analysis (i.e. implicit bias towards correction over multiple rounds of challenges when incorrect) coupled with less than ideal prompt selection for the methods. And given that we should expect every new generation of models to have new and different nuances to what works in practice and what doesn't, I don't know that this is a problem that's going to get better before it gets worse.

tanvach 3y ago

Alternative explanation - maybe the training data is full of internet discussions, and it's just that we we really bad at self correction online.

MuffinFlavored 3y ago

I wonder in LLMs either need to be rethought/retrained to say "I don't know" instead of guessing (I have no clue what would go into this) or... if we somehow are able to take a "test-driven-approach" with them. Upfront, we tell it what criteria must pass. Connect it to a "test/task" runner of some sort per prompt. Then, it keeps trying behind the scenes to come up with an answer that passes the criteria (to prevent humans from having to waste iterations on 'no, you got it wrong, try again') However, I will admit... most of the time when the LLM can't solve it, retrying and retrying over and over slightly different ways really doesn't typically end up in success. If the task is "too complex" for the LLM, I have yet to find a way to break it down context wise and overcome the limitation.

zitterbewegung 3y ago
In some situations you can make the prompt tell you. "I don't know". For example "You are a friendly D&D dungeon master and know all of the rules of D&D and run the game. If you interact with a player you should observe all of the rules. If you don't know if they are following the rules then say "I don't know"
viraptor 3y ago
> I wonder in LLMs either need to be rethought/retrained to say "I don't know" instead of guessing (I have no clue what would go into this) You can already do this yourself to some extent by providing examples in the prompt. Ask your question, explicitly add that you're allowing the "don't know" answer and add that for this similar question the answer is X, for that the answer is don't know, for that the answer is Y. It's a bit of work, but it works in practice. Check out the concept of priming. Yeah, it would be nice if this was pre-trained - maybe in the future.
trifurcate 3y ago
Non-instruct-tuned LLMs are far more likely to say "I don't know".

kazinator 3y ago

What chat LLMs do is correct their answers by cribbing from the information you give in the conversation, and then take credit for it.

RecycledEle 3y ago

LLMs reason like we autistic people do. I almost never correct myself after someone pointed out my errors. I either find the error myself or it takes my many hours of contemplation to see my error(s).

delecti 3y ago
LLMs don't reason, and autistic people have an internal state. We have thoughts and memories, and understand context. If you were trying to fit a pegs into holes, you'd correct pretty quickly when the square peg didn't fit a circle hole.

jeffbee 3y ago

"For GPT-4, to reduce the cost, we randomly sampled 200 questions" Wow, DeepMind research budgets must be nothing.

madsbuch 3y ago

I asked ChatGPT to explain fast exponentiation. It did 9 iterations until it came up with the right algorithm. Why would this not be considered self-correction?

m3kw9 3y ago

It seems like they self correct when you explain to the LLM something is incorrect. But in reality they take the original question, plus the “incorrect” statement and get the most likely responses from the model

dansalvato 3y ago

I would really like for LLMs to reach a point where they can actually tell you if they're not quite sure about something, rather than confidently outputting their BS like it's fact. Too many people interpret LLM output as though it's guaranteed correct. Certainly there's some confidence value metric that they can understand internally and use when wording their response.

munchler 3y ago
The irony of this comment is rich. There’s no such “confidence value metric” known, no matter how “certain” you are. This is why techniques like chain-of-thought or self-correction are used instead. LLM’s predict the next token in a sequence. That’s it. The final softmax layer gives a per-token probability, but that is not the same thing as overall confidence in the response.
- famouswaffles 3y ago
  The irony of this comment is rich. >The final softmax layer gives a per-token probability, but that is not the same thing as overall confidence in the response. Clearly not. https://imgur.com/a/3gYel9r https://imgur.com/a/3gYel9r
  - munchler 3y ago
    Hah, I wasn’t aware of that. Does it say how the base model computes its confidence? Is it requested in the prompt?
    - famouswaffles 3y ago
      They didn't specify.
  - singularity2001 3y ago
    This may be philosophically one of the most important graphs regarding the current state of AI! It shows that large language models already DO have an internal model of confidence in their statements, all that is needed is to train the models to output this confidence alongside their hallucinations if the confidence is low.

deleted 3y ago

[deleted]

tangentstorm 3y ago

I tested their first example in ChatGPT 4. Interestingly, it gives the correct answer on the first try: https://chat.openai.com/share/d86fe16a-9dfd-4753-8eaf-6d2948096ea3 I then gave GPT4 a "chain-of-thought" flavored prompt, telling it to treat the problem like a geometry proof. It gave the same incorrect answer as GPT3 did in the paper. I then told it to "Review your work and check for mistakes." With this follow-up, it checked each line of the proof and was able to find and explain the error: https://chat.openai.com/share/c4ce6e98-43e3-4547-a4c8-380c1d1cc5fe GPT3.5, given the same prompts, was still confident in its incorrect answer: https://chat.openai.com/share/1a0be419-092d-4dbb-a6c3-79e61914fd0d (edited to update links and ask them each version to check its work twice)

YeGoblynQueenne 3y ago
>> I then told it to "Review your work and check for mistakes." With this follow-up, it checked each line of the proof and was able to find and explain the error: And after that? Did you tell it that it may be wrong and ask it to check again?
- tangentstorm 3y ago
  Good point. I did now. Both models stuck with their second answer. (I see now I didn't know how to share ChatGPT links properly. Updating the links now...)

jaidhyani 3y ago

I am begging people to stop confusing "I was unable to get LLM X to do Y using strategy Z" with "All LLMs are categorically unable to do Y".

happytiger 3y ago

If you add “This generation of” to the beginning of all of this research on existing LLMs you’ll have a clearer understanding. Like all young technologies it’s a constant failure in so many important ways, and will continue to be until it’s not or another superior model built on what we’ve learned replaces it. But the important thing to remember is that this process is fairly standard for any technology innovation wave. They suck at the beginning and take a longer time to mature than people realize even while they’re going through them. The early Internet was mindblowingly amateur. Early cell phones were basically pager quality and the smart phone revolution started with things like palm pilots that couldn’t even connect to one another or barely could. You have to look at LLMs as cell phones from 1991 or maybe 1993, and when you do you realize how freekin’ crazy these technology direction could be. This generation of LLMs are simply designed to regurgitate information in a coherent fashion, not be right about the information or maintain consistency (they literally dont have the memory capacity — it’s just like early computers were lol). People are poor predictors of technology progress because they tend to extend current reality forward and don’t realize that technology works in leaps and bounds and isn’t linearly predictable with any substantive reliability either.

anigbrowl 3y ago

Nature shows us that bilaterality works. Connecting two LLM instances together and and giving them slightly different prompts to appraise each others' output is going to be a lot more effective than trying to iterate some monolithic super-reasoning device.

kristianp 3y ago

I've taken to adding guidance to my original prompt instead of trying to correct the model. Perhaps having the incorrect answer plus the correction in the context causes it to be dumber?

p5a0u9l 3y ago

People forget that getting answers is not what makes a good solution. Computers are still dumb, they only give answers. Asking the right question is everything. With the right question the uncertainty of a problem is resolved. Finishing is a mere exercise. whether hand-crafted assembly or an LLM’s regurgitation and synthesis of things it’s seen before is just a matter of efficiency.