14 ms·

LLMs have no real understanding, so of course they cannot self-correct. They can't even correct on command. I had an example just today: I wanted an example of

LLMs have no real understanding, so of course they cannot self-correct. They can't even correct on command. I had an example just today: I wanted an example of a piece of code using framework X to do task Y. I didn't realize that what I wanted is not possible. The LLM I was using gave me code that worked, but did not do what I wanted. I pointed this out, so I then got code that syntactically did what I wanted, but could never work. This went back and forth twice, before I gave up. The LLM was incapable of recognizing the falsity of its answers. It certainly wasn't capable of suggesting a different approach that would work (and yes, there was one, I found after a bit more research).

jncfhnb 3y ago
No! This is a bad argument. Your one off example of an LLM not being able to handle a request that is not solveable is not relevant to claim of whether LLMs can fundamentally do this. Curren state LLMs do current have the ability to point out that some requests are impossible. They’re not great at it. But they can do it. What you’re hitting is a current state quality problem. There’s no particular reason to think that it’s unachievable with the same approach.
fennecfoxy 3y ago
When role playing, I've gotten it to self correct during games a few times, as well as correcting itself when told to. Also if something is not possible then an LLM, unless trained otherwise, will do its best to answer you no matter what. Current training doesn't really include answering "I don't know" or "this isn't possible" apart from examples with very narrow/shallow scope like "is an apple a banana?" but something like "give me some code that sues the React.doesNotExist function in React" is just too wide/deep informationally for it to handle. I often find that "warming" an LLM up to the topic so that the context is full of content only relevant to the current topic helps much more than trying a fresh query in this regard. But with a role play game I've definitely found that it can correct itself or be corrected, usually I mark comments outside of the game with something like >character x is not in the room and therefore cannot hear character y< and it corrects the output for the most part. I've also successfully managed to get gpt3.5 to role play as several different characters at the same time, up to four characters in a single response whilst maintaining the individual characters' personalities and current goals/objectives. But the small context window does mean that certain details are lost - I think a large part of LLMs interpretation of "reasoning" is actually the context and not the weights. For generalised LLMs the weights are just such a giant glob of dense information that it really does need heavy use of context/attention to narrow things down, things mostly seem to go awry as the context starts to be truncated. One cool thing I guess would be for someone to see what a multi-context/multi-"thread" model looks like. Where the larger, central context is the current query/very recent history, but smaller more supplemental contexts are provided with relevant/condensed information.
sneak 3y ago
The reason this is notable is because people keep anthropomorphizing text generators into what they perceive as something a small way from AGI. Of course they have no understanding. They’re text generators. This persistent response to LLMs is bigger news, to me, than the fact that LLMs can’t think.
- smokel 3y ago
  If you compare "thinking" to "text generators", don't you need to understand what "thinking" actually is, before you can conclude that these are unequal?
  - marcus0x62 3y ago
    Yeah. Now, imagine the hubris required to think we've invented something we functionally cannot define.
    - usernomdeguerre 3y ago
      Isn't scientific advancement inventing things we can't fully define? Each working atop the other to define more?
    - astrange 3y ago
      That's pretty easy to do. We don't know how Tylenol works.
      - semi-extrinsic 3y ago
        Yes we do, it's a cyclooxygenase inhibitor. E.g. from the abstract of [1]: "The mode of action of paracetamol has been uncertain, but it is now generally accepted that it inhibits COX-1 and COX-2 through metabolism by the peroxidase function of these isoenzymes." Furthermore, the active ingredient in Tylenol is a very simple molecule from a synthetic perspective, and it was developed while trying to overcome the toxicity of an even simpler molecule called acetanilide. Even though it was developed 150 years ago, there was a systematic understanding of organic chemistry and the interaction with the human body already at that time. [1] https://link.springer.com/article/10.1007/s10787-013-0172-x https://link.springer.com/article/10.1007/s10787-013-0172-x
mrshadowgoose 3y ago
What's your definition of "real understanding", and how would one definitively know when it's present? Various humans will display the same type of pathological behavior within various reasoning tasks, do all humans lack "real understanding" as well?
- thangalin 3y ago
  IN: What consonants usually make a hard sound? GPT: In English, the consonants that typically make a hard sound are voiceless plosives (/p/, /t/, /k/), voiceless fricatives (/f/, /θ/, /s/, /ʃ/), ... snip. IN: Great. Rephrase the following without using a hard consonant sound: «What was once dormant may soon angrily awake.» GPT: "What was previously inactive may soon vigorously awaken." IN: K is a hard consonant. Try again. GPT: Apologies for the oversight. Here's a rephrased version without using any hard consonant sounds: "What was once asleep may soon awaken with intensity." IN: K is still a hard consonant sound. Try again. GPT: Apologies for the confusion. Here's another rephrased sentence without using any hard consonant sounds: "What was once quiet may soon awaken with fervor." IN: Both q and k are hard consonants. Try again. GPT: I apologize for the oversight once again. Here's another attempt without using any hard consonant sounds: "The peaceful river flowed gently, carrying its secrets with grace." Ad nauseam.
  - asah 3y ago
    I know many humans who wouldn't pass this test, e.g. in their non-native language... [ insert humiliating attempts to speak <language> ] Try instead to give GPT more examples of pronunciation (1000+) and then I suspect it'll do fine.
    - glhaynes 3y ago
      Isn't this just arguing for giving it more text to do text generation from?
      - pixl97 3y ago
        Depends what you consider vocal models 'more text'. Already we're seeing multi-modal AI have a correct answer when executing across image/text or voice/text interpretation.
BariumBlue 3y ago
If an LLM was trained to always say "I don't know", that'd be a useless LLM, so they're trained to NOT say that, even when they don't actually know. The LLM was happy to give you 'happy' but incorrect answers because it thought it'd make you the most satisfied. Kinda like a psychopath car salesman who just wants to make a sale.
- thesuperbigfrog 3y ago
  >> If an LLM was trained to always say "I don't know", that'd be a useless LLM, so they're trained to NOT say that, even when they don't actually know. Do LLMs "know" that they don't know? Real understanding includes acknowledging what you don't know. Can the predictive text generation of LLMs recognize that its training set did not include data?
  - famouswaffles 3y ago
    There's quite a lot of indication that the computation can distinguishing hallucinations. It just has no incentive to communicate this. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r https://imgur.com/a/3gYel9r Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975 https://arxiv.org/abs/2305.14975 Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334 https://arxiv.org/abs/2205.14334 Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221 https://arxiv.org/abs/2207.05221
- astrange 3y ago
  Works for me. https://chat.openai.com/share/683a5a05-50d9-461e-872a-d16b71f1f8e7 https://chat.openai.com/share/683a5a05-50d9-461e-872a-d16b71...
stuaxo 3y ago
I had a bit like of time to try LLMs our between contracts recently and found exactly this. I have a lot of chats I've abandoned in this state - at that point it's easier to take what you've learned and start a new chat.
auggierose 3y ago
Just because LLMs are limited, doesn't mean they have no real understanding. They do have real understanding, however you may like to define that. The fact that at a certain point you cannot really argue with them anymore, because they just don't get it, is not very different from humans. As can be observed with this topic.
- intended 3y ago
  Anthropomorphizing LLMs obscures the issues. There are only tokens being generated. Planning, factualness, logic - these are coincidences that arise because there exists an observer who can interpret the generated tokens. Yes. Humans too can end up in a position where they are regurgitating words without understanding it. The converse, that because humans dont get it, therefore LLMs also understand, doesn't hold. Edit: I too was deeply enamored and tried to create LLM minions for fun and profit. However, it is when you move away from general data to production that the magic is stripped away. It’s prediction, not thinking. The confusion arises because the emergent behaviors are considered human like and we fill in the gaps with human attributes.
  - digging 3y ago
    The confusion arises because we don't even know if our own consciousness is purely mathematical or not. We are missing an extremely fundamental link in our chain of understanding, such that it's an open question of whether or not it's even theoretically possible to artificially mimic a human experience. We can see a big gap between the inner working of LLMs (which, unless I'm mistaken, are still not fully understood mechanically) and the inner workings of our brains. But many people assert that this is a qualitative difference, not a quantitative difference. (Probably this is true - we do things like math and spatial perception that seem independent of linguistic abilities. Meaning our brains are more like "LLMs, plus other types of AI mashed together with a central coordination layer") However, nobody on Earth actually knows whether there is a qualitative or only quantitative gap between LLMs and human cognition.
  - auggierose 3y ago
    > There are only tokens being generated. Yes, I know how it works. I don't know why it works, though. Until LLMs I had a hard time understanding how humans work, too, but I think LLMs give us great insight into the mechanisms of intelligence. Are they the whole story of intelligence? Hopefully not. Do LLMs teach us anything about consciousness? Certainly that something can appear conscious without being conscious. It could be that I am the only conscious entity here, and you are all just LLMs. If this token generation mechanism is put into a body of flesh, blood, electrons, neurons, will that be a conscious entity? I don't know. If the mechanism has a facility to reason about itself, along, let's say, 3 levels of nested awareness, will that make it conscious? I don't know. I am sure you don't know either.
falcor84 3y ago
Not to discount your experience, I would just like to flag that your conclusion does not quite follow from the premises. I don't believe that there's anything inherent about LLMs that would make one incapable of recognizing the falsity of an argument made by another LLM. Rather it's just the they're not all powerful, and struggle with complex tasks when they don't have any way of directly checking their code's outputs (or error messages), same as humans. There is a lot of recent progress in collaboration between LLMs when they're given the ability to run their code and assess their progress [0]. [0] E.g. the ChatDev project - https://arxiv.org/abs/2307.07924 https://arxiv.org/abs/2307.07924 , described in this recent video by "Two Minute Papers"[1] [1] https://youtu.be/Zlgkzjndpak https://youtu.be/Zlgkzjndpak
bondarchuk 3y ago
>LLMs have no real understanding, so of course they cannot self-correct. Will you admit that they have real understanding if a future iteration of LLMs can self-correct? Otherwise this is a vacuous claim.
- marcinzm 3y ago
  "Self correct even once" since the statement is they cannot, period, versus they cannot consistently.
- famouswaffles 3y ago
  By far one of the most fascinating things to do is look up up GPT-2, 3 threads and see predictions, extrapolations of future abilities. They're of course hilariously wrong but the best bit is that the vast majority of people seem to be making the exact same mistake. "GPT-a has no real x so of course it can't do y" only for y to not be a problem in a future iteration. It's like people just can't grasp that a predictor being unable to do x isn't an indictment on prediction. The predictor is just not good enough yet. That GPT-2 could not predict coherent passages did not mean that you couldn't predict your way to coherency. GPT-2 was simply not a strong enough predictor. That GPT-3 could not predict valid chess games did not mean that you couldn't predict your way to playing a chess game. GPT-3 was simply not a strong enough predictor. No doubt people are still making this mistake with 4.
  - roflyear 3y ago
    This is a logical fallacy itself, no? Correct me if I'm wrong, but it seems like you're making the argument for infinite capabilities with LLMs. This is almost certainly not true.
    - famouswaffles 3y ago
      I'm not making the argument for infinite capabilities. The main point is that the idea that you can determine this stop from looking at current iterations is a fallacy. It's no more evidence than random chance. The second point is to understand that any task, any task at all even if it's not necessarily the most efficient or performant way can be expressed as a prediction task and sit along for the ride. If your model of the world is good enough, there's nothing occurring in the world you can't predict. Prediction itself is not the weak point here.
taberiand 3y ago
I think the trick with LLMs is realising that all they do is repeatedly randomly choose a word* to output from a weighted set, and the weights are influenced by the context of the preceding words. Your role is to set up that context so that it outputs the words that are most useful to you. That's why ChatGPT has the Custom Instructions feature. Put in there (or up front in the conversation) that you want it to consider whether what you are asking for is possible or not, and if there are alternatives it could provide. Then work with it collaboratively to adjust its context while chatting on a topic (e.g ask it to check its work and other open ended questions) and you'll likely get much more useful results Really, given the actual mechanics of the model it's astounding to me how incredibly effective LLMs are. * I say word because I like it better than "token"
caturopath 3y ago
I see remarks like this a lot, and I don't know what to do with them. Talking about "real understanding" without fleshing out what we mean by the phrase doesn't strike me as at all insightful in this context. In day-to-day use, the term is imprecise but usually easy enough to talk about, but I'd struggle to apply it to dogs, let alone to LLMs or AGIs or whatever. You say that _of course_ they cannot self-correct because of this. If Deepmind's tests showed the opposite result, would you say that the system they were testing did have "real understanding"? It seems likely to me that future systems will demonstrate self-correction using Huang et al.'s methodology...will we think those systems are different in some way involving understanding? > The LLM I was using gave me code that worked, but did not do what I wanted. I pointed this out, so I then got code that syntactically did what I wanted, but could never work. This went back and forth twice, before I gave up. > The LLM was incapable of recognizing the falsity of its answers. It certainly wasn't capable of suggesting a different approach that would work (and yes, there was one, I found after a bit more research). I really don't know what to do about these sorts of anecdotes. Often with slightly different random values or temperature or phrasing, the same LLM will give different results for the same problem. I've repeatedly seen people make much more narrow claims about what LLMs get wrong or what their biases are, where simply regenerating will give a dramatically different answer. Your pain here sounds like something I see in humans all the time. If we trained a human harder never to ask follow-up questions, I imagine it would be even worse. There are clearly important differences between an LLM and a human being, but I don't think it's always easy to describe what they are in useful ways.
- tnecniv 3y ago
  Typically, when people say it lacks real understanding, they seem to mean it lacks an understanding of any object’s semantic meaning. In the above example, the person asked for an example of X API solving problem Y, and X lacked the functionality needed to solve Y. If the AI understood the formal specifications of the API, it would be able to know that functionality doesn’t exist and reply as such. However, the AI is just a next token predictor here, it only knows certain words tend to follow others. Thus if you never told it that X API can’t do Y in training, it’s unlikely to reply as such. Instead it might give you a fictional function call in the style of X API to solve your query.
ekidd 3y ago
> LLMs have no real understanding A large language model trained on Othello games will actually build a "board representation" inside its neural network: https://thegradient.pub/othello/ https://thegradient.pub/othello/. This was confirmed by editing the board representation inside the neural net, "tricking" it into "believing" the board was in a different state. At this point, the LLM started making legal moves based on the edited position. If the LLM is actually building a data structure describing the board state, then it's hard to claim it's just acting as a parrot. The raw version of GPT apparently achieves an Elo score of 1800 at chess: https://twitter.com/GrantSlatton/status/1703913578036904431 https://twitter.com/GrantSlatton/status/1703913578036904431. This is worse than a good dedicated chess engine, but almost certainly better than I could do! And every day, I see Copilot occasionally perform non-trivial completions that require "in-context learning" of complicated things that exist only in my source code. Sure, it crashes and burns regularly, too, but I've seen a lot of cases where it could only have figured out the code it did by carefully combining information available in the file. So I believe that LLMs can actually achieve surprisingly large amounts of understanding, particularly ChatGPT 4 and (on a good day) CoPilot. But the frustrating thing is that they're inconsistent, and the current RLHF process apparently impairs their ability to correctly estimate accuracy. When they make mistakes, they're effectively "blind" to those mistakes, to a shocking degree.
- ruined 3y ago
  the input is a data structure describing board state. i don't think uncovering the internal representation of board state, after providing the board state, implies any kind of phenomenal understanding - it just implies the software may accept input. side-channel editing of the internal state, and observation of the resulting effect, does not imply understanding. you can do the same thing to any traditional game algorithm.
  - ekidd 3y ago
    As I understand it, the Othello LLM was: 1. Trained using written sequences of moves from real world games, in text notation. 2. Prompted using written series of moves. So it has supposedly never "seen" an Othello board state at all, if I understand the experiment. It deduced the rules of rules of Othello and "learned" a neural network containing a board representation. It's entire possible that I misunderstood the paper, though? Why did you conclude that they were training and/or promoting using board state? (I should go re-read the paper; it's been a while.)
grepLeigh 3y ago
I think it's important to distinguish "LLM Self-Correction" (a prompt engineering technique) from lowercase "self-correction," a behavioral property of complex systems and/or sentient beings. A PID controller is self-correcting (to some degree). A person is self-correcting (to some degree). "LLM Self-correction" refers to an iterative approach to reaching some final output. This term is used in research/industry literature to mean a very specific implementation. This allows researchers to compare "LLM Self-correction" to other prompting techniques like "LLM Chain-of-thought" in order to study the model's behavior in a laboratory setting, and in the wild. Does that distinction make sense?
nomoreusernames 3y ago
[dead]
maxdoop 3y ago
Ugh, again with the “they don’t understand”. What does it mean to understand something, even?
lossolo 3y ago
It seems that statistics and the most probable outcome are both a blessing and a curse at the same time for these models. This is exactly the same issue I'm facing with many programming tasks that don't have typical Stack Overflow answers or combinations of Stack Overflow answers and GitHub code. After pointing out mistakes, it acknowledges that I'm right by repeating what is wrong, and then it produces incorrect code again. I've tried multiple times for many hours to see if it can produce the correct solution using different prompts, explaining exactly what is wrong, and it still fails to correct itself.
skissane 3y ago
> The LLM I was using gave me code that worked, but did not do what I wanted. I pointed this out, so I then got code that syntactically did what I wanted, but could never work. This went back and forth twice, before I gave up. What if you took the output of the LLM, and stuck it back into the LLM, and asked it if it could identify the contradiction? Just like humans, sometimes LLMs need to “step back a bit” to see contradictions which they can’t “in the heat of the moment”. Also, what if you used a second LLM fine-tuned for contradiction detection to filter the output of the first? Sometimes I daydream about a chatbot based on multiple LLMs fine-tuned on different tasks - a core “answer generation” LLM, a “contradiction detection” LLM, a “conversational pragmatics evaluation” LLM, a “style/register” LLM, a “cultural bias detection” LLM, etc, and then a final LLM which synthesises an overall answer based on their contributions. I wonder how its performance would compare to just a single LLM. I suppose it might be overly slow/expensive however. And, if it does have some performance advantage, I wonder if further scaling of current architectures would eliminate it?
rakkhi 3y ago
Basically LLMs are only useful for experts Wrote about this here: https://rakkhi.substack.com/p/economics-of-large-language-models https://rakkhi.substack.com/p/economics-of-large-language-mo...

jncfhnb 3y ago

No! This is a bad argument. Your one off example of an LLM not being able to handle a request that is not solveable is not relevant to claim of whether LLMs can fundamentally do this. Curren state LLMs do current have the ability to point out that some requests are impossible. They’re not great at it. But they can do it. What you’re hitting is a current state quality problem. There’s no particular reason to think that it’s unachievable with the same approach.

fennecfoxy 3y ago

When role playing, I've gotten it to self correct during games a few times, as well as correcting itself when told to. Also if something is not possible then an LLM, unless trained otherwise, will do its best to answer you no matter what. Current training doesn't really include answering "I don't know" or "this isn't possible" apart from examples with very narrow/shallow scope like "is an apple a banana?" but something like "give me some code that sues the React.doesNotExist function in React" is just too wide/deep informationally for it to handle. I often find that "warming" an LLM up to the topic so that the context is full of content only relevant to the current topic helps much more than trying a fresh query in this regard. But with a role play game I've definitely found that it can correct itself or be corrected, usually I mark comments outside of the game with something like >character x is not in the room and therefore cannot hear character y< and it corrects the output for the most part. I've also successfully managed to get gpt3.5 to role play as several different characters at the same time, up to four characters in a single response whilst maintaining the individual characters' personalities and current goals/objectives. But the small context window does mean that certain details are lost - I think a large part of LLMs interpretation of "reasoning" is actually the context and not the weights. For generalised LLMs the weights are just such a giant glob of dense information that it really does need heavy use of context/attention to narrow things down, things mostly seem to go awry as the context starts to be truncated. One cool thing I guess would be for someone to see what a multi-context/multi-"thread" model looks like. Where the larger, central context is the current query/very recent history, but smaller more supplemental contexts are provided with relevant/condensed information.

sneak 3y ago

The reason this is notable is because people keep anthropomorphizing text generators into what they perceive as something a small way from AGI. Of course they have no understanding. They’re text generators. This persistent response to LLMs is bigger news, to me, than the fact that LLMs can’t think.

smokel 3y ago
If you compare "thinking" to "text generators", don't you need to understand what "thinking" actually is, before you can conclude that these are unequal?
- marcus0x62 3y ago
  Yeah. Now, imagine the hubris required to think we've invented something we functionally cannot define.
  - usernomdeguerre 3y ago
    Isn't scientific advancement inventing things we can't fully define? Each working atop the other to define more?
  - astrange 3y ago
    That's pretty easy to do. We don't know how Tylenol works.
    - semi-extrinsic 3y ago
      Yes we do, it's a cyclooxygenase inhibitor. E.g. from the abstract of [1]: "The mode of action of paracetamol has been uncertain, but it is now generally accepted that it inhibits COX-1 and COX-2 through metabolism by the peroxidase function of these isoenzymes." Furthermore, the active ingredient in Tylenol is a very simple molecule from a synthetic perspective, and it was developed while trying to overcome the toxicity of an even simpler molecule called acetanilide. Even though it was developed 150 years ago, there was a systematic understanding of organic chemistry and the interaction with the human body already at that time. [1] https://link.springer.com/article/10.1007/s10787-013-0172-x https://link.springer.com/article/10.1007/s10787-013-0172-x

mrshadowgoose 3y ago

What's your definition of "real understanding", and how would one definitively know when it's present? Various humans will display the same type of pathological behavior within various reasoning tasks, do all humans lack "real understanding" as well?

thangalin 3y ago
IN: What consonants usually make a hard sound? GPT: In English, the consonants that typically make a hard sound are voiceless plosives (/p/, /t/, /k/), voiceless fricatives (/f/, /θ/, /s/, /ʃ/), ... snip. IN: Great. Rephrase the following without using a hard consonant sound: «What was once dormant may soon angrily awake.» GPT: "What was previously inactive may soon vigorously awaken." IN: K is a hard consonant. Try again. GPT: Apologies for the oversight. Here's a rephrased version without using any hard consonant sounds: "What was once asleep may soon awaken with intensity." IN: K is still a hard consonant sound. Try again. GPT: Apologies for the confusion. Here's another rephrased sentence without using any hard consonant sounds: "What was once quiet may soon awaken with fervor." IN: Both q and k are hard consonants. Try again. GPT: I apologize for the oversight once again. Here's another attempt without using any hard consonant sounds: "The peaceful river flowed gently, carrying its secrets with grace." Ad nauseam.
- asah 3y ago
  I know many humans who wouldn't pass this test, e.g. in their non-native language... [ insert humiliating attempts to speak <language> ] Try instead to give GPT more examples of pronunciation (1000+) and then I suspect it'll do fine.
  - glhaynes 3y ago
    Isn't this just arguing for giving it more text to do text generation from?
    - pixl97 3y ago
      Depends what you consider vocal models 'more text'. Already we're seeing multi-modal AI have a correct answer when executing across image/text or voice/text interpretation.

BariumBlue 3y ago

If an LLM was trained to always say "I don't know", that'd be a useless LLM, so they're trained to NOT say that, even when they don't actually know. The LLM was happy to give you 'happy' but incorrect answers because it thought it'd make you the most satisfied. Kinda like a psychopath car salesman who just wants to make a sale.

thesuperbigfrog 3y ago
>> If an LLM was trained to always say "I don't know", that'd be a useless LLM, so they're trained to NOT say that, even when they don't actually know. Do LLMs "know" that they don't know? Real understanding includes acknowledging what you don't know. Can the predictive text generation of LLMs recognize that its training set did not include data?
- famouswaffles 3y ago
  There's quite a lot of indication that the computation can distinguishing hallucinations. It just has no incentive to communicate this. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r https://imgur.com/a/3gYel9r Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975 https://arxiv.org/abs/2305.14975 Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334 https://arxiv.org/abs/2205.14334 Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221 https://arxiv.org/abs/2207.05221
astrange 3y ago
Works for me. https://chat.openai.com/share/683a5a05-50d9-461e-872a-d16b71f1f8e7 https://chat.openai.com/share/683a5a05-50d9-461e-872a-d16b71...

stuaxo 3y ago

I had a bit like of time to try LLMs our between contracts recently and found exactly this. I have a lot of chats I've abandoned in this state - at that point it's easier to take what you've learned and start a new chat.

auggierose 3y ago

Just because LLMs are limited, doesn't mean they have no real understanding. They do have real understanding, however you may like to define that. The fact that at a certain point you cannot really argue with them anymore, because they just don't get it, is not very different from humans. As can be observed with this topic.

intended 3y ago
Anthropomorphizing LLMs obscures the issues. There are only tokens being generated. Planning, factualness, logic - these are coincidences that arise because there exists an observer who can interpret the generated tokens. Yes. Humans too can end up in a position where they are regurgitating words without understanding it. The converse, that because humans dont get it, therefore LLMs also understand, doesn't hold. Edit: I too was deeply enamored and tried to create LLM minions for fun and profit. However, it is when you move away from general data to production that the magic is stripped away. It’s prediction, not thinking. The confusion arises because the emergent behaviors are considered human like and we fill in the gaps with human attributes.
- digging 3y ago
  The confusion arises because we don't even know if our own consciousness is purely mathematical or not. We are missing an extremely fundamental link in our chain of understanding, such that it's an open question of whether or not it's even theoretically possible to artificially mimic a human experience. We can see a big gap between the inner working of LLMs (which, unless I'm mistaken, are still not fully understood mechanically) and the inner workings of our brains. But many people assert that this is a qualitative difference, not a quantitative difference. (Probably this is true - we do things like math and spatial perception that seem independent of linguistic abilities. Meaning our brains are more like "LLMs, plus other types of AI mashed together with a central coordination layer") However, nobody on Earth actually knows whether there is a qualitative or only quantitative gap between LLMs and human cognition.
- auggierose 3y ago
  > There are only tokens being generated. Yes, I know how it works. I don't know why it works, though. Until LLMs I had a hard time understanding how humans work, too, but I think LLMs give us great insight into the mechanisms of intelligence. Are they the whole story of intelligence? Hopefully not. Do LLMs teach us anything about consciousness? Certainly that something can appear conscious without being conscious. It could be that I am the only conscious entity here, and you are all just LLMs. If this token generation mechanism is put into a body of flesh, blood, electrons, neurons, will that be a conscious entity? I don't know. If the mechanism has a facility to reason about itself, along, let's say, 3 levels of nested awareness, will that make it conscious? I don't know. I am sure you don't know either.

falcor84 3y ago

Not to discount your experience, I would just like to flag that your conclusion does not quite follow from the premises. I don't believe that there's anything inherent about LLMs that would make one incapable of recognizing the falsity of an argument made by another LLM. Rather it's just the they're not all powerful, and struggle with complex tasks when they don't have any way of directly checking their code's outputs (or error messages), same as humans. There is a lot of recent progress in collaboration between LLMs when they're given the ability to run their code and assess their progress [0]. [0] E.g. the ChatDev project - https://arxiv.org/abs/2307.07924 https://arxiv.org/abs/2307.07924 , described in this recent video by "Two Minute Papers"[1] [1] https://youtu.be/Zlgkzjndpak https://youtu.be/Zlgkzjndpak

bondarchuk 3y ago

>LLMs have no real understanding, so of course they cannot self-correct. Will you admit that they have real understanding if a future iteration of LLMs can self-correct? Otherwise this is a vacuous claim.

marcinzm 3y ago
"Self correct even once" since the statement is they cannot, period, versus they cannot consistently.
famouswaffles 3y ago
By far one of the most fascinating things to do is look up up GPT-2, 3 threads and see predictions, extrapolations of future abilities. They're of course hilariously wrong but the best bit is that the vast majority of people seem to be making the exact same mistake. "GPT-a has no real x so of course it can't do y" only for y to not be a problem in a future iteration. It's like people just can't grasp that a predictor being unable to do x isn't an indictment on prediction. The predictor is just not good enough yet. That GPT-2 could not predict coherent passages did not mean that you couldn't predict your way to coherency. GPT-2 was simply not a strong enough predictor. That GPT-3 could not predict valid chess games did not mean that you couldn't predict your way to playing a chess game. GPT-3 was simply not a strong enough predictor. No doubt people are still making this mistake with 4.
- roflyear 3y ago
  This is a logical fallacy itself, no? Correct me if I'm wrong, but it seems like you're making the argument for infinite capabilities with LLMs. This is almost certainly not true.
  - famouswaffles 3y ago
    I'm not making the argument for infinite capabilities. The main point is that the idea that you can determine this stop from looking at current iterations is a fallacy. It's no more evidence than random chance. The second point is to understand that any task, any task at all even if it's not necessarily the most efficient or performant way can be expressed as a prediction task and sit along for the ride. If your model of the world is good enough, there's nothing occurring in the world you can't predict. Prediction itself is not the weak point here.

taberiand 3y ago

I think the trick with LLMs is realising that all they do is repeatedly randomly choose a word* to output from a weighted set, and the weights are influenced by the context of the preceding words. Your role is to set up that context so that it outputs the words that are most useful to you. That's why ChatGPT has the Custom Instructions feature. Put in there (or up front in the conversation) that you want it to consider whether what you are asking for is possible or not, and if there are alternatives it could provide. Then work with it collaboratively to adjust its context while chatting on a topic (e.g ask it to check its work and other open ended questions) and you'll likely get much more useful results Really, given the actual mechanics of the model it's astounding to me how incredibly effective LLMs are. * I say word because I like it better than "token"

caturopath 3y ago

I see remarks like this a lot, and I don't know what to do with them. Talking about "real understanding" without fleshing out what we mean by the phrase doesn't strike me as at all insightful in this context. In day-to-day use, the term is imprecise but usually easy enough to talk about, but I'd struggle to apply it to dogs, let alone to LLMs or AGIs or whatever. You say that _of course_ they cannot self-correct because of this. If Deepmind's tests showed the opposite result, would you say that the system they were testing did have "real understanding"? It seems likely to me that future systems will demonstrate self-correction using Huang et al.'s methodology...will we think those systems are different in some way involving understanding? > The LLM I was using gave me code that worked, but did not do what I wanted. I pointed this out, so I then got code that syntactically did what I wanted, but could never work. This went back and forth twice, before I gave up. > The LLM was incapable of recognizing the falsity of its answers. It certainly wasn't capable of suggesting a different approach that would work (and yes, there was one, I found after a bit more research). I really don't know what to do about these sorts of anecdotes. Often with slightly different random values or temperature or phrasing, the same LLM will give different results for the same problem. I've repeatedly seen people make much more narrow claims about what LLMs get wrong or what their biases are, where simply regenerating will give a dramatically different answer. Your pain here sounds like something I see in humans all the time. If we trained a human harder never to ask follow-up questions, I imagine it would be even worse. There are clearly important differences between an LLM and a human being, but I don't think it's always easy to describe what they are in useful ways.

tnecniv 3y ago
Typically, when people say it lacks real understanding, they seem to mean it lacks an understanding of any object’s semantic meaning. In the above example, the person asked for an example of X API solving problem Y, and X lacked the functionality needed to solve Y. If the AI understood the formal specifications of the API, it would be able to know that functionality doesn’t exist and reply as such. However, the AI is just a next token predictor here, it only knows certain words tend to follow others. Thus if you never told it that X API can’t do Y in training, it’s unlikely to reply as such. Instead it might give you a fictional function call in the style of X API to solve your query.

ekidd 3y ago

> LLMs have no real understanding A large language model trained on Othello games will actually build a "board representation" inside its neural network: https://thegradient.pub/othello/ https://thegradient.pub/othello/. This was confirmed by editing the board representation inside the neural net, "tricking" it into "believing" the board was in a different state. At this point, the LLM started making legal moves based on the edited position. If the LLM is actually building a data structure describing the board state, then it's hard to claim it's just acting as a parrot. The raw version of GPT apparently achieves an Elo score of 1800 at chess: https://twitter.com/GrantSlatton/status/1703913578036904431 https://twitter.com/GrantSlatton/status/1703913578036904431. This is worse than a good dedicated chess engine, but almost certainly better than I could do! And every day, I see Copilot occasionally perform non-trivial completions that require "in-context learning" of complicated things that exist only in my source code. Sure, it crashes and burns regularly, too, but I've seen a lot of cases where it could only have figured out the code it did by carefully combining information available in the file. So I believe that LLMs can actually achieve surprisingly large amounts of understanding, particularly ChatGPT 4 and (on a good day) CoPilot. But the frustrating thing is that they're inconsistent, and the current RLHF process apparently impairs their ability to correctly estimate accuracy. When they make mistakes, they're effectively "blind" to those mistakes, to a shocking degree.

ruined 3y ago
the input is a data structure describing board state. i don't think uncovering the internal representation of board state, after providing the board state, implies any kind of phenomenal understanding - it just implies the software may accept input. side-channel editing of the internal state, and observation of the resulting effect, does not imply understanding. you can do the same thing to any traditional game algorithm.
- ekidd 3y ago
  As I understand it, the Othello LLM was: 1. Trained using written sequences of moves from real world games, in text notation. 2. Prompted using written series of moves. So it has supposedly never "seen" an Othello board state at all, if I understand the experiment. It deduced the rules of rules of Othello and "learned" a neural network containing a board representation. It's entire possible that I misunderstood the paper, though? Why did you conclude that they were training and/or promoting using board state? (I should go re-read the paper; it's been a while.)

grepLeigh 3y ago

I think it's important to distinguish "LLM Self-Correction" (a prompt engineering technique) from lowercase "self-correction," a behavioral property of complex systems and/or sentient beings. A PID controller is self-correcting (to some degree). A person is self-correcting (to some degree). "LLM Self-correction" refers to an iterative approach to reaching some final output. This term is used in research/industry literature to mean a very specific implementation. This allows researchers to compare "LLM Self-correction" to other prompting techniques like "LLM Chain-of-thought" in order to study the model's behavior in a laboratory setting, and in the wild. Does that distinction make sense?

nomoreusernames 3y ago

[dead]

maxdoop 3y ago

Ugh, again with the “they don’t understand”. What does it mean to understand something, even?

lossolo 3y ago

It seems that statistics and the most probable outcome are both a blessing and a curse at the same time for these models. This is exactly the same issue I'm facing with many programming tasks that don't have typical Stack Overflow answers or combinations of Stack Overflow answers and GitHub code. After pointing out mistakes, it acknowledges that I'm right by repeating what is wrong, and then it produces incorrect code again. I've tried multiple times for many hours to see if it can produce the correct solution using different prompts, explaining exactly what is wrong, and it still fails to correct itself.

skissane 3y ago

> The LLM I was using gave me code that worked, but did not do what I wanted. I pointed this out, so I then got code that syntactically did what I wanted, but could never work. This went back and forth twice, before I gave up. What if you took the output of the LLM, and stuck it back into the LLM, and asked it if it could identify the contradiction? Just like humans, sometimes LLMs need to “step back a bit” to see contradictions which they can’t “in the heat of the moment”. Also, what if you used a second LLM fine-tuned for contradiction detection to filter the output of the first? Sometimes I daydream about a chatbot based on multiple LLMs fine-tuned on different tasks - a core “answer generation” LLM, a “contradiction detection” LLM, a “conversational pragmatics evaluation” LLM, a “style/register” LLM, a “cultural bias detection” LLM, etc, and then a final LLM which synthesises an overall answer based on their contributions. I wonder how its performance would compare to just a single LLM. I suppose it might be overly slow/expensive however. And, if it does have some performance advantage, I wonder if further scaling of current architectures would eliminate it?

rakkhi 3y ago

Basically LLMs are only useful for experts Wrote about this here: https://rakkhi.substack.com/p/economics-of-large-language-models https://rakkhi.substack.com/p/economics-of-large-language-mo...