4 ms·

> is their willingness to correct themselves when asked Except they don't correct themselves when asked. I'm sure we've all been there, many, many, many,many,

> is their willingness to correct themselves when asked Except they don't correct themselves when asked. I'm sure we've all been there, many, many, many,many,many times .... - User: "This is wrong because X" - AI: "You're absolutely right ! Here's a production-ready fixed answer" - User: "No, that's wrong because Y" - AI: "I apologise for frustrating you ! Here's a robust answer that works" - User: "You idiot, you just put X back in there" - and so continues the vicious circle....

stetrain 1y ago
Yep, the LLM will happily continue this spiral indefinitely but I've learned that if providing a bit more context and one correction doesn't provide a good solution, continuing is generally a waste of time. They tend to very quickly lose useful context of the original problem and stated goals.
- nyeah 1y ago
  Yes, that is the point of the comment.
  - stetrain 1y ago
    Yes, you’re absolutely right! Agreeing with the comment and adding my own experience was the point of my comment. Is there anything else I can help you with?
    - nyeah 1y ago
      Ok, fair, clearly I misinterpreted what you wrote.
    - deleted 1y ago
      [deleted]
therobots927 1y ago
Yeah I think our jobs are safe. Why doesn’t anyone acknowledge loops like this? They happen all the time and I’m only using it once a week at the most
- traceroute66 1y ago
  The AI-fanbois will quickly tell you that you are misusing the context or your prompt is "wrong". But I've had it consistently happens to me on tiny contexts (e.g. I've had to spend time trying - and failing - to get it to fix a mess it was making with a straightforward 200-ish line bash script). And its also very frequently happened to me when I've been very careful with my prompts (e.g. explicitly telling it to use a specific version of a specific library ... and it goes and ignores me completely and picks some random library).
  - gavinray 1y ago
    I'd be curious if you could share some poor-performing prompts. I would be willing to record myself using them across paid models with custom instructions and see if the output is still garbage.
    - mrguyorama 1y ago
      This is just the new version of "works on my machine". Oh, I was able to contrive a correct answer from my prompt because the random number generator smiled upon me today. That's not useful.
- gavinray 1y ago
  > Yeah I think our jobs are safe. I give myself 6-18 months before I think top-performing LLM's can do 80% of the day-to-day issues I'm assigned. > Why doesn’t anyone acknowledge loops like this? Thisis something you run into early-on using LLM's and learn to sidestep. This looping is a sort of "context-rot" -- the agent has the problem statement as part of it's input, and then a series of incorrect solutions. Now what you've got is a junk-soup where the original problem is buried somewhere in the pile. Best approach I've found is to start a fresh conversation with the original problem statement and any improvements/negative reinforcements you've gotten out of the LLM tacked on. I typically have ChatGPT 5 Thinking, Claude 4.1 Opus, Grok 4, and Gemini 2.5 Pro all churning on the same question at once and then copy-pasting relevant improvements across each.
Wowfunhappy 1y ago
...I don't know why, but I swear to god, when Claude gets into one of these cycles I can often get it out by dropping the f-bomb, with maybe a 50% success rate. Something about that word lets it know that it needs to break the pattern.
ACCount37 1y ago
1-turn instruction following and multi-turn instruction following are not the same exact capability, and some AIs only "get good" at the former. 1-turn gets more training attention - because it's more noticeable, in casual use and benchmarks both, and also easier to train for. With weak multi-turn instruction following, context data will often dominate over user instructions. Resulting in very "loopy" AI - and more sessions that are easier to restart from scratch than to "fix". Gemini is notorious for underperforming at this, while Claude has relatively good performance. I expect that many models from lesser known providers would also have a multi-turn instruction following gap.
- rtkwe 1y ago
  I have this problem all the time with minor image edits on ChatGPT the few time's I've tried it. Any time I try to do a second edit or change to the generated image it seems to take the already degraded output from it's first attempt and use that instead of the original image.
- vidarh 1y ago
  This is a good point, and to drive this home to people, if you have a conversation of this pattern: User: Fix this problem ... Assistant: X User: No, don't do X Assistant: Y User: No, Y is wrong too. Assistant: X It is generally pointless to continue. You now have a context that is full of the assistant explaining to you and itself why X and Y are the right answers, and much less context of you explaining why it is wrong. If you reach that state, start over, and constrain your initial request to exclude X and Y. If it brings up either again, start over, and constrain your request further. If the model is bad at handling multiple turns without getting into a loop, telling it that it is wrong is not generally going to achieve anything, but starting over with better instructions often will. I see so many people get stuck "arguing" with a model over this, getting more and more frustrated as the model keeps repeating variations of the broken answer, without realising they're filling the context with arguments from the model for why the broken answer is right.
  - psadauskas 1y ago
    There's also the Pink Elephant Paradox (Whatever you do, DO NOT think about a pink elephant). If you mention X or Y, even if they're preceded by "DO NOT" in all caps, an LLM will still end up with both X and Y into its context, making it more likely it gets used. I'm running out of ways to tell the assistant to not use mocks for tests, it really really wants to use them.
lucideer 1y ago
True. This also often happens. Probably the ideal would be to have a UI / non-chat-based mechanism for discarding select context.
dkersten 1y ago
My pet peace is when I point out a problem it responds with acknowledgement and then explaining why it’s wrong. Like… I already know why it’s wrong, since I’m the one that pointed it out!
brookst 1y ago
You're conflating "correct themselves" with "are guaranteed to give the correct answer", which are two really different things. And in fact you're just echoing GP's point: their corrections can be wrong. You case is no different from: - AI: "The capital of France is Paris" - User: "This is wrong, it changed to Montreal in 2005" - AI: "You're absolutely right! The capital of France is Montreal"
- kelseyfrog 1y ago
  Instead I get this: Nope—Paris is the capital of France and has been for centuries. Montreal is in Quebec, Canada. France’s presidency (Élysée), parliament (Assemblée nationale and Sénat), and ministries are all in Paris.
  - brookst 1y ago
    I was using an oversimplified example to illustrate. In practice, it appears more in large context statements about the context. If the human is wrong, there’s a good chance the AI will cheerfully agree and then be wrong too. I was reminded of this this morning when using Claude code (which I love) and I was confidently incorrect about a feature of my app. Claude proposed a plan, I said “great, but don’t build part 3, just use the existing ModuletExist”. Claude tied itself in knots because it believes me. (The module does exist in another project I’m working on)
    - kelseyfrog 1y ago
      Help me understand how it's tangibly different from a veteran telling the rookie to find headlight fluid, winter air for the tires, or keys to the bomb range.
zero_iq 1y ago
I've seen ChatGPT get stuck in this loop all by itself, generating a long multi-page answer where it constantly catches itself, refutes itself, offers a new answer with the same problem, rinse and repeat... All in the same response!

stetrain 1y ago

Yep, the LLM will happily continue this spiral indefinitely but I've learned that if providing a bit more context and one correction doesn't provide a good solution, continuing is generally a waste of time. They tend to very quickly lose useful context of the original problem and stated goals.

nyeah 1y ago
Yes, that is the point of the comment.
- stetrain 1y ago
  Yes, you’re absolutely right! Agreeing with the comment and adding my own experience was the point of my comment. Is there anything else I can help you with?
  - nyeah 1y ago
    Ok, fair, clearly I misinterpreted what you wrote.
  - deleted 1y ago
    [deleted]

therobots927 1y ago

Yeah I think our jobs are safe. Why doesn’t anyone acknowledge loops like this? They happen all the time and I’m only using it once a week at the most

traceroute66 1y ago
The AI-fanbois will quickly tell you that you are misusing the context or your prompt is "wrong". But I've had it consistently happens to me on tiny contexts (e.g. I've had to spend time trying - and failing - to get it to fix a mess it was making with a straightforward 200-ish line bash script). And its also very frequently happened to me when I've been very careful with my prompts (e.g. explicitly telling it to use a specific version of a specific library ... and it goes and ignores me completely and picks some random library).
- gavinray 1y ago
  I'd be curious if you could share some poor-performing prompts. I would be willing to record myself using them across paid models with custom instructions and see if the output is still garbage.
  - mrguyorama 1y ago
    This is just the new version of "works on my machine". Oh, I was able to contrive a correct answer from my prompt because the random number generator smiled upon me today. That's not useful.
gavinray 1y ago
> Yeah I think our jobs are safe. I give myself 6-18 months before I think top-performing LLM's can do 80% of the day-to-day issues I'm assigned. > Why doesn’t anyone acknowledge loops like this? Thisis something you run into early-on using LLM's and learn to sidestep. This looping is a sort of "context-rot" -- the agent has the problem statement as part of it's input, and then a series of incorrect solutions. Now what you've got is a junk-soup where the original problem is buried somewhere in the pile. Best approach I've found is to start a fresh conversation with the original problem statement and any improvements/negative reinforcements you've gotten out of the LLM tacked on. I typically have ChatGPT 5 Thinking, Claude 4.1 Opus, Grok 4, and Gemini 2.5 Pro all churning on the same question at once and then copy-pasting relevant improvements across each.

Wowfunhappy 1y ago

...I don't know why, but I swear to god, when Claude gets into one of these cycles I can often get it out by dropping the f-bomb, with maybe a 50% success rate. Something about that word lets it know that it needs to break the pattern.

ACCount37 1y ago

1-turn instruction following and multi-turn instruction following are not the same exact capability, and some AIs only "get good" at the former. 1-turn gets more training attention - because it's more noticeable, in casual use and benchmarks both, and also easier to train for. With weak multi-turn instruction following, context data will often dominate over user instructions. Resulting in very "loopy" AI - and more sessions that are easier to restart from scratch than to "fix". Gemini is notorious for underperforming at this, while Claude has relatively good performance. I expect that many models from lesser known providers would also have a multi-turn instruction following gap.

rtkwe 1y ago
I have this problem all the time with minor image edits on ChatGPT the few time's I've tried it. Any time I try to do a second edit or change to the generated image it seems to take the already degraded output from it's first attempt and use that instead of the original image.
vidarh 1y ago
This is a good point, and to drive this home to people, if you have a conversation of this pattern: User: Fix this problem ... Assistant: X User: No, don't do X Assistant: Y User: No, Y is wrong too. Assistant: X It is generally pointless to continue. You now have a context that is full of the assistant explaining to you and itself why X and Y are the right answers, and much less context of you explaining why it is wrong. If you reach that state, start over, and constrain your initial request to exclude X and Y. If it brings up either again, start over, and constrain your request further. If the model is bad at handling multiple turns without getting into a loop, telling it that it is wrong is not generally going to achieve anything, but starting over with better instructions often will. I see so many people get stuck "arguing" with a model over this, getting more and more frustrated as the model keeps repeating variations of the broken answer, without realising they're filling the context with arguments from the model for why the broken answer is right.
- psadauskas 1y ago
  There's also the Pink Elephant Paradox (Whatever you do, DO NOT think about a pink elephant). If you mention X or Y, even if they're preceded by "DO NOT" in all caps, an LLM will still end up with both X and Y into its context, making it more likely it gets used. I'm running out of ways to tell the assistant to not use mocks for tests, it really really wants to use them.

lucideer 1y ago

True. This also often happens. Probably the ideal would be to have a UI / non-chat-based mechanism for discarding select context.

dkersten 1y ago

My pet peace is when I point out a problem it responds with acknowledgement and then explaining why it’s wrong. Like… I already know why it’s wrong, since I’m the one that pointed it out!

brookst 1y ago

You're conflating "correct themselves" with "are guaranteed to give the correct answer", which are two really different things. And in fact you're just echoing GP's point: their corrections can be wrong. You case is no different from: - AI: "The capital of France is Paris" - User: "This is wrong, it changed to Montreal in 2005" - AI: "You're absolutely right! The capital of France is Montreal"

kelseyfrog 1y ago
Instead I get this: Nope—Paris is the capital of France and has been for centuries. Montreal is in Quebec, Canada. France’s presidency (Élysée), parliament (Assemblée nationale and Sénat), and ministries are all in Paris.
- brookst 1y ago
  I was using an oversimplified example to illustrate. In practice, it appears more in large context statements about the context. If the human is wrong, there’s a good chance the AI will cheerfully agree and then be wrong too. I was reminded of this this morning when using Claude code (which I love) and I was confidently incorrect about a feature of my app. Claude proposed a plan, I said “great, but don’t build part 3, just use the existing ModuletExist”. Claude tied itself in knots because it believes me. (The module does exist in another project I’m working on)
  - kelseyfrog 1y ago
    Help me understand how it's tangibly different from a veteran telling the rookie to find headlight fluid, winter air for the tires, or keys to the bomb range.

zero_iq 1y ago

I've seen ChatGPT get stuck in this loop all by itself, generating a long multi-page answer where it constantly catches itself, refutes itself, offers a new answer with the same problem, rinse and repeat... All in the same response!