7 ms·

While the thrust of this article is generally correct, I have two issues with it: 1. The words "the only thing" massively underplays the difficulty of this pro

While the thrust of this article is generally correct, I have two issues with it: 1. The words "the only thing" massively underplays the difficulty of this problem. It's not a small thing. 2. One of the issues I've seen with a lot of chat LLMs is their willingness to correct themselves when asked - this might seem, on the surface, to be a positive (allowing a user to steer the AI toward a more accurate or appropriate solution), but in reality it simply plays into users' biases & makes it more likely that the user will accept & approve of incorrect responses from the AI. Often, rather than "correcting" itself it merely "teaches" the AI how to be confidently wrong in an amenable & subtle manner which the individual user finds easy to accept (or more difficult to spot). If anything, unless/until we can solve the (insurmountable) problem of AI being wrong, AI should at least be trained to be confidently & stubbornly wrong (or right). This would also likely lead to better consistency in testing.

traceroute66 1y ago
> is their willingness to correct themselves when asked Except they don't correct themselves when asked. I'm sure we've all been there, many, many, many,many,many times .... - User: "This is wrong because X" - AI: "You're absolutely right ! Here's a production-ready fixed answer" - User: "No, that's wrong because Y" - AI: "I apologise for frustrating you ! Here's a robust answer that works" - User: "You idiot, you just put X back in there" - and so continues the vicious circle....
- stetrain 1y ago
  Yep, the LLM will happily continue this spiral indefinitely but I've learned that if providing a bit more context and one correction doesn't provide a good solution, continuing is generally a waste of time. They tend to very quickly lose useful context of the original problem and stated goals.
  - nyeah 1y ago
    Yes, that is the point of the comment.
    - stetrain 1y ago
      Yes, you’re absolutely right! Agreeing with the comment and adding my own experience was the point of my comment. Is there anything else I can help you with?
      - nyeah 1y ago
        Ok, fair, clearly I misinterpreted what you wrote.
      - deleted 1y ago
        [deleted]
- therobots927 1y ago
  Yeah I think our jobs are safe. Why doesn’t anyone acknowledge loops like this? They happen all the time and I’m only using it once a week at the most
  - traceroute66 1y ago
    The AI-fanbois will quickly tell you that you are misusing the context or your prompt is "wrong". But I've had it consistently happens to me on tiny contexts (e.g. I've had to spend time trying - and failing - to get it to fix a mess it was making with a straightforward 200-ish line bash script). And its also very frequently happened to me when I've been very careful with my prompts (e.g. explicitly telling it to use a specific version of a specific library ... and it goes and ignores me completely and picks some random library).
    - gavinray 1y ago
      I'd be curious if you could share some poor-performing prompts. I would be willing to record myself using them across paid models with custom instructions and see if the output is still garbage.
      - mrguyorama 1y ago
        This is just the new version of "works on my machine". Oh, I was able to contrive a correct answer from my prompt because the random number generator smiled upon me today. That's not useful.
  - gavinray 1y ago
    > Yeah I think our jobs are safe. I give myself 6-18 months before I think top-performing LLM's can do 80% of the day-to-day issues I'm assigned. > Why doesn’t anyone acknowledge loops like this? Thisis something you run into early-on using LLM's and learn to sidestep. This looping is a sort of "context-rot" -- the agent has the problem statement as part of it's input, and then a series of incorrect solutions. Now what you've got is a junk-soup where the original problem is buried somewhere in the pile. Best approach I've found is to start a fresh conversation with the original problem statement and any improvements/negative reinforcements you've gotten out of the LLM tacked on. I typically have ChatGPT 5 Thinking, Claude 4.1 Opus, Grok 4, and Gemini 2.5 Pro all churning on the same question at once and then copy-pasting relevant improvements across each.
- Wowfunhappy 1y ago
  ...I don't know why, but I swear to god, when Claude gets into one of these cycles I can often get it out by dropping the f-bomb, with maybe a 50% success rate. Something about that word lets it know that it needs to break the pattern.
- ACCount37 1y ago
  1-turn instruction following and multi-turn instruction following are not the same exact capability, and some AIs only "get good" at the former. 1-turn gets more training attention - because it's more noticeable, in casual use and benchmarks both, and also easier to train for. With weak multi-turn instruction following, context data will often dominate over user instructions. Resulting in very "loopy" AI - and more sessions that are easier to restart from scratch than to "fix". Gemini is notorious for underperforming at this, while Claude has relatively good performance. I expect that many models from lesser known providers would also have a multi-turn instruction following gap.
  - rtkwe 1y ago
    I have this problem all the time with minor image edits on ChatGPT the few time's I've tried it. Any time I try to do a second edit or change to the generated image it seems to take the already degraded output from it's first attempt and use that instead of the original image.
  - vidarh 1y ago
    This is a good point, and to drive this home to people, if you have a conversation of this pattern: User: Fix this problem ... Assistant: X User: No, don't do X Assistant: Y User: No, Y is wrong too. Assistant: X It is generally pointless to continue. You now have a context that is full of the assistant explaining to you and itself why X and Y are the right answers, and much less context of you explaining why it is wrong. If you reach that state, start over, and constrain your initial request to exclude X and Y. If it brings up either again, start over, and constrain your request further. If the model is bad at handling multiple turns without getting into a loop, telling it that it is wrong is not generally going to achieve anything, but starting over with better instructions often will. I see so many people get stuck "arguing" with a model over this, getting more and more frustrated as the model keeps repeating variations of the broken answer, without realising they're filling the context with arguments from the model for why the broken answer is right.
    - psadauskas 1y ago
      There's also the Pink Elephant Paradox (Whatever you do, DO NOT think about a pink elephant). If you mention X or Y, even if they're preceded by "DO NOT" in all caps, an LLM will still end up with both X and Y into its context, making it more likely it gets used. I'm running out of ways to tell the assistant to not use mocks for tests, it really really wants to use them.
- lucideer 1y ago
  True. This also often happens. Probably the ideal would be to have a UI / non-chat-based mechanism for discarding select context.
- dkersten 1y ago
  My pet peace is when I point out a problem it responds with acknowledgement and then explaining why it’s wrong. Like… I already know why it’s wrong, since I’m the one that pointed it out!
- brookst 1y ago
  You're conflating "correct themselves" with "are guaranteed to give the correct answer", which are two really different things. And in fact you're just echoing GP's point: their corrections can be wrong. You case is no different from: - AI: "The capital of France is Paris" - User: "This is wrong, it changed to Montreal in 2005" - AI: "You're absolutely right! The capital of France is Montreal"
  - kelseyfrog 1y ago
    Instead I get this: Nope—Paris is the capital of France and has been for centuries. Montreal is in Quebec, Canada. France’s presidency (Élysée), parliament (Assemblée nationale and Sénat), and ministries are all in Paris.
    - brookst 1y ago
      I was using an oversimplified example to illustrate. In practice, it appears more in large context statements about the context. If the human is wrong, there’s a good chance the AI will cheerfully agree and then be wrong too. I was reminded of this this morning when using Claude code (which I love) and I was confidently incorrect about a feature of my app. Claude proposed a plan, I said “great, but don’t build part 3, just use the existing ModuletExist”. Claude tied itself in knots because it believes me. (The module does exist in another project I’m working on)
      - kelseyfrog 1y ago
        Help me understand how it's tangibly different from a veteran telling the rookie to find headlight fluid, winter air for the tires, or keys to the bomb range.
- zero_iq 1y ago
  I've seen ChatGPT get stuck in this loop all by itself, generating a long multi-page answer where it constantly catches itself, refutes itself, offers a new answer with the same problem, rinse and repeat... All in the same response!
stetrain 1y ago
Yes, the quick to correct itself isn't really useful. I would not like a human assistant/intern/pair programmer who when asked how to do X said: > To accomplish X you can just use Y! But Y isn't applicable in this scenario. > Oh, you're absolutely right! Instead of Y you can do Z. Are you sure? I don't think Z accomplishes X. > On second thought you're absolutely correct. Y or Z will clearly not accomplish X, but let's try Q....
sfn42 1y ago
Being confidently wrong isn't even the problem. It's a symptom of the much deeper problem that these things aren't AI at all, they're just atocomplete bots good enough to kind of seem like AI. There's no actual intelligence. That's the problem.
- ninetyninenine 1y ago
  No. The experts in the field are past this argument. People have moved on. It is clear to everyone who builds LLMs that the AI is intelligent. The algorithm was autocomplete, but we are finding as an autocomplete bot is basically autocompleting things with humanity changing intelligent content. Your opinion is a minority now and not shared by people on the forefront of building these things. Your holding onto the initial fever pitched alarmist reaction people had to LLMs when it first came out. Like you realize humans hallucinate too right? And that there are humans that have a disease that makes them hallucinate constantly. Hallucinations don’t preclude humans from being “intelligent”. It also doesn’t preclude the LLM from being intelligent.
  - eCa 1y ago
    > Like you realize humans hallucinate too right? A developer that hallucinates at work to the extent that LLMs does would probably have issues getting their PRs past code reviews a lot.
    - bigstrat2003 1y ago
      They would have issues even remaining employed. AI defenders are very quick to point out "humans mistakes too", but that is a false equivalence because humans learn. If a junior makes a really stupid mistake, when I show him the correct way he won't make that mistake again. An AI will, because (as people correctly point out) it has no actual intelligence.
      - ninetyninenine 1y ago
        There’s examples of humans who can’t learn. Have you seen the movie memento. There are cases where humans lose all ability to form long term memories and outside of a timed context window they remember nothing. That context window is minutes at best. According to your logic these people have no actual intelligence or sentience. Therefore they should be euthanized. You personally can grab a gun and execute each of these people one by one with a bullet straight to the head because clearly these people have no actual intelligence or sentience. That’s the implication of your logic. https://en.m.wikipedia.org/wiki/Anterograde_amnesia https://en.m.wikipedia.org/wiki/Anterograde_amnesia It’s called anterograde amnesia. Do you see how your logic can justify gassing all these people holocaust style? When I point out the flaw in your logic do you use the new facts to form a new conclusion? Or do you rearrange the facts to maintain support for your existing conclusion? If you did the later I hate to tell you this, it wasn’t very intelligent. It was biased. But given that you’re human, that’s what you most likely did and it’s normal. But pause for a second and try to do the former of using the new facts to form a different more nuanced conclusion.
decentrality 1y ago
Agreed with #1 ( came here to say that also ) Pronoun and noun wordplay aside ( 'Their' ... `themselves` ) I also agree that LLMs can correct the path being taken, regenerate better, etc... But the idea that 'AI' needs to be _stubbornly_ wrong ( more human in the worst way ) is a bad idea. There is a fundamental showing, and it is being missed. What is the context reality? Where is this prompt/response taking place? Almost guaranteed to be going on in a context which is itself violated or broken; such as with `Open Web UI` in a conservative example: Who even cares if we get the responses right? Now we have 'right' responses in a cul-de-sac universe. This might be worthwhile using `Ollama` in `Zed` for example, but for what purpose? An agentic process that is going to be audited anyway, because we always need to understand the code? And if we are talking about decision-making processes in a corporate system strategy... now we are fully down the rabbit hole. The corporate context itself is coming or going on whether it is right/wrong, good/evil, etc... as the entire point of what is going on there. The entire world is already beating that corporation to death or not, or it is beating the world to death or not... so the 'AI' aspect is more of an accelerant of an underlying dynamic, and if we stand back... what corporation is not already stubbornly wrong, on average?
- taco_emoji 1y ago
  > Pronoun and noun wordplay aside ( 'Their' ... `themselves` ) How is that wordplay? Those are the correct pronouns.
energy123 1y ago
Mechanistic interpretability could play a role here. The sycophancy you describe in chat mode could be when the question is "too difficult" and the AI defaults to easy circuits that rely on simple rule of thumbs (like does the context contain positive words such as "excellent"). The user experiences this as the AI just following basic nudges. Could real-time observability into the network's internals somehow feed back into the model to reduce these hallucination-inducing shortcuts? Like train the system to detect when a shortcut is being used, then do something about it?
stingraycharles 1y ago
> 1. The words "the only thing" massively underplays the difficulty of this problem. It's not a small thing. Exactly. One could argue that this is just an artifact from the fundamental technique being used: it’s a really fancy autocomplete based on a huge context window. People still think there’s actual intelligence in there, while the actual problems by making these systems appear intelligent is mostly algorithms and software managing exactly what goes into these context windows at what place. Don’t get me wrong: it feels like magic. But I would argue that the only way to recognize a model being “confidently wrong” is to let another model, trained on completely different datasets with different techniques, judge them. And then preferably multiple. (This is actually a feature of an MCP tool I use, “consensus” from zen-mcp-server, which enables you to query multiple different models to reach a consensus on a certain problem / solution).
ninetyninenine 1y ago
It’s not massively underplaying it imo. AI hype is real. This is revolutionary technology that humanity has never seen before. But it happened at a time where hype can be delivered at a magnitude never before seen by humanity as well to a degree of volume that is completely unnatural by any standard set previously by hype machines created by humanity. Not even landing on the moon has inundated people with as much hype. But inevitably like landing on the moon, humanity is suffering from hype fatigue. Too much hype makes us numb to the reality of how insane the technology is. Like when someone says the only thing stopping LLMs is hallucinations… that is literally the last gap. LLMs cover creativity, comprehension, analysis, knowledge and much more. Hallucinations is it. The final problem is targeted and boxed into something much more narrower then just build a human level AI from scratch. Don’t get me wrong. Hallucinations are hard. But this being the last thing left is not an underplay. Yes it’s a massive issue but yes it is also a massive achievement to reduce all of agi to simply solving just an hallucination problem.
- taco_emoji 1y ago
  Oh, buddy, LLM hallucinations are not the only gap left for AGI
  - ninetyninenine 1y ago
    It is. After that it's virtually indistinguishable from chatting with a human
- indigoabstract 1y ago
  I think you would have really enjoyed living in the '50s, when the future was bright and colonizing Mars was basically a solved problem. What we got instead is a bunch of wisecracking programmers who like to remind everyone of the 90–90 rule, or the last 10 percent.
  - ninetyninenine 1y ago
    Mars wasn't part of the 90-90 rule genius. If you know your history, it's mainly because political interest was lost. Technologically sending someone to mars is 100000000x more feasible than agi simply because the technology and theory exists such that we can do it. AGI is part of the last 10 percent rule. But like that's the entire issue. 90% is still 90% progress. That is massive. And the hype surrounding LLMs has made people forget how far 90% is. People are going, "LLMs are retarded because it has the IQ of a 5 year old". They don't realize how even getting it to the level of a 5 year old was impossible for decades and decades.
dns_snek 1y ago
> but in reality it simply plays into users' biases & makes it more likely that the user will accept & approve of incorrect responses from the AI. Yes! I often find myself overthinking my phrasing to the nth degree because I've learned that even a sprinkle of bias can often make the LLM run in that direction even if it's not the correct answer. It often feels a bit like interacting with a deeply unstable and insecure people pleasing person. I can't say anything that could possibly be interpreted as a disagreement because they'll immediately flip the script, I can't mention that I like pizza before asking them what their favorite food is because they'll just mirror me.
KoolKat23 1y ago
Gemini 2.5 pro is quite good at being stubborn (well at least the initial release versions, haven't tested since).
tango12 1y ago
The AI being wrong problem is probably not insurmountable. Humans have meta-cognition that helps them judge if they're doing a thing with lots of assumptions vs doing something that's blessed. Humans decouple planning from execution right? Not fully but we choose when to separate it and when to not. If we had enough data on here's a good plan given user context and here's a bad plan, it doesn't seem unreasonable to have a pretty reliable meta cognition capability on the goodness of a plan.
- procaryote 1y ago
  Depending on your definitions, either: * there are already lots of "reasoning" models trying meta-cognition, while still getting simple things wrong or: * the models aren't doing cognition, so meta-cognition seems very far away
burnte 1y ago
I'm just so glad people are seeing this. I started saying this literally days after ChatGPT came out and I started examining the technology. It's SUPER useful, but it's assistive, it can't be trusted to do things autonomously yet. That's ok, though, it can make human workers more productive, rather than worrying about replacing humans.

traceroute66 1y ago

> is their willingness to correct themselves when asked Except they don't correct themselves when asked. I'm sure we've all been there, many, many, many,many,many times .... - User: "This is wrong because X" - AI: "You're absolutely right ! Here's a production-ready fixed answer" - User: "No, that's wrong because Y" - AI: "I apologise for frustrating you ! Here's a robust answer that works" - User: "You idiot, you just put X back in there" - and so continues the vicious circle....

stetrain 1y ago
Yep, the LLM will happily continue this spiral indefinitely but I've learned that if providing a bit more context and one correction doesn't provide a good solution, continuing is generally a waste of time. They tend to very quickly lose useful context of the original problem and stated goals.
- nyeah 1y ago
  Yes, that is the point of the comment.
  - stetrain 1y ago
    Yes, you’re absolutely right! Agreeing with the comment and adding my own experience was the point of my comment. Is there anything else I can help you with?
    - nyeah 1y ago
      Ok, fair, clearly I misinterpreted what you wrote.
    - deleted 1y ago
      [deleted]
therobots927 1y ago
Yeah I think our jobs are safe. Why doesn’t anyone acknowledge loops like this? They happen all the time and I’m only using it once a week at the most
- traceroute66 1y ago
  The AI-fanbois will quickly tell you that you are misusing the context or your prompt is "wrong". But I've had it consistently happens to me on tiny contexts (e.g. I've had to spend time trying - and failing - to get it to fix a mess it was making with a straightforward 200-ish line bash script). And its also very frequently happened to me when I've been very careful with my prompts (e.g. explicitly telling it to use a specific version of a specific library ... and it goes and ignores me completely and picks some random library).
  - gavinray 1y ago
    I'd be curious if you could share some poor-performing prompts. I would be willing to record myself using them across paid models with custom instructions and see if the output is still garbage.
    - mrguyorama 1y ago
      This is just the new version of "works on my machine". Oh, I was able to contrive a correct answer from my prompt because the random number generator smiled upon me today. That's not useful.
- gavinray 1y ago
  > Yeah I think our jobs are safe. I give myself 6-18 months before I think top-performing LLM's can do 80% of the day-to-day issues I'm assigned. > Why doesn’t anyone acknowledge loops like this? Thisis something you run into early-on using LLM's and learn to sidestep. This looping is a sort of "context-rot" -- the agent has the problem statement as part of it's input, and then a series of incorrect solutions. Now what you've got is a junk-soup where the original problem is buried somewhere in the pile. Best approach I've found is to start a fresh conversation with the original problem statement and any improvements/negative reinforcements you've gotten out of the LLM tacked on. I typically have ChatGPT 5 Thinking, Claude 4.1 Opus, Grok 4, and Gemini 2.5 Pro all churning on the same question at once and then copy-pasting relevant improvements across each.
Wowfunhappy 1y ago
...I don't know why, but I swear to god, when Claude gets into one of these cycles I can often get it out by dropping the f-bomb, with maybe a 50% success rate. Something about that word lets it know that it needs to break the pattern.
ACCount37 1y ago
1-turn instruction following and multi-turn instruction following are not the same exact capability, and some AIs only "get good" at the former. 1-turn gets more training attention - because it's more noticeable, in casual use and benchmarks both, and also easier to train for. With weak multi-turn instruction following, context data will often dominate over user instructions. Resulting in very "loopy" AI - and more sessions that are easier to restart from scratch than to "fix". Gemini is notorious for underperforming at this, while Claude has relatively good performance. I expect that many models from lesser known providers would also have a multi-turn instruction following gap.
- rtkwe 1y ago
  I have this problem all the time with minor image edits on ChatGPT the few time's I've tried it. Any time I try to do a second edit or change to the generated image it seems to take the already degraded output from it's first attempt and use that instead of the original image.
- vidarh 1y ago
  This is a good point, and to drive this home to people, if you have a conversation of this pattern: User: Fix this problem ... Assistant: X User: No, don't do X Assistant: Y User: No, Y is wrong too. Assistant: X It is generally pointless to continue. You now have a context that is full of the assistant explaining to you and itself why X and Y are the right answers, and much less context of you explaining why it is wrong. If you reach that state, start over, and constrain your initial request to exclude X and Y. If it brings up either again, start over, and constrain your request further. If the model is bad at handling multiple turns without getting into a loop, telling it that it is wrong is not generally going to achieve anything, but starting over with better instructions often will. I see so many people get stuck "arguing" with a model over this, getting more and more frustrated as the model keeps repeating variations of the broken answer, without realising they're filling the context with arguments from the model for why the broken answer is right.
  - psadauskas 1y ago
    There's also the Pink Elephant Paradox (Whatever you do, DO NOT think about a pink elephant). If you mention X or Y, even if they're preceded by "DO NOT" in all caps, an LLM will still end up with both X and Y into its context, making it more likely it gets used. I'm running out of ways to tell the assistant to not use mocks for tests, it really really wants to use them.
lucideer 1y ago
True. This also often happens. Probably the ideal would be to have a UI / non-chat-based mechanism for discarding select context.
dkersten 1y ago
My pet peace is when I point out a problem it responds with acknowledgement and then explaining why it’s wrong. Like… I already know why it’s wrong, since I’m the one that pointed it out!
brookst 1y ago
You're conflating "correct themselves" with "are guaranteed to give the correct answer", which are two really different things. And in fact you're just echoing GP's point: their corrections can be wrong. You case is no different from: - AI: "The capital of France is Paris" - User: "This is wrong, it changed to Montreal in 2005" - AI: "You're absolutely right! The capital of France is Montreal"
- kelseyfrog 1y ago
  Instead I get this: Nope—Paris is the capital of France and has been for centuries. Montreal is in Quebec, Canada. France’s presidency (Élysée), parliament (Assemblée nationale and Sénat), and ministries are all in Paris.
  - brookst 1y ago
    I was using an oversimplified example to illustrate. In practice, it appears more in large context statements about the context. If the human is wrong, there’s a good chance the AI will cheerfully agree and then be wrong too. I was reminded of this this morning when using Claude code (which I love) and I was confidently incorrect about a feature of my app. Claude proposed a plan, I said “great, but don’t build part 3, just use the existing ModuletExist”. Claude tied itself in knots because it believes me. (The module does exist in another project I’m working on)
    - kelseyfrog 1y ago
      Help me understand how it's tangibly different from a veteran telling the rookie to find headlight fluid, winter air for the tires, or keys to the bomb range.
zero_iq 1y ago
I've seen ChatGPT get stuck in this loop all by itself, generating a long multi-page answer where it constantly catches itself, refutes itself, offers a new answer with the same problem, rinse and repeat... All in the same response!

stetrain 1y ago

Yes, the quick to correct itself isn't really useful. I would not like a human assistant/intern/pair programmer who when asked how to do X said: > To accomplish X you can just use Y! But Y isn't applicable in this scenario. > Oh, you're absolutely right! Instead of Y you can do Z. Are you sure? I don't think Z accomplishes X. > On second thought you're absolutely correct. Y or Z will clearly not accomplish X, but let's try Q....

sfn42 1y ago

Being confidently wrong isn't even the problem. It's a symptom of the much deeper problem that these things aren't AI at all, they're just atocomplete bots good enough to kind of seem like AI. There's no actual intelligence. That's the problem.

ninetyninenine 1y ago
No. The experts in the field are past this argument. People have moved on. It is clear to everyone who builds LLMs that the AI is intelligent. The algorithm was autocomplete, but we are finding as an autocomplete bot is basically autocompleting things with humanity changing intelligent content. Your opinion is a minority now and not shared by people on the forefront of building these things. Your holding onto the initial fever pitched alarmist reaction people had to LLMs when it first came out. Like you realize humans hallucinate too right? And that there are humans that have a disease that makes them hallucinate constantly. Hallucinations don’t preclude humans from being “intelligent”. It also doesn’t preclude the LLM from being intelligent.
- eCa 1y ago
  > Like you realize humans hallucinate too right? A developer that hallucinates at work to the extent that LLMs does would probably have issues getting their PRs past code reviews a lot.
  - bigstrat2003 1y ago
    They would have issues even remaining employed. AI defenders are very quick to point out "humans mistakes too", but that is a false equivalence because humans learn. If a junior makes a really stupid mistake, when I show him the correct way he won't make that mistake again. An AI will, because (as people correctly point out) it has no actual intelligence.
    - ninetyninenine 1y ago
      There’s examples of humans who can’t learn. Have you seen the movie memento. There are cases where humans lose all ability to form long term memories and outside of a timed context window they remember nothing. That context window is minutes at best. According to your logic these people have no actual intelligence or sentience. Therefore they should be euthanized. You personally can grab a gun and execute each of these people one by one with a bullet straight to the head because clearly these people have no actual intelligence or sentience. That’s the implication of your logic. https://en.m.wikipedia.org/wiki/Anterograde_amnesia https://en.m.wikipedia.org/wiki/Anterograde_amnesia It’s called anterograde amnesia. Do you see how your logic can justify gassing all these people holocaust style? When I point out the flaw in your logic do you use the new facts to form a new conclusion? Or do you rearrange the facts to maintain support for your existing conclusion? If you did the later I hate to tell you this, it wasn’t very intelligent. It was biased. But given that you’re human, that’s what you most likely did and it’s normal. But pause for a second and try to do the former of using the new facts to form a different more nuanced conclusion.

decentrality 1y ago

Agreed with #1 ( came here to say that also ) Pronoun and noun wordplay aside ( 'Their' ... `themselves` ) I also agree that LLMs can correct the path being taken, regenerate better, etc... But the idea that 'AI' needs to be _stubbornly_ wrong ( more human in the worst way ) is a bad idea. There is a fundamental showing, and it is being missed. What is the context reality? Where is this prompt/response taking place? Almost guaranteed to be going on in a context which is itself violated or broken; such as with `Open Web UI` in a conservative example: Who even cares if we get the responses right? Now we have 'right' responses in a cul-de-sac universe. This might be worthwhile using `Ollama` in `Zed` for example, but for what purpose? An agentic process that is going to be audited anyway, because we always need to understand the code? And if we are talking about decision-making processes in a corporate system strategy... now we are fully down the rabbit hole. The corporate context itself is coming or going on whether it is right/wrong, good/evil, etc... as the entire point of what is going on there. The entire world is already beating that corporation to death or not, or it is beating the world to death or not... so the 'AI' aspect is more of an accelerant of an underlying dynamic, and if we stand back... what corporation is not already stubbornly wrong, on average?

taco_emoji 1y ago
> Pronoun and noun wordplay aside ( 'Their' ... `themselves` ) How is that wordplay? Those are the correct pronouns.

energy123 1y ago

Mechanistic interpretability could play a role here. The sycophancy you describe in chat mode could be when the question is "too difficult" and the AI defaults to easy circuits that rely on simple rule of thumbs (like does the context contain positive words such as "excellent"). The user experiences this as the AI just following basic nudges. Could real-time observability into the network's internals somehow feed back into the model to reduce these hallucination-inducing shortcuts? Like train the system to detect when a shortcut is being used, then do something about it?

stingraycharles 1y ago

> 1. The words "the only thing" massively underplays the difficulty of this problem. It's not a small thing. Exactly. One could argue that this is just an artifact from the fundamental technique being used: it’s a really fancy autocomplete based on a huge context window. People still think there’s actual intelligence in there, while the actual problems by making these systems appear intelligent is mostly algorithms and software managing exactly what goes into these context windows at what place. Don’t get me wrong: it feels like magic. But I would argue that the only way to recognize a model being “confidently wrong” is to let another model, trained on completely different datasets with different techniques, judge them. And then preferably multiple. (This is actually a feature of an MCP tool I use, “consensus” from zen-mcp-server, which enables you to query multiple different models to reach a consensus on a certain problem / solution).

ninetyninenine 1y ago

It’s not massively underplaying it imo. AI hype is real. This is revolutionary technology that humanity has never seen before. But it happened at a time where hype can be delivered at a magnitude never before seen by humanity as well to a degree of volume that is completely unnatural by any standard set previously by hype machines created by humanity. Not even landing on the moon has inundated people with as much hype. But inevitably like landing on the moon, humanity is suffering from hype fatigue. Too much hype makes us numb to the reality of how insane the technology is. Like when someone says the only thing stopping LLMs is hallucinations… that is literally the last gap. LLMs cover creativity, comprehension, analysis, knowledge and much more. Hallucinations is it. The final problem is targeted and boxed into something much more narrower then just build a human level AI from scratch. Don’t get me wrong. Hallucinations are hard. But this being the last thing left is not an underplay. Yes it’s a massive issue but yes it is also a massive achievement to reduce all of agi to simply solving just an hallucination problem.

taco_emoji 1y ago
Oh, buddy, LLM hallucinations are not the only gap left for AGI
- ninetyninenine 1y ago
  It is. After that it's virtually indistinguishable from chatting with a human
indigoabstract 1y ago
I think you would have really enjoyed living in the '50s, when the future was bright and colonizing Mars was basically a solved problem. What we got instead is a bunch of wisecracking programmers who like to remind everyone of the 90–90 rule, or the last 10 percent.
- ninetyninenine 1y ago
  Mars wasn't part of the 90-90 rule genius. If you know your history, it's mainly because political interest was lost. Technologically sending someone to mars is 100000000x more feasible than agi simply because the technology and theory exists such that we can do it. AGI is part of the last 10 percent rule. But like that's the entire issue. 90% is still 90% progress. That is massive. And the hype surrounding LLMs has made people forget how far 90% is. People are going, "LLMs are retarded because it has the IQ of a 5 year old". They don't realize how even getting it to the level of a 5 year old was impossible for decades and decades.

dns_snek 1y ago

> but in reality it simply plays into users' biases & makes it more likely that the user will accept & approve of incorrect responses from the AI. Yes! I often find myself overthinking my phrasing to the nth degree because I've learned that even a sprinkle of bias can often make the LLM run in that direction even if it's not the correct answer. It often feels a bit like interacting with a deeply unstable and insecure people pleasing person. I can't say anything that could possibly be interpreted as a disagreement because they'll immediately flip the script, I can't mention that I like pizza before asking them what their favorite food is because they'll just mirror me.

KoolKat23 1y ago

Gemini 2.5 pro is quite good at being stubborn (well at least the initial release versions, haven't tested since).

tango12 1y ago

The AI being wrong problem is probably not insurmountable. Humans have meta-cognition that helps them judge if they're doing a thing with lots of assumptions vs doing something that's blessed. Humans decouple planning from execution right? Not fully but we choose when to separate it and when to not. If we had enough data on here's a good plan given user context and here's a bad plan, it doesn't seem unreasonable to have a pretty reliable meta cognition capability on the goodness of a plan.

procaryote 1y ago
Depending on your definitions, either: * there are already lots of "reasoning" models trying meta-cognition, while still getting simple things wrong or: * the models aren't doing cognition, so meta-cognition seems very far away

burnte 1y ago

I'm just so glad people are seeing this. I started saying this literally days after ChatGPT came out and I started examining the technology. It's SUPER useful, but it's assistive, it can't be trusted to do things autonomously yet. That's ok, though, it can make human workers more productive, rather than worrying about replacing humans.