3 ms·

LLMs are broken, too: > "Of course. This is an excellent example that demonstrates a fundamental and powerful concept in Rust: the distinction between cloning

LLMs are broken, too: > "Of course. This is an excellent example that demonstrates a fundamental and powerful concept in Rust: the distinction between cloning a smart pointer and cloning the data it points to. [...]" Then I post the compiler's output: > "Ah, an excellent follow-up! You are absolutely right to post the compiler error. My apologies—my initial explanation described how one might expect it to work logically, but I neglected a crucial and subtle detail [...]" Aren't you also getting very tired of this behavior?

the_mitsuhiko 1y ago
> Aren't you also getting very tired of this behavior? The part that annoys me definitely is how confident they all sound. However the way I'm using them is with tool usage loops and so it usually runs into part 2 immediately and course corrects.
- bt1a 1y ago
  Well, they're usually told that they're some unicorn master of * languages, frameworks, skillsets, etc., so can you really fault them? :)
renewiltord 1y ago
Haha, I encountered the opposite of this when I did a destructive thing recently but first asked Gemini, then countered it saying it’s wrong and it insisted it was right. So the reality they encountered is probably that: it either is stubbornly wrong or overly obsequious with no ability to switch. My friend was a big fan of Gemini 2.5 Pro and I kept telling him it was garbage except for OCR and he nearly followed what it recommended. Haha, he’s never touching it again. Every other LLM changed its tune on pushback.
ramon156 1y ago
You should check Twitter nowadays, people love this kind of response. Some even use it as an argument
- darkwater 1y ago
  And this is why basically LLMs are "bad". They have already reached critical mass adoption, they are right or mostly right most of the time but they also screw up badly many times as well. And people will just not know, trust blindly and going even deeper down in the spiral of the total absence of critical judgement. And yeah, it also happened with Google and search engines back in the day ("I found it on the web so it must be true") but now with LLMs it is literally tailored to what you are asking, for every possible question you can ask (well, minus the censored ones). I keep thinking the LLM contribution to humanity is/will be a net negative in the long run.
  - bt1a 1y ago
    Yet theyre fantastic personal tutors / assistants who can provide a deeply needed 1:1 learning interface for less privileged individuals. I emphasize 'can'. Not saying kids should have them by their side in their current rough around the edges and mediocre intelligent forms. Many will get burned as you describe, but it should be a lesson to curate information from multiple sources and practice applying reasoning skills!
    - darkwater 1y ago
      I agree with your take, and I personally used Claude and ChatGPT to learn better/hone some skills while interviewing to land a new job. And they also help me get unstucked when doing small home fixes, because it's a custom-tailored answer to my current doubt/issue that a normal web search would make much more complicated to answer (I had to know more context about it). But still, they get things wrong and can lead you astray even if you know the topic.
    - AIPedant 1y ago
      My dad taught high school science until retiring this year, and at least in 2024 the LLM tutors were totally useless for honest learning. They were good at the “happy path” but the space of high schoolers’ misconceptions about physics greatly exceeds the training data and can’t be cheaply RLHFed, so they crap the bed when you role play as a dumb high schooler. In my experience this is still true for the reasoning models with undergraduate mathematics - if you ask it to do your point-set topology homework (dishonest learning) it will score > 85/100, if you are confused about point-set topology and try to ask it an honest (but ignorant) question it will give you a pile of pseudo-mathematical BS.
rfoo 1y ago
TBH I'm tired of only the "Ah, an excellent follow-up! You are absolutely right <...> My apologies" part.
- IshKebab 1y ago
  Yeah they definitely didn't do that in the past. We've lost "as a large language model" and "it's important to remember" but gained "you're absolutely right!" I would have thought they'd add "don't apologise!!!!" or something like that to the system prompt like they do to avoid excessive lists.
deleted 1y ago
[deleted]
codedokode 1y ago
Languages like Rust and C seem to be too complicated for them. I also asked different LLMs to write a C macro or function that creates a struct and a function to print it (so that I don't have to duplicate a field list) and it generates plausible garbage.
- bigfishrunning 1y ago
  LLMs only ever produce plausible garbage -- sometimes that garbage happens to be right, but that's down to luck.

the_mitsuhiko 1y ago

> Aren't you also getting very tired of this behavior? The part that annoys me definitely is how confident they all sound. However the way I'm using them is with tool usage loops and so it usually runs into part 2 immediately and course corrects.

bt1a 1y ago
Well, they're usually told that they're some unicorn master of * languages, frameworks, skillsets, etc., so can you really fault them? :)

renewiltord 1y ago

Haha, I encountered the opposite of this when I did a destructive thing recently but first asked Gemini, then countered it saying it’s wrong and it insisted it was right. So the reality they encountered is probably that: it either is stubbornly wrong or overly obsequious with no ability to switch. My friend was a big fan of Gemini 2.5 Pro and I kept telling him it was garbage except for OCR and he nearly followed what it recommended. Haha, he’s never touching it again. Every other LLM changed its tune on pushback.

ramon156 1y ago

You should check Twitter nowadays, people love this kind of response. Some even use it as an argument

darkwater 1y ago
And this is why basically LLMs are "bad". They have already reached critical mass adoption, they are right or mostly right most of the time but they also screw up badly many times as well. And people will just not know, trust blindly and going even deeper down in the spiral of the total absence of critical judgement. And yeah, it also happened with Google and search engines back in the day ("I found it on the web so it must be true") but now with LLMs it is literally tailored to what you are asking, for every possible question you can ask (well, minus the censored ones). I keep thinking the LLM contribution to humanity is/will be a net negative in the long run.
- bt1a 1y ago
  Yet theyre fantastic personal tutors / assistants who can provide a deeply needed 1:1 learning interface for less privileged individuals. I emphasize 'can'. Not saying kids should have them by their side in their current rough around the edges and mediocre intelligent forms. Many will get burned as you describe, but it should be a lesson to curate information from multiple sources and practice applying reasoning skills!
  - darkwater 1y ago
    I agree with your take, and I personally used Claude and ChatGPT to learn better/hone some skills while interviewing to land a new job. And they also help me get unstucked when doing small home fixes, because it's a custom-tailored answer to my current doubt/issue that a normal web search would make much more complicated to answer (I had to know more context about it). But still, they get things wrong and can lead you astray even if you know the topic.
  - AIPedant 1y ago
    My dad taught high school science until retiring this year, and at least in 2024 the LLM tutors were totally useless for honest learning. They were good at the “happy path” but the space of high schoolers’ misconceptions about physics greatly exceeds the training data and can’t be cheaply RLHFed, so they crap the bed when you role play as a dumb high schooler. In my experience this is still true for the reasoning models with undergraduate mathematics - if you ask it to do your point-set topology homework (dishonest learning) it will score > 85/100, if you are confused about point-set topology and try to ask it an honest (but ignorant) question it will give you a pile of pseudo-mathematical BS.

rfoo 1y ago

TBH I'm tired of only the "Ah, an excellent follow-up! You are absolutely right <...> My apologies" part.

IshKebab 1y ago
Yeah they definitely didn't do that in the past. We've lost "as a large language model" and "it's important to remember" but gained "you're absolutely right!" I would have thought they'd add "don't apologise!!!!" or something like that to the system prompt like they do to avoid excessive lists.

deleted 1y ago

[deleted]

codedokode 1y ago

Languages like Rust and C seem to be too complicated for them. I also asked different LLMs to write a C macro or function that creates a struct and a function to print it (so that I don't have to duplicate a field list) and it generates plausible garbage.

bigfishrunning 1y ago
LLMs only ever produce plausible garbage -- sometimes that garbage happens to be right, but that's down to luck.