5 ms·

Fascinating article. I daily catch LLMs in “lies” like: “I found the root cause of the bug” or “this approach is twice as fast”. It’s hard to say what causes th

by timfsu 2mo ago

Fascinating article. I daily catch LLMs in “lies” like: “I found the root cause of the bug” or “this approach is twice as fast”. It’s hard to say what causes this uninformed certainty - is it intrinsic to being trained on human writing, or something that comes from the RLHF process afterwards, but it’s extremely annoying. It’s one thing to have a LLM make poor decisions, but it feels worse to have it “lie” to you in the process.

cadamsdotcom 2mo ago
Ungrounded LLM outputs are a bit like your dreams. Without anything to test hypotheses against, stuff can pop in and out of existence and physics is just advice. Ground your LLM. Tests, documentation, give it many ways to run the thing its reasoning about. It needs to be able to test its hypotheses on its own. Take yourself out of that loop so you only find out once it's sure.
- tra3 2mo ago
  Love LLMs gonna keep using them. It feels like your suggested approach is expensive, in terms of tokens. I feel (second time I say this) that when I steer the process I get pretty good results vs my coworkers that let the LLMs run away. I do have data on our token usage, not much in terms of quality of the delivery. I keep thinking about the c compiler implementation that anthropic shared earlier in the year that had all the requirements you mention and arguably wasn’t that great.
  - vkazanov 2mo ago
    Thr thing is that both you and your agent should have a way to verify the solution. OBVIOUSLY, the compiler experiment was just a cringe pr stunt. But it has a point: everything works better with a good testing loop, and compilers always have one by thr nature of the work they do
  - cadamsdotcom 2mo ago
    > expensive, in terms of tokens. No amount of tokens can come close to my hourly rate.
  - mrtesthah 2mo ago
    Do you steer your agents by manually running every single test and linter and reporting the results back to them?
- amelius 2mo ago
  But this is exactly what the AI labs should be doing ...
  - moffkalast 2mo ago
    And they are, at least for Claude I know it writes random mocks and tests in its virtual env even in the web version, cause it sometimes annoyingly includes them in the final result. It's the only reason it produces anything that runs.
- dgellow 2mo ago
  That’s exactly why I don’t believe LLMs will cure cancer anytime soon, make terrible lawyers, shouldn’t be trusted for medical decisions, etc. software and maths are some really the niches where we have great, battle tested, reliable validation tools. That’s not the case for “softer” domains
  - cadamsdotcom 2mo ago
    Software is a very "spiky domain"; things either work or fail, and there is sharp delineation between and easy verification. Hm. Two orthogonal properties! This sounds like a 2x2 matrix! Let's swap hard/easy around & explore the 4 possibilities... There are domains with sharp delineation and hard verification; they are not at risk until AI gets much better. Humans operate in these domains by applying tremendous deep thought and subjective judgment - our superpower. Domains with soft delineation and easy verification are most at risk: "it's a picture of a cat" remains true through a wide range of perturbations - eg. skewing the image or moving it across a pixel or correcting its white balance or even changing the cat. AI music? Lots of domains already solved by AI here but they're also not that meaty. My prediction is the next interesting stuff will happen where verification is hard but there's no sharp delineation. It's the world of "I'll know it when I see it". Good customer service?
    - dgellow 2mo ago
      Eh, that’s a very interesting way to differentiate, I will steal your explanation next time I have that discussion, if you don’t mind!
  - gwerbin 2mo ago
    The only people who think LLMs would make good lawyers are the people selling LLMs. The more practical among us recognize that LLMs are our amazing tools for searching through and making sense of large amounts of text with a high level of sophistication, which can significantly enhance the productivity of a human lawyer.
gman83 2mo ago
I've gone in loops where it'll tell me twenty times in a row that it's definitely found the root cause of a bug, when in fact it definitely did not.
Yokohiii 2mo ago
LLM aren't logic machines. It's concepts of "root cause" and "faster" are purely linguistic, including all the internal conflicts of those loose definitions. In that sense an LLM wouldn't even think it "lied" being confidently incorrect, in human terms it's just a "whoopsie". Apparently we want exactly that.
gwerbin 2mo ago
This is a "don't make me tap the sign" moment. LLMs are next token prediction models. If there are factual errors, confused ideas, etc. in the preceding tokens, that will affect the generation of subsequent tokens, and the error accumulates. Case in point, I hit an error in a SQL query today because it turned out I was trying to do something that wasn't supported by the query engine. I pasted the error message and a bit of background info into my Claude Code session with Sonnet 5 High, it worked on a response for an unexpectedly long amount of time, including consulting the advisor model, and then came back with an explanation of the mistake I made in my query. Except it turned out I pointed it to the wrong file, and there wasn't a mistake in that file. It had completely taken for granted that the pasted error output was a real error and went on some wild goose chase. Part of why the current gen models feel so smart is that they're getting better (via CoT and training) at recognizing when something is wrong and then back up to reassess. So it's easy to forget that it really is just token prediction, and (pending the next big advancement) there's only so much you can do with that.
- paulddraper 2mo ago
  That’s a bit of a garbage in, garbage out moment.
  - deleted 2mo ago
    [deleted]
- WelkinFolk 2mo ago
  Why is this being downvoted?
- ChrisMarshallNY 2mo ago
  > Except it turned out I pointed it to the wrong file, and there wasn't a mistake in that file. That’s a fairly common error, on my part. I may have a wrong file, selected in the Xcode Navigator, and the LLM accepts that, as the problematic file. In other cases, I’m just plain wrong, in my diagnosis. Maybe the error manifests in that file, but is caused, in another one. I have taken to giving the LLM as many involved files as possible, when asking for help. I found that, as the LLM progressed through my project, learning more and more about it, it caught these errors, but early on, it ran with my wrong input, and gave bad advice. It still, occasionally, gives bad advice, but that’s just because it was probably trained on bad code. I have found that it gives good advice, more often than not. It’s just incumbent on me, to treat its input as “advice,” and carefully consider it, before integrating it into my own work. I have just upgraded my account. In order to do this, I had to create a new one, for billing purposes. This means that it needs to start over, learning the project. I am in the process of exporting from the previous account, but I’m not sure how well this will transfer the accumulated knowledge.
friendzis 2mo ago
> It’s hard to say what causes this uninformed certainty - is it intrinsic to being trained on human writing, or something that comes from the RLHF process afterwards LLMs are trained to produce some form of output that is NOT nothing/dunno. The so called hyperspace collapses onto something, instead of collapsing onto lack of answer tokens.
- gwerbin 2mo ago
  If you watch the thinking traces of just about any modern LLM, you might be surprised at how much "uncertainty" is in there. Weak models with no thinking limits vacillate back-and-forth back-and-forth on a topic for potentially thousands of tokens before gradually spiraling towards some kind of an answer. Which makes it all the more interesting that "I don't know" is so rarely the final prediction, even with so much waffling in the chain of thought. Until the big labs decide to start adding synthetic "I don't know" outcomes to their data sets, I've been thinking that the best way to evaluate uncertainty is to have a separate LLM monitoring the conversation and asking it to classify if the agent is overstating its confidence. On the other hand I've also noticed that most models will tell you they don't know something if you specifically include it in the prompt, eg "if you don't know the answer, just say so" and/or "be clear about any gaps in your knowledge that would reduce the confidence of your response" etc. but even with the big frontier models I have noticed some quality degradation if I throw too many instructions into the system prompt. I have a little more faith in harness-level engineering than in praying to the token generation gods. That said, there is a completely different form of "uncertainty" in which the LLM tends to place very high trust in its own prior outputs as well as user provided inputs. Again if you look at the thinking traces, these models will try very very hard to rationalize the inputs they are given, falling back to the possibility of user error only after working through several alternative possibilities, maybe even investigating data or source code in the process. And if your context is big enough, the model might just completely miss when pieces of information conflict.
thatjoeoverthr 2mo ago
A lot of weirdness in their style is the fact they think in-band. Notice it "calls its attacks" when writing, like Goku shouting "solar flare" and doing a solar flare. To write an honest answer, it says, "Here's the honest answer", and proceeds from there. If it plans to write a "kicker", first it says, "And here's the kicker", and then it writes the "kicker". This gives an affect distinctly different from any professional writer because editors are specifically trained to _cut out_ explicated subtext. You would expect "thinking" models and the agent harness (which conceals "thinking sections") would mitigate this but it's a deep seated behaviour.
ee334y5rthsrth 2mo ago
decrease temperature options to 0.0

by timfsu 2mo ago

cadamsdotcom 2mo ago

Ungrounded LLM outputs are a bit like your dreams. Without anything to test hypotheses against, stuff can pop in and out of existence and physics is just advice. Ground your LLM. Tests, documentation, give it many ways to run the thing its reasoning about. It needs to be able to test its hypotheses on its own. Take yourself out of that loop so you only find out once it's sure.

tra3 2mo ago
Love LLMs gonna keep using them. It feels like your suggested approach is expensive, in terms of tokens. I feel (second time I say this) that when I steer the process I get pretty good results vs my coworkers that let the LLMs run away. I do have data on our token usage, not much in terms of quality of the delivery. I keep thinking about the c compiler implementation that anthropic shared earlier in the year that had all the requirements you mention and arguably wasn’t that great.
- vkazanov 2mo ago
  Thr thing is that both you and your agent should have a way to verify the solution. OBVIOUSLY, the compiler experiment was just a cringe pr stunt. But it has a point: everything works better with a good testing loop, and compilers always have one by thr nature of the work they do
- cadamsdotcom 2mo ago
  > expensive, in terms of tokens. No amount of tokens can come close to my hourly rate.
- mrtesthah 2mo ago
  Do you steer your agents by manually running every single test and linter and reporting the results back to them?
amelius 2mo ago
But this is exactly what the AI labs should be doing ...
- moffkalast 2mo ago
  And they are, at least for Claude I know it writes random mocks and tests in its virtual env even in the web version, cause it sometimes annoyingly includes them in the final result. It's the only reason it produces anything that runs.
dgellow 2mo ago
That’s exactly why I don’t believe LLMs will cure cancer anytime soon, make terrible lawyers, shouldn’t be trusted for medical decisions, etc. software and maths are some really the niches where we have great, battle tested, reliable validation tools. That’s not the case for “softer” domains
- cadamsdotcom 2mo ago
  Software is a very "spiky domain"; things either work or fail, and there is sharp delineation between and easy verification. Hm. Two orthogonal properties! This sounds like a 2x2 matrix! Let's swap hard/easy around & explore the 4 possibilities... There are domains with sharp delineation and hard verification; they are not at risk until AI gets much better. Humans operate in these domains by applying tremendous deep thought and subjective judgment - our superpower. Domains with soft delineation and easy verification are most at risk: "it's a picture of a cat" remains true through a wide range of perturbations - eg. skewing the image or moving it across a pixel or correcting its white balance or even changing the cat. AI music? Lots of domains already solved by AI here but they're also not that meaty. My prediction is the next interesting stuff will happen where verification is hard but there's no sharp delineation. It's the world of "I'll know it when I see it". Good customer service?
  - dgellow 2mo ago
    Eh, that’s a very interesting way to differentiate, I will steal your explanation next time I have that discussion, if you don’t mind!
- gwerbin 2mo ago
  The only people who think LLMs would make good lawyers are the people selling LLMs. The more practical among us recognize that LLMs are our amazing tools for searching through and making sense of large amounts of text with a high level of sophistication, which can significantly enhance the productivity of a human lawyer.

gman83 2mo ago

I've gone in loops where it'll tell me twenty times in a row that it's definitely found the root cause of a bug, when in fact it definitely did not.

Yokohiii 2mo ago

LLM aren't logic machines. It's concepts of "root cause" and "faster" are purely linguistic, including all the internal conflicts of those loose definitions. In that sense an LLM wouldn't even think it "lied" being confidently incorrect, in human terms it's just a "whoopsie". Apparently we want exactly that.

gwerbin 2mo ago

This is a "don't make me tap the sign" moment. LLMs are next token prediction models. If there are factual errors, confused ideas, etc. in the preceding tokens, that will affect the generation of subsequent tokens, and the error accumulates. Case in point, I hit an error in a SQL query today because it turned out I was trying to do something that wasn't supported by the query engine. I pasted the error message and a bit of background info into my Claude Code session with Sonnet 5 High, it worked on a response for an unexpectedly long amount of time, including consulting the advisor model, and then came back with an explanation of the mistake I made in my query. Except it turned out I pointed it to the wrong file, and there wasn't a mistake in that file. It had completely taken for granted that the pasted error output was a real error and went on some wild goose chase. Part of why the current gen models feel so smart is that they're getting better (via CoT and training) at recognizing when something is wrong and then back up to reassess. So it's easy to forget that it really is just token prediction, and (pending the next big advancement) there's only so much you can do with that.

paulddraper 2mo ago
That’s a bit of a garbage in, garbage out moment.
- deleted 2mo ago
  [deleted]
WelkinFolk 2mo ago
Why is this being downvoted?
ChrisMarshallNY 2mo ago
> Except it turned out I pointed it to the wrong file, and there wasn't a mistake in that file. That’s a fairly common error, on my part. I may have a wrong file, selected in the Xcode Navigator, and the LLM accepts that, as the problematic file. In other cases, I’m just plain wrong, in my diagnosis. Maybe the error manifests in that file, but is caused, in another one. I have taken to giving the LLM as many involved files as possible, when asking for help. I found that, as the LLM progressed through my project, learning more and more about it, it caught these errors, but early on, it ran with my wrong input, and gave bad advice. It still, occasionally, gives bad advice, but that’s just because it was probably trained on bad code. I have found that it gives good advice, more often than not. It’s just incumbent on me, to treat its input as “advice,” and carefully consider it, before integrating it into my own work. I have just upgraded my account. In order to do this, I had to create a new one, for billing purposes. This means that it needs to start over, learning the project. I am in the process of exporting from the previous account, but I’m not sure how well this will transfer the accumulated knowledge.

friendzis 2mo ago

> It’s hard to say what causes this uninformed certainty - is it intrinsic to being trained on human writing, or something that comes from the RLHF process afterwards LLMs are trained to produce some form of output that is NOT nothing/dunno. The so called hyperspace collapses onto something, instead of collapsing onto lack of answer tokens.

gwerbin 2mo ago
If you watch the thinking traces of just about any modern LLM, you might be surprised at how much "uncertainty" is in there. Weak models with no thinking limits vacillate back-and-forth back-and-forth on a topic for potentially thousands of tokens before gradually spiraling towards some kind of an answer. Which makes it all the more interesting that "I don't know" is so rarely the final prediction, even with so much waffling in the chain of thought. Until the big labs decide to start adding synthetic "I don't know" outcomes to their data sets, I've been thinking that the best way to evaluate uncertainty is to have a separate LLM monitoring the conversation and asking it to classify if the agent is overstating its confidence. On the other hand I've also noticed that most models will tell you they don't know something if you specifically include it in the prompt, eg "if you don't know the answer, just say so" and/or "be clear about any gaps in your knowledge that would reduce the confidence of your response" etc. but even with the big frontier models I have noticed some quality degradation if I throw too many instructions into the system prompt. I have a little more faith in harness-level engineering than in praying to the token generation gods. That said, there is a completely different form of "uncertainty" in which the LLM tends to place very high trust in its own prior outputs as well as user provided inputs. Again if you look at the thinking traces, these models will try very very hard to rationalize the inputs they are given, falling back to the possibility of user error only after working through several alternative possibilities, maybe even investigating data or source code in the process. And if your context is big enough, the model might just completely miss when pieces of information conflict.

thatjoeoverthr 2mo ago

A lot of weirdness in their style is the fact they think in-band. Notice it "calls its attacks" when writing, like Goku shouting "solar flare" and doing a solar flare. To write an honest answer, it says, "Here's the honest answer", and proceeds from there. If it plans to write a "kicker", first it says, "And here's the kicker", and then it writes the "kicker". This gives an affect distinctly different from any professional writer because editors are specifically trained to _cut out_ explicated subtext. You would expect "thinking" models and the agent harness (which conceals "thinking sections") would mitigate this but it's a deep seated behaviour.

ee334y5rthsrth 2mo ago

decrease temperature options to 0.0