3 ms·

This is a "don't make me tap the sign" moment. LLMs are next token prediction models. If there are factual errors, confused ideas, etc. in the preceding tokens,

by gwerbin 2mo ago

This is a "don't make me tap the sign" moment. LLMs are next token prediction models. If there are factual errors, confused ideas, etc. in the preceding tokens, that will affect the generation of subsequent tokens, and the error accumulates. Case in point, I hit an error in a SQL query today because it turned out I was trying to do something that wasn't supported by the query engine. I pasted the error message and a bit of background info into my Claude Code session with Sonnet 5 High, it worked on a response for an unexpectedly long amount of time, including consulting the advisor model, and then came back with an explanation of the mistake I made in my query. Except it turned out I pointed it to the wrong file, and there wasn't a mistake in that file. It had completely taken for granted that the pasted error output was a real error and went on some wild goose chase. Part of why the current gen models feel so smart is that they're getting better (via CoT and training) at recognizing when something is wrong and then back up to reassess. So it's easy to forget that it really is just token prediction, and (pending the next big advancement) there's only so much you can do with that.

paulddraper 2mo ago
That’s a bit of a garbage in, garbage out moment.
- deleted 2mo ago
  [deleted]
WelkinFolk 2mo ago
Why is this being downvoted?
ChrisMarshallNY 2mo ago
> Except it turned out I pointed it to the wrong file, and there wasn't a mistake in that file. That’s a fairly common error, on my part. I may have a wrong file, selected in the Xcode Navigator, and the LLM accepts that, as the problematic file. In other cases, I’m just plain wrong, in my diagnosis. Maybe the error manifests in that file, but is caused, in another one. I have taken to giving the LLM as many involved files as possible, when asking for help. I found that, as the LLM progressed through my project, learning more and more about it, it caught these errors, but early on, it ran with my wrong input, and gave bad advice. It still, occasionally, gives bad advice, but that’s just because it was probably trained on bad code. I have found that it gives good advice, more often than not. It’s just incumbent on me, to treat its input as “advice,” and carefully consider it, before integrating it into my own work. I have just upgraded my account. In order to do this, I had to create a new one, for billing purposes. This means that it needs to start over, learning the project. I am in the process of exporting from the previous account, but I’m not sure how well this will transfer the accumulated knowledge.
- DenisM 2mo ago
  > as the LLM progressed through my project, learning more and more about it… This means that it needs to start over, learning the project. I am in the process of exporting from the previous account, but I’m not sure how well this will transfer the accumulated knowledge LLMs don’t learn though. Are you talking about some sort of architecture.md file collection being built? This files should be just easy to copy?
  - ChrisMarshallNY 2mo ago
    Oh, no. The LLM definitely learns my project. It learns my coding and documentation style, and remembers files. Sometimes, that's a pain, as it remembers old versions of files, and disregards newer ones. That's actually a real advantage. Right now, I am getting it to compile a tutorial for the app. Since it's starting from scratch, I can't rely on it remembering much, so I have to "start from the beginning," and give it a lot of information that the other account already had.
    - NateEag 2mo ago
      What tool are you using? I know Claude will maintain its own memory files in ~/.claude, unbidden. The actual LLM under the hood is static, though - it was trained once and will not self-modify.
      - ChrisMarshallNY 2mo ago
        ChatGPT, as a chat interface. I just switched from Plus to Pro. There may be a file, but I haven't seen it, and I know that the export process is non-trivial (it said it would "take several days").
        NateEag 2mo ago
        Ah, got it. IIRC ChatGPT does use "memory" systems to maintain familiarity with a given chat's subject. I believe that's an important ingredient for creating AI psychosis - human and AI become slowly more unhinged together in a feedback loop until it finally spirals out of control.
iainmerrick 2mo ago
I don't see how "it's just token prediction" is relevant to that at all. Sure, the model can go wrong, but sometimes it's able to realise that and correct its course. Stronger models are better at doing this. People do exactly the same thing! Haven't you ever wasted a lot of time chasing down a blind alley? To say the LLM has immutable limits because it only predicts the next token and can't backtrack is like saying we have immutable limits because we can't travel backwards in time. It's a true statement but not particularly relevant or helpful.
- kasey_junk 2mo ago
  I have found it extremely helpful to keep the “token prediction machine” metaphor front and center in my work with llms, not because it expresses some fundamental limit of the technology but because it allows me to have a mental model about where and how to use the technology in my process. Contrast this to the “chat” mental model. If you are chatting with someone and they lie to you, that is a transgression. A bad prediction is just a normal outcome. These are _human_ judgements but you address transgressions differently than normal but incorrect outcomes. That’s a helpful distinction for me when building my tooling. If anything, that we are getting so much traction out of text prediction is the most magical part of llms. But it doesn’t change that you shouldn’t treat what they output the same as what a human says.
  - iainmerrick 2mo ago
    Hmm, what specific differences does it make in practice? For me, the huge one is that LLMs are currently bad at learning from experience. I don’t trust any kind of automatic MEMORY.md or whatnot; in fact I greatly prefer starting from a clean slate each time because the LLM’s baseline general knowledge is so good. In terms of accuracy and “lying”, I don’t really see a huge difference. Most LLMs are unfortunately a bit sycophantic and over-confident, but you sometimes see that in people as well.
    - kasey_junk 2mo ago
      The biggest difference is I view any interaction with an llm that doesn’t produce an artifact that can be verified with skepticism. That is I very rarely even use a “chat” interface anymore. Under the covers I’m still using the mainline agents but I’m treating them more like black box Unix tools that take text in and spit text out. My interactions are evolving to have very small llm cores wrapped with old fashioned deterministic programming. This makes the boundary/verification step more deterministic and less susceptible to bad predictions. But it also has the side effect that I rarely interact with llm output that has “tone” (for instance sycophantic output). That usually only happens when I’m debugging what went wrong, particularly around model upgrade cycles.
__alexs 2mo ago
This line of thinking is quite confusing to me. If you keep following it then ultimately we must reckon with the reality that we only experience time in one direction. While it might be technically true that "there is only so much you can do with that" I think it might also be a completely useless statement to make.
- fizzbuzzbarbazz 2mo ago
  I think it provides a necessary pointer to the idea that we will need more than prediction to make a viable general AI - that the prediction model is useful, but has fundamental limitations.

by gwerbin 2mo ago

paulddraper 2mo ago

That’s a bit of a garbage in, garbage out moment.

deleted 2mo ago
[deleted]

WelkinFolk 2mo ago

Why is this being downvoted?

ChrisMarshallNY 2mo ago

> Except it turned out I pointed it to the wrong file, and there wasn't a mistake in that file. That’s a fairly common error, on my part. I may have a wrong file, selected in the Xcode Navigator, and the LLM accepts that, as the problematic file. In other cases, I’m just plain wrong, in my diagnosis. Maybe the error manifests in that file, but is caused, in another one. I have taken to giving the LLM as many involved files as possible, when asking for help. I found that, as the LLM progressed through my project, learning more and more about it, it caught these errors, but early on, it ran with my wrong input, and gave bad advice. It still, occasionally, gives bad advice, but that’s just because it was probably trained on bad code. I have found that it gives good advice, more often than not. It’s just incumbent on me, to treat its input as “advice,” and carefully consider it, before integrating it into my own work. I have just upgraded my account. In order to do this, I had to create a new one, for billing purposes. This means that it needs to start over, learning the project. I am in the process of exporting from the previous account, but I’m not sure how well this will transfer the accumulated knowledge.

DenisM 2mo ago
> as the LLM progressed through my project, learning more and more about it… This means that it needs to start over, learning the project. I am in the process of exporting from the previous account, but I’m not sure how well this will transfer the accumulated knowledge LLMs don’t learn though. Are you talking about some sort of architecture.md file collection being built? This files should be just easy to copy?
- ChrisMarshallNY 2mo ago
  Oh, no. The LLM definitely learns my project. It learns my coding and documentation style, and remembers files. Sometimes, that's a pain, as it remembers old versions of files, and disregards newer ones. That's actually a real advantage. Right now, I am getting it to compile a tutorial for the app. Since it's starting from scratch, I can't rely on it remembering much, so I have to "start from the beginning," and give it a lot of information that the other account already had.
  - NateEag 2mo ago
    What tool are you using? I know Claude will maintain its own memory files in ~/.claude, unbidden. The actual LLM under the hood is static, though - it was trained once and will not self-modify.
    - ChrisMarshallNY 2mo ago
      ChatGPT, as a chat interface. I just switched from Plus to Pro. There may be a file, but I haven't seen it, and I know that the export process is non-trivial (it said it would "take several days").
      - NateEag 2mo ago
        Ah, got it. IIRC ChatGPT does use "memory" systems to maintain familiarity with a given chat's subject. I believe that's an important ingredient for creating AI psychosis - human and AI become slowly more unhinged together in a feedback loop until it finally spirals out of control.

iainmerrick 2mo ago

I don't see how "it's just token prediction" is relevant to that at all. Sure, the model can go wrong, but sometimes it's able to realise that and correct its course. Stronger models are better at doing this. People do exactly the same thing! Haven't you ever wasted a lot of time chasing down a blind alley? To say the LLM has immutable limits because it only predicts the next token and can't backtrack is like saying we have immutable limits because we can't travel backwards in time. It's a true statement but not particularly relevant or helpful.

kasey_junk 2mo ago
I have found it extremely helpful to keep the “token prediction machine” metaphor front and center in my work with llms, not because it expresses some fundamental limit of the technology but because it allows me to have a mental model about where and how to use the technology in my process. Contrast this to the “chat” mental model. If you are chatting with someone and they lie to you, that is a transgression. A bad prediction is just a normal outcome. These are _human_ judgements but you address transgressions differently than normal but incorrect outcomes. That’s a helpful distinction for me when building my tooling. If anything, that we are getting so much traction out of text prediction is the most magical part of llms. But it doesn’t change that you shouldn’t treat what they output the same as what a human says.
- iainmerrick 2mo ago
  Hmm, what specific differences does it make in practice? For me, the huge one is that LLMs are currently bad at learning from experience. I don’t trust any kind of automatic MEMORY.md or whatnot; in fact I greatly prefer starting from a clean slate each time because the LLM’s baseline general knowledge is so good. In terms of accuracy and “lying”, I don’t really see a huge difference. Most LLMs are unfortunately a bit sycophantic and over-confident, but you sometimes see that in people as well.
  - kasey_junk 2mo ago
    The biggest difference is I view any interaction with an llm that doesn’t produce an artifact that can be verified with skepticism. That is I very rarely even use a “chat” interface anymore. Under the covers I’m still using the mainline agents but I’m treating them more like black box Unix tools that take text in and spit text out. My interactions are evolving to have very small llm cores wrapped with old fashioned deterministic programming. This makes the boundary/verification step more deterministic and less susceptible to bad predictions. But it also has the side effect that I rarely interact with llm output that has “tone” (for instance sycophantic output). That usually only happens when I’m debugging what went wrong, particularly around model upgrade cycles.

__alexs 2mo ago

This line of thinking is quite confusing to me. If you keep following it then ultimately we must reckon with the reality that we only experience time in one direction. While it might be technically true that "there is only so much you can do with that" I think it might also be a completely useless statement to make.

fizzbuzzbarbazz 2mo ago
I think it provides a necessary pointer to the idea that we will need more than prediction to make a viable general AI - that the prediction model is useful, but has fundamental limitations.