4 ms·

> LLMs … approximate correctness for varying amounts of time. Once that time runs out there is a sharp drop off in model accuracy, it simply cannot continue to

> LLMs … approximate correctness for varying amounts of time. Once that time runs out there is a sharp drop off in model accuracy, it simply cannot continue to offer you an output that even approximates something workable. I have taken to calling this phenomenon the "AI Cliff," as it is very sharp and very sudden I’ve never heard of this cliff before. Has anyone else experienced this?

sandspar 1y ago
I'm not sure. Is he talking about context poisoning?
Kuinox 1y ago
I'm doing my own procedurally generated benchmark. I can make the problem input bigger as I want. Each LLM have a different thresholf for each problem, when crossed the performance of the LLM collapse.
Paradigma11 1y ago
If the context gets to big or otherwise poisoned you have to restart the chat/agent. A bit like windows of old. This trains you to document the current state of your work so the new agent can get up to speed.
bubblyworld 1y ago
I've only experienced this while vibe coding through chat interfaces, i.e. in the complete absence of feedback loops. This is much less of a problem with agentic tools like claude code/codex/gemini cli, where they manage their own context windows and can run your dev tooling to sanity check themselves as they go.
Syzygies 1y ago
One can find opinions that Claude Code Opus 4 is worth the monthly $200 I pay for Anthropic's Max plan. Opus 4 is smarter; one either can't afford to use it, or can't afford not to use it. I'm in the latter group. One feature others have noted is that the Opus 4 context buffer rarely "wears out" in a work session. It can, and one needs to recognize this and start over. With other agents, it was my routine experience that I'd be lucky to get an hour before having to restart my agent. A reliable way to induce this "cliff" is to let AI take on a much too hard problem in one step, then flail helplessly trying to fix their mess. Vibe-coding an unsuitable problem. One can even kill Opus 4 this way, but that's no way to run a race horse. Some "persistence of memory" harness is as important as one's testing harness, for effective AI coding. With the right care having AI edit its own context prompts for orienting new sessions, this all matters less. AI is spectacularly bad at breaking problems into small steps without our guidance, and small steps done right can be different sessions. I'll regularly start new sessions when I have a hunch that this will get me better focus for the next step. So the cliff isn't so important. But Opus 4 is smarter in other ways.
- suddenlybananas 1y ago
  >can't afford not to use it. I'm in the latter group. People love to justify big expenses as necessary.
  - Syzygies 1y ago
    $200 is a small expense and you don't know why I need AI. The online dialog about AI is mostly noise, and even at HN it is badly distorted by people who wince at $20 a month, and complain AI isn't that smart.
- fwip 1y ago
  Sometimes after it flails for a while, but I think it's on the right path, I'll rewind the context to just before it started trying to solve the problem (but keep the code changes). And I'll tell it "I got this other guy to attempt what we just talked about, but it still has some problems." Snipping out the flailing in this way seems to help.
gwd 1y ago
I experience it pretty regularly -- once the complexity of the code passes a certain threshold, the LLM can't keep everything in its head and starts thrashing around. Part of my job working with the LLM is to manage the complexity it sees. And one of the things with current generators is that they tend to make things more complex over time, rather than less. It's always me prompting the LLM to refactor things to make it simpler, or doing the refactoring once it's gotten to complex for the LLM to deal with. So at least with the current generation of LLMs, it seems rather inevitable that if you just "give LLMs their head" and let them do what they want, eventually they'll create a giant Rube Goldberg mess that you'll have to try to clean up. ETA: And to the point of the article -- if you're an old salt, you'll be able to recognize when the LLM is taking you out to sea early, and be able to navigate your way back into shallower waters even if you go out a bit too far. If you're a new hand, you'll be out of your depth and lost at sea before you know it's happened.
windward 1y ago
I've seen it referred to as 'context drunk'. Imagine that you have your input to the context, 10000 tokens that are 99% correct. Each time the LLM replies it adds 1000 tokens that are 90% correct. After some back-and-forth of you correcting the LLM, its context window is mostly its own backwash^Woutput. Worse, the error compounds because the 90% that is correct is just correct extrapolation of an argument about incorrect code, and because the LLM ranks more recent tokens as more important. The same problem also shows up in prose.
Workaccount2 1y ago
I call it context rot. As the context fills up the quality of output erodes with it. The rot gets even worse or progresses faster the more spurious or tangential discussion is in context. This is also can be made much worse by thinking models, as their CoT is all in context, and if there thoughts really wander it just plants seeds of poison feeding the rot. I really wish they can implement some form of context pruning, so you can nip irrelevant context when it forms. In the meantime, I make summaries and carry it to a fresh instance when I notice the rot forming.
lubujackson 1y ago
I definitely hit this vibe coding a large-ish backend. Well defined data structures, good modularity, etc. But at a point, Cursor started to lose the plot and rewrite or duplicate functions, recreate or misue data structures, etc. The solve was to define several Cursor rules files for different views of the codebase - here's the structure, here's the validation logic, etc. That and using o3 has at least gotten me to the next level.
impure 1y ago
This sounds a lot like accuracy collapse as discussed in that Apple paper. That paper clearly showed that there is some point where AI accuracy collapses extremely quickly. I suspect it has something more to do with the model producing too many tokens and becoming fixated on what it said before. You'll often see this in long conversations. The only way to fix it is to start a new conversation.
npteljes 1y ago
I reset "work" AI sessions quite frequently, so I didn't see that there. I experienced it though with storytelling. In my storytelling scenario, context and length was important. And the AI at one late point forgot how my characters should behave in the developing situation, and just had them react to it in a very different way. And there was no going back from that. Very weird experience.

sandspar 1y ago

I'm not sure. Is he talking about context poisoning?

Kuinox 1y ago

I'm doing my own procedurally generated benchmark. I can make the problem input bigger as I want. Each LLM have a different thresholf for each problem, when crossed the performance of the LLM collapse.

Paradigma11 1y ago

If the context gets to big or otherwise poisoned you have to restart the chat/agent. A bit like windows of old. This trains you to document the current state of your work so the new agent can get up to speed.

bubblyworld 1y ago

I've only experienced this while vibe coding through chat interfaces, i.e. in the complete absence of feedback loops. This is much less of a problem with agentic tools like claude code/codex/gemini cli, where they manage their own context windows and can run your dev tooling to sanity check themselves as they go.

Syzygies 1y ago

One can find opinions that Claude Code Opus 4 is worth the monthly $200 I pay for Anthropic's Max plan. Opus 4 is smarter; one either can't afford to use it, or can't afford not to use it. I'm in the latter group. One feature others have noted is that the Opus 4 context buffer rarely "wears out" in a work session. It can, and one needs to recognize this and start over. With other agents, it was my routine experience that I'd be lucky to get an hour before having to restart my agent. A reliable way to induce this "cliff" is to let AI take on a much too hard problem in one step, then flail helplessly trying to fix their mess. Vibe-coding an unsuitable problem. One can even kill Opus 4 this way, but that's no way to run a race horse. Some "persistence of memory" harness is as important as one's testing harness, for effective AI coding. With the right care having AI edit its own context prompts for orienting new sessions, this all matters less. AI is spectacularly bad at breaking problems into small steps without our guidance, and small steps done right can be different sessions. I'll regularly start new sessions when I have a hunch that this will get me better focus for the next step. So the cliff isn't so important. But Opus 4 is smarter in other ways.

suddenlybananas 1y ago
>can't afford not to use it. I'm in the latter group. People love to justify big expenses as necessary.
- Syzygies 1y ago
  $200 is a small expense and you don't know why I need AI. The online dialog about AI is mostly noise, and even at HN it is badly distorted by people who wince at $20 a month, and complain AI isn't that smart.
fwip 1y ago
Sometimes after it flails for a while, but I think it's on the right path, I'll rewind the context to just before it started trying to solve the problem (but keep the code changes). And I'll tell it "I got this other guy to attempt what we just talked about, but it still has some problems." Snipping out the flailing in this way seems to help.

gwd 1y ago

I experience it pretty regularly -- once the complexity of the code passes a certain threshold, the LLM can't keep everything in its head and starts thrashing around. Part of my job working with the LLM is to manage the complexity it sees. And one of the things with current generators is that they tend to make things more complex over time, rather than less. It's always me prompting the LLM to refactor things to make it simpler, or doing the refactoring once it's gotten to complex for the LLM to deal with. So at least with the current generation of LLMs, it seems rather inevitable that if you just "give LLMs their head" and let them do what they want, eventually they'll create a giant Rube Goldberg mess that you'll have to try to clean up. ETA: And to the point of the article -- if you're an old salt, you'll be able to recognize when the LLM is taking you out to sea early, and be able to navigate your way back into shallower waters even if you go out a bit too far. If you're a new hand, you'll be out of your depth and lost at sea before you know it's happened.

windward 1y ago

I've seen it referred to as 'context drunk'. Imagine that you have your input to the context, 10000 tokens that are 99% correct. Each time the LLM replies it adds 1000 tokens that are 90% correct. After some back-and-forth of you correcting the LLM, its context window is mostly its own backwash^Woutput. Worse, the error compounds because the 90% that is correct is just correct extrapolation of an argument about incorrect code, and because the LLM ranks more recent tokens as more important. The same problem also shows up in prose.

Workaccount2 1y ago

I call it context rot. As the context fills up the quality of output erodes with it. The rot gets even worse or progresses faster the more spurious or tangential discussion is in context. This is also can be made much worse by thinking models, as their CoT is all in context, and if there thoughts really wander it just plants seeds of poison feeding the rot. I really wish they can implement some form of context pruning, so you can nip irrelevant context when it forms. In the meantime, I make summaries and carry it to a fresh instance when I notice the rot forming.

lubujackson 1y ago

I definitely hit this vibe coding a large-ish backend. Well defined data structures, good modularity, etc. But at a point, Cursor started to lose the plot and rewrite or duplicate functions, recreate or misue data structures, etc. The solve was to define several Cursor rules files for different views of the codebase - here's the structure, here's the validation logic, etc. That and using o3 has at least gotten me to the next level.

impure 1y ago

This sounds a lot like accuracy collapse as discussed in that Apple paper. That paper clearly showed that there is some point where AI accuracy collapses extremely quickly. I suspect it has something more to do with the model producing too many tokens and becoming fixated on what it said before. You'll often see this in long conversations. The only way to fix it is to start a new conversation.

npteljes 1y ago

I reset "work" AI sessions quite frequently, so I didn't see that there. I experienced it though with storytelling. In my storytelling scenario, context and length was important. And the AI at one late point forgot how my characters should behave in the developing situation, and just had them react to it in a very different way. And there was no going back from that. Very weird experience.