10 ms·

LLMs get lost in multi-turn conversation

374 points by simonpure 1y ago

Benjammer 1y ago
It's nice to see a paper that confirms what anyone who has practiced using LLM tools already knows very well, heuristically. Keeping your context clean matters, "conversations" are only a construct of product interfaces, they hurt the quality of responses from the LLM itself, and once your context is "poisoned" it will not recover, you need to start fresh with a new chat.
- MattGaiser 1y ago
  Yep. I regretted leaving on memory as it is poisoned my conversations with irrelevant junk.
  - neom 1y ago
    You can go in and delete memory items
- morsecodist 1y ago
  This matches my experience exactly. "poisoned" is a great way to put it. I find once something has gone wrong all subsequent responses are bad. This is why I am iffy on ChatGPT's memory features. I don't notice it causing any huge problems but I don't love how it pollutes my context in ways I don't fully understand.
  - AstroBen 1y ago
    good point on the memory feature. Wow that sounds terrible
    - distances 1y ago
      The memory is easy to turn off. It sounded like a very bad idea to cross-contaminate chats so I disabled it as soon as ChatGPT introduced it.
  - somenameforme 1y ago
    It's interesting how much the nature of LLMs fundamentally being self recursive next token predictors aligns with the Chinese Room experiment. [1] In such experiment it also makes perfect sense that a single wrong response would cascade into a series of subsequent ever more drifting errors. I think it all emphasizes the relevance of the otherwise unqualifiable concept of 'understanding.' In many ways this issue could make the Chinese Room thought experiment even more compelling. Because it's a very practical and inescapable issue. [1] - https://en.wikipedia.org/wiki/Chinese_room https://en.wikipedia.org/wiki/Chinese_room
    - keiferski 1y ago
      Great comment on the Chinese room. That idea seems to be dismissed nowadays but the concept of “cascading failure to understand context” is absolutely relevant to LLMs. I often find myself needing to explain basic details over and over again to an LLM; when with a person it would be a five second, “no, I mean like this way, not that way” explanation.
    - jampekka 1y ago
      I don't think the Chinese room thought experiment is about this, or performance of LLMs in general. Searle explicitly argues that a program can't induce "understanding" even if it mimicked human understanding perfectly because programs don't have "causal powers" to generate "mental states". This is mentioned in the Wikipedia page too: "Although its proponents originally presented the argument in reaction to statements of artificial intelligence (AI) researchers, it is not an argument against the goals of mainstream AI research because it does not show a limit in the amount of intelligent behavior a machine can display."
  - OtherShrezzing 1y ago
    I find using tools like LMStudio, which lets you edit your chat history on the fly, really helps deal with this problem. The models you can host locally are much weaker, but they perform a little better than the really big models once you need to factor in these poisoning problems. A nice middle-ground I'm finding is to ask Claude an initial conversation starter in its "thinking" mode, and then copy/paste that conversation into LMStudio and have a weaker model like Gemma pick-up from where Claude left off.
  - shrewduser 1y ago
    I have very limited experience with llms but i've always thought of it as a compounding errors problem, once you get a small error early on it can compound and go completely off track later.
- Helmut10001 1y ago
  My experiences somewhat confirm these observations, but I also had one that was different. Two weeks of debugging IPSEC issues with Gemini. Initially, I imported all the IPSEC documentation from OPNsense and pfSense into Gemini and informed it of the general context in which I was operating (in reference to 'keeping your context clean'). Then I added my initial settings for both sides (sensitive information redacted!). Afterwards, I entered a long feedback loop, posting logs and asking and answering questions. At the end of the two weeks, I observed that: The LLM was much less likely to become distracted. Sometimes, I would dump whole forum threads or SO posts into it, when it said "this is not what we are seeing here, because of [earlier context or finding]. I eliminated all dead ends logically and informed it of this (yes, it can help with the reflection, but I had to make the decisions). In the end, I found the cause of my issues. This somewhat confirms what some user here on HN said a few days ago. LLMs are good at compressing complex information into simple one, but not at expanding simple ideas into complex ones. As long as my input was larger than the output (either complexity or length), I was happy with the results. I could have done this without the LLM. However, it was helpful in that it stored facts from the outset that I had either forgotten or been unable to retrieve quickly in new contexts. It also made it easier to identify time patterns in large log files, which helped me debug my site-to-site connection. I also optimized many other settings along the way, resolving not only the most problematic issue. This meant, in addition to fixing my problem, I learned quite a bit. The 'state' was only occasionally incorrect about my current parameter settings, but this was always easy to correct. This confirms what others already saw: If you know where you are going and treat it as a tool, it is helpful. However, don't try to offload decisions or let it direct you in the wrong direction. Overall, 350k Tokens used (about 300k words). Here's a related blog post [1] with my overall path, but not directly corresponding to this specific issue. (please don't recommend wireguard; I am aware of it) [1]: https://du.nkel.dev/blog/2021-11-19_pfsense_opnsense_ipsec_cgnat/
  - Benjammer 1y ago
    That's some impressive prompt engineering skills to keep it on track for that long, nice work! I'll have to try out some longer-form chats with Gemini and see what I get. I totally agree that LLMs are great at compressing information; I've set up the docs feature in Cursor to index several entire large documentation websites for major libraries and it's able to distill relevant information very quickly.
- unshavedyak 1y ago
  Has any interface implemented a .. history cleaning mechanism? Ie with every chat message focus on cleaning up dead ends in the conversation or irrelevant details. Like summation but organic for the topic at hand? Most history would remain, it wouldn’t try to summarize exactly, just prune and organize the history relative to the conversation path?
  - nosefurhairdo 1y ago
    I've had success having a conversation about requirements, asking the model to summarize the requirements as a spec to feed into a model for implementation, then pass that spec into a fresh context. Haven't seen any UI to do this automatically but fairly trivial/natural to perform with existing tools.
    - dep_b 1y ago
      Doing the same. Though I wish there was some kind of optimization of text generated by an LLM for an LLM. Just mentioning it’s for an LLM instead of Juan consumption yields no observably different results.
  - Benjammer 1y ago
    I mean, you could build this, but it would just be a feature on top of a product abstraction of a "conversation". Each time you press enter, you are spinning up a new instance of the LLM and passing in the entire previous chat text plus your new message, and asking it to predict the next tokens. It does this iteratively until the model produces a <stop> token, and then it returns the text to you and the PRODUCT parses it back into separate chat messages and displays it in your UI. What you are asking the PRODUCT to now do is to edit your and its chat messages in the history of the chat, and then send that as the new history with your latest message. This is the only way to clean the context because the context is nothing more than your messages and its previous responses, plus anything that tools have pulled in. I think it would be sort of a weird feature to add to a chat bot to have the chat bot, each time you send a new message, go back through the entire history of your chat and just start editing the messages to prune out details. You would scroll up and see a different conversation, it would be confusing. IMO, this is just part of prompt engineering skills to keep your context clean or know how to "clean" it by branching/summarizing conversations.
- CompoundEyes 1y ago
  Agreed poisoned is a good term. I’d like to see “version control” for conversations via the API and UI that lets you rollback to a previous place or clone from that spot into a new conversation. Even a typo or having to clarify a previous message skews the probabilities of future responses due to the accident.
  - mh- 1y ago
    "Forking" or "branching" (probably better received outside of SWEs) a conversation really ought to be a first class feature of ChatGPT et Al.
    - HaZeust 1y ago
      It is in Google Gemini, which I really hate to say - but I've been using a lot more than GPT. I reckon I'll be cancelling my Pro if Gemini stays with this lead for my everyday workflows.
      - energy123 1y ago
        How? I use the Gemini web app and don't see it.
        voidspark 1y ago
        http://aistudio.google.com http://aistudio.google.com
        crooked-v 1y ago
        AI Studio is borderline unusable for long conversations. I don't know what in the world it's doing but it sure looks like a catastrophic memory leak in the basic design.
        voidspark 1y ago
        I have been using it up to 100k tokens so far without issues. Never needed to go further than that. But much of that was in uploaded documents.
- djmips 1y ago
  Happens with people too if you think about it.
  - kfarr 1y ago
    Who gets lost in multi-turn conversations?
    - TheOtherHobbes 1y ago
      Everyone? How often in meetings does everyone maintain a running context of the entire conversation, instead of responding to the last thing that was said with a comment that has an outstanding chance of being forgotten as soon as the next person starts speaking?
      - djmips 1y ago
        Indeed - and since human's are susceptible to injection prompt, all it needs is one derailing comment to take things off course.
- CobrastanJorji 1y ago
  An interesting little example of this problem is initial prompting, which is effectively just a permanent, hidden context that can't be cleared. On Twitter right now, the "Grok" bot has recently begun frequently mentioning "White Genocide," which is, y'know, odd. This is almost certainly because someone recently adjusted its prompt to tell it what its views on white genocide are meant to be, which for a perfect chatbot wouldn't matter when you ask it about other topics, but it DOES matter. It's part of the context. It's gonna talk about that now.
  - ezst 1y ago
    The heck??
    - CobrastanJorji 1y ago
      Yeah, things are a little weird on Twitter these days. https://www.nbcnews.com/tech/tech-news/elon-musks-ai-chatbot-grok-brings-south-african-white-genocide-claims-rcna206838 https://www.nbcnews.com/tech/tech-news/elon-musks-ai-chatbot...
  - 9dev 1y ago
    Well, telling an AI chatbot to insist on discussing a white genocide seems like a perfectly Elon thing to do!
  - M4v3R 1y ago
    > This is almost certainly because someone recently adjusted its prompt to tell it what its views on white genocide are Do you have any source on this? System prompts get leaked/extracted all the time so imagine someone would notice this Edit: just realized you’re talking about the Grok bot, not Grok the LLM available on X or grok.com. With the bot it’s probably harder to extract its exact instructions since it only replies via tweets. For reference here’s the current Grok the LLM system prompt: https://github.com/asgeirtj/system_prompts_leaks/blob/main/grok-3.md https://github.com/asgeirtj/system_prompts_leaks/blob/main/g...
  - dragonwriter 1y ago
    > This is almost certainly because someone recently adjusted its prompt to tell it what its views on white genocide are meant to be Well, someone did something to it; whether it was training, feature boosting the way Golden Gate Claude [0] was done, adjusting the system prompt, or assuring that it's internet search for contextual information would always return material about that, or some combination of those, is neither obvious nor, if someone had a conjecture as to which one or combination it was, easily falsifiable/verifiable. [0] https://www.anthropic.com/news/golden-gate-claude https://www.anthropic.com/news/golden-gate-claude
- b800h 1y ago
  I've been saying for ages that I want to be able to fork conversations so I can experiment with the direction an exchange takes without irrevocably poisoning a promising well. I can't do this with ChatGPT, is anyone aware of a provider that offers this as a feature?
  - anonexpat 1y ago
    I believe Claude has forking in their web interface.
  - granra 1y ago
    Some 3rd party UIs offer this, I use typingmind sometimes that does but AFAIK some open source ones do too.
  - stuffoverflow 1y ago
    Google AI studio, ChatGPT and Claude all support this. Google AI studio is the only one that let's you branch to a separate chat though. For ChatGPT and claude you just edit the message you want to branch from.
    - Garlef 1y ago
      Support: Yes. But the UX is not optimized for this. Imagine trying to find a specific output/input that was good in the conversation tree.
      - layer8 1y ago
        Yes, it would be nice if you could at least bookmark a particular branch.
    - giordanol 1y ago
      Feels like a semi-simple UX fix could make this a lot more natural. Git-style forks but for chats.
  - deleted 1y ago
    [deleted]
- veunes 1y ago
  What surprised me is how early the models start locking into wrong assumptions
- amelius 1y ago
  I suppose that the chain-of-thought style of prompting that is used by AI chat applications internally also breaks down because of this phenomenon.
- Adambuilds 1y ago
  I agree—once the context is "poisoned," it’s tough to recover. A potential improvement could be having the LLM periodically clean or reset certain parts of the context without starting from scratch. However, the challenge would be determining which parts of the context need resetting without losing essential information. Smarter context management could help maintain coherence in longer conversations, but it’s a tricky balance to strike.Perhaps using another agent to do the job?
- freehorse 1y ago
  Which is why I really like zed's chat UX experience: being able to edit the full prior conversation like a text file, I can go back and clean it up, do small adjustments, delete turns etc and then continue the discussion with a cleaner and more relevant context. I have made zed one of my main llm chat interfaces even for non-programming tasks, because being able to do that is great.
- jimmySixDOF 1y ago
  >"conversations" are only a construct of product interfaces This seems to be in flux now due to RL training on multiturn eval datasets so while the context window is evergreen every time, there will be some bias towards interpreting each prompt as part of a longer conversation. Mutliturn post training is not scaled out yet in public but I think it may be the way to keep on the 'double time spent on goal every 7 months curve'
- bentt 1y ago
  Yes even when coding and not conversing I often start new conversations where I take the current code and explain it new. This often gives better results than hammering on one conversation. This feels like something that can be fixed with manual instructions which prompt the model to summarize and forget. This might even map appropriately to human psychology. Working Memory vs Narrative/Episodic Memory.
- pseudocomposer 1y ago
  I mostly just use LLMs for autocomplete (not chat), but wouldn’t this be fixed by adding a “delete message” button/context option in LLM chat UIs? If you delete the last message from the LLM (so now, you sent the last message), it would then generate a new response. (This would be particularly useful with high-temperature/more “randomly” configured LLMs.) If you delete any other message, it just updates the LLM context for any future responses it sends (the real problem at hand, context cleanup). I think seeing it work this way would also really help end users who think LLMs are “intelligent” to better understand that it’s just a big, complex autocomplete (and that’s still very useful). Maybe this is standard already, or used in some LLM UI? If not, consider this comment as putting it in the public domain. Now that I’m thinking about it, it seems like it might be practical to use “sub-contextual LLMs” to manage the context of your main LLM chat. Basically, if an LLM response in your chat/context is very long, you could ask the “sub-contextual LLM” to shorten/summarize that response, thus trimming down/cleaning the context for your overall conversation. (Also, more simply, an “edit message” button could do the same, just with you, the human, editing the context instead of an LLM…)
  - dr_dshiv 1y ago
    This is how Claude’s UI used to work, in practice, where you could edit the context directly.
- dr_dshiv 1y ago
  The #1 tip I teach is to make extensive use of the teeny-tiny mostly hidden “edit” button in ChatGPT and Claude. When you get a bad response, stop and edit to get a better one, rather than letting crap start to multiply crap.
  - diggan 1y ago
    Hear hear! Basically if the first reply isn't good/didnt understand/got something wrong, restart from the beginning with a better prompt, explaining more/better. Rinse and repeat.
    - forgotTheLast 1y ago
      You can do even better by asking it to ask clarifying questions before generating anything, then editing your initial prompt with those clarifications.
  - cruffle_duffle 1y ago
    It is also a great way to branch conversations from some shared “initial context”. They really need to make that edit feature much more prominent. It is such an important way to interact with the model.
- yaur 1y ago
  One of the most frustrating features of ChatGPT is “memories” which can cause that poisoning to follow you around between chats.
- aleksituk 1y ago
  Yarp! And "poisoning" can be done with "off-topic" questions and answers as well as just sort of "dilution". Have noticed this when doing content generation repeatedly, tight instructions get diluted over time.
- bredren 1y ago
  This is why I created FileKitty, which lets you quickly concatenate multiple source code files into markdown-formatted copy-pasta: https://github.com/banagale/FileKitty https://github.com/banagale/FileKitty When getting software development assistance, relying on LLM products to search code bases etc leaves too much room for error. Throw in what amounts to lossy compression of that context to save the service provider on token costs and the LLM is serving watered down results. Getting the specific context right up front and updating that context as the conversation unfolds leads to superior results. Even then, you do need to mind the length of conversations. I have a prompt designed to capture conversational context, and transfer it into a new session. It identifies files that should be included in the new initial prompt, etc. For a bit more discussion on this, see this thread and its ancestry: https://news.ycombinator.com/item?id=43711216 https://news.ycombinator.com/item?id=43711216
- QuantumGood 1y ago
  " 'conversations' are only a construct of product interface" is so helpful maintain top-of-mind, but difficult because of all the "conversational" cues
- oaeirjtlj 1y ago
  And now that chatgpt has a "memory" and can access previous conversations, it might be poisoned permanently. It gets one really bad idea, and forever after it insists on dumping that bad idea into every subsequent response ever after you repeatedly tell it "THAT'S A SHIT IDEA DON'T EVER MENTION THAT AGAIN". Sometimes it'll accidentally include some of its internal prompting, "user is very unhappy, make sure to not include xyz", and then it'll give you a response that is entirely focused around xyz.
  - rcdwealth 1y ago
    [dead]
- Macuyiko 1y ago
  Weirdly it has gotten so far that I have embedded this into my workflow and will often prompt: > "Good work so far, now I want to take it to another step (somewhat related but feeling it too hard): <short description>. Do you think we can do it in this conversation or is it better to start fresh? If so, prepare an initial prompt for your next fresh instantiation." Sometimes the model says that it might be better to start fresh, and prepares a good summary prompt (including a final 'see you later'), whereas in other cases it assures me it can continue. I have a lot of notebooks with "initial prompts to explore forward". But given the sycophancy going on as well as one-step RL (sigh) post-training [1], it indeed seems AI platforms would like to keep the conversation going. [1] RL in post-training has little to do with real RL and just uses one shot preference mechanisms with an RL inspired training loop. There is very little work in terms of long-term preferences slash conversations, as that would increase requirements exponentially.
  - senordevnyc 1y ago
    Is there any reason to think that LLMs have the introspection ability to be able to answer your question effectively? I just default to having them provide a summary that I can use to start the next conversation, because I’m unclear on how an LLM would know it’s losing the plot due to long context window.
alganet 1y ago
Humans also often get lost in multi-turn conversation. I have experienced that in person many, many times. Jumps in context that seem easy for one person to follow, but very hard for others. So, assuming the paper is legit (arxiv, you never know...), its more like something that could be improved than a difference from human beings.
- morsecodist 1y ago
  Subjectively the "getting lost" feels totally different than human conversations. Once there is something bad in the context it seems almost impossible to get back on track. All subsequent responses become get a lot worse and it starts contradicting itself. It is possible that with more training this problem can be improved, but what is interesting to me isn't it's worse than humans in this way but that this sort of difficulty scales differently than it does in humans. I would love to get some more objective descriptions of these subjective notions.
  - alganet 1y ago
    Contradictions are normal. Humans make them all the time. They're even easy to induce, due to the simplistic nature of our communication (lots of ambiguities, semantic disputes, etc). I don't see how that's a problem. Subjectivity is part of human communication.
    - westurner 1y ago
      Algorithmic convergence and caching :: Consensus in conversational human communication
      - alganet 1y ago
        Any sufficiently large amount of information exchange could be interpreted as computational if you see it as separated parts. It doesn't mean that it is intrinsically computational. Seeing human interactions as computer-like is a side effect of our most recent shiny toy. In the last century, people saw everything as gears and pulleys. All of these perspectives are essentially the same reductionist thinking, recycled over and over again. We've seen men promising that they would build a gear-man, resurrect the dead with electricity, and all sorts of (now) crazy talk. People believed it for some time.
dontreact 1y ago
My take: multi turn evals are hard because to do it really correctly you have to simulate a user. This is not yet modeled well enough for multi turn to work as well as it could.
deleted 1y ago
[deleted]
Sharlin 1y ago
Seems like this is an aspect of their well-known overconfidence and the inability to self-reflect and recognize they have to ask for more details because their priors are too low. If you look at the output of reasoning models, it’s clear that the idea of asking for clarification very rarely occurs to them – when they’re confused, it’s just endless speculation of what the user might have meant. This, of course, has certain implications as to the wisdom of the idea of “replacing human programmers”, given that one of the hard parts of the trade is trying to turn vague and often confused ideas into precise specifications by interacting with the shareholders.
- Terr_ 1y ago
  > inability to self-reflect IMO the One Weird Trick for LLMs is recognizing that there's no real entity, and that users are being tricked into a suspended-disbelief story. In most cases cases you're contributing text-lines for a User-character in a movie-script document, and the LLM algorithm is periodically triggered to autocomplete incomplete lines for a Chatbot character. You can have an interview with a vampire DraculaBot, but that character can only "self-reflect" in the same shallow/fictional way that it can "thirst for blood" or "turn into a cloud of bats."
  - Sharlin 1y ago
    This is a tired semantic argument that does not bring any insight into the discussion. A token-predictor could still be trained to predict the tokens “I’m not sure what you mean because of points x, y, and z; could you elaborate?”
    - Terr_ 1y ago
      It means if you want something resembling a self-introspective theory of mind, you need to arrange the overall document to cohere to documents where such things are/appear-to-be happening. This leads us to new questions: How can we characterize and identify real-world documents which fit? How can we determine what features may be significant, and which of those can be easily transplanted to our use-case?
Caelus9 1y ago
[dead]
zacksiri 1y ago
I've been working on solving this with quite a bit of success, I'll be sharing more on this soon. It involves having 2 systems 1st system is the LLM itself and another system which acts like a 'curator' of thoughts you could say. It dynamically swaps in / out portions of the context. This system is also not based on explicit definitions it relies on LLMs 'filling the gaps'. The system helps the llm break down problems into small tasks which then eventually aggregate into the full task.
- adiadd 1y ago
  would be great to get more info on what you’re building - seems interesting!
  - zacksiri 1y ago
    I publish my findings on my youtube channel and my blog, you're welcome to have a look. Both links are in my profile.
- cadamsdotcom 1y ago
  Sounds like an exciting idea. May I suggest - put what you have out there in the world, even if it’s barely more than a couple of prompts. If people see it and improve on it, and it’s a good idea, it’ll get picked up & worked on by others - might even take on a life of its own!
  - zacksiri 1y ago
    Have a look here, it's an early preview https://x.com/zacksiri/status/1922500206127349958 https://x.com/zacksiri/status/1922500206127349958 You can see it's going from introduction, asking me for my name, and then able to answer question about some topic. There is also another example in the thread you can see. Behind the scenes, the system prompt is being modified dynamically based on the user's request. All the information about movies is also being loaded into context dynamically. I'm also working on some technique to unload stuff from context when the subject matter of a given thread has changed dramatically. Imagine having a long thread of conversation with your friend, and along the way you 'context switch' multiple times as time progresses, you probably don't even remember what you said to your friend 4 years ago. There is a concept of 'main thread' and 'sub threads' involved as well that I'm exploring. I will be releasing the code base in the coming months. I need to take this demo further than just a few prompt replies.
permo-w 1y ago
I feel like at this point the LLM space is just filled with people solving and resolving the same problems over and over
- meroes 1y ago
  It’s herding cats, not “learning”, which is a fine situation for some parts of workflows.
- dankwizard 1y ago
  And everyone loves to chime in with their own excellence in prompt engineering
- deleted 1y ago
  [deleted]
- kristianp 1y ago
  Just like the llms in multi-turn conversations.
ranyume 1y ago
I'd like more research done on context understanding other than NIAH. I don't believe LLMs support the context length companies say they support. But I need to know this to effectively use the tools. At least for coding. Stuff like this: 1. Do: Best practice for X model is to include at max 10k lines of code + task + CONVENTIONS.md + architecture guidance. Only queue tasks for components that are fairly decoupled from the rest of the codebase (e.g. small modules). 2. Don't: Start a project without a clearly defined architecture in this format. Don't ask for tasks that require X amount of reading hops to understand the logic. I find it frustrating that companies release their benchmaxxing without helping developers actually use their models. It's more ironic that some people think of these AIs as employees. Employees can work with their boss about the best way to achieve things! With LLMs you don't even know how to communicate with them and as a result their output is unreliable.
- skydhash 1y ago
  You could swap those recommendations for programming without LLMs. Open any software engineering books and you’ll see a lot of good recommendations for building software.
coderatlarge 1y ago
i’ve see deepseek-coder local get into an infinite loop generating the same line over and over. which i assume without evidence is some sort of feedback from the generated line back into the generation process. so kind of getting lost in thought and going off topic from the simple .h api that my prompt asked for.
- silisili 1y ago
  Yes! Deepseek does this to me all the time. I had 20 something files I wanted it to check and change something. The first 5 or so it did, then the sixth it rightly said everything is correct moving on. It said that for the rest of the 20, the same text over and over. I checked, and file 6 was the only correct one. It like, learned to just repeat itself after that and did nothing.
  - TheOtherHobbes 1y ago
    Claude does this too. It gets into a death spiral where it repeats the entire previous output instead of changing parts and moving on.
tsunamifury 1y ago
Have you seen a bunch of humans in a room?
- xyzal 1y ago
  I see lots of humans in an echo chamber
airylizard 1y ago
Why I came up with TSCE(Two-Step Contextual Enrichment). +30pp uplift when using GPT-35-turbo on a mix of 300 tasks. Free open framework, check the repo try it yourself https://github.com/AutomationOptimization/tsce_demo https://github.com/AutomationOptimization/tsce_demo I tested this another 300 times with gpt-4.1 to remove those obtrusive "em-dashes" everyone hates. Tested a single-pass baseline vs TSCE, same exact instructions and prompt "Remove the em-dashes from my linkedin post. . .". Out of the 300 tests, baseline failed to remove the em-dashes 149/300 times. TSCE failed to remove the em-dashes 18/300 times. It works, all the data as well as the entire script used for testing is in the repo.
- arnaudsm 1y ago
  That's a lot of kilo-watt-hours wasted for a find and replace operation. Have you heard of text.replace("—", "-") ?
  - airylizard 1y ago
    The test isn't for how well an LLM can find or replace a string. It's for how well it can carry out given instructions... Is that not obvious?
- thegeomaster 1y ago
  I slightly tweaked your baseline em dash example and got 100% success rate with GPT-4.1 without any additional calls, token spend, or technobabble. System prompt: "Remove every em-dash (—) from the following text while leaving other characters unchanged.\n\nReturn only the cleaned text." User prompt: <prompt from tsce_chat.py filled with em dashes> Temperature: 0.0
  - deleted 1y ago
    [deleted]
  - airylizard 1y ago
    Hey, thanks for kicking the tires! The run you’re describing was done in mid-April, right after GPT-4.1 went live. Since then OpenAI has refreshed the weights behind the “gpt-4.1” alias a couple of times, and one of those updates fixed the em-dash miss. If you reran today you’d see the same improved pass rate I’m getting now. That’s the downside of benchmarking against latest model names; behaviour changes quietly unless you pin to a dated snapshot. For bigger, noisier prompts (or on GPT-3.5-turbo, which hasn’t changed) TSCE still gives a solid uplift, so the framework’s value stands. Appreciate you checking it out!
dr_dshiv 1y ago
This is the best paper on machine psychology [1] I’ve yet seen. Rigorous, empirical, insightful — and very practical. [1] http://ui.adsabs.harvard.edu/abs/2023arXiv230313988H/abstract http://ui.adsabs.harvard.edu/abs/2023arXiv230313988H/abstrac...
jsemrau 1y ago
That's no surprise. When I was working on game theory and agent reasoning I reached the same conclusion a year ago. My conclusion was that context needs to be managed well for the LLMs to manage accuracy in replies. Also, it helps to have a planning process ("graph reasoning") before task execution because it guardrails the models thought process. This also introduces a discussion on general use vs workflow agent implementations as in the former it is much more difficult to generalize all components in structuring effective ReAct patterns.
- veunes 1y ago
  It's probably why workflow agents feel more reliable: they're built around structure, not just raw prediction
  - jsemrau 1y ago
    Also you have more control points. It's not just a brain a vat.
jumploops 1y ago
It's amazing that branching/forking isn't a core aspect of the main chat tools. You can edit responses, sure, but then a bunch of other context is lost. My flow is basically: 1. plan 2. build 3. branch (into some feature/esoteric dependency issue) 4. goto #2 Prompt pruning/branching should be a first-class tool for any LLM usage.
- Capricorn2481 1y ago
  I've been kicking around making this for a while. BetterChatGPT at least has some good ergonomics around deleting history. But I agree that branching is the next step.
- jampekka 1y ago
  Google AI studio at least has this. I found at least that implementation quite confusing though, which may be a reason it's not implemented in more "consumer oriented" tools.
SamPatt 1y ago
I always felt the derision around the term "prompt engineering" was partially due to people overestimating the importance of the initial prompt and underestimating the importance of managing the ongoing context. You develop a knack for how to steer the models or start a new conversation through experience. The system or initial prompt are important, but nothing will save you if you naively keep a conversation going too long.
- veunes 1y ago
  Yeah, totally. Prompt engineering isn't just about crafting the perfect opener, it's more like conversation management. You start to develop a feel for when things are going off the rails and it's time to reset
veunes 1y ago
Kind of wild how even the best models still struggle with keeping context straight over time. Definitely feels like a big challenge if we want these things to hold real conversations.
guardiang 1y ago
Exactly why expert steering should be valued.
badmonster 1y ago
Why do LLMs struggle so much with recovering from early wrong turns in multi-turn conversations — even when all prior context is available and tokenized? Is it due to the model's training distribution (mostly single-shot completions), the way context windows are encoded, or an architectural bottleneck? Feels like there's no dynamic internal state that evolves over the conversation — only a repeated re-parsing of static history. Has anyone seen work on integrating memory/state mechanisms that allow belief revision within a session, not just regurgitation of past tokens?
- JohnKemeny 1y ago
  We shouldn’t anthropomorphize LLMs—they don’t “struggle.” A better framing is: why is the most likely next token, given the prior context, one that reinforces the earlier wrong turn?
- bandrami 1y ago
  Because Markov chains propagate forward in time
- vjerancrnjak 1y ago
  Imagine optimizing/training on a happy path. When you generate future tokens, you're looking at history tokens that are happy. So how can a model, given sad tokens, generate future happy tokens if it did not learn to do so? The work you're looking for is already here, it's "thinking". I assume they include sad tokens in the dataset, produce "thinking", which should result in happy tokens coming after thinking tokens. If thinking is bad (by looking at following happy tokens), then it's punished, if good, then descent.
- mountainriver 1y ago
  It’s a problem specific to autoregressive LLMs, the early tokens bias the output
podgorniy 1y ago
There is a noticable issue when one builds LLMs interfaces around single turn conversations. Majority people expect linear conversations. I've built telegram bot http://t.me/experai_bot http://t.me/experai_bot as univresal UI to LLMs (with somewhat reduced functionality) exactly around idea "non-reply message means new conversation". Wanna keep context? Keep replying to replies of bot. Non-power user strugge with this idea. -- Also I observed that OpenAI models performed worse replying to the same questions (for example list of options in reply got shorter) even with smallest system message. That was the case with 3.5, 4o. Don't know how modern ones behave. That made me decide not to include any system messages by default Still I give option to add ones if you need. You can even toggle them to mix-and-match.
sky2224 1y ago
Ha, kind of funny to see this right now. I've been fighting copilot in vscode in trying to get it to output anything once I take the context down to a very specific problem. It feels like I have to reset and almost reground the model into what I'm trying to accomplish at a certain point.
debuggerson 1y ago
The more we chat, the more irrelevant details pile up. For example, a small mention early on might get repeated or build on itself, leading to a lot of unnecessary context. As the conversation continues, it becomes harder for the model to focus on the main point because it gets tangled in all the extra information. Unlike humans, who can intuitively filter out the noise, LLMs struggle to keep track of what’s truly important in longer, more complex exchanges.
deleted 1y ago
[deleted]
overflow897 1y ago
I believe we're already using llms to evaluate llm output for training, I wonder if there's some variation of that which could be used to identify when one llm gets "stuck". I guess chain of thought in theory should do that but having variations on prompt and context might behave differently?
t-kalinowski 1y ago
This was the main reason I wrote promptdown. I want to be able to edit the full chat history every turn, and the append-only standard chat interfaces don't make that easy. https://github.com/t-kalinowski/promptdown https://github.com/t-kalinowski/promptdown
tmountain 1y ago
I often ask the LLM for a concise summary of the discussion so far—formatted as a prompt. I then edit it appropriately and use it to start a new conversation without the baggage. I have found this to be a very effective technique, but I imagine it will be automated sometime soon.
- Drunkfoowl 1y ago
  [dead]
- maleldil 1y ago
  Claude Code has a /compact command that summarises the conversation so far to save on context tokens.
- drewbitt 1y ago
  Cursor tried doing this automatically - it may still if you're not on a large context model like gemini 2.5 pro - but I found the summary was just missing too many details to use out of the box.
aleksituk 1y ago
This is very interesting and I like the conversation about not only the technology itself, but also about the importance of thinking about the interface as a user experience and where / how it fits the paradigm. We've been working on a lot of data processing and generation tasks. We've been doing this using an API primarily, but sometimes I end up testing creating data in a chat window and I first chat through what the requirements are for the data analysis / processing and then once I'm done I would like the whole conversation to be then summarised into basically a one-prompt process so that I can re-use it (because I can't really process new inputs via the chat). Even when you do manage to get it down to a single prompt you can use in a chat and then ask the chat to just keep producing new data (like imagine a blog post in certain style if the base content is given as input and I'm making like 20 of them). If you produce these in the chat, there's notable benefits in that if something is wrong with the blog post the chat suggests, you can immediately edit it. The trouble is that the context window starts becoming so big that the chat starts to forget what the original instruction is and eventually you do have to just create a new chat. One way to solve for this is having a chat with selective memory where you keep a task in memory, but you have the chat forget/not-include all the generated data in the context so that it stays clean, but only bring it to the context if the user refers to it. Has anyone else done data processing types of tasks in chats and had issues like this? Are there some other tools to use or tricks to do in chats?
WhitneyLand 1y ago
Any reason to not call bullshit on this paper? One of the biggest developments in language models over the last year has been test-time reasoning (aka inference scaling or “thinking”). Most vendors tested offer such a model. It’s plausible it could make a huge difference here, and they did not bother to test it or even mention it? Things like COT and planning can really affect this and those are just a couple of things that happen automatically in more advanced models. Seems like it wouldn’t have been hard to add this to the experiment, but they could’ve called it out in a “Limitations” or “Future Work” section. Or at least a single sentence like “We did not test chain-of-thought prompting, which may mitigate some of these issues”.
giordanol 1y ago
Would love to see metrics that isolate recovery behaviour (if any)
curtisszmania 1y ago
[dead]
Workaccount2 1y ago
Reminds me of Claude plays pokemon, where it would note something insignificant, and then fixate on it for hours.
Zobat 1y ago
This must mean that LLMs really are like genies in bottles. You get three questions answered, anything after that will be nonsense.
sattard 1y ago
Why haven't AI code editors not built this at their core yet, to automatically consolidate previous conversational turns into a more structured context summary. Instead of relying solely on the model’s memory of all prior exchanges, surely these tools should take responsibility for intermittently “restating” the clarified requirements so the model doesn’t have to reconstruct context from scratch (or worse, pick up mistakes). This might mitigate compounding errors and reduce verbosity.
RandyOrion 1y ago
One problem with this paper is that authors didn't conduct experiments on popular LLMs from Qwen and Mistral. Why?

374 points by simonpure 1y ago

It's nice to see a paper that confirms what anyone who has practiced using LLM tools already knows very well, heuristically. Keeping your context clean matters, "conversations" are only a construct of product interfaces, they hurt the quality of responses from the LLM itself, and once your context is "poisoned" it will not recover, you need to start fresh with a new chat.

MattGaiser 1y ago
Yep. I regretted leaving on memory as it is poisoned my conversations with irrelevant junk.
- neom 1y ago
  You can go in and delete memory items
morsecodist 1y ago
This matches my experience exactly. "poisoned" is a great way to put it. I find once something has gone wrong all subsequent responses are bad. This is why I am iffy on ChatGPT's memory features. I don't notice it causing any huge problems but I don't love how it pollutes my context in ways I don't fully understand.
- AstroBen 1y ago
  good point on the memory feature. Wow that sounds terrible
  - distances 1y ago
    The memory is easy to turn off. It sounded like a very bad idea to cross-contaminate chats so I disabled it as soon as ChatGPT introduced it.
- somenameforme 1y ago
  It's interesting how much the nature of LLMs fundamentally being self recursive next token predictors aligns with the Chinese Room experiment. [1] In such experiment it also makes perfect sense that a single wrong response would cascade into a series of subsequent ever more drifting errors. I think it all emphasizes the relevance of the otherwise unqualifiable concept of 'understanding.' In many ways this issue could make the Chinese Room thought experiment even more compelling. Because it's a very practical and inescapable issue. [1] - https://en.wikipedia.org/wiki/Chinese_room https://en.wikipedia.org/wiki/Chinese_room
  - keiferski 1y ago
    Great comment on the Chinese room. That idea seems to be dismissed nowadays but the concept of “cascading failure to understand context” is absolutely relevant to LLMs. I often find myself needing to explain basic details over and over again to an LLM; when with a person it would be a five second, “no, I mean like this way, not that way” explanation.
  - jampekka 1y ago
    I don't think the Chinese room thought experiment is about this, or performance of LLMs in general. Searle explicitly argues that a program can't induce "understanding" even if it mimicked human understanding perfectly because programs don't have "causal powers" to generate "mental states". This is mentioned in the Wikipedia page too: "Although its proponents originally presented the argument in reaction to statements of artificial intelligence (AI) researchers, it is not an argument against the goals of mainstream AI research because it does not show a limit in the amount of intelligent behavior a machine can display."
- OtherShrezzing 1y ago
  I find using tools like LMStudio, which lets you edit your chat history on the fly, really helps deal with this problem. The models you can host locally are much weaker, but they perform a little better than the really big models once you need to factor in these poisoning problems. A nice middle-ground I'm finding is to ask Claude an initial conversation starter in its "thinking" mode, and then copy/paste that conversation into LMStudio and have a weaker model like Gemma pick-up from where Claude left off.
- shrewduser 1y ago
  I have very limited experience with llms but i've always thought of it as a compounding errors problem, once you get a small error early on it can compound and go completely off track later.
Helmut10001 1y ago
My experiences somewhat confirm these observations, but I also had one that was different. Two weeks of debugging IPSEC issues with Gemini. Initially, I imported all the IPSEC documentation from OPNsense and pfSense into Gemini and informed it of the general context in which I was operating (in reference to 'keeping your context clean'). Then I added my initial settings for both sides (sensitive information redacted!). Afterwards, I entered a long feedback loop, posting logs and asking and answering questions. At the end of the two weeks, I observed that: The LLM was much less likely to become distracted. Sometimes, I would dump whole forum threads or SO posts into it, when it said "this is not what we are seeing here, because of [earlier context or finding]. I eliminated all dead ends logically and informed it of this (yes, it can help with the reflection, but I had to make the decisions). In the end, I found the cause of my issues. This somewhat confirms what some user here on HN said a few days ago. LLMs are good at compressing complex information into simple one, but not at expanding simple ideas into complex ones. As long as my input was larger than the output (either complexity or length), I was happy with the results. I could have done this without the LLM. However, it was helpful in that it stored facts from the outset that I had either forgotten or been unable to retrieve quickly in new contexts. It also made it easier to identify time patterns in large log files, which helped me debug my site-to-site connection. I also optimized many other settings along the way, resolving not only the most problematic issue. This meant, in addition to fixing my problem, I learned quite a bit. The 'state' was only occasionally incorrect about my current parameter settings, but this was always easy to correct. This confirms what others already saw: If you know where you are going and treat it as a tool, it is helpful. However, don't try to offload decisions or let it direct you in the wrong direction. Overall, 350k Tokens used (about 300k words). Here's a related blog post [1] with my overall path, but not directly corresponding to this specific issue. (please don't recommend wireguard; I am aware of it) [1]: https://du.nkel.dev/blog/2021-11-19_pfsense_opnsense_ipsec_cgnat/
- Benjammer 1y ago
  That's some impressive prompt engineering skills to keep it on track for that long, nice work! I'll have to try out some longer-form chats with Gemini and see what I get. I totally agree that LLMs are great at compressing information; I've set up the docs feature in Cursor to index several entire large documentation websites for major libraries and it's able to distill relevant information very quickly.
unshavedyak 1y ago
Has any interface implemented a .. history cleaning mechanism? Ie with every chat message focus on cleaning up dead ends in the conversation or irrelevant details. Like summation but organic for the topic at hand? Most history would remain, it wouldn’t try to summarize exactly, just prune and organize the history relative to the conversation path?
- nosefurhairdo 1y ago
  I've had success having a conversation about requirements, asking the model to summarize the requirements as a spec to feed into a model for implementation, then pass that spec into a fresh context. Haven't seen any UI to do this automatically but fairly trivial/natural to perform with existing tools.
  - dep_b 1y ago
    Doing the same. Though I wish there was some kind of optimization of text generated by an LLM for an LLM. Just mentioning it’s for an LLM instead of Juan consumption yields no observably different results.
- Benjammer 1y ago
  I mean, you could build this, but it would just be a feature on top of a product abstraction of a "conversation". Each time you press enter, you are spinning up a new instance of the LLM and passing in the entire previous chat text plus your new message, and asking it to predict the next tokens. It does this iteratively until the model produces a <stop> token, and then it returns the text to you and the PRODUCT parses it back into separate chat messages and displays it in your UI. What you are asking the PRODUCT to now do is to edit your and its chat messages in the history of the chat, and then send that as the new history with your latest message. This is the only way to clean the context because the context is nothing more than your messages and its previous responses, plus anything that tools have pulled in. I think it would be sort of a weird feature to add to a chat bot to have the chat bot, each time you send a new message, go back through the entire history of your chat and just start editing the messages to prune out details. You would scroll up and see a different conversation, it would be confusing. IMO, this is just part of prompt engineering skills to keep your context clean or know how to "clean" it by branching/summarizing conversations.
CompoundEyes 1y ago
Agreed poisoned is a good term. I’d like to see “version control” for conversations via the API and UI that lets you rollback to a previous place or clone from that spot into a new conversation. Even a typo or having to clarify a previous message skews the probabilities of future responses due to the accident.
- mh- 1y ago
  "Forking" or "branching" (probably better received outside of SWEs) a conversation really ought to be a first class feature of ChatGPT et Al.
  - HaZeust 1y ago
    It is in Google Gemini, which I really hate to say - but I've been using a lot more than GPT. I reckon I'll be cancelling my Pro if Gemini stays with this lead for my everyday workflows.
    - energy123 1y ago
      How? I use the Gemini web app and don't see it.
      - voidspark 1y ago
        http://aistudio.google.com http://aistudio.google.com
        crooked-v 1y ago
        AI Studio is borderline unusable for long conversations. I don't know what in the world it's doing but it sure looks like a catastrophic memory leak in the basic design.
        voidspark 1y ago
        I have been using it up to 100k tokens so far without issues. Never needed to go further than that. But much of that was in uploaded documents.
djmips 1y ago
Happens with people too if you think about it.
- kfarr 1y ago
  Who gets lost in multi-turn conversations?
  - TheOtherHobbes 1y ago
    Everyone? How often in meetings does everyone maintain a running context of the entire conversation, instead of responding to the last thing that was said with a comment that has an outstanding chance of being forgotten as soon as the next person starts speaking?
    - djmips 1y ago
      Indeed - and since human's are susceptible to injection prompt, all it needs is one derailing comment to take things off course.
CobrastanJorji 1y ago
An interesting little example of this problem is initial prompting, which is effectively just a permanent, hidden context that can't be cleared. On Twitter right now, the "Grok" bot has recently begun frequently mentioning "White Genocide," which is, y'know, odd. This is almost certainly because someone recently adjusted its prompt to tell it what its views on white genocide are meant to be, which for a perfect chatbot wouldn't matter when you ask it about other topics, but it DOES matter. It's part of the context. It's gonna talk about that now.
- ezst 1y ago
  The heck??
  - CobrastanJorji 1y ago
    Yeah, things are a little weird on Twitter these days. https://www.nbcnews.com/tech/tech-news/elon-musks-ai-chatbot-grok-brings-south-african-white-genocide-claims-rcna206838 https://www.nbcnews.com/tech/tech-news/elon-musks-ai-chatbot...
- 9dev 1y ago
  Well, telling an AI chatbot to insist on discussing a white genocide seems like a perfectly Elon thing to do!
- M4v3R 1y ago
  > This is almost certainly because someone recently adjusted its prompt to tell it what its views on white genocide are Do you have any source on this? System prompts get leaked/extracted all the time so imagine someone would notice this Edit: just realized you’re talking about the Grok bot, not Grok the LLM available on X or grok.com. With the bot it’s probably harder to extract its exact instructions since it only replies via tweets. For reference here’s the current Grok the LLM system prompt: https://github.com/asgeirtj/system_prompts_leaks/blob/main/grok-3.md https://github.com/asgeirtj/system_prompts_leaks/blob/main/g...
- dragonwriter 1y ago
  > This is almost certainly because someone recently adjusted its prompt to tell it what its views on white genocide are meant to be Well, someone did something to it; whether it was training, feature boosting the way Golden Gate Claude [0] was done, adjusting the system prompt, or assuring that it's internet search for contextual information would always return material about that, or some combination of those, is neither obvious nor, if someone had a conjecture as to which one or combination it was, easily falsifiable/verifiable. [0] https://www.anthropic.com/news/golden-gate-claude https://www.anthropic.com/news/golden-gate-claude
b800h 1y ago
I've been saying for ages that I want to be able to fork conversations so I can experiment with the direction an exchange takes without irrevocably poisoning a promising well. I can't do this with ChatGPT, is anyone aware of a provider that offers this as a feature?
- anonexpat 1y ago
  I believe Claude has forking in their web interface.
- granra 1y ago
  Some 3rd party UIs offer this, I use typingmind sometimes that does but AFAIK some open source ones do too.
- stuffoverflow 1y ago
  Google AI studio, ChatGPT and Claude all support this. Google AI studio is the only one that let's you branch to a separate chat though. For ChatGPT and claude you just edit the message you want to branch from.
  - Garlef 1y ago
    Support: Yes. But the UX is not optimized for this. Imagine trying to find a specific output/input that was good in the conversation tree.
    - layer8 1y ago
      Yes, it would be nice if you could at least bookmark a particular branch.
  - giordanol 1y ago
    Feels like a semi-simple UX fix could make this a lot more natural. Git-style forks but for chats.
- deleted 1y ago
  [deleted]
veunes 1y ago
What surprised me is how early the models start locking into wrong assumptions
amelius 1y ago
I suppose that the chain-of-thought style of prompting that is used by AI chat applications internally also breaks down because of this phenomenon.
Adambuilds 1y ago
I agree—once the context is "poisoned," it’s tough to recover. A potential improvement could be having the LLM periodically clean or reset certain parts of the context without starting from scratch. However, the challenge would be determining which parts of the context need resetting without losing essential information. Smarter context management could help maintain coherence in longer conversations, but it’s a tricky balance to strike.Perhaps using another agent to do the job?
freehorse 1y ago
Which is why I really like zed's chat UX experience: being able to edit the full prior conversation like a text file, I can go back and clean it up, do small adjustments, delete turns etc and then continue the discussion with a cleaner and more relevant context. I have made zed one of my main llm chat interfaces even for non-programming tasks, because being able to do that is great.
jimmySixDOF 1y ago
>"conversations" are only a construct of product interfaces This seems to be in flux now due to RL training on multiturn eval datasets so while the context window is evergreen every time, there will be some bias towards interpreting each prompt as part of a longer conversation. Mutliturn post training is not scaled out yet in public but I think it may be the way to keep on the 'double time spent on goal every 7 months curve'
bentt 1y ago
Yes even when coding and not conversing I often start new conversations where I take the current code and explain it new. This often gives better results than hammering on one conversation. This feels like something that can be fixed with manual instructions which prompt the model to summarize and forget. This might even map appropriately to human psychology. Working Memory vs Narrative/Episodic Memory.
pseudocomposer 1y ago
I mostly just use LLMs for autocomplete (not chat), but wouldn’t this be fixed by adding a “delete message” button/context option in LLM chat UIs? If you delete the last message from the LLM (so now, you sent the last message), it would then generate a new response. (This would be particularly useful with high-temperature/more “randomly” configured LLMs.) If you delete any other message, it just updates the LLM context for any future responses it sends (the real problem at hand, context cleanup). I think seeing it work this way would also really help end users who think LLMs are “intelligent” to better understand that it’s just a big, complex autocomplete (and that’s still very useful). Maybe this is standard already, or used in some LLM UI? If not, consider this comment as putting it in the public domain. Now that I’m thinking about it, it seems like it might be practical to use “sub-contextual LLMs” to manage the context of your main LLM chat. Basically, if an LLM response in your chat/context is very long, you could ask the “sub-contextual LLM” to shorten/summarize that response, thus trimming down/cleaning the context for your overall conversation. (Also, more simply, an “edit message” button could do the same, just with you, the human, editing the context instead of an LLM…)
- dr_dshiv 1y ago
  This is how Claude’s UI used to work, in practice, where you could edit the context directly.
dr_dshiv 1y ago
The #1 tip I teach is to make extensive use of the teeny-tiny mostly hidden “edit” button in ChatGPT and Claude. When you get a bad response, stop and edit to get a better one, rather than letting crap start to multiply crap.
- diggan 1y ago
  Hear hear! Basically if the first reply isn't good/didnt understand/got something wrong, restart from the beginning with a better prompt, explaining more/better. Rinse and repeat.
  - forgotTheLast 1y ago
    You can do even better by asking it to ask clarifying questions before generating anything, then editing your initial prompt with those clarifications.
- cruffle_duffle 1y ago
  It is also a great way to branch conversations from some shared “initial context”. They really need to make that edit feature much more prominent. It is such an important way to interact with the model.
yaur 1y ago
One of the most frustrating features of ChatGPT is “memories” which can cause that poisoning to follow you around between chats.
aleksituk 1y ago
Yarp! And "poisoning" can be done with "off-topic" questions and answers as well as just sort of "dilution". Have noticed this when doing content generation repeatedly, tight instructions get diluted over time.
bredren 1y ago
This is why I created FileKitty, which lets you quickly concatenate multiple source code files into markdown-formatted copy-pasta: https://github.com/banagale/FileKitty https://github.com/banagale/FileKitty When getting software development assistance, relying on LLM products to search code bases etc leaves too much room for error. Throw in what amounts to lossy compression of that context to save the service provider on token costs and the LLM is serving watered down results. Getting the specific context right up front and updating that context as the conversation unfolds leads to superior results. Even then, you do need to mind the length of conversations. I have a prompt designed to capture conversational context, and transfer it into a new session. It identifies files that should be included in the new initial prompt, etc. For a bit more discussion on this, see this thread and its ancestry: https://news.ycombinator.com/item?id=43711216 https://news.ycombinator.com/item?id=43711216
QuantumGood 1y ago
" 'conversations' are only a construct of product interface" is so helpful maintain top-of-mind, but difficult because of all the "conversational" cues
oaeirjtlj 1y ago
And now that chatgpt has a "memory" and can access previous conversations, it might be poisoned permanently. It gets one really bad idea, and forever after it insists on dumping that bad idea into every subsequent response ever after you repeatedly tell it "THAT'S A SHIT IDEA DON'T EVER MENTION THAT AGAIN". Sometimes it'll accidentally include some of its internal prompting, "user is very unhappy, make sure to not include xyz", and then it'll give you a response that is entirely focused around xyz.
- rcdwealth 1y ago
  [dead]
Macuyiko 1y ago
Weirdly it has gotten so far that I have embedded this into my workflow and will often prompt: > "Good work so far, now I want to take it to another step (somewhat related but feeling it too hard): <short description>. Do you think we can do it in this conversation or is it better to start fresh? If so, prepare an initial prompt for your next fresh instantiation." Sometimes the model says that it might be better to start fresh, and prepares a good summary prompt (including a final 'see you later'), whereas in other cases it assures me it can continue. I have a lot of notebooks with "initial prompts to explore forward". But given the sycophancy going on as well as one-step RL (sigh) post-training [1], it indeed seems AI platforms would like to keep the conversation going. [1] RL in post-training has little to do with real RL and just uses one shot preference mechanisms with an RL inspired training loop. There is very little work in terms of long-term preferences slash conversations, as that would increase requirements exponentially.
- senordevnyc 1y ago
  Is there any reason to think that LLMs have the introspection ability to be able to answer your question effectively? I just default to having them provide a summary that I can use to start the next conversation, because I’m unclear on how an LLM would know it’s losing the plot due to long context window.

alganet 1y ago

Humans also often get lost in multi-turn conversation. I have experienced that in person many, many times. Jumps in context that seem easy for one person to follow, but very hard for others. So, assuming the paper is legit (arxiv, you never know...), its more like something that could be improved than a difference from human beings.

morsecodist 1y ago
Subjectively the "getting lost" feels totally different than human conversations. Once there is something bad in the context it seems almost impossible to get back on track. All subsequent responses become get a lot worse and it starts contradicting itself. It is possible that with more training this problem can be improved, but what is interesting to me isn't it's worse than humans in this way but that this sort of difficulty scales differently than it does in humans. I would love to get some more objective descriptions of these subjective notions.
- alganet 1y ago
  Contradictions are normal. Humans make them all the time. They're even easy to induce, due to the simplistic nature of our communication (lots of ambiguities, semantic disputes, etc). I don't see how that's a problem. Subjectivity is part of human communication.
  - westurner 1y ago
    Algorithmic convergence and caching :: Consensus in conversational human communication
    - alganet 1y ago
      Any sufficiently large amount of information exchange could be interpreted as computational if you see it as separated parts. It doesn't mean that it is intrinsically computational. Seeing human interactions as computer-like is a side effect of our most recent shiny toy. In the last century, people saw everything as gears and pulleys. All of these perspectives are essentially the same reductionist thinking, recycled over and over again. We've seen men promising that they would build a gear-man, resurrect the dead with electricity, and all sorts of (now) crazy talk. People believed it for some time.

dontreact 1y ago

My take: multi turn evals are hard because to do it really correctly you have to simulate a user. This is not yet modeled well enough for multi turn to work as well as it could.

deleted 1y ago

[deleted]

Sharlin 1y ago

Seems like this is an aspect of their well-known overconfidence and the inability to self-reflect and recognize they have to ask for more details because their priors are too low. If you look at the output of reasoning models, it’s clear that the idea of asking for clarification very rarely occurs to them – when they’re confused, it’s just endless speculation of what the user might have meant. This, of course, has certain implications as to the wisdom of the idea of “replacing human programmers”, given that one of the hard parts of the trade is trying to turn vague and often confused ideas into precise specifications by interacting with the shareholders.

Terr_ 1y ago
> inability to self-reflect IMO the One Weird Trick for LLMs is recognizing that there's no real entity, and that users are being tricked into a suspended-disbelief story. In most cases cases you're contributing text-lines for a User-character in a movie-script document, and the LLM algorithm is periodically triggered to autocomplete incomplete lines for a Chatbot character. You can have an interview with a vampire DraculaBot, but that character can only "self-reflect" in the same shallow/fictional way that it can "thirst for blood" or "turn into a cloud of bats."
- Sharlin 1y ago
  This is a tired semantic argument that does not bring any insight into the discussion. A token-predictor could still be trained to predict the tokens “I’m not sure what you mean because of points x, y, and z; could you elaborate?”
  - Terr_ 1y ago
    It means if you want something resembling a self-introspective theory of mind, you need to arrange the overall document to cohere to documents where such things are/appear-to-be happening. This leads us to new questions: How can we characterize and identify real-world documents which fit? How can we determine what features may be significant, and which of those can be easily transplanted to our use-case?

Caelus9 1y ago

[dead]

zacksiri 1y ago

I've been working on solving this with quite a bit of success, I'll be sharing more on this soon. It involves having 2 systems 1st system is the LLM itself and another system which acts like a 'curator' of thoughts you could say. It dynamically swaps in / out portions of the context. This system is also not based on explicit definitions it relies on LLMs 'filling the gaps'. The system helps the llm break down problems into small tasks which then eventually aggregate into the full task.

adiadd 1y ago
would be great to get more info on what you’re building - seems interesting!
- zacksiri 1y ago
  I publish my findings on my youtube channel and my blog, you're welcome to have a look. Both links are in my profile.
cadamsdotcom 1y ago
Sounds like an exciting idea. May I suggest - put what you have out there in the world, even if it’s barely more than a couple of prompts. If people see it and improve on it, and it’s a good idea, it’ll get picked up & worked on by others - might even take on a life of its own!
- zacksiri 1y ago
  Have a look here, it's an early preview https://x.com/zacksiri/status/1922500206127349958 https://x.com/zacksiri/status/1922500206127349958 You can see it's going from introduction, asking me for my name, and then able to answer question about some topic. There is also another example in the thread you can see. Behind the scenes, the system prompt is being modified dynamically based on the user's request. All the information about movies is also being loaded into context dynamically. I'm also working on some technique to unload stuff from context when the subject matter of a given thread has changed dramatically. Imagine having a long thread of conversation with your friend, and along the way you 'context switch' multiple times as time progresses, you probably don't even remember what you said to your friend 4 years ago. There is a concept of 'main thread' and 'sub threads' involved as well that I'm exploring. I will be releasing the code base in the coming months. I need to take this demo further than just a few prompt replies.

permo-w 1y ago

I feel like at this point the LLM space is just filled with people solving and resolving the same problems over and over

meroes 1y ago
It’s herding cats, not “learning”, which is a fine situation for some parts of workflows.
dankwizard 1y ago
And everyone loves to chime in with their own excellence in prompt engineering
deleted 1y ago
[deleted]
kristianp 1y ago
Just like the llms in multi-turn conversations.

ranyume 1y ago

I'd like more research done on context understanding other than NIAH. I don't believe LLMs support the context length companies say they support. But I need to know this to effectively use the tools. At least for coding. Stuff like this: 1. Do: Best practice for X model is to include at max 10k lines of code + task + CONVENTIONS.md + architecture guidance. Only queue tasks for components that are fairly decoupled from the rest of the codebase (e.g. small modules). 2. Don't: Start a project without a clearly defined architecture in this format. Don't ask for tasks that require X amount of reading hops to understand the logic. I find it frustrating that companies release their benchmaxxing without helping developers actually use their models. It's more ironic that some people think of these AIs as employees. Employees can work with their boss about the best way to achieve things! With LLMs you don't even know how to communicate with them and as a result their output is unreliable.

skydhash 1y ago
You could swap those recommendations for programming without LLMs. Open any software engineering books and you’ll see a lot of good recommendations for building software.

coderatlarge 1y ago

i’ve see deepseek-coder local get into an infinite loop generating the same line over and over. which i assume without evidence is some sort of feedback from the generated line back into the generation process. so kind of getting lost in thought and going off topic from the simple .h api that my prompt asked for.

silisili 1y ago
Yes! Deepseek does this to me all the time. I had 20 something files I wanted it to check and change something. The first 5 or so it did, then the sixth it rightly said everything is correct moving on. It said that for the rest of the 20, the same text over and over. I checked, and file 6 was the only correct one. It like, learned to just repeat itself after that and did nothing.
- TheOtherHobbes 1y ago
  Claude does this too. It gets into a death spiral where it repeats the entire previous output instead of changing parts and moving on.

tsunamifury 1y ago

Have you seen a bunch of humans in a room?

xyzal 1y ago
I see lots of humans in an echo chamber

airylizard 1y ago

Why I came up with TSCE(Two-Step Contextual Enrichment). +30pp uplift when using GPT-35-turbo on a mix of 300 tasks. Free open framework, check the repo try it yourself https://github.com/AutomationOptimization/tsce_demo https://github.com/AutomationOptimization/tsce_demo I tested this another 300 times with gpt-4.1 to remove those obtrusive "em-dashes" everyone hates. Tested a single-pass baseline vs TSCE, same exact instructions and prompt "Remove the em-dashes from my linkedin post. . .". Out of the 300 tests, baseline failed to remove the em-dashes 149/300 times. TSCE failed to remove the em-dashes 18/300 times. It works, all the data as well as the entire script used for testing is in the repo.

arnaudsm 1y ago
That's a lot of kilo-watt-hours wasted for a find and replace operation. Have you heard of text.replace("—", "-") ?
- airylizard 1y ago
  The test isn't for how well an LLM can find or replace a string. It's for how well it can carry out given instructions... Is that not obvious?
thegeomaster 1y ago
I slightly tweaked your baseline em dash example and got 100% success rate with GPT-4.1 without any additional calls, token spend, or technobabble. System prompt: "Remove every em-dash (—) from the following text while leaving other characters unchanged.\n\nReturn only the cleaned text." User prompt: <prompt from tsce_chat.py filled with em dashes> Temperature: 0.0
- deleted 1y ago
  [deleted]
- airylizard 1y ago
  Hey, thanks for kicking the tires! The run you’re describing was done in mid-April, right after GPT-4.1 went live. Since then OpenAI has refreshed the weights behind the “gpt-4.1” alias a couple of times, and one of those updates fixed the em-dash miss. If you reran today you’d see the same improved pass rate I’m getting now. That’s the downside of benchmarking against latest model names; behaviour changes quietly unless you pin to a dated snapshot. For bigger, noisier prompts (or on GPT-3.5-turbo, which hasn’t changed) TSCE still gives a solid uplift, so the framework’s value stands. Appreciate you checking it out!

dr_dshiv 1y ago

This is the best paper on machine psychology [1] I’ve yet seen. Rigorous, empirical, insightful — and very practical. [1] http://ui.adsabs.harvard.edu/abs/2023arXiv230313988H/abstract http://ui.adsabs.harvard.edu/abs/2023arXiv230313988H/abstrac...

jsemrau 1y ago

That's no surprise. When I was working on game theory and agent reasoning I reached the same conclusion a year ago. My conclusion was that context needs to be managed well for the LLMs to manage accuracy in replies. Also, it helps to have a planning process ("graph reasoning") before task execution because it guardrails the models thought process. This also introduces a discussion on general use vs workflow agent implementations as in the former it is much more difficult to generalize all components in structuring effective ReAct patterns.

veunes 1y ago
It's probably why workflow agents feel more reliable: they're built around structure, not just raw prediction
- jsemrau 1y ago
  Also you have more control points. It's not just a brain a vat.

jumploops 1y ago

It's amazing that branching/forking isn't a core aspect of the main chat tools. You can edit responses, sure, but then a bunch of other context is lost. My flow is basically: 1. plan 2. build 3. branch (into some feature/esoteric dependency issue) 4. goto #2 Prompt pruning/branching should be a first-class tool for any LLM usage.

Capricorn2481 1y ago
I've been kicking around making this for a while. BetterChatGPT at least has some good ergonomics around deleting history. But I agree that branching is the next step.
jampekka 1y ago
Google AI studio at least has this. I found at least that implementation quite confusing though, which may be a reason it's not implemented in more "consumer oriented" tools.

SamPatt 1y ago

I always felt the derision around the term "prompt engineering" was partially due to people overestimating the importance of the initial prompt and underestimating the importance of managing the ongoing context. You develop a knack for how to steer the models or start a new conversation through experience. The system or initial prompt are important, but nothing will save you if you naively keep a conversation going too long.

veunes 1y ago
Yeah, totally. Prompt engineering isn't just about crafting the perfect opener, it's more like conversation management. You start to develop a feel for when things are going off the rails and it's time to reset

veunes 1y ago

Kind of wild how even the best models still struggle with keeping context straight over time. Definitely feels like a big challenge if we want these things to hold real conversations.

guardiang 1y ago

Exactly why expert steering should be valued.

badmonster 1y ago

Why do LLMs struggle so much with recovering from early wrong turns in multi-turn conversations — even when all prior context is available and tokenized? Is it due to the model's training distribution (mostly single-shot completions), the way context windows are encoded, or an architectural bottleneck? Feels like there's no dynamic internal state that evolves over the conversation — only a repeated re-parsing of static history. Has anyone seen work on integrating memory/state mechanisms that allow belief revision within a session, not just regurgitation of past tokens?

JohnKemeny 1y ago
We shouldn’t anthropomorphize LLMs—they don’t “struggle.” A better framing is: why is the most likely next token, given the prior context, one that reinforces the earlier wrong turn?
bandrami 1y ago
Because Markov chains propagate forward in time
vjerancrnjak 1y ago
Imagine optimizing/training on a happy path. When you generate future tokens, you're looking at history tokens that are happy. So how can a model, given sad tokens, generate future happy tokens if it did not learn to do so? The work you're looking for is already here, it's "thinking". I assume they include sad tokens in the dataset, produce "thinking", which should result in happy tokens coming after thinking tokens. If thinking is bad (by looking at following happy tokens), then it's punished, if good, then descent.
mountainriver 1y ago
It’s a problem specific to autoregressive LLMs, the early tokens bias the output

podgorniy 1y ago

There is a noticable issue when one builds LLMs interfaces around single turn conversations. Majority people expect linear conversations. I've built telegram bot http://t.me/experai_bot http://t.me/experai_bot as univresal UI to LLMs (with somewhat reduced functionality) exactly around idea "non-reply message means new conversation". Wanna keep context? Keep replying to replies of bot. Non-power user strugge with this idea. -- Also I observed that OpenAI models performed worse replying to the same questions (for example list of options in reply got shorter) even with smallest system message. That was the case with 3.5, 4o. Don't know how modern ones behave. That made me decide not to include any system messages by default Still I give option to add ones if you need. You can even toggle them to mix-and-match.

sky2224 1y ago

Ha, kind of funny to see this right now. I've been fighting copilot in vscode in trying to get it to output anything once I take the context down to a very specific problem. It feels like I have to reset and almost reground the model into what I'm trying to accomplish at a certain point.

debuggerson 1y ago

The more we chat, the more irrelevant details pile up. For example, a small mention early on might get repeated or build on itself, leading to a lot of unnecessary context. As the conversation continues, it becomes harder for the model to focus on the main point because it gets tangled in all the extra information. Unlike humans, who can intuitively filter out the noise, LLMs struggle to keep track of what’s truly important in longer, more complex exchanges.

deleted 1y ago

[deleted]

overflow897 1y ago

I believe we're already using llms to evaluate llm output for training, I wonder if there's some variation of that which could be used to identify when one llm gets "stuck". I guess chain of thought in theory should do that but having variations on prompt and context might behave differently?

t-kalinowski 1y ago

This was the main reason I wrote promptdown. I want to be able to edit the full chat history every turn, and the append-only standard chat interfaces don't make that easy. https://github.com/t-kalinowski/promptdown https://github.com/t-kalinowski/promptdown

tmountain 1y ago

I often ask the LLM for a concise summary of the discussion so far—formatted as a prompt. I then edit it appropriately and use it to start a new conversation without the baggage. I have found this to be a very effective technique, but I imagine it will be automated sometime soon.

Drunkfoowl 1y ago
[dead]
maleldil 1y ago
Claude Code has a /compact command that summarises the conversation so far to save on context tokens.
drewbitt 1y ago
Cursor tried doing this automatically - it may still if you're not on a large context model like gemini 2.5 pro - but I found the summary was just missing too many details to use out of the box.

aleksituk 1y ago

This is very interesting and I like the conversation about not only the technology itself, but also about the importance of thinking about the interface as a user experience and where / how it fits the paradigm. We've been working on a lot of data processing and generation tasks. We've been doing this using an API primarily, but sometimes I end up testing creating data in a chat window and I first chat through what the requirements are for the data analysis / processing and then once I'm done I would like the whole conversation to be then summarised into basically a one-prompt process so that I can re-use it (because I can't really process new inputs via the chat). Even when you do manage to get it down to a single prompt you can use in a chat and then ask the chat to just keep producing new data (like imagine a blog post in certain style if the base content is given as input and I'm making like 20 of them). If you produce these in the chat, there's notable benefits in that if something is wrong with the blog post the chat suggests, you can immediately edit it. The trouble is that the context window starts becoming so big that the chat starts to forget what the original instruction is and eventually you do have to just create a new chat. One way to solve for this is having a chat with selective memory where you keep a task in memory, but you have the chat forget/not-include all the generated data in the context so that it stays clean, but only bring it to the context if the user refers to it. Has anyone else done data processing types of tasks in chats and had issues like this? Are there some other tools to use or tricks to do in chats?

WhitneyLand 1y ago

Any reason to not call bullshit on this paper? One of the biggest developments in language models over the last year has been test-time reasoning (aka inference scaling or “thinking”). Most vendors tested offer such a model. It’s plausible it could make a huge difference here, and they did not bother to test it or even mention it? Things like COT and planning can really affect this and those are just a couple of things that happen automatically in more advanced models. Seems like it wouldn’t have been hard to add this to the experiment, but they could’ve called it out in a “Limitations” or “Future Work” section. Or at least a single sentence like “We did not test chain-of-thought prompting, which may mitigate some of these issues”.

giordanol 1y ago

Would love to see metrics that isolate recovery behaviour (if any)

curtisszmania 1y ago

[dead]

Workaccount2 1y ago

Reminds me of Claude plays pokemon, where it would note something insignificant, and then fixate on it for hours.

Zobat 1y ago

This must mean that LLMs really are like genies in bottles. You get three questions answered, anything after that will be nonsense.

sattard 1y ago

Why haven't AI code editors not built this at their core yet, to automatically consolidate previous conversational turns into a more structured context summary. Instead of relying solely on the model’s memory of all prior exchanges, surely these tools should take responsibility for intermittently “restating” the clarified requirements so the model doesn’t have to reconstruct context from scratch (or worse, pick up mistakes). This might mitigate compounding errors and reduce verbosity.

RandyOrion 1y ago

One problem with this paper is that authors didn't conduct experiments on popular LLMs from Qwen and Mistral. Why?