4 ms·

A mistake I see people repeating over and over, is never restarting their conversations with a edited initial message. Instead of doing what the author is doin

by diggan 2y ago

A mistake I see people repeating over and over, is never restarting their conversations with a edited initial message. Instead of doing what the author is doing here, and sending messages back and forward, leading to a longer and longer conversation, where each messages leads to worse and worse quality replies, until the LLM seems like a dumb rock, rewrite your initial message with everything that went wrong/was misunderstood, and aim to have whatever you want solved in the first message, and you'll get a lot higher quality answers. If the LLM misunderstood, don't reply "No, what I mean was..." but instead rewrite the first message so it's clearer. This is at least true for all ChatGPT, Claude and DeepSeek models, YMMV with other models.

bpodgursky 2y ago
100% To be honest, this would help a lot of person-implemented iteration too, if it was biologically feasible to erase a conversation from a brain.
- dingnuts 2y ago
  alright, time for you to go watch Eternal Sunshine of the Spotless Mind so that you can disabuse yourself of that notion
swatcoder 2y ago
Yup. Inasmuch as these are collaborative document generators at their core, "minimally ambiguous prompt and conforming reply" is a strongly represented document structure and so we benefit by setting them up to complete one. Likewise, "tragi-comic dialog between increasingly frustrated instructor and bumbling pupil" is also a widely represented document structure that we benefit by trying to avoid. Chatbot training works to minimize the chance of an LLM engaging in the latter, because dialog is a intuitive interface that users enjoy, but we can avoid the problem more successfully by just providing a new and less ambiguous prompt in a new session, as you suggest.
- dartos 2y ago
  > dialog is a intuitive interface that users enjoy Do people enjoy chat interfaces in their workflows? I always thought that cursor/copilot/copy.ai/v0.dev were so popular because they break away from the chat UI. Dialog is cool when exploring but, imo, really painful when trying to accomplish a task. LLMs are far too slow to make a real fluid conversation.
01100011 2y ago
I tried this approach when attempting to get Deepseek-r1 and GrokV3 to create a simple CUDA application. It was necessary because the iterative approach kept leading to hangs and divergent behaviors. I still wasn't able to get a working application, however.
yuvalr1 2y ago
This means the leading UI for LLMs - the chat - is the wrong UI, at least for some of the tasks. We should instead have a single query text field, like in search engines, that you continue to edit and refine, just like in complex search queries.
- sgillen 2y ago
  I still think there is value in chats and retaining context. But there is also value in starting clean when necessary. Giving users control and teaching people how to use it is the way IMO.
  - nomel 2y ago
    The problem with retaining context is that it gets polluted. That pollution gets you into a latent space with errors, which probably not where you want your next token prediction to be sourced. The reasonable alternative is a chat interface that lets you edit any text, the AI response or your prompts, and regenerate from any point. This is why I use the API "playground" interfaces or something like LibreChat. Deepseek at least has prompt editing/regeneration.
- freehorse 2y ago
  I like zed's approach, where the whole discussion is a plain text file you can edit as any text, which gives you the ability to change anything in the "discussion" regardless if it was generated by you or the llm. It makes stuff like that much simpler, ie you can correct simple stuff in the llm's response without unecessary back and forths, you can just cut parts out of the discussion to reduce context size or guide the discussion where you actually want removing distractions etc. I don't understand why the dominant approach is an actual, realistic chat interface where you can only add a new response, or in best case create "threads".
  - diggan 2y ago
    > I don't understand why the dominant approach is an actual, realistic chat interface where you can only add a new response, or in best case create "threads". I'm not 100% sure either, I think it might just be a first-iteration UX that is generally useful, but not specifically useful for use cases like coding. To kind of work around this, I generally keep my prompts as .md files on disk, treat them like templates where I have variables like $SRC that gets replaced with the actual code when I "compile" them. So I write a prompt, paste it into ChatGPT, notice something is wrong, edit my template on disk then paste it into a new conversation. Iterate until it works. I ended up putting the CLI I use for this here, in case others wanna try the same approach: https://github.com/victorb/prompta https://github.com/victorb/prompta
chatmasta 2y ago
That’s too much work. I’d rather ask the LLM to rewrite my first message for me. And the UI should then give me an option to “start new chat from suggested prompt.”
- diggan 2y ago
  > I’d rather ask the LLM to rewrite my first message for me I guess you can do that too, as long as you start a new conversation afterwards. Personally I found it much easier to keep prompts in .md files on disk, and paste them into the various interfaces when needed, and then I iterate on my local files if I notice the first answer misunderstood/got something wrong. Also lets you compose prompts which is useful if you deal with many different languages/technologies and so on.
danenania 2y ago
I built Plandex[1], an open source AI coding agent, partly to enable this workflow. It has `log` and `rewind` commands that allow you to easily back up to any previous point in the conversation and start again from there with an updated prompt. Plandex also has branches, which can be helpful for not losing history when using this approach. You’re right that it’s often a way to get superior results. Having mistakes or bad output in the conversation history tends to beget more mistakes and bad output, even if you are specifically directing the LLM to fix those things. Trial and error with a new prompt and clean context avoids this problem. 1 - https://plandex.ai https://plandex.ai P.S. I wrote a bit about the pros and cons of this approach vs. continuing to prompt iteratively in Plandex’s docs here: https://docs.plandex.ai/core-concepts/prompts#which-is-better https://docs.plandex.ai/core-concepts/prompts#which-is-bette...
th0ma5 2y ago
In my experience this only marginally improves things. It constantly offers new ways to be wrong.
godelski 2y ago
It seems like the author in fact did do this. They asked Claud the same message. I really doubt they repeated the entire conversation to get to that point, but I may be wrong. From personal experience, I agree with you, but I wouldn't make the critique here as it is far from a magic bullet. Honestly, with the first stuff it seems faster to learn mermaid and implement it yourself. Mermaid can be learned in a rather short time, the basic syntax is fairly trivial and essentially obvious. As an added benefit, you then get to have this knowledge and use it later on. This will certainly feel slower than the iterative back and forth with a LLM -- either by follow-up conversations or refining your one shot -- but I'm not convinced it will be a huge difference in time as measured by the clock on the wall[0] [0] idk, going back and forth with an LLM and refining my initial messages feels slow to me. It reminds me of print statement debugging in a compiled language. Lots of empty time.
- diggan 2y ago
  > It seems like the author in fact did do this. It doesn't seem like that to me. At one point in the article: "There are also a few issues [...] Let’s fix with the prompt" and then a prompt that is referring the previous message. Almost all prompts after that seem to depend on the context before them. My point is that instead of doing that, revise the original initially message so the very first response from the LLM doesn't contain any errors, because (in my experience) that's way easier and faster than trying to correct errors by adding more messages, since they all (even O1 Pro) seem to lose track of what's important in the conversation really fast.
  - godelski 2y ago
    I'm just saying I don't think they repeated that whole process for Claud.
throwaway519 2y ago
Markov Chain system doesn't like Markov Chain input.
barnas2 2y ago
I've also started adding "Ask any questions you think are relevant before starting" to the end of my prompts. It usually results in at least one question that addresses something I didn't think to add to my prompt.
- kordlessagain 2y ago
  I’ve been saying “stop writing code until we agree what needs to be done”.
kordlessagain 2y ago
I love Claude, but whomever works on their UI needs to be slapped a bit. Code output covering the stop button on my laptop, page lockups on iPhone/Chrome with certain artifacts (even after reload), crazy slow typing on the computer and refusal to “continue” chat with a cheaper model. Simply providing a summary of the chat on running out of tokens would let me start another conversation, or at least a warning I was getting close.

by diggan 2y ago

bpodgursky 2y ago

100% To be honest, this would help a lot of person-implemented iteration too, if it was biologically feasible to erase a conversation from a brain.

dingnuts 2y ago
alright, time for you to go watch Eternal Sunshine of the Spotless Mind so that you can disabuse yourself of that notion

swatcoder 2y ago

Yup. Inasmuch as these are collaborative document generators at their core, "minimally ambiguous prompt and conforming reply" is a strongly represented document structure and so we benefit by setting them up to complete one. Likewise, "tragi-comic dialog between increasingly frustrated instructor and bumbling pupil" is also a widely represented document structure that we benefit by trying to avoid. Chatbot training works to minimize the chance of an LLM engaging in the latter, because dialog is a intuitive interface that users enjoy, but we can avoid the problem more successfully by just providing a new and less ambiguous prompt in a new session, as you suggest.

dartos 2y ago
> dialog is a intuitive interface that users enjoy Do people enjoy chat interfaces in their workflows? I always thought that cursor/copilot/copy.ai/v0.dev were so popular because they break away from the chat UI. Dialog is cool when exploring but, imo, really painful when trying to accomplish a task. LLMs are far too slow to make a real fluid conversation.

01100011 2y ago

I tried this approach when attempting to get Deepseek-r1 and GrokV3 to create a simple CUDA application. It was necessary because the iterative approach kept leading to hangs and divergent behaviors. I still wasn't able to get a working application, however.

yuvalr1 2y ago

This means the leading UI for LLMs - the chat - is the wrong UI, at least for some of the tasks. We should instead have a single query text field, like in search engines, that you continue to edit and refine, just like in complex search queries.

sgillen 2y ago
I still think there is value in chats and retaining context. But there is also value in starting clean when necessary. Giving users control and teaching people how to use it is the way IMO.
- nomel 2y ago
  The problem with retaining context is that it gets polluted. That pollution gets you into a latent space with errors, which probably not where you want your next token prediction to be sourced. The reasonable alternative is a chat interface that lets you edit any text, the AI response or your prompts, and regenerate from any point. This is why I use the API "playground" interfaces or something like LibreChat. Deepseek at least has prompt editing/regeneration.
freehorse 2y ago
I like zed's approach, where the whole discussion is a plain text file you can edit as any text, which gives you the ability to change anything in the "discussion" regardless if it was generated by you or the llm. It makes stuff like that much simpler, ie you can correct simple stuff in the llm's response without unecessary back and forths, you can just cut parts out of the discussion to reduce context size or guide the discussion where you actually want removing distractions etc. I don't understand why the dominant approach is an actual, realistic chat interface where you can only add a new response, or in best case create "threads".
- diggan 2y ago
  > I don't understand why the dominant approach is an actual, realistic chat interface where you can only add a new response, or in best case create "threads". I'm not 100% sure either, I think it might just be a first-iteration UX that is generally useful, but not specifically useful for use cases like coding. To kind of work around this, I generally keep my prompts as .md files on disk, treat them like templates where I have variables like $SRC that gets replaced with the actual code when I "compile" them. So I write a prompt, paste it into ChatGPT, notice something is wrong, edit my template on disk then paste it into a new conversation. Iterate until it works. I ended up putting the CLI I use for this here, in case others wanna try the same approach: https://github.com/victorb/prompta https://github.com/victorb/prompta

chatmasta 2y ago

That’s too much work. I’d rather ask the LLM to rewrite my first message for me. And the UI should then give me an option to “start new chat from suggested prompt.”

diggan 2y ago
> I’d rather ask the LLM to rewrite my first message for me I guess you can do that too, as long as you start a new conversation afterwards. Personally I found it much easier to keep prompts in .md files on disk, and paste them into the various interfaces when needed, and then I iterate on my local files if I notice the first answer misunderstood/got something wrong. Also lets you compose prompts which is useful if you deal with many different languages/technologies and so on.

danenania 2y ago

I built Plandex[1], an open source AI coding agent, partly to enable this workflow. It has `log` and `rewind` commands that allow you to easily back up to any previous point in the conversation and start again from there with an updated prompt. Plandex also has branches, which can be helpful for not losing history when using this approach. You’re right that it’s often a way to get superior results. Having mistakes or bad output in the conversation history tends to beget more mistakes and bad output, even if you are specifically directing the LLM to fix those things. Trial and error with a new prompt and clean context avoids this problem. 1 - https://plandex.ai https://plandex.ai P.S. I wrote a bit about the pros and cons of this approach vs. continuing to prompt iteratively in Plandex’s docs here: https://docs.plandex.ai/core-concepts/prompts#which-is-better https://docs.plandex.ai/core-concepts/prompts#which-is-bette...

th0ma5 2y ago

In my experience this only marginally improves things. It constantly offers new ways to be wrong.

godelski 2y ago

It seems like the author in fact did do this. They asked Claud the same message. I really doubt they repeated the entire conversation to get to that point, but I may be wrong. From personal experience, I agree with you, but I wouldn't make the critique here as it is far from a magic bullet. Honestly, with the first stuff it seems faster to learn mermaid and implement it yourself. Mermaid can be learned in a rather short time, the basic syntax is fairly trivial and essentially obvious. As an added benefit, you then get to have this knowledge and use it later on. This will certainly feel slower than the iterative back and forth with a LLM -- either by follow-up conversations or refining your one shot -- but I'm not convinced it will be a huge difference in time as measured by the clock on the wall[0] [0] idk, going back and forth with an LLM and refining my initial messages feels slow to me. It reminds me of print statement debugging in a compiled language. Lots of empty time.

diggan 2y ago
> It seems like the author in fact did do this. It doesn't seem like that to me. At one point in the article: "There are also a few issues [...] Let’s fix with the prompt" and then a prompt that is referring the previous message. Almost all prompts after that seem to depend on the context before them. My point is that instead of doing that, revise the original initially message so the very first response from the LLM doesn't contain any errors, because (in my experience) that's way easier and faster than trying to correct errors by adding more messages, since they all (even O1 Pro) seem to lose track of what's important in the conversation really fast.
- godelski 2y ago
  I'm just saying I don't think they repeated that whole process for Claud.

throwaway519 2y ago

Markov Chain system doesn't like Markov Chain input.

barnas2 2y ago

I've also started adding "Ask any questions you think are relevant before starting" to the end of my prompts. It usually results in at least one question that addresses something I didn't think to add to my prompt.

kordlessagain 2y ago
I’ve been saying “stop writing code until we agree what needs to be done”.

kordlessagain 2y ago

I love Claude, but whomever works on their UI needs to be slapped a bit. Code output covering the stop button on my laptop, page lockups on iPhone/Chrome with certain artifacts (even after reload), crazy slow typing on the computer and refusal to “continue” chat with a cheaper model. Simply providing a summary of the chat on running out of tokens would let me start another conversation, or at least a warning I was getting close.