3 ms·
I find it interesting that no one has created a better interface for these LLMs. Two things that should be available by now in conversations are branching and
by sureglymop 1y ago
I find it interesting that no one has created a better interface for these LLMs.
Two things that should be available by now in conversations are branching and history editing. Branching is somewhat trivial so let's focus on the history.
Now when I last used an LLM API, the whole context was still managed manually. Meaning I had to send the API the whole conversation history as one long text for every new query.
This means that I could technically change a part of the history directly. Manipulating the history though is not really a trivial problem. The LLM would need to re evaluate starting from that point.
But, the re-evaluation may result in something completely different... If there are branches, perhaps it would also be desirable to let it propagate into the branches.
Next, re-evaluate until where? We can assume a conversation happened until the present moment and the user may have changed their reality/state during the conversation before that point. For example, I may have changed some function based on a suggestion of the LLM. Now, for re-evaluation it would actually be nice if the LLM could also take that state change into consideration.
Here it would be nice if the LLM had the concept of certain logical facts and pieces of information and how they relate to each other but with an interface so that we could see that. If such a piece of information in the conversation is then changed, that would affect the information that is related to it. We could follow a sort of sequence of logical conclusions being made to verify what happened.
Just some thoughts with no conclusion. I think current LLM interfaces could be a lot better.
- dmd 1y agoI'm confused by what you mean by this, because most LLM interfaces other than the very very basic ones do already have branching and history editing? E.g. pretty much every third party LLM interface? Are you talking about something more than that?
- sureglymop 1y agoYes. Here is a scenario: Let's say you ask an LLM to help you with a programming task. The LLM gives you a solution and you copy that into your code. While getting to the result of this solution, spanned over multiple messages, you give it more context about your code base. You ask it to help with multiple sub problems and it answers with suitable responses containing code snippets. Now, you change your mind and want to actually write your program in a different language. You could at this point just tell that to the LLM and it would change course from here on out. Everytime it is queried the full history including the original problem is sent to it. The change in programming language is taken into account because it eventually gets to the point where you changed your mind. But instead what you'd probably want is the solution it gave you initially and all the little solutions to all the sub problems but in the newly chosen programming language! You'd like to simulate or re-evaluate the conversation but with the newly chosen programming language. I haven't seen that implemented in a usable way but as I said, this is not a trivial feature either.
- mattnewton 1y agoCheckout cursor. It doesn't have the old branch "rebasing" you are describing, but going back into the history reverts changes made after that point by default. (And honestly I am not convinced automatically rebasing that way is better than rerunning the prompts with changes in most cases, it seems too likely to go off the rails to me)
- MangoToupe 1y agoYou'd also want to have some mechanism to ensure deterministic generation if you add rebasing, I think? Though I might be behind the times in my understanding of how LLMs work.
- hombre_fatal 1y agoNeither Claude, OpenAI, nor Gemini let you do either, so it's not ubiquitous. You can't delete messages, you can't branch, and you can't edit the LLM's messages. These are basic, obvious, day-1 features I implemented in my own LLM UI that I built over 3 years ago, yet I'm still waiting for flagship providers to implement them. It's pathetic how rare good UX is in software. Can you list some UIs with these features?
- jstanley 1y agoI think you may be confused about how LLMs work. Editing the history you send to the API works perfectly fine. You don't need the LLM to specifically "re-evaluate" the history from the point you edited. In a way it is "re-evaluating" the entire history for every token it generates.
- sureglymop 1y agoWhat I'm saying is: If, earlier in the history, I change something, that thing may also still be used further down in the history which I specifically don't want. So yes, I know that I can just do a simple text replacement somewhere in the history but that's not really useful. I want the conversation to be re-evaluated from that point because it might have gone differently with that changed message.
- haiku2077 1y agoI can do this with my assistants in Zed and Kagi just fine? Did a quick test and it works exactly how you describe.
- sureglymop 1y agoNo. When you edit one of your queries in Kagi Assistant, any queries you made in the conversation after that are discarded. Again, I don't just mean simple re-evaluation of a single query.
- haiku2077 1y agoMy previous queries aren't discarded. I can flip between them like branches in a conversation tree.
- justinrubek 1y agoPrecisely. They are discarded from the new query.
- 1y ago
- amelius 1y agoOne simple thing I'm missing is a way to point at a word, and let the LLM know that that's where it went wrong (so it can generate an alternative output).
- sureglymop 1y agoYup, that's exactly what I mean!
- blharr 1y agoLM studio allows you to edit the context of an existing message like this, and you can even insert whatever writing to continue a thought, then have it continue generating.
- quantadev 1y agoThe first chatbot I wrote back in early 2023, had exactly this (branching and history editing), and I was highly active on the OpenAI discord showing them it, and trying to influence them, to go in this direction. Tree-based editors can be hard to implement, and can be confusing to users, so that's one reason GUIs tend to not attempt it, and just go with linear (although often nowadays "editable") chats. Related to this I've learned that a lot of the time it's best to just use a "Context File" which is a big file of instructions explaining goals, architecture, and constraints so I can just tell the AI to read it and then say "Do Step 21 Now" for example, if step 21 is laid out in the file. This way I sort of can micromanage the context, rather than trying to reuse old discussions which are usually 90% throw away content, no longer appliable.
- RossBencina 1y ago> Two things that should be available by now in conversations are branching and history editing. Prapti can do editable history no problem, the entire history is just the markdown file. We experimented with branching using git to automatically switch branches based on how you edit the history. I still use it frequently. I prefer keeping my chats in local markdown files. https://github.com/prapti-ai/prapti/ https://github.com/prapti-ai/prapti/