4 ms·
Is it possible to roll back the output and insert essentially into the previous line of the LLM output? But not losing what they're writing now. Here's the flo
by IanCal 3y ago
Is it possible to roll back the output and insert essentially into the previous line of the LLM output? But not losing what they're writing now.
Here's the flow I've been picturing while writing code writing models. Feel free to take this idea anyone, would appreciate a heads up if it works and you publish sota before I get around to it :)
Let's give llms ide level info. As they type, if there's a function they're starting to call use a language server to get tooltip docs and put it in a comment just above the line it's writing. Put auto complete help, type docs, etc in their context while they code.
Edit
Second thought, this kind of thing feels very low in compute compared to the LLM calculations, is this the kind of thing that could be passed up to a remote service as a wasm bundle/similar, to control the streaming output?
- lbeurerkellner 3y agoIf I understand correctly what you are imagining is some sort of local retrieval enhancement to generate a specific part of the overall response, which later is removed, once e.g. the generated piece of code (e.g. a function call) has completed generation. Indeed, this is a form of LLM prompting that we are also exploring in our preview release channel with something called in-context functions. See https://next.lmql.ai https://next.lmql.ai and choose the "In-Context Functions" example in the New Features showcase screen. With in-context functions, you can provide additional instructions/data to the LLM, that will apply locally only. Once such an in-context function returns, additional instructions/retrieved info is removed and only the LLM-generated end-result remains.