4 ms·
Running it locally lets you INTERJECT IN IT'S THINKING IN REALTIME and I cannot stress enough how useful that is.
by thot_experiment 2y ago
Running it locally lets you INTERJECT IN IT'S THINKING IN REALTIME and I cannot stress enough how useful that is.
- amarcheschi 2y agoOh this is so cool
- Gooblebrai 2y agoYou mean it reacts to you writing something while it's thinking of that you can stop it while it's thinking?
- hmottestad 2y agoYou can stop it at any time, then modify what it's written so far...then press continue and let it continue thinking and answering.
- thot_experiment 2y agoFundamentally the UI is up to you, I have a "typing-pauses-inference-and-starts-gaslighting" feature in my homebrew frontend, but in OpenWebUI/Sillytavern you can just pause it and edit the chain of thought and then have it continue from the edit.
- Gracana 2y agoThat's a great idea. In your frontend, do you write in the same text entry field as the bot? I use oobabooga/text-generation-webui and I findit's a little awkward to edit the bot responses.
- thot_experiment 2y agoNo, but the chat divs are all contenteditable.
- Gracana 2y agoOh! That is an excellent solution. I wish it was that easy in every UI.
- thot_experiment 2y agoThanks, for what it's worth unless you particularly need to use exl2 ollama works great for local inference and you can prompt together a half decent chat UI for yourself in a matter of minutes these days which gives you full control over everything. I also lean a lot on https://www.npmjs.com/package/amallo https://www.npmjs.com/package/amallo which is a api wrapper i wrote for ollama which makes this sort of hacking very very easy. (not that the default lib is bad, i just didn't like the ergonomics)
- thenameless7741 2y agoInteresting.. In the official API [1], there's no way to prefill the reasoning_content: > Please note that if the reasoning_content field is included in the sequence of input messages, the API will return a 400 error. Therefore, you should remove the reasoning_content field from the API response before making the API request So the best I can do is pass the reasoning as part of the context (which means starting over from the beginning). [1] https://api-docs.deepseek.com/guides/reasoning_model https://api-docs.deepseek.com/guides/reasoning_model
- arresin 2y agoHow are you running it locally??
- thot_experiment 2y agoI am running a 4bit imatrix quant of the 70b distill with quantized context. It fits in the 43gb of vram I have.