3 ms·
Anyone able to get llama.cpp into some sort chat-bot style mode via a script or similar? What do you need to wrap around it to get it to act that way? Right n
by novok 4y ago
Anyone able to get llama.cpp into some sort chat-bot style mode via a script or similar? What do you need to wrap around it to get it to act that way? Right now it will just go on forever in response to prompts.
- leobg 4y ago“Chat mode” just means that 1) you structure the prompt given to the model for completion as a chat, i.e. a list of lives in the form “NAME: UTTERANCE” and 2) specify as “end” sequence the “NAME:” part for the user (i.e. prevent the model from going on in both roles). On top of that, you may want to do fine-tuning with reinforcement learning. But the above is “all you need” to get into chat mode with a LLM (or any text generation model, really).
- mk_stjames 4y agoGeorge Hotz did a stream yesterday building with Llama and got some good functionality. It's done with some few-shot pre-prompting to give it understanding of 'who it is' and how to answer questions: https://www.youtube.com/watch?v=nctqc8FBJ2U https://www.youtube.com/watch?v=nctqc8FBJ2U