4 ms·
That’s the point I’m trying to make though, you’re not using it wrong in the sense like your application is wrong or you’re doing something dumb. By “wrong” I m
by syntaxing 3y ago
That’s the point I’m trying to make though, you’re not using it wrong in the sense like your application is wrong or you’re doing something dumb. By “wrong” I mean the structure you’re sending it is off. Like some local LLM use <INST></INST>, some use USER, Some use HUMAN. Miss a \n for a context and your results is garbage. That’s why I recommend using ollama instead of llama cpp directly because I have not been able to find a reliable way to define this. When you use llama cpp and just send a prompt, by default it directly sends what you send to llama cpp. Ollama has a layer that abstracts this away.
Please give Ollama a go! Would love to hear if it works out! Feel free to contact my email in my profile if you need some help.
- wokwokwok 3y agoEvery model on hugging face defines the input context. For mistral it is "<s>[INST] ... [/INST]" It's pretty obvious if you're writing prompts you have to use the correct prompt syntax. ? ollama seems unrelated to the problems I'm having. There's no way you can define an arbitrary mapping between prompt formats where some have eg. SYSTEM and some don't. It's simply not possible. You have to update your prompts for different models. I keep a separate list of prompts for each model. It's no big deal.
- v3ss0n 3y agoOllama have prompts properly defined for you in the library.but back to OP that is the problem I am facing too even with ollama.