4 ms·

Thanks for the tip with ollama

• behnamoh 3y ago
  ollama is actually not a great way to run these models as it makes it difficult to change server parameters and doesn't use `mlock` to keep the models in memory.
  • bravura 3y ago
    What do you suggest?
    • behnamoh 3y ago
      vanilla llama.cpp (run a `/server`)
• discordance 3y ago
  If you do: 1. ollama run deepseek-coder:6.7b 2. pip install litellm 3. litellm --model deepseek-coder:6.7b You will have a local OpenAI compatible API for it.