4 ms·
Local LLM support? (Llama/Ollama)
by smcleod 3y ago
Local LLM support? (Llama/Ollama)
- okwhateverdude 3y agoOllama folks just announced api compat with openai's stuff https://ollama.ai/blog/openai-compatibility https://ollama.ai/blog/openai-compatibility So apparently, yeah.
- smcleod 3y agoWould be nice if rather than adhering to a specific very-closed source company first things were developed for standards, or at least used something like litellm.
- kevinlu1248 3y agoIt probably makes it easier for many companies to move off of OpenAI since they won’t need to drastically alter their codebase.
- kevinlu1248 3y agoWe have tried many of the current open source models but unfortunately the only model whose capability is close to GPT-4 is Deepseek and unfortunately Deepseek can’t follow our specified format and is very sensitive to prompt changes.
- kevinlu1248 3y agoThe other problem is with latency: Deepseek 34B on A100s seem slower than GPT-4 but perhaps it will be better on H100s.