4 ms·

How do I use it with ollama models?

• MKuykendall 1y ago
  To use Shimmy (instead of Ollama): 1. Install Shimmy: cargo install shimmy 2. Get GGUF models (same models you'd use with Ollama): # Download to ./models/ directory huggingface-cli download microsoft/Phi-3-mini-4k-instruct-gguf --local-dir ./models/ # Or use existing Ollama models from ~/.ollama/models/ 3. Start serving: ./shimmy serve 4. Use with any OpenAI-compatible client at http://localhost:11435
  • sunscream89 1y ago
    I am trying to use ~/.ollama/models/, even linked it to ~/models. I don’t have phi-3, it may be possible none of my models are supported. It acts as though it sees nothing. How do I know for sure it is checking ~/.ollama/models/ (if linking isn’t the right approach.)
    • MKuykendall 1y ago
      I didn't have that path set to autodiscover; pull the newest version this is fixed now!!