37 ms·
To use Shimmy (instead of Ollama): 1. Install Shimmy: cargo install shimmy 2. Get GGUF models (same models you'd use with Ollama): # Download to ./mode
by MKuykendall 1y ago
To use Shimmy (instead of Ollama):
1. Install Shimmy:
cargo install shimmy
2. Get GGUF models (same models you'd use with Ollama):
# Download to ./models/ directory
huggingface-cli download microsoft/Phi-3-mini-4k-instruct-gguf --local-dir
./models/
# Or use existing Ollama models from ~/.ollama/models/
3. Start serving:
./shimmy serve
4. Use with any OpenAI-compatible client at http://localhost:11435
- sunscream89 1y agoI am trying to use ~/.ollama/models/, even linked it to ~/models. I don’t have phi-3, it may be possible none of my models are supported. It acts as though it sees nothing. How do I know for sure it is checking ~/.ollama/models/ (if linking isn’t the right approach.)
- MKuykendall 1y agoI didn't have that path set to autodiscover; pull the newest version this is fixed now!!