3 ms·

I’ve seen on the localllama subreddit that some GGUFs have bugs in them. The one recommended was by unsloth. However, I don’t know how the Ollama GGUF holds up.

compumetrika 2y ago
Ollama can pull directly from HF, you just provide the URL and add to the end :Q8_0 (or whatever) to specify your desired quant. Bonus: use the short form url of `hf` instead of `huggingface` to shorten the model name a little in the ollama list table. Edit: so for example of you want the unsloth "debugged" version of Phi4, you would run: `$ollama pull hf.co/unsloth/phi-4-GGUF:Q8_0` (check on the right side of the hf.co/unsloth/phi-4-GGUF page for the available quants)
- jimmySixDOF 2y ago
  You still need to make sure the modelfile works so this method will not run out of the box on a vision GGUF or anything with special schemas. Thats why mostly a good idea to pull from ollama directly.
- wruza 2y ago
  Is it true that non-gguf models are basically all Q4-equivalent? I'm always not sure which one to download to get the "default score".
magicalhippo 2y ago
Here's[1] a recent submission on that. [1]: https://news.ycombinator.com/item?id=42660335 https://news.ycombinator.com/item?id=42660335 Phi-4 Bug Fixes
jmorgan 2y ago
Phi-4's architecture changed slightly from Phi-3.5 (it no longer uses a sliding window of 2,048 tokens [1]), causing a change in the hyperparameters (and ultimately an error at inference time for some published GGUF files on Hugging Face, since the same architecture name/identifier was re-used between the two models). For the Phi-4 uploaded to Ollama, the hyperparameters were set to avoid the error. The error should stop occurring in the next version of Ollama [2] for imported GGUF files as well In retrospect, a new architecture name should probably have been used entirely, instead of re-using "phi3". [1] https://arxiv.org/html/2412.08905v1 https://arxiv.org/html/2412.08905v1 [2] https://github.com/ollama/ollama/releases/tag/v0.5.5 https://github.com/ollama/ollama/releases/tag/v0.5.5

compumetrika 2y ago

Ollama can pull directly from HF, you just provide the URL and add to the end :Q8_0 (or whatever) to specify your desired quant. Bonus: use the short form url of `hf` instead of `huggingface` to shorten the model name a little in the ollama list table. Edit: so for example of you want the unsloth "debugged" version of Phi4, you would run: `$ollama pull hf.co/unsloth/phi-4-GGUF:Q8_0` (check on the right side of the hf.co/unsloth/phi-4-GGUF page for the available quants)

jimmySixDOF 2y ago
You still need to make sure the modelfile works so this method will not run out of the box on a vision GGUF or anything with special schemas. Thats why mostly a good idea to pull from ollama directly.
wruza 2y ago
Is it true that non-gguf models are basically all Q4-equivalent? I'm always not sure which one to download to get the "default score".

magicalhippo 2y ago

Here's[1] a recent submission on that. [1]: https://news.ycombinator.com/item?id=42660335 https://news.ycombinator.com/item?id=42660335 Phi-4 Bug Fixes

jmorgan 2y ago

Phi-4's architecture changed slightly from Phi-3.5 (it no longer uses a sliding window of 2,048 tokens [1]), causing a change in the hyperparameters (and ultimately an error at inference time for some published GGUF files on Hugging Face, since the same architecture name/identifier was re-used between the two models). For the Phi-4 uploaded to Ollama, the hyperparameters were set to avoid the error. The error should stop occurring in the next version of Ollama [2] for imported GGUF files as well In retrospect, a new architecture name should probably have been used entirely, instead of re-using "phi3". [1] https://arxiv.org/html/2412.08905v1 https://arxiv.org/html/2412.08905v1 [2] https://github.com/ollama/ollama/releases/tag/v0.5.5 https://github.com/ollama/ollama/releases/tag/v0.5.5