2 ms·

> Oh the dynamic 4bit quants sadly are not GGUF compatible yet Ah bummer. Is this a GGUF file-format issue or mostly "just" a code-doesn't-exist issue? > Also

> Oh the dynamic 4bit quants sadly are not GGUF compatible yet Ah bummer. Is this a GGUF file-format issue or mostly "just" a code-doesn't-exist issue? > Also I'm unsure yet if llama.cpp as of 11th Jan 2025 supports Llama Vision (or maybe it's new?) Ah, I totally forgot Ollama did that on their own and didn't merge upstream. I'm using Ollama because was so easy to get running on my main Windows rig so I can take advantage of my GPU there, I still do a bit of gaming, while all the stuff which uses Ollama for inference I run on my server. Anyway thanks for the response.

danielhanchen 2y ago
> Ah bummer. Is this a GGUF file-format issue or mostly "just" a code-doesn't-exist issue? Just code! Technically I was working on dynamic quants for DeepSeek V3 (200GB in size), which will increase accuracy by a lot for a 2bit model (if you leave attention in 4bit), and just use 20GB more. -> But I'm still working on it! > Ah, I totally forgot Ollama did that on their own and didn't merge upstream. Yep they have Llama vision support! Llama.cpp has Qwen, Llava support - I think Llama V support is coming, but it'll take much more time - the arch is vastly different than normal transformers due to cross attention