4 ms·
> Ah bummer. Is this a GGUF file-format issue or mostly "just" a code-doesn't-exist issue? Just code! Technically I was working on dynamic quants for DeepSeek
by danielhanchen 2y ago
> Ah bummer. Is this a GGUF file-format issue or mostly "just" a code-doesn't-exist issue?
Just code! Technically I was working on dynamic quants for DeepSeek V3 (200GB in size), which will increase accuracy by a lot for a 2bit model (if you leave attention in 4bit), and just use 20GB more. -> But I'm still working on it!
> Ah, I totally forgot Ollama did that on their own and didn't merge upstream.
Yep they have Llama vision support! Llama.cpp has Qwen, Llava support - I think Llama V support is coming, but it'll take much more time - the arch is vastly different than normal transformers due to cross attention