4 ms·
n.b. the image processing is by a separate model, basically has to load the image and generate ~1000 tokens (source: vision was available in llama.cpp but Very
by refulgentis 1y ago
n.b. the image processing is by a separate model, basically has to load the image and generate ~1000 tokens
(source: vision was available in llama.cpp but Very Hard, been maintaining an implementation)
(n.b. it's great work, extremely welcome, and new in that the vision code badly needed a rebase and refactoring after a year or two of each model adding in more stuff)
- brrrrrm 1y agowait sorry, can you explain how this works? I thought gemma3 used siglip, which can output all 256 embeddings in parallel (also, would you mind sharing a code pointer if you have any handy? I found this https://github.com/ggml-org/llama.cpp/blob/master/tools/mtmd/mtmd.cpp#L443 https://github.com/ggml-org/llama.cpp/blob/master/tools/mtmd... but not sure if that's the codepath taken)