3 ms·
It seems pretty impressive - I'm running it on my CPU (16 core AMD 3950x) and it's very very impressive at translation, and the image description is very impres
by trebligdivad 1y ago
It seems pretty impressive - I'm running it on my CPU (16 core AMD 3950x) and it's very very impressive at translation, and the image description is very impressive as well. I'm getting about 2.3token/s on it (compared to under 1/s on the Calme-3.2 I was previously using).
It does tend to be a bit chatty unless you tell it not to be; pretty much everything it'll give you a 'breakdown' unless you tell it not to - so for traslation my prompt is 'Translate the input to English, only output the translation' to stop it giving a breakdown of the input language.
- simonw 1y agoWhat are you using to run it? I haven't got image input working yet myself.
- trebligdivad 1y agoI'm using llama.cpp - built last night from head; to do image stuff you have to run a separate client they provide, with something like: ./build/bin/llama-gemma3-cli -m /discs/fast/ai/gemma-3-27b-it-q4_0.gguf --mmproj /discs/fast/ai/mmproj-model-f16-27B.gguf -p "Describe this image." --image ~/Downloads/surprise.png Note the 2nd gguf in there - I'm not sure, but I think that's for encoding the image.
- terhechte 1y agoImage input has been working with LM Studio for quite some time
- tough 1y agoneed it here for cli usage! https://github.com/agustif/llm-lmstudio https://github.com/agustif/llm-lmstudio
- Havoc 1y agoThe upcoming qwen3 series is supposed to be MoE...likely to give better tk/s on CPU
- slekker 1y agoWhat's MoE?
- zamalek 1y agoMixture of Experts. Very broadly speaking, there are a bunch of mini networks (experts) which can be independently activated.
- Havoc 1y agoMixture of experts like other guy said - everything gets loaded into mem but not every byte is needed to generate a token (unlike classic LLMs like gemma). So for devices that have lots of mem but weaker processing power it can get you similar output quality but faster. So tends to do better on CPU and APU like setups
- trebligdivad 1y agoI'm not even sure they're loading everything into memory for MoE; maybe they can get away with only the relevant experts being paged in.