2 ms·
Those would use more ram than most rpi have wouldn't they? Gemma uses 4GB right?
by galangalalgol 1y ago
Those would use more ram than most rpi have wouldn't they? Gemma uses 4GB right?
- antirez 1y agoNope, gemma3 and qwen3 exist of many sizes, including very small ones, that 4-bit quantized can run on very small systems. Qwen3-0.6B, 1.7B, ... imagine if you quantize those to 4 bit. But there is the space for the KV cache, if we don't want to limit the runs to very small prompts.
- nolist_policy 1y agoGemma 3 4B QAT int4 quantized from bartowsky should barely fit in a 4GB Raspberry Pi, but without the vision encoder. However the brand-new Gemma 3n E2B and E4B models might fit with vision.
- antirez 1y agoYep, the Gemma 3 1B would be 815MB, with enough margin for a longer prompt. Probably more realistic.