4 ms·
Made some GGUFs if anyone wants to run them! ./llama.cpp/llama-cli -hf unsloth/gemma-3n-E4B-it-GGUF:UD-Q4_K_XL -ngl 99 --jinja --temp 0.0 ./llama.cpp/llama-cl
by danielhanchen 1y ago
Made some GGUFs if anyone wants to run them!
./llama.cpp/llama-cli -hf unsloth/gemma-3n-E4B-it-GGUF:UD-Q4_K_XL -ngl 99 --jinja --temp 0.0
./llama.cpp/llama-cli -hf unsloth/gemma-3n-E2B-it-GGUF:UD-Q4_K_XL -ngl 99 --jinja --temp 0.0
I'm also working on an inference + finetuning Colab demo! I'm very impressed since Gemma 3N has audio, text and vision! https://docs.unsloth.ai/basics/gemma-3n-how-to-run-and-fine-tune https://docs.unsloth.ai/basics/gemma-3n-how-to-run-and-fine-...
- upghost 1y agoLiterally was typing out "Unsloth, do your thing!!" but you are way ahead of me. You rock <3 <3 <3 Thank you!
- danielhanchen 1y ago:) Thanks!
- deleted 1y ago[deleted]
- knowaveragejoe 1y agoWhat is `jinja` in this context?
- bilsbie 1y agoThanks! What kind of rig do I need?
- jszymborski 1y agoLikely nothing crazy. My RTX 2080 is pumping out 45 tok/s.
- magicalhippo 1y agoTried the E4B model in Ollama and it's totally broken when interpreting images. The output depends only on the text and is consistent in that way, but otherwise completely wrong. Works fine with regular Gemma 3 4B, so I'll assume it's something on Ollama's side. edit: yep, text-only for now[1], would be nice if that was a bit more prominent than burried in a ticket... Don't feel like compiling llama.cpp myself, so I'll have to wait to try your GGUFs there. [1]: https://github.com/ollama/ollama/issues/10792#issuecomment-3009619264 https://github.com/ollama/ollama/issues/10792#issuecomment-3...
- danielhanchen 1y agoOh I don't think multimodal works yet - it's text only for now!