4 ms·
Good call, I'm trying that one just now in LM Studio (by clicking "Use this model -> LM Studio" on https://huggingface.co/unsloth/gemma-3-270m-it-GGUF https://h
by simonw 1y ago
Good call, I'm trying that one just now in LM Studio (by clicking "Use this model -> LM Studio" on https://huggingface.co/unsloth/gemma-3-270m-it-GGUF https://huggingface.co/unsloth/gemma-3-270m-it-GGUF and selecting the F16 one).
(It did not do noticeably better at my pelican test).
Actually it's worse than that, several of my attempts resulted in infinite loops spitting out the same text. Maybe that GGUF is a bit broken?
- danielhanchen 1y agoOh :( Maybe the settings? Could you try temperature = 1.0, top_k = 64, top_p = 0.95, min_p = 0.0
- canyon289 1y agoDaniel, thanks for being here providing technical support as well. Cannot express enough how much we appreciate your all work and partnership.
- danielhanchen 1y agoThank you and fantastic work with Gemma models!
- simonw 1y agoMy topping only lets me set temperature and top_p but setting them to those values did seem to avoid the infinite loops, thanks.
- danielhanchen 1y agoOh fantastic it worked! I was actually trying to see if we can auto set these within LM Studio (Ollama for eg has params, template) - not sure if you know how that can be done? :)
- JLCarveth 1y agoI ran into the same looping issue with that model.
- danielhanchen 1y agoDefinitely give temperature = 1.0, top_k = 64, top_p = 0.95, min_p = 0.0 a try, and maybe repeat_penalty = 1.1