3 ms·
> Models like ornith:9b, mistral:7b or cogito:14b can get the job done sometimes, but they are not fast and reliable enough for general use, specially if you ha
by kouteiheika 1mo ago
> Models like ornith:9b, mistral:7b or cogito:14b can get the job done sometimes, but they are not fast and reliable enough for general use, specially if you have only 4GB of VRAM.
Have you considered/tried using a model that's, well, more appropriate size-wise for an use case like this? These are relatively big. Something like FunctionGemma [1] finetuned for a given set of tasks would be a lot more speedy.
[1] https://blog.google/innovation-and-ai/technology/developers-tools/functiongemma/ https://blog.google/innovation-and-ai/technology/developers-...
- gioscarab 1mo agoI tried functiongemma, it is for sure faster than those models, the problem is that is not reliable enough for a terminal assistant. I would say that no LLM is good for a terminal assistant, if you take into account the operational cost and the risk of damage. Even if it fails only 1 time out of 10 becomes useless. That's why I developed FlintParser!
- coder543 1mo agoFunctionGemma never worked well for me (without fine tuning). Liquid has released 230M and 350M models that work far, far better in my testing: https://huggingface.co/LiquidAI/LFM2.5-230M https://huggingface.co/LiquidAI/LFM2.5-230M I really look forward to a hypothetical LFM3-230M, because LFM2.5-230M is so close to being usable, while FunctionGemma is miles away from being usable. But, yes, still tangential to TERMy.
- kennywinker 1mo agohttps://github.com/ThorOdinson246/whatisit-nl2sh https://github.com/ThorOdinson246/whatisit-nl2sh uses a finetune of Qwen2.5-Coder-1.5B-Instruct. It works pretty well, tho it will misunderstand things from time to time