3 ms·

I'm just playing around with these models, first time using them locally. I don't know about huggingface, but with ollama I was up and running in about 10 minut

I'm just playing around with these models, first time using them locally. I don't know about huggingface, but with ollama I was up and running in about 10 minutes. Shockingly easy! Also, ollama claims to distribute the load automatically over CPU and GPU. Gemma 2B and 7B are both very fast, work well in common domains, but have some holes in their knowledge. Mixtral impresses with completeness, but is noticeably slower.

mchinen 3y ago
Interesting, I have had a different experience with Gemma 7B in being able to be generally useful for basic questions after a couple prompts. It frequently gets stuck and starts outputting grammatically incorrect sentences and answers unrelated to the prompt. I also found it answering everything with "Sure, " and wasn't able to stop even if prompted. Another thing was that every response was a list of items. Feels like a temperature issue, but not sure. With mistral or llama I get a better session in every case so far (Only about 5 conversations with Gemma).