2 ms·

Is there a reason people seem to exclude google models? I see lots of talk about llama, mistral, deepseek, and then ofc gpt and claude, but not much about gemma

Is there a reason people seem to exclude google models? I see lots of talk about llama, mistral, deepseek, and then ofc gpt and claude, but not much about gemma and gemini despite them benchmarking well.

anotherpaulg 2y ago
The main aider leaderboard includes gemini models. I benchmarked gemma-2-27b-it, but it did so badly that it wasn't worth adding to the leaderboard. https://aider.chat/docs/leaderboards/ https://aider.chat/docs/leaderboards/
- Lash-L 2y ago
  Interesting - why do you think that is? can-ai-code seems to have Gemma 2-27b as pretty capable - beating out some of the other models that perform really well on aider: https://huggingface.co/spaces/mike-ravkine/can-ai-code-results https://huggingface.co/spaces/mike-ravkine/can-ai-code-resul...
  - 22c 2y ago
    Not an expert, but I believe it's related to this difference: > LLMs which are good at editing code, not just good at writing code So it may be that Gemma can write new code, but is not particularly good at editing or refactoring code.