7 ms·

It's exceptionally strong. In LMSys Chatbot Arena, the 27B version scores above LLama-3-70B, at the level of OpenAI GPT-4 and Claude-3 Sonnet!

screye 2y ago
What's the most obvious standouts? In my experience, smaller models tend to do well on benchmarks and fail at generalization. Phi-2 comes to mind.
- moffkalast 2y ago
  It's multilingual. Genuinely. Compared my results with some people on reddit and the consensus is that the 27B is near perfect in a few obscure languages and likely perfect in most common ones. The 9B is not as good but it's still coherent enough to use in a pinch. It's literally the first omni-translation tool that actually works that you can run offline at home. I'm amazed that Google mentioned absolutely nothing about this in their paper.
  - norealmet 2y ago
    [dead]
  - jug 2y ago
    Wow, that's very impressive and indeed a game changer. I've previously had trouble with various Scandinavian languages, but last I checked with was Llama 2 and I kind of gave up on it. I had expected we were going to need special purpose small models for these uses as a crutch, like SW-GPT3. So I guess Gemma 2 is going to become Gemini 2.0 in their truly large and closed variants then? Or is it the open version of Gemini 1.5?
typpo 2y ago
If anyone is interested in evaling Gemma locally, this can be done pretty easily using ollama[0] and promptfoo[1] with the following config: prompts: - 'Answer this coding problem in Python: {{ask}}' providers: - ollama:chat:gemma2:9b - ollama:chat:llama3:8b tests: - vars: ask: function to find the nth fibonacci number - vars: ask: calculate pi to the nth digit - # ... One small thing I've always appreciated about Gemma is that it doesn't include a "Sure, I can help you" preamble. It just gets right into the code, and follows it with an explanation. The training seems to emphasize response structure and ease of comprehension. Also, best to run evals that don't rely on rote memorization of public code... so please substitute with your personal tests :) [0] https://ollama.com/library/gemma2 https://ollama.com/library/gemma2 [1] https://github.com/promptfoo/promptfoo https://github.com/promptfoo/promptfoo
- deleted 2y ago
  [deleted]
- roywiggins 2y ago
  In Ollama, Gemma:9b works fine, but 27b seems to be producing a lot of nonsense for me. Asking for a bit of python or JavaScript code rapidly devolves into producing code-like gobbledegook, extending for hundreds of lines.
  - bugglebeetle 2y ago
    The tokenizer in llama.cpp probably needs fixing then or it has some other bug.
    - 0x7cfe 2y ago
      Definitely. I tried gemma2:27B model with phrases like "translate the following sentence to language X" and it even failed to understand the task and spat out completely irrelevant things, like math formulas. OTOH, smaller model did it perfectly.
  - brandall10 2y ago
    27b is working fine for me, hosted on ollama w/ continue.dev in VSCode.
  - thot_experiment 2y ago
    Had a chance to do some testing and it seems quite good on oneshot tasks with a small context window but as you approach context saturation it starts to go way off the rails. Maybe this is an implementation issue? I'm using Q6_K quants of both sizes in ollama. I'll report back if I figure it out. A larger context window really helps on RAG tasks, it's frustrating that a lot of the foundational models have such small windows.
    - jmorgan 2y ago
      Sorry about this – working on fixing the issue with hitting the context limit. Gemma 2 supports a 8192 context limit – which can be selected if you provide the `num_ctx` parameter in the API or via `ollama run` with `/set parameter num_ctx 8192`
deleted 2y ago
[deleted]
resource_waste 2y ago
Do we believe that? I've been told Google's AI was going to be great 4 times now, and its consistently #4 behind OpenAI, Facebook, and Claude.
- aubanel 2y ago
  LMSys Chatbot Arena is a crowd-sourced ranking with an ELO system: basically users a presented with 2 hidden models, they get the answers of the 2 models when presenting their request, and they vote which one performed bests, which realized one marche and updates the ELO scores. This is the closest thing that we have to a gold truth for LLM evaluation: and Gemma2-27B performs extremely well in Chatbot Arena ELO.
usaar333 2y ago
I think this is just due to better non-English training data. It's 15 ELO under Llama-3-70B on english hard prompts and 41 ELO under Llama-3-70B (the latter is actually stat sig) for general English.
lhl 2y ago
I'd encourage people to test for themselves (and to let the Chatbot Arena scores to settle) before getting caught up in too much hype. I just did a personal eval and I found gemma-2-27b-it (tested on AI Studio) performed far worse in my testing than Llama 3 70B, especially for reasoning and basic world understanding queries.
- nacs 2y ago
  Same. I tried 27B and found it to be not even close to llama3-70b. Even llama-8b did better in some of my tests than Gemma 27b.
- WiSaGaN 2y ago
  I also prefer to use "Coding" or "Hard Prompts (Overall)" instead of default "Overall" in Chatbot Arena scores to determine the actual performance level of LLMs. Seems much more align to my vibe test in terms reasoning. I guess the "Overall" contains a lot of creative tasks, which is not what I use the most in the daily tasks.
lhl 2y ago
Just saw this, might get lost in the noise, but just for posterity, apparently the Gemma 2 models were specifically RL’d to index on Chat Arena performance: https://x.com/natolambert/status/1806384821826109597 https://x.com/natolambert/status/1806384821826109597 (Relevant sections of the paper highlighted.)
- occamrazor 2y ago
  On prompts only, with answers presumably from the teacher model (Gemini). It was not trained or RLHFd on Arena replies or user preferences.
  - lhl 2y ago
    Yes, answers were distilled from a much stronger model. On the one hand, you can argue that this is exactly what the LMSYS, WildBench etc datasets are for (to improve performance/alignment on real-world use cases), but on the other hand, it's clear that training on the questions (most of which are repeatedly used by the (largely non-representative of general population) users of the ChatArena for comparing/testing models) makes ChatArena ELO less useful as a model comparison tool and artificially elevates Gemma 2's ChatArena score relative to its OOD performance. At the end of the day, by optimizing for leaderboard scoring, it makes the leaderboard ranking less useful as a benchmark (Goodhart's law strikes again). The Gemma team obviously isn't the only one doing it, but it's important to be clear-eyed about the consequences.

screye 2y ago

What's the most obvious standouts? In my experience, smaller models tend to do well on benchmarks and fail at generalization. Phi-2 comes to mind.

moffkalast 2y ago
It's multilingual. Genuinely. Compared my results with some people on reddit and the consensus is that the 27B is near perfect in a few obscure languages and likely perfect in most common ones. The 9B is not as good but it's still coherent enough to use in a pinch. It's literally the first omni-translation tool that actually works that you can run offline at home. I'm amazed that Google mentioned absolutely nothing about this in their paper.
- norealmet 2y ago
  [dead]
- jug 2y ago
  Wow, that's very impressive and indeed a game changer. I've previously had trouble with various Scandinavian languages, but last I checked with was Llama 2 and I kind of gave up on it. I had expected we were going to need special purpose small models for these uses as a crutch, like SW-GPT3. So I guess Gemma 2 is going to become Gemini 2.0 in their truly large and closed variants then? Or is it the open version of Gemini 1.5?

typpo 2y ago

If anyone is interested in evaling Gemma locally, this can be done pretty easily using ollama[0] and promptfoo[1] with the following config: prompts: - 'Answer this coding problem in Python: {{ask}}' providers: - ollama:chat:gemma2:9b - ollama:chat:llama3:8b tests: - vars: ask: function to find the nth fibonacci number - vars: ask: calculate pi to the nth digit - # ... One small thing I've always appreciated about Gemma is that it doesn't include a "Sure, I can help you" preamble. It just gets right into the code, and follows it with an explanation. The training seems to emphasize response structure and ease of comprehension. Also, best to run evals that don't rely on rote memorization of public code... so please substitute with your personal tests :) [0] https://ollama.com/library/gemma2 https://ollama.com/library/gemma2 [1] https://github.com/promptfoo/promptfoo https://github.com/promptfoo/promptfoo

deleted 2y ago
[deleted]
roywiggins 2y ago
In Ollama, Gemma:9b works fine, but 27b seems to be producing a lot of nonsense for me. Asking for a bit of python or JavaScript code rapidly devolves into producing code-like gobbledegook, extending for hundreds of lines.
- bugglebeetle 2y ago
  The tokenizer in llama.cpp probably needs fixing then or it has some other bug.
  - 0x7cfe 2y ago
    Definitely. I tried gemma2:27B model with phrases like "translate the following sentence to language X" and it even failed to understand the task and spat out completely irrelevant things, like math formulas. OTOH, smaller model did it perfectly.
- brandall10 2y ago
  27b is working fine for me, hosted on ollama w/ continue.dev in VSCode.
- thot_experiment 2y ago
  Had a chance to do some testing and it seems quite good on oneshot tasks with a small context window but as you approach context saturation it starts to go way off the rails. Maybe this is an implementation issue? I'm using Q6_K quants of both sizes in ollama. I'll report back if I figure it out. A larger context window really helps on RAG tasks, it's frustrating that a lot of the foundational models have such small windows.
  - jmorgan 2y ago
    Sorry about this – working on fixing the issue with hitting the context limit. Gemma 2 supports a 8192 context limit – which can be selected if you provide the `num_ctx` parameter in the API or via `ollama run` with `/set parameter num_ctx 8192`

deleted 2y ago

[deleted]

resource_waste 2y ago

Do we believe that? I've been told Google's AI was going to be great 4 times now, and its consistently #4 behind OpenAI, Facebook, and Claude.

aubanel 2y ago
LMSys Chatbot Arena is a crowd-sourced ranking with an ELO system: basically users a presented with 2 hidden models, they get the answers of the 2 models when presenting their request, and they vote which one performed bests, which realized one marche and updates the ELO scores. This is the closest thing that we have to a gold truth for LLM evaluation: and Gemma2-27B performs extremely well in Chatbot Arena ELO.

usaar333 2y ago

I think this is just due to better non-English training data. It's 15 ELO under Llama-3-70B on english hard prompts and 41 ELO under Llama-3-70B (the latter is actually stat sig) for general English.

lhl 2y ago

I'd encourage people to test for themselves (and to let the Chatbot Arena scores to settle) before getting caught up in too much hype. I just did a personal eval and I found gemma-2-27b-it (tested on AI Studio) performed far worse in my testing than Llama 3 70B, especially for reasoning and basic world understanding queries.

nacs 2y ago
Same. I tried 27B and found it to be not even close to llama3-70b. Even llama-8b did better in some of my tests than Gemma 27b.
WiSaGaN 2y ago
I also prefer to use "Coding" or "Hard Prompts (Overall)" instead of default "Overall" in Chatbot Arena scores to determine the actual performance level of LLMs. Seems much more align to my vibe test in terms reasoning. I guess the "Overall" contains a lot of creative tasks, which is not what I use the most in the daily tasks.

lhl 2y ago

Just saw this, might get lost in the noise, but just for posterity, apparently the Gemma 2 models were specifically RL’d to index on Chat Arena performance: https://x.com/natolambert/status/1806384821826109597 https://x.com/natolambert/status/1806384821826109597 (Relevant sections of the paper highlighted.)

occamrazor 2y ago
On prompts only, with answers presumably from the teacher model (Gemini). It was not trained or RLHFd on Arena replies or user preferences.
- lhl 2y ago
  Yes, answers were distilled from a much stronger model. On the one hand, you can argue that this is exactly what the LMSYS, WildBench etc datasets are for (to improve performance/alignment on real-world use cases), but on the other hand, it's clear that training on the questions (most of which are repeatedly used by the (largely non-representative of general population) users of the ChatArena for comparing/testing models) makes ChatArena ELO less useful as a model comparison tool and artificially elevates Gemma 2's ChatArena score relative to its OOD performance. At the end of the day, by optimizing for leaderboard scoring, it makes the leaderboard ranking less useful as a benchmark (Goodhart's law strikes again). The Gemma team obviously isn't the only one doing it, but it's important to be clear-eyed about the consequences.