3 ms·

...Also, we have eval'd Gemma 7B internally in a deterministic, zero temperature test, and its error rate is like double Mistral Instruct 0.2. Well below most o

...Also, we have eval'd Gemma 7B internally in a deterministic, zero temperature test, and its error rate is like double Mistral Instruct 0.2. Well below most other 7Bs. Was not very impressed with the chat either. So maybe this is neat for embedded projects, but if it's Gemma only, that would be quite a sticking point for me.

Havoc 3y ago
That does seem to be the consensus unfortunately. Would have been better for everyone if google’s foray into open model a la FB made a splash
- brucethemoose2 3y ago
  Yeah, especially with how much Google is hyping it. It could have been long context? Or a little bigger, to fill the relative gap in the 13B-30B area? Even if the model itself was mediocre (which you can't know until after training), it would have been more interesting.
Vetch 3y ago
Was it via gemma.cpp or some other library? I've seen a few people note that gemma performance via gemma.cpp is much better than llama.cpp, possible that the non-google implementations are still not quite right?
- brucethemoose2 3y ago
  I eval'd it with vllm. One thing I do suspect people are running into is sampling issues. Gemma probably doesn't like llama defaults with its 256K vocab. Many Chinese llms have a similar "default sampling" issue. But our testing was done with zero temperature and constrained single token responses, so that shouldnt be an issue.
trisfromgoogle 3y ago
Any chance you can share more details on your measurement setup and eval protocols? You're likely seeing some config snafus, which we're trying to track down.
- brucethemoose2 3y ago
  I just loaded it in vllm with default settings. I can't share the eval, but it's pretty simple: it asks a question about some data, and is restricted to only answer yes/no (based on the output logits and suggested in the prompt). It's called with 0 temperature and only 1 output token, so sampling shouldn't be an issue.

Havoc 3y ago

That does seem to be the consensus unfortunately. Would have been better for everyone if google’s foray into open model a la FB made a splash

brucethemoose2 3y ago
Yeah, especially with how much Google is hyping it. It could have been long context? Or a little bigger, to fill the relative gap in the 13B-30B area? Even if the model itself was mediocre (which you can't know until after training), it would have been more interesting.

Vetch 3y ago

Was it via gemma.cpp or some other library? I've seen a few people note that gemma performance via gemma.cpp is much better than llama.cpp, possible that the non-google implementations are still not quite right?

brucethemoose2 3y ago
I eval'd it with vllm. One thing I do suspect people are running into is sampling issues. Gemma probably doesn't like llama defaults with its 256K vocab. Many Chinese llms have a similar "default sampling" issue. But our testing was done with zero temperature and constrained single token responses, so that shouldnt be an issue.

trisfromgoogle 3y ago

Any chance you can share more details on your measurement setup and eval protocols? You're likely seeing some config snafus, which we're trying to track down.

brucethemoose2 3y ago
I just loaded it in vllm with default settings. I can't share the eval, but it's pretty simple: it asks a question about some data, and is restricted to only answer yes/no (based on the output logits and suggested in the prompt). It's called with 0 temperature and only 1 output token, so sampling shouldn't be an issue.