5 ms·
Benchmarks for Gemma 7B seem to be in the ballpark of Mistral 7B +-------------+----------+-------------+-------------+ | Benchmark | Gemma 7B | Mistral
by tosh 3y ago
Benchmarks for Gemma 7B seem to be in the ballpark of Mistral 7B
+-------------+----------+-------------+-------------+
| Benchmark | Gemma 7B | Mistral 7B | Llama-2 7B |
+-------------+----------+-------------+-------------+
| MMLU | 64.3 | 60.1 | 45.3 |
| HellaSwag | 81.2 | 81.3 | 77.2 |
| HumanEval | 32.3 | 30.5 | 12.8 |
+-------------+----------+-------------+-------------+
via https://mistral.ai/news/announcing-mistral-7b/ https://mistral.ai/news/announcing-mistral-7b/
- brucethemoose2 3y agoOnly 8K context as well, like Mistral. Also, as always, take these benchmarks with a huge grain of salt. Even base model releases are frequently (seemingly) contaminated these days.
- tosh 3y agoAgree: will be interesting how Gemma does on ChatBot Arena
- DreamGen 3y agoMistral Instruct v0.2 is 32K.
- Kydlaw 3y agoThey state in their report that they filter evaluation data off their training data, see p.3 - Filtering: "Further, we filter all evaluation sets from our pre-training data mixture, run targeted contamination analyses to check against evaluation set leakage, and reduce the risk of recitation by minimizing proliferation of sensitive outputs."
- jcuenod 3y agoCame here to post the same thing for Phi-2: +-------------+----------+-------------+ | Benchmark | Gemma 2B | Phi-2 2.7B | +-------------+----------+-------------+ | MMLU | 42.3 | 56.7 | | MBPP | 29.2 | 59.1 | | BoolQ | 69.4 | 83.3 | +-------------+----------+-------------+ [0] https://www.kaggle.com/models/google/gemma https://www.kaggle.com/models/google/gemma [1] https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/ https://www.microsoft.com/en-us/research/blog/phi-2-the-surp...
- daemonologist 3y agoReally looking forward to the day someone puts out an open model which outperforms Flan-T5 on BoolQ.
- rfw300 3y agoA caveat: my impression of Phi-2, based on my own use and others’ experiences online, is that these benchmarks do not remotely resemble reality. The model is a paper tiger that is unable to perform almost any real-world task because it’s been fed so heavily with almost exclusively synthetic data targeted towards improving benchmark performance.
- refulgentis 3y agoHear hear! I don't understand why it has persistent mindshare, it's not even trained for chat. Meanwhile StableLM 3B runs RAG in my browser, on my iPhone, on my Pixel ..
- djsavvy 3y agoHow have you been using RAG in your browser/on your phones?
- refulgentis 3y agoTo be released, someday [sobs in engineer] Idea is usage-based charging for non-local and a $5/month sub for syncing. keep an eye on @jpohhhh on Twitter if you're interested now that I got it on web, I'm hoping to at least get a PoC up soon. I've open-sourced the consitutent parts as FONNX and FLLAMA, Flutter libraries that work on all platforms. FONNX has embeddings, FLLAMA has llama. https://github.com/Telosnex/fonnx https://github.com/Telosnex/fonnx https://github.com/Telosnex/fllama https://github.com/Telosnex/fllama
- FergusArgyll 3y agothe real gold will be when this gets finetuned. (maybe by mistral...)
- brucethemoose2 3y agoTBH the community has largely outrun Mistral's own finetuning. The 7B model in particular is such a popular target because its so practical to train.
- whimsicalism 3y agoStrong disagree - a Mistral fine tune of llama 70b was the top performing llama fine tune. They have lots of data the community simply does not.
- brucethemoose2 3y agoMiqu was (allegedly) an internal continued pretrain Mistral did as a test, that was leaked as a GGUF. Maybe its just semantics, it is technically a finetune... But to me theres a big difference between expensive "continuation training" (like Solar 10.7B or Mistral 70B) and a much less intense finetuning. The former is almost like releasing a whole new base model. It would be awesome if Mistral did that with their data, but thats very different than releasing a Gemma Instruct finetune.
- whimsicalism 3y agoThere’s typically a difference in LR between a ‘continued pretrain’ and ‘fine tune.’ I don’t have the details around miqu, but was merely trying to say that Mistral could produce a better version of these models than the OSS community might. If the size of the corpora they use means we are no longer in fine tuning territory, then okay.
- sanjiwatsuki 3y agoNo shot. Mistral Medium's outputs from API were virtually identical. Miqu really was Mistral Medium which happened to be a continued pretrain
- sa-code 3y agoThank you. I thought it was weird for them to release a 7B model and not mention Mistral in their release.
- mirekrusin 3y agoThey forgot. Also phi-2.
- mochomocha 3y agoThe technical report (linked in the 2nd paragraph of the blog post) mentions it, and compares against it: https://storage.googleapis.com/deepmind-media/gemma/gemma-report.pdf https://storage.googleapis.com/deepmind-media/gemma/gemma-re...
- deleted 3y ago[deleted]
- nl 3y agoThe release page has comparisons to Mistral everywhere: https://ai.google.dev/gemma https://ai.google.dev/gemma
- sa-code 3y agoGood to know, although cmd+f for "mistral" returns 0 hits on the original link
- lawxls 3y agoHonestly, this is more of a PR stunt to advertise the Google Dev ecosystem than a contribution to open-source. I'm not complaining, just calling it what it is. Barely an improvement over the 5-month-old Mistral model, with the same context length of 8k. And this is a release after their announcement of Gemini Pro 1.5, which had an exponential increase in context length.
- scarmig 3y agoWho cares if it's a PR stunt to improve developer good will? It's still a good thing, and it's now the most open model out there.
- moffkalast 3y agoHow is it more open than Mistral with Apache 2.0? Google wants people to sign a waiver to even download it.
- scarmig 3y agoFair enough; that was more directed at LLaMA and derivatives, which have commercial restrictions.
- observationist 3y agoHow exactly is it the "most open model" ? It's more like a masterclass in corporate doublespeak. Google’s "transparency" is as clear as mud, with pretraining details thinner than their privacy protections. Diving into Google’s tech means auctioning off your privacy (and your users' privacy) to the highest bidder. Their "open source" embrace is more of a chokehold, with their tech biases and monopolistic strategies baked into every line of code. Think of it as Google's way of marking territory - every developer is a fire hydrant. These megacorps aren’t benevolent patrons of open source; they're self-serving giants cloaking power grabs under the guise of "progress". Use these products at your own risk. If these companies wanted to engage in good faith, they'd use Apache or MIT licensing and grant people the agency and responsibility for their own use and development of software. Their licenses are designed to mitigate liability, handcuff potential competitors, and eke every last drop of value from users, with informed consent frequently being an optional afterthought. That doesn't even get into the Goodharting of metrics and actual performance of the models; I highly doubt they're anywhere near as good as Mistral. The UAE is a notoriously illiberal authoritarian state, yet even they have released AI models far more free and open than Google or Meta. https://huggingface.co/tiiuae/falcon-40b/blob/main/README.md https://huggingface.co/tiiuae/falcon-40b/blob/main/README.md If it’s not Apache or MIT, (or even some flavor of GPL,) it’s not open source; it’s a trojan horse. These "free" models come at the cost of your privacy and freedoms. These models aren't Open or Open Access or Free unless you perform the requisite mental gymnastics cooked up by their marketing and legal teams. Oceania has always been at war with Eastasia. Gemma is doubleplusgood.
- YetAnotherNick 3y agoAccording to their paper, average of standard task of Mistral is 54.0 and for Gemma it's 56.4, so 4.4% relative better. Not as big as you would expect for the company which invented transformers and probably has 2-3 order more compute for training it vs few month old French startup. Also for note on their human evaluations, Gemma 7B IT has a 51.7% win rate against Mistral v0.2 7B Instruct.
- deleted 3y ago[deleted]
- attentive 3y agoIn my subjective tests it's not even close to Mistral. While my local gemma is quantized, so is mistral. But I also tried gemma on huggingface.co/chat which I assume isn't quantized.
- heckl239u 3y agohttps://www.youtube.com/watch?v=1Mn0U6HGLeg https://www.youtube.com/watch?v=1Mn0U6HGLeg some test vids came out on the 7b model. Shock it doesn't perform well at all.