15 ms·
So it's twice the size of phi 3 and considerably worse? What am I missing
by iamronaldo 2y ago
So it's twice the size of phi 3 and considerably worse? What am I missing
- ertgbnm 2y agoThey used two non-mutually exclusive techniques. Phi-3 is mostly a curriculum training breakthrough. By filtering training set for high quality tokens and training on synthetic data, they were able to achieve great results. Gemma-2 is a distillation breakthrough. By training LLMs with guidance from larger teacher LLMs, they were able to achieve great results too. Porque no los dos?
- m00x 2y agoWorse in some aspects, better in other. Small models are never going to be generalists, so having several small models allows you to pick the one that best fits your needs.
- k__ 2y agoWhen would you use which?
- Aerbil313 2y agoObviously another small model would be specialized in determining that.
- k__ 2y agoIs it models all the way down?
- m00x 2y agoWhichever model works better for your use. It's hard to know without testing it at the moment. I've found Gemini to be better at some use-cases, and GPT-4 better at others for my specific taste and use-case. You can kind of go by the benchmark scores to have an idea if it's good at logic, creativity, etc.
- azeirah 2y agoHave you tried Phi 3? It's smart which makes it perform well on benchmarks, but it's not great at conversation or as a chatbot. I imagine Gemma 2 is a better general-purpose assistant for most people, whereas Phi 3 is a solid small LLM (SLM?) for more specific use-cases like summarization, RAG, learning about math and stuff.
- floridianfisher 2y agoWhy not try it here and make your comparisons that way? https://aistudio.google.com/app/prompts/new_chat?model=gemma-2-27b-it https://aistudio.google.com/app/prompts/new_chat?model=gemma...
- pona-a 2y agoOne compelling reason not to would be a region block... [0] https://ai.google.dev/gemini-api/docs/available-regions https://ai.google.dev/gemini-api/docs/available-regions
- ferretj 2y agoAnother take on this: phi-3 small has 1100 ELO on LMSYS (ranked #52) while the confidence interval for Gemma 2 9B is [1170, 1200] ELO (ranked btw #15 and #25).
- reissbaker 2y agoPhi-3 does well in benchmarks but underperforms IRL; for example, Phi-3-Medium gets beaten badly by Llama-3-8b on the LMSYS Chatbot Arena despite doing better on benchmarks. Gemma's performance if anything seems understated on benchmarks: the 27b is currently ahead of Llama3-70b on the Chatbot Arena leaderboard.
- ertgbnm 2y agoI suspect Phi-3 is not robust to normal human input like typos and strange grammar since it's only trained on filtered "high quality" tokens and synthetic data. Since it doesn't need to waste a ton of parameters learning how to error correct input, it's much smarter on well curated benchmarks compared to its weight class. However, it can't operate out of distribution at all.
- reissbaker 2y agoPersonally vibe checking Phi-3-Medium is worse in my experience, no matter how well you spell — it just isn't good at all compared to Llama3-8b, despite being significantly larger in param count. I suspect the "high quality tokens" were "high quality" in the sense that they resembled tokens one might encounter in benchmarks, and not "high quality" in the sense of representing human-like input/output.