20 ms·
What is up with that eval @32? Am I reading it correctly that they are generating 32 responses and taking majority? Who will use the API like that? That feels l
by rolisz 3y ago
What is up with that eval @32? Am I reading it correctly that they are generating 32 responses and taking majority? Who will use the API like that? That feels like such a fake way to improve metrics
- technics256 3y agoThis also jumped out at me. It also seems that they are selectively choosing different promoting strategies too, one lists "CoT@32". Makes it seem like they really needed to get creative to have it beat GPT4. Not a good sign imho
- bryanh 3y agoPage 7 of their technical report [0] has a better apples to apples comparison. Why they choose to show apples to oranges on their landing page is odd to me. [0] https://storage.googleapis.com/deepmind-media/gemini/gemini_1_report.pdf https://storage.googleapis.com/deepmind-media/gemini/gemini_...
- polygamous_bat 3y agoI assume these landing pages are made for wall st analysts rather than people who understand LLM eval methods.
- bryanh 3y agoTrue, but even some of the apples to apples is favorable to Gemini Ultra 90.04% CoT@32 vs. GPT-4 87.29% CoT@32 (via API).
- dongobread 3y agoThis isn't apples to apples - they're taking the optimal prompting technique for their own model, then using that technique for both models. They should be comparing it against the optimal prompting technique for GPT-4.
- rockinghigh 3y agoShowing dominance in AI is also targeted at their entreprise customers who spend millions on Google Cloud services.