2 ms·
In the Gemini paper they already demonstrated that they beat GPT-4 on most important benchmarks, HellaSwag being an exception. See https://storage.googleapis.c
by cubefox 3y ago
In the Gemini paper they already demonstrated that they beat GPT-4 on most important benchmarks, HellaSwag being an exception.
See https://storage.googleapis.com/deepmind-media/gemini/gemini_1_report.pdf#page=7 https://storage.googleapis.com/deepmind-media/gemini/gemini_...
By the way, I linked to this report when it came out, but was labeled a "dupe" because someone else had already posted a link to the flashy website without any technical details.
- pants2 3y agoGoogle's misleading demonstration video shows they can't be trusted to honestly evaluate their own model. Until we can actually get our hands on Ultra and try for ourselves, I wouldn't give too much credit to their numbers.
- replwoacause 3y agoIt’s like Big Tobacco doing safety research into their cigarettes, cannot be trusted. Google is untrustworthy and has been for a long time.
- sumedh 3y ago> Google's misleading demonstration video I cannot believe someone in Google approved that video but then again I cannot believe that a company would be launching 10 different chat apps.
- ianbicking 3y ago"Important benchmarks" aren't the same as being useful and reliable. I tried using Gemini Pro for tasks that GPT-3.5 does very well on and it fell apart completely. From reading the reviews of people using it, I get the impression Gemini Pro is good at writing marketing copy and other stuff that GPT-3 was good at, and I suspect a lot of benchmarks are over focused on that same text completion modality. Honestly Gemini Pro is so unimpressive I expect even GPT-3.5 will beat Gemini Ultra in terms of utility, at least for the next 6 months.