4 ms·

I'm surprised to see them reference the LMSO leaderboard here. Did we ever get an explanation as to how Gemini Pro had such a large increase in rating so sudde

I'm surprised to see them reference the LMSO leaderboard here. Did we ever get an explanation as to how Gemini Pro had such a large increase in rating so suddenly? And is there an explanation as to why people will get a correct answer from this API but Bard will give you hallucinated, incorrect answers? I think it's very important for Google to be competitive here so my hopes are high, but the Gemini launch been kind of an inconsistent mess.

siva7 3y ago
> Did we ever get an explanation as to how Gemini Pro had such a large increase in rating so suddenly? Easy, they bought it. https://huggingface.co/blog/gcp-partnership https://huggingface.co/blog/gcp-partnership
- swyx 3y ago
  1) lmsys is not affiliated with huggingface 2) this fails basic sniff test of how research is done. google overmarkets but it doesn't lie. to answer GP's question - the #2 rated bard is an "online" llm, presumably people are rating more recent knowledge more favorably. its sad that pplx-api as the only other "online llm" does not do better, but people are recognizing it is unfair to compare "online LLMs" with not-online https://twitter.com/lmsysorg/status/1752126690476863684 https://twitter.com/lmsysorg/status/1752126690476863684
- whimsicalism 3y ago
  HN quality never recovered after the pandemic.
  - a_vanderbilt 3y ago
    The orange equivalent of the Eternal September?
hatenberg 3y ago
32x CoT benchmark juicing? Safety bullshit on the app because that’s consumer space
- whimsicalism 3y ago
  you dont know what you are talking about
thefourthchime 3y ago
i'm wondering the same thing it seemed to pop up out of nowhere and anecdotal doesn't seem much better to me.
- whimsicalism 3y ago
  People are beginning to over-index on the lm-sys leaderboard. It is doing well because it is a decent model and it also has internet access.
austinkhale 3y ago
> Did we ever get an explanation as to how Gemini Pro had such a large increase in rating so suddenly? Different fine-tune and gave it access to the Internet. Source: https://x.com/asadovsky/status/1750983142041911412?s=20 https://x.com/asadovsky/status/1750983142041911412?s=20

siva7 3y ago

> Did we ever get an explanation as to how Gemini Pro had such a large increase in rating so suddenly? Easy, they bought it. https://huggingface.co/blog/gcp-partnership https://huggingface.co/blog/gcp-partnership

swyx 3y ago
1) lmsys is not affiliated with huggingface 2) this fails basic sniff test of how research is done. google overmarkets but it doesn't lie. to answer GP's question - the #2 rated bard is an "online" llm, presumably people are rating more recent knowledge more favorably. its sad that pplx-api as the only other "online llm" does not do better, but people are recognizing it is unfair to compare "online LLMs" with not-online https://twitter.com/lmsysorg/status/1752126690476863684 https://twitter.com/lmsysorg/status/1752126690476863684
whimsicalism 3y ago
HN quality never recovered after the pandemic.
- a_vanderbilt 3y ago
  The orange equivalent of the Eternal September?

hatenberg 3y ago

32x CoT benchmark juicing? Safety bullshit on the app because that’s consumer space

whimsicalism 3y ago
you dont know what you are talking about

thefourthchime 3y ago

i'm wondering the same thing it seemed to pop up out of nowhere and anecdotal doesn't seem much better to me.

whimsicalism 3y ago
People are beginning to over-index on the lm-sys leaderboard. It is doing well because it is a decent model and it also has internet access.

austinkhale 3y ago

> Did we ever get an explanation as to how Gemini Pro had such a large increase in rating so suddenly? Different fine-tune and gave it access to the Internet. Source: https://x.com/asadovsky/status/1750983142041911412?s=20 https://x.com/asadovsky/status/1750983142041911412?s=20