4 ms·

What's everyone's favorite LLM leaderboard? Gemini 2 seems to be edging out 4o on chatbot arena(https://lmarena.ai/?leaderboard https://lmarena.ai/?leaderboard)

SV_BubbleTime 2y ago
AI benchmarks and leaderboards are complete nonsense though. Find something you like, use it, be ready to look again in a month or two.
- falcor84 2y ago
  With the accelerating progress, the "be ready to look again" is becoming a full time job that we need to be able to delegate in some way, and I haven't found anything better than benchmarks, leaderboards and reviews. EDIT: Typo
- siliconc0w 2y ago
  FWIW I've found the 'coding' 'category' of the leaderboard to be reasonably accurate. Claude was the best, o1-mini then was typically stronger, now the Gemini Exp 1206 is at the top. I find myself just paying a la carte via the API rather than paying the $20/mo so I can switch between the models.
- hombre_fatal 2y ago
  poe.com has a decent model where you buy credits and spend them talking to any LLM which makes it nice to swap between them even during the same conversation instead of paying for multiple subscriptions. Though gpt-4o could say "David Mayer" on poe.com but not on chat.openai.com which makes me wonder if they sometimes cheat and sneak in different models.
manishsharan 2y ago
Leaderboards are not that useful for measuring real-life effectiveness of the models atleast in my day-today usage. I am currently struggling to diagnose an ipv6 mis-configuration in my enormous aws cloudformation yaml code. I gave the same input to Claude Opus, Gemini and ChatGPT ( o1 and 4o). 4o was the worst. verbose and waste of my time. Claude completely went off-tangent and began recommending fixes for ipv4 while I specifically asked for ipv6 issues o1 made a suggestion which I tried out and it fixed it. It literally found a needle in the haystack. The solution is working well now. Gemini made a suggestion which almost got it right but it was not a full solution. I must clarify diagnosing network issues on AWS VPC is not my expertise and I use the LLMs to supplement my knowledge.
- blastbking 2y ago
  Sonnet 3.5 as of today is superior to Opus, curious if sonnet could have solved your problem
  - nunodonato 2y ago
    Yes. I find it a bit funny how much people care about leaderboards. I see models going up and down, winning this or that benchmark and yet, for me, Sonnet 3.5 still beats the crap out of all of them.
  - manishsharan 2y ago
    I too was puzzled by the response from Claude. I am using the Anthropic workbench with claude-3-5-sonnet-20241022 (latest) But it think it has to do more with the freshness of training data. AWS IPV6 Egress is a new technology from AWS which was introduced only recently. Previously, we had to deploy NAT gateway which supported IPV4. I am assuming claude-3-5-sonnet-20241022 (latest) was not trained on this data.
zhyder 2y ago
I like that https://artificialanalysis.ai/leaderboards/models https://artificialanalysis.ai/leaderboards/models describes both quality and speed (tokens/s and first chunk s). Not sure how accurate it is; anyone know? Speed and variance of it in particular seems difficult to pin down because providers obviously vary it with load to control their costs.
IAmGraydon 2y ago
https://aider.chat/docs/leaderboards/ https://aider.chat/docs/leaderboards/
lossolo 2y ago
https://livebench.ai/#/ https://livebench.ai/#/
danpalmer 2y ago
Notably, GPT-4o is a "full size" model, whereas Gemini 2 Flash is the small and efficient variant in that family as far as I understand it.
- jug 2y ago
  True that - and I think Gemini-Exp-1206 is Gemini 2.0 Pro in testing. I noticed how they only replaced the "experimental" moniker for one of their experimental models, and it turned into 2.0 Flash. And that still experimental model is currently leading across all categories.

SV_BubbleTime 2y ago

AI benchmarks and leaderboards are complete nonsense though. Find something you like, use it, be ready to look again in a month or two.

falcor84 2y ago
With the accelerating progress, the "be ready to look again" is becoming a full time job that we need to be able to delegate in some way, and I haven't found anything better than benchmarks, leaderboards and reviews. EDIT: Typo
siliconc0w 2y ago
FWIW I've found the 'coding' 'category' of the leaderboard to be reasonably accurate. Claude was the best, o1-mini then was typically stronger, now the Gemini Exp 1206 is at the top. I find myself just paying a la carte via the API rather than paying the $20/mo so I can switch between the models.
hombre_fatal 2y ago
poe.com has a decent model where you buy credits and spend them talking to any LLM which makes it nice to swap between them even during the same conversation instead of paying for multiple subscriptions. Though gpt-4o could say "David Mayer" on poe.com but not on chat.openai.com which makes me wonder if they sometimes cheat and sneak in different models.

manishsharan 2y ago

Leaderboards are not that useful for measuring real-life effectiveness of the models atleast in my day-today usage. I am currently struggling to diagnose an ipv6 mis-configuration in my enormous aws cloudformation yaml code. I gave the same input to Claude Opus, Gemini and ChatGPT ( o1 and 4o). 4o was the worst. verbose and waste of my time. Claude completely went off-tangent and began recommending fixes for ipv4 while I specifically asked for ipv6 issues o1 made a suggestion which I tried out and it fixed it. It literally found a needle in the haystack. The solution is working well now. Gemini made a suggestion which almost got it right but it was not a full solution. I must clarify diagnosing network issues on AWS VPC is not my expertise and I use the LLMs to supplement my knowledge.

blastbking 2y ago
Sonnet 3.5 as of today is superior to Opus, curious if sonnet could have solved your problem
- nunodonato 2y ago
  Yes. I find it a bit funny how much people care about leaderboards. I see models going up and down, winning this or that benchmark and yet, for me, Sonnet 3.5 still beats the crap out of all of them.
- manishsharan 2y ago
  I too was puzzled by the response from Claude. I am using the Anthropic workbench with claude-3-5-sonnet-20241022 (latest) But it think it has to do more with the freshness of training data. AWS IPV6 Egress is a new technology from AWS which was introduced only recently. Previously, we had to deploy NAT gateway which supported IPV4. I am assuming claude-3-5-sonnet-20241022 (latest) was not trained on this data.

zhyder 2y ago

I like that https://artificialanalysis.ai/leaderboards/models https://artificialanalysis.ai/leaderboards/models describes both quality and speed (tokens/s and first chunk s). Not sure how accurate it is; anyone know? Speed and variance of it in particular seems difficult to pin down because providers obviously vary it with load to control their costs.

IAmGraydon 2y ago

https://aider.chat/docs/leaderboards/ https://aider.chat/docs/leaderboards/

lossolo 2y ago

https://livebench.ai/#/ https://livebench.ai/#/

danpalmer 2y ago

Notably, GPT-4o is a "full size" model, whereas Gemini 2 Flash is the small and efficient variant in that family as far as I understand it.

jug 2y ago
True that - and I think Gemini-Exp-1206 is Gemini 2.0 Pro in testing. I noticed how they only replaced the "experimental" moniker for one of their experimental models, and it turned into 2.0 Flash. And that still experimental model is currently leading across all categories.