21 ms·
Kagi LLM benchmark updated with general purpose and thinking mode for Sonnet 3.7. https://help.kagi.com/kagi/ai/llm-benchmark.html https://help.kagi.com/kagi/a
by freediver 2y ago
Kagi LLM benchmark updated with general purpose and thinking mode for Sonnet 3.7.
https://help.kagi.com/kagi/ai/llm-benchmark.html https://help.kagi.com/kagi/ai/llm-benchmark.html
Appears to be second most capable general purpose LLM we tried (second to gemini 2.0 pro, in front of gpt-4o). Less impressive in thinking mode, about at the same level as o1-mini and o3-mini (with 8192 token thinking budget).
Overall a very nice update, you get higher quality and higher speed model at same price.
Hope to enable it in Kagi Assistant within 24h!
- jjice 2y agoThank you to the Kagi team for such fast turn around on new LLMs being accessible via the Assistant! The value of Kagi Assistant has been a no-brainer for me.
- hackernewds 2y ago[flagged]
- jjice 2y agoI find that giving encouraging messages when you're grateful is a good thing for everyone involved. I want the devs to know that their work is appreciated. Not everything is a tactical operation to get more subscription purchases - sometimes people like the things they use and want to say thanks and let others know.
- zaphod420 2y agoSome of us just actally really like kagi...
- flixing 2y agoDo you think kagi is the right Eval tool? If so,why?
- freediver 2y agoThe right eval tool depends on your evaluation task. Kagi LLM benchmark focuses on using LLMS in the context of information retrieval (which is what Kagi does) which includes measuring reasoning and instruction following capabilities.
- thefourthchime 2y agoNice, but where is Grok?
- pertymcpert 2y agoPerhaps they're waiting for the Grok API to be public?
- Squarex 2y agoI'm surprised that Gemini 2.0 is first now. I remember that Google models were under performing on kagi benchmarks.
- manmal 2y agoGemini 2 is really good, and insanely fast.
- Workaccount2 2y agoHaving your own hardware to run LLMs will pay dividends. Despite getting off on the wrong foot, I still believe Google is best positioned to run away with the AI lead, solely because they are not beholden to Nvidia and not stuck with a 3rd party cloud provider. They are the only AI team that is top to bottom in-house.
- Squarex 2y agoI've used gemini for it's large context window before. It's a great model. But specifically in this benchmark it has always scored very low. So I wonder what has changed.
- SubiculumCode 2y agoI don't know, but very recent Gemini models have certainly seemed much more impressive...and became my daily.
- ripped_britches 2y ago
- guelo 2y agoHow did you chose the 8192 token thinking budget? I've often seen Deepseek R1 use way more than that.
- freediver 2y agoArbitrary, and even with this budget it is already more verbose (and slower) overall than all the other thinking models - check tokens and latency in the table.
- KTibow 2y agoOne thing I don't understand is why Claude 3.5 Haiku, a non thinking model in the non thinking section, says it has a 8192 thinking budget.
- baobabKoodaa 2y agoI see it in Kagi Assistant already and it's not even 24 hours! Nice.
- Kye 2y agoI thought o3-mini was o1-mini. OpenAI's naming gets confusing.