3 ms·

By what metrics are they 99% as good? There are a lot of benchmarks out there. Please share them. I think the answer lies in the "we actually care a lot abou

By what metrics are they 99% as good? There are a lot of benchmarks out there. Please share them. I think the answer lies in the "we actually care a lot about that 1% (which is actually a lot more than 1%)".

Topfi 9mo ago
I'm still testing myself and cannot make a confident statement yet, but Artifical Analysis is a solid and independent, though also to be fair somewhat imperfect source for a general overview: https://artificialanalysis.ai/ https://artificialanalysis.ai/ Purely according to Artificial Analysis, Kimi K2.5 is rather competitive in regard to pure output quality, agentic evals are also close to or beating US made frontier models and, lest we forget, the model is far more affordable than said competitors, to a point where it is frankly silly that we are actually comparing them. For what it's worth, of the models I have been able to test as of yet, when focusing purely on raw performance (meaning solely task adherence, output quality and agentic capabilities; so discounting price, speed, hosting flexibility), I have personally found the prior Kimi K2 Thinking model to be overall more usable and reliable than Gemini 3 Pro and Flash. Purely on output quality in very specific coding tasks, Opus 4.5 was in my testing leaps and bounds superior of both the Gemini models and K2 Thinking however, though task adherence was surprisingly less reliable than Haiku 4.5 or K2 Thinking. Being many times more expensive and in some cases less reliably adhering to tasks, I really cannot say that Opus 4.5 is superior or Kimi K2 Thinking is inferior here. The latter is certainly better in my specific usage than any Gemini model and again, I haven't yet gone through this with K2.5. I try not to just presume from the outset that K2.5 is better than K2 Thinking, though even if K2.5 remains at the same level of quality and reliability, just with multi modal input, that'd make the model very competitive.
samuelknight 9mo ago
Open models have been about 6 to 9 months behind frontier models, and this has been the case since 2024. That is a very long time for this technology at it's current rate of development. If fast takeoff theory is right, this should widen (although with Kimi K2.5 it might have actually shortened). If we consider what typically happens with other technologies, we would expect open models to match others on general intelligence benchmarks in time. Sort of like how every brand of battery-powered drill you find at the store is very similar, despite being head and shoulders better than the best drill from 25 years ago.
- hobofan 9mo ago
  > That is a very long time for this technology at it's current rate of development. Yes, as long as that gap stays consistent, there is no problem with building on ~9 months old tech from a business perspective. Heck, many companies are lagging behind tech advancements by decades and are doing fine.
- SecretDreams 9mo ago
  > Sort of like how every brand of battery-powered drill you find at the store is very similar, despite being head and shoulders better than the best drill from 25 years ago. They all get made in China, mostly all in the same facilities. Designs tend to converge under such conditions. Especially since design is not open loop - you talk to the supplier that will make your drill and the supplier might communicate how they already make drills for others.

Topfi 9mo ago

I'm still testing myself and cannot make a confident statement yet, but Artifical Analysis is a solid and independent, though also to be fair somewhat imperfect source for a general overview: https://artificialanalysis.ai/ https://artificialanalysis.ai/ Purely according to Artificial Analysis, Kimi K2.5 is rather competitive in regard to pure output quality, agentic evals are also close to or beating US made frontier models and, lest we forget, the model is far more affordable than said competitors, to a point where it is frankly silly that we are actually comparing them. For what it's worth, of the models I have been able to test as of yet, when focusing purely on raw performance (meaning solely task adherence, output quality and agentic capabilities; so discounting price, speed, hosting flexibility), I have personally found the prior Kimi K2 Thinking model to be overall more usable and reliable than Gemini 3 Pro and Flash. Purely on output quality in very specific coding tasks, Opus 4.5 was in my testing leaps and bounds superior of both the Gemini models and K2 Thinking however, though task adherence was surprisingly less reliable than Haiku 4.5 or K2 Thinking. Being many times more expensive and in some cases less reliably adhering to tasks, I really cannot say that Opus 4.5 is superior or Kimi K2 Thinking is inferior here. The latter is certainly better in my specific usage than any Gemini model and again, I haven't yet gone through this with K2.5. I try not to just presume from the outset that K2.5 is better than K2 Thinking, though even if K2.5 remains at the same level of quality and reliability, just with multi modal input, that'd make the model very competitive.

samuelknight 9mo ago

Open models have been about 6 to 9 months behind frontier models, and this has been the case since 2024. That is a very long time for this technology at it's current rate of development. If fast takeoff theory is right, this should widen (although with Kimi K2.5 it might have actually shortened). If we consider what typically happens with other technologies, we would expect open models to match others on general intelligence benchmarks in time. Sort of like how every brand of battery-powered drill you find at the store is very similar, despite being head and shoulders better than the best drill from 25 years ago.

hobofan 9mo ago
> That is a very long time for this technology at it's current rate of development. Yes, as long as that gap stays consistent, there is no problem with building on ~9 months old tech from a business perspective. Heck, many companies are lagging behind tech advancements by decades and are doing fine.
SecretDreams 9mo ago
> Sort of like how every brand of battery-powered drill you find at the store is very similar, despite being head and shoulders better than the best drill from 25 years ago. They all get made in China, mostly all in the same facilities. Designs tend to converge under such conditions. Especially since design is not open loop - you talk to the supplier that will make your drill and the supplier might communicate how they already make drills for others.