4 ms·
Weird they're comparing it to really old deepseek v1 models, even v2 has been out a long time now.
by smcleod 2y ago
Weird they're comparing it to really old deepseek v1 models, even v2 has been out a long time now.
- butterfly42069 2y agoMy exact thoughts, especially because DeepseekV2 is meant to be a massive improvement. It seems to be an emerging trend people should look out for that model release sheets often contain comparisons with out of date models and don't inform so much as just try to make the model look "best." It's an annoying trend. Untrustworthy metrics betray untrustworthy morals.
- bubblyworld 2y agoMy barely-informed guess is that they don't have the resources to run it (it's a 200b+ model).
- regularfry 2y agoThey could compare to DeepSeek-Coder-V2-Lite-Instruct. That's a 16B model, and it comes out at 24.3 on LiveCodeBench. Given the size delta they're respectably close - they're only just behind at 23.4. The full V2 is way ahead.
- smcleod 2y agoThat’s for the larger model, most people running it locally use the -lite model (both of which has lots of benchmarks published)