5 ms·
For reference, here is the terminal-bench leaderboard: https://www.tbench.ai/leaderboard https://www.tbench.ai/leaderboard Looks like it doesn't get close to
by hodgehog11 1y ago
For reference, here is the terminal-bench leaderboard:
https://www.tbench.ai/leaderboard https://www.tbench.ai/leaderboard
Looks like it doesn't get close to GPT-5, Claude 4, or GLM-4.5, but still does reasonably well compared to other open weight models. Benchmarks are rarely the full story though, so time will tell how good it is in practice.
- seunosewa 1y agoThe DeepSeek R1 in that list is the old model that's been replaced. Update: Understood.
- yorwba 1y agoYes, and 31.3% is given in the announcement as the performance of the new v3.1, which would put it in sixteenth place.
- deleted 1y ago[deleted]
- coliveira 1y agoMy personal experience is that it produces high quality results.
- amrrs 1y agoAny example or prompt you use to make this statment?
- imachine1980_ 1y agoI remember asking for quotes about the Spanish conquest of South America because I couldn't remember who said a specific thing. The GPT model started hallucinating quotes on the topic, while DeepSeek responded with, "I don't know a quote about that specific topic, but you might mean this other thing." or something like that then cited a real quote in the same topic, after acknowledging that it wasn't able to find the one I had read in an old book. i don't use it for coding, but for things that are more unique i feel is more precise.
- sync 1y agoI'm doing coreference resolution and this model (w/o thinking) performs at the Gemini 2.5-Pro level (w/ thinking_budget set to -1) at a fraction of the cost.
- SV_BubbleTime 1y agoVine is about the only benchmark I think is real. We made objective systems turn out subjective answers… why the shit would anyone think objective tests would be able to grade them?
- YetAnotherNick 1y agoDepends on the agent. Rank 5 and 15 are claude 4 sonnet, and this stands close to 15th.
- guluarte 1y agotbh companies like anthopic, openai, create custom agents for specific benchmarks
- bazmattaz 1y agoDo you have a source for this? I’m intrigued
- guluarte 1y agohttps://www-cdn.anthropic.com/07b2a3f9902ee19fe39a36ca638e5ae987bc64dd.pdf https://www-cdn.anthropic.com/07b2a3f9902ee19fe39a36ca638e5a... "we iteratively refine prompting by analyzing failure cases and developing prompts to address them."
- amelius 1y agoAren't good benchmarks supposed to be secret?
- wkat4242 1y agoThis industry is currently burning billions a month. With that much money around I don't think any secrets can exist.
- noodletheworld 1y agoHow can a benchmark be secret if you post it to an API to test a model on it? "We totally promise that when we run your benchmark against our API we won't take the data from it and use to be better at your benchmark next time" :P If you want to do it properly you have to avoid any 3rd party hosted model when you test your benchmark, which means you can't have GPT5, claude, etc. on it; and none of the benchmarks want to be 'that guy' who doesn't have all the best models on it. So no. They're not secret.
- dmos62 1y agoHow do you propose that would work? A pipeline that goes through query-response pairs to deduce response quality and then uses the low-quality responses for further training? Wouldn't you need a model that's already smart enough to tell that previous model's responses weren't smart enough? Sounds like a chicken and egg problem.
- segmondy 1y agogarbage benchmark, inconsistent mix of "agent tools" and models. if you wanted to present a meaningful benchmark, the agent tools will stay the same and then we can really compare the models. there are plenty of other benchmarks that disagree with these, with that said. from my experience most of these benchmarks are trash. use the model yourself, apply your own set of problems and see how well it fairs.
- paradite 1y agoHey. I like your roast on benchmarks. I also publish my own evals on new models (using coding tasks that I curated myself, without tools, rated by human with rubrics). Would love you to check out and give your thoughts: Example recent one on GPT-5: https://eval.16x.engineer/blog/gpt-5-coding-evaluation-underwhelming-performance https://eval.16x.engineer/blog/gpt-5-coding-evaluation-under... All results: https://eval.16x.engineer/evals/coding https://eval.16x.engineer/evals/coding
- jstummbillig 1y agoWhich benchmarks are not garbage? I don't consider myself super special. I think it should be doable to create a benchmark that beats me having to test every single new model.
- tonyhart7 1y agoYeah but the pricing is insane, I don't care about SOTA if its not break my bank