4 ms·
How credible is this benchmark? does it correlated with others real world experience?
by zzleeper 4mo ago
How credible is this benchmark? does it correlated with others real world experience?
- Catloafdev 4mo agoIt's a relatively new benchmark but from what I can tell it has serious cred behind it. I assume it will be picked up as part of the standard suite of CS-related benchmarks soon enough.
- emp17344 4mo agoSeems like it literally popped up yesterday with the express purpose of building hype for this release.
- anthonypasq 4mo agowhat incentive does Cognition have for doing this? seems like complete nonsense speculation on your part.
- bel8 4mo agoWith billions/trillions of dollars floating around, is it hard to imagine benchmarks could be biased? I think it's safe to assume everything AI related is heavily biased until proven otherwise. Just like in pharma.
- camdenreslink 4mo agoPeople game benchmarks for fake internet points to get their favorite web framework to the top of the list. I'm pretty sure they will do it for billions of dollars.
- anthonypasq 4mo agoyou didnt answer my question. Why would cognition be biased towards making anthropic look good?
- gloosx 4mo agoBecause Cognition is a major customer of Anthropic?
- anthonypasq 4mo agothey are also a major customer of OpenAI and every other model maker. whats your point?
- vanuatu 4mo agoi doubt it, cog wants coding agents to be better because it directly improves their product they aren't married to a particular lab, most of their usage is their in house model i believe
- osti 4mo agoAnd notable absence of DeepSWE benchmark where they do badly, but somehow a benchmark that was published yesterday is in this announcement.
- zzleeper 4mo agoExactly.. a bit of a red flag for me..
- swyx 4mo agoteam member here - we had been working on frontiercode for ~6-7months. timing just lined up
- emp17344 4mo agoYeah, right. If this benchmark was truly developed in an independent manner, and the timing just “lined up”, how did Anthropic even know to include results in their model release documentation the day after the benchmark is revealed? It seems like there must have been some collaboration or influence from Anthropic behind the scenes.
- oblio 4mo agoCome on, why are you a jerk about this? Nobody would have 800+ billion reasons to lie by commission or omission here.
- bfeynman 4mo agoGiven it was made by cognition (team behind devin flop) who now just got to wait out until claude and gpt5 basically do all of the work for them - not very. When you read about it, the framework is highly subjective. Which very quickly becomes a problem because its based on heuristics that probably change a bunch with a better code model.
- vanuatu 4mo agothe subjective framework is exactly why its good prior bms relied mostly on unit tests or synthetic judges which are easily benchmaxxed, which leads to nobody trusting benchmarks we need people manually checking the data for good code quality
- vanuatu 4mo agoi worked on one of the benchmarks typically found in new model releases this benchmark looks very good from the methodology. a cog researcher checking the data themselves is very high signal (not scaleable so don't take the benchmark as gospel, but directionally good)
- schipperai 4mo agoCognition did well in documenting their approach [1]. TL;DR - they worked with OSS project maintainers to build tasks. They score models based on whether a PR is mergeable. All tasks are graded by a human researcher. SoTA models have hill-climbing to do which raises the bar and inspires confidence. I'd say it's legit. [1]: https://x.com/cognition/status/2064061031912288715 https://x.com/cognition/status/2064061031912288715
- CSMastermind 4mo agoDeepSWE is the benchmark you want to actually look out for. Only one that aligns with actual user reported results from trying the models.
- ryeguy 4mo agoDid you read the blog post? They compare to deepswe and call it out as the worst one for false positives (failed, but the benchmark assessed it as correct). It also has less language variance.
- CSMastermind 4mo agoI mean yes that is what you'd say if you were writing a blog post about your new benchmark.
- ryeguy 4mo agoSure, but they at least quantified it with data. It's not like they just dropped a sentence saying the above, they showed numbers.
- shimman 4mo agoIt's an unacademic benchmark by a failed VC startup clawing for relevancy.
- piphf 4mo ago[dead]