6 ms·
From GPT 5.1 Thinking: ARC AGI v2: 17.6% -> 52.9% SWE Verified: 76.3% -> 80% That's pretty good!
by josalhor 10mo ago
From GPT 5.1 Thinking:
ARC AGI v2: 17.6% -> 52.9%
SWE Verified: 76.3% -> 80%
That's pretty good!
- verdverm 10mo agoWe're also in benchmark saturation territory. I heard it speculated that Anthropic emphasizes benchmarks less in their publications because internally they don't care about them nearly as much as making a model that works well on the day-to-day
- quantumHazer 10mo agoSeems pretty false if you look at the model card and web site of Opus 4.5 that is… (check notes) their latest model.
- verdverm 10mo agoBuilding a good model generally means it will do well on benchmarks too. The point of the speculation is that Anthropic is not focused on benchmaxxing which is why they have models people like to use for their day-to-day. I use Gemini, Anthropic stole $50 from me (expired and kept my prepaid credits) and I have not forgiven them yet for it, but people rave about claude for coding so I may try the model again through Vertex Ai... The person who made the speculation I believe was more talking about blog posts and media statements than model cards. Most ai announcements come with benchmark touting, Anthropic supposedly does less / little of this in their announcements. I haven't seen or gathered the data to know what is truth
- Mistletoe 10mo agoHow do you measure whether it works better day to day without benchmarks?
- standardUser 10mo agoSubscriptions.
- mrguyorama 10mo agoAh yes, humans are famously empirical in their behavior and we definitely do not have direct evidence of the "best" sports players being much more likely than the average to be superstitious or do things like wear "lucky underwear" or buy right into scam bracelets that "give you more balance" using a holographic sticker.
- standardUser 10mo agoIt's all the shareholders care about. These are not research institutions.
- bulbar 10mo agoManually labeling answers maybe? There exist a lot of infrastructure built around and as it's heavily used for 2 decades and it's relatively cheap. That's still benchmarking of course, but not utilizing any of the well known / public ones.
- verdverm 10mo agoInternal evals, Big AI certainly has good, proprietary training and eval data, it's one reason why their models are better
- aydyn 10mo agoThen publish the results of those internal evals. Public benchmark saturation isn't an excuse to be un-quantitative.
- brokensegue 10mo agohow do you quantitatively measure day-to-day quality? only thing i can think is A/B tests which take a while to evaluate
- verdverm 10mo agomore or less this, but also synthetic if you think about GANs, it's all the same concept 1. train model (agent) 2. train another model (agent) to do something interesting with/to the main model 3. gain new capabilities 4. iterate You can use a mix of both real and synthetic chat sessions or whatever you want your model to be good at. Mid/late training seems to be where you start crafting personality and expertises. Getting into the guts of agentic systems has me believing we have quite a bit of runway for iteration here, especially as we move beyond single model / LLM training. I still need to get into what all is de jour in the RL / late training, that's where a lot of opportunity lies from my understanding so far Nathan Lambert (https://bsky.app/profile/natolambert.bsky.social https://bsky.app/profile/natolambert.bsky.social) from Ai2 (https://allenai.org/ https://allenai.org/) & RLHF Book (https://rlhfbook.com/ https://rlhfbook.com/) has a really great video out yesterday about the experience training Olmo 3 Think https://www.youtube.com/watch?v=uaZ3yRdYg8A https://www.youtube.com/watch?v=uaZ3yRdYg8A
- HDThoreaun 10mo agoArc-AGI is just an iq test. I don’t see the problem with training it to be good at iq tests because that’s a skill that translates well.
- CamperBob2 10mo agoExactly. In principle, at least, the only way to overfit to Arc-AGI is to actually be that smart. Edit: if you disagree, try actually TAKING the Arc-AGI 2 test, then post.
- npinsker 10mo agoCompletely false. This is like saying being good at chess is equivalent to being smart. Look no farther than the hodgepodge of independent teams running cheaper models (and no doubt thousands of their own puzzles, many of which surely overlap with the private set) that somehow keep up with SotA, to see how impactful proper practice can be. The benchmark isn’t particularly strong against gaming, especially with private data.
- CamperBob2 10mo agoCompletely false. This is like saying being good at chess is equivalent to being smart. No, it isn't. Go take the test yourself and you'll understand how wrong that is. Arc-AGI is intentionally unlike any other benchmark.
- fwip 10mo agoTook a couple just now. It seems like a straight-forward generalization of the IQ tests I've taken before, reformatted into an explicit grid to be a little bit friendlier to machines. Not to humble-brag, but I also outperform on IQ tests well beyond my actual intelligence, because "find the pattern" is fun for me and I'm relatively good at visual-spatial logic. I don't find their ability to measure 'intelligence' very compelling.
- 10mo ago
- stego-tech 10mo agoThese models still consistently fail the only benchmark that matters: if I give you a task, can you complete it successfully without making shit up? Thus far they all fail. Code outputs don’t run, or variables aren’t captured correctly, or hallucinations are stated as factual rather than suspect or “I don’t know.” It’s 2000’s PC gaming all over again (“gotta game the benchmark!”).
- verdverm 10mo agoI'm not sure, here's my anecdotal counter example, was able to get gemini-2.5-flash, in two turns, to understand and implement something I had done separately first, and it found another bug (also that I had fixed, but forgot was in this path) That I was able to have a flash model replicate the same solution I had, to two problems in two turns, it's just the opposite experience of your consistency argument. I'm using tasks I've already solved as the evals while developing my custom agentic setup (prompts/tools/envs). They are able to do more of them today then they were even 6-12 months ago (pre-thinking models). https://bsky.app/profile/verdverm.com/post/3m7p7gtwo5c2v https://bsky.app/profile/verdverm.com/post/3m7p7gtwo5c2v
- stego-tech 10mo agoAnd therein lies the rub for why I still approach this technology with caution, rather than charge in full steam ahead: variable outputs based on immensely variable inputs. I read stories like yours all the time, and it encourages me to keep trying LLMs from almost all the major vendors (Google being a noteworthy exception while I try and get off their platform). I want to see the magic others see, but when my IT-brain starts digging in the guts of these things, I’m always disappointed at how unstructured and random they ultimately are. Getting back to the benchmark angle though, we’re firmly in the era of benchmark gaming - hence my quip about these things failing “the only benchmark that matters.” I meant for that to be interpreted along the lines of, “trust your own results rather than a spreadsheet matrix of other published benchmarks”, but I clearly missed the mark in making that clear. That’s on me.
- verdverm 10mo ago
- poormathskills 10mo agoFor a minor version update (5.1 -> 5.2) that's a way bigger improvement than I would have guessed.
- beering 10mo agoModel capability improvements are very uneven. Changes between one model and the next tend to benefit certain areas substantially without moving the needle on others. You see this across all frontier labs’ model releases. Also the version numbering is BS (remember GPT-4.5 followed by GPT-4.1?).
- catigula 10mo agoYes, but it's not good enough. They needed to surpass Opus 4.5.
- mikairpods 10mo agothat is better...?
- minimaxir 10mo agoNote that GPT 5.2 newly supports a "xhigh" reasoning level, which could explain the better benchmarks. It'll be noteworthy to see the cost-per-task on ARC AGI v2.
- granzymes 10mo ago> It'll be noteworthy to see the cost-per-task on ARC AGI v2. Already live. gpt-5.2-pro scores a new high of 54.2% with a cost/task of $15.72. The previous best was Gemini 3 Pro (54% with a cost/task of $30.57). The best bang-for-your-buck is the new xhigh on gpt-5.2, which is 52.9% for $1.90, a big improvement on the previous best in this category which was Opus 4.5 (37.6% for $2.40). https://arcprize.org/leaderboard https://arcprize.org/leaderboard
- minimaxir 10mo agoHuh, that is indeed up and to left of Opus.
- walletdrainer 10mo ago5.1-codex supports that too, no? Pretty sure I’ve been using xhigh for at least a week now
- causal 10mo agoThat ARC AGI score is a little suspicious. That's a really tough for AI benchmark. Curious if there were improvements to the test harness because that's a wild jump in general problem solving ability for an incremental update.
- taurath 10mo agoI don’t think their words mean just about anything, only the behavior of the models. Still waiting of Full Self Driving myself.
- woeirua 10mo agoThey're clearly building better training datasets and doing extensive RL on these benchmarks over time. The out of distribution performance is still awful.
- thinkingtoilet 10mo agoOpen AI has already been busted for getting benchmark information and training the models on that. At this point if you believe Sam Altman, I have a bridge to sell you.
- fuddle 10mo agoI don't think SWE Verified is an ideal benchmark, as the solutions are in the training dataset.
- joshuahedlund 10mo agoI would love for SWE Verified to put out a set of fresh but comparable problems and see how the top performing models do, to test against overfitting.