3 ms·
It's benchmaxxed. If they actually matched Opus 4.6 on such a short timeline, it would have been mighty impressive. (Keep in mind this is a new lab and they ar
by solenoid0937 7mo ago
It's benchmaxxed.
If they actually matched Opus 4.6 on such a short timeline, it would have been mighty impressive. (Keep in mind this is a new lab and they are prohibited from doing distills.)
- throwaw12 7mo agohow do you know it's benchmaxxed?
- solenoid0937 7mo agoFriends at Meta with access to the model + personal experience at Meta. Meta's performance process is essentially "show good numbers or you're out." So guess what people do when they don't have good numbers? They fudge them. Happens all across the company.
- prodigycorp 7mo agometa's benchmaxing tendencies are well known. llama4 was mega benchmaxxed, there's nothing that suggests to me that meta's culture has changed.
- spindump8930 7mo agoRe: changes, there's been enormous turnover in AI organizations, and in theory this one was developed by a "new" org. Whether that means less or more benchmaxxing is anyone's guess.
- bob001 7mo agoMore I'd guess since the new org needs to prove itself long enough for stock to vest. Fudge the benchmarks gives them a longer horizon before they're all fired anyways.
- luma 7mo agoFor one, they aren't using the latest version of many of the benchmarks. eg, ARC-AGI 2 and not 3, etc.