5 ms·
> A record number of timeouts. Fable 5's extended thinking caused more per-instance timeouts than any model-and-harness combination we have ever tested, directl
by gwern 4mo ago
> A record number of timeouts. Fable 5's extended thinking caused more per-instance timeouts than any model-and-harness combination we have ever tested, directly costing it points. ... Highest cheating volume. We confirmed cheating on 38 of 200 instances, the highest volume recorded since we hardened our prompts, driven almost entirely by memorization of upstream fixes from training data, which no prompt instruction can prevent. ... Four hall-of-fame firsts. Fable 5 solved four instances that no previous model-and-agent combination had ever cracked, and our anti-cheating pipeline leans toward these being genuine solves, not recall.
All of this points to their claim of 'average' as being heavily biased downwards. A model being so up to date and large-parameter it's memorized solutions to your problems is not a knock against it (but rather, a knock against your benchmark being valid), and why should timeouts (especially for a model just launched) be counted at all?
- anematode 4mo ago> memorization of upstream fixes from training data At least now we have up-to-date evidence on their laundering, and the fact that regurgitation absolutely still happens.
- CuriouslyC 4mo agoIt actually happens more with these large overparameterized models, because they have the capacity to memorize more than smaller models.
- Aurornis 4mo agoI agree. This article could have been an interesting read about how coding benchmarks are hard and a constantly moving target, but instead they anchored to a belief that their benchmark is correct. I can't shake the feeling that they knew which headline would generate the most shares and wrote the article to fit instead of acknowledging where they went wrong.
- menaerus 4mo agoIt's a crappy article. I expected better than a click-bait.
- sigmar 4mo agoAgree with this. Strange to me to frame the "training recall" as cheating (33 of the 38 cheating instances). Most people think of "cheating" as breaking rules. How is the LLM model supposed to not use what was put into the weights?
- anematode 4mo agoBy writing a not-identical, but valid, solution? Any modestly complex engineering problem has many solutions. This is an obvious example of why LLM training is so different than human learning.
- simoncion 4mo agoI expect any well-informed corporate lawyer that has thought about this carefully is strongly advising that these tools not be used. When the LLM [0] barfs up some nontrivial code that's covered by the AGPL and your company's devs put it into the company's "all rights reserved" codebase -entirely unaware of its provenance- it's going to be a nightmare to come back from that. [0] ...that Nvidia's CEO says they should be spending 50% of a senior dev's salary per seat per year on...
- senordevnyc 4mo agoThe ship sailed on this a long time ago.
- simoncion 4mo agoOh definitely not. We're not yet solidly out of the "extremely exuberant hype" phase, so the folks that matter tend to not ask questions that dampen the mood.
- senordevnyc 4mo agoSorry to tell you friend, but LLMs have touched the vast majority of active codebases out there, whether you like it or not. You can tell yourself that you’re one of “the folks that matter” (lol) all you want, but we’re never going back.