2 ms·
ARC has their own writeup on the result, which offers some nuance. https://arcprize.org/blog/astra https://arcprize.org/blog/astra tl;dr it's 62% when apples-t
by minimaxir 1mo ago
ARC has their own writeup on the result, which offers some nuance. https://arcprize.org/blog/astra https://arcprize.org/blog/astra
tl;dr it's 62% when apples-to-apples to other models, which is still notable.
- ciefa 1mo agoWoah, that is a crazy interesting read!
- debazel 1mo agoARC's harness is just straight up broken. No serious harness removes reasoning context between each step. Not only does this significantly lower performance over all reasoning LLMs, but it also increase cost as you destroy the cache on every turn. Tossing the oldest entry when context fills up instead of using compaction is equally bad with the same issues.