3 ms·
This workshop contribution is OK, and the benchmark is somewhat valuable even without the rephrasing part of the problems, but the rephrasing (of only a small n
by pama 2y ago
This workshop contribution is OK, and the benchmark is somewhat valuable even without the rephrasing part of the problems, but the rephrasing (of only a small number of problems) sometimes genuinely makes the problem more confusing to humans as well by either poor phrasing (fig 3), or unneeded breaking of convention (fig 4; points in 2D are often P, with coordinates x,y). It would have been nice to see effects on the rephrasing of the latest/post-training date problems as a function of the increased noising, to delineate part of this confusion. I wonder how much better o3 is on the same benchmark.
Also, the correct title of this contribution is: Putnam-AXIOM: A Functional and Static Benchmark for Measuring Higher Level Mathematical Reasoning