3 ms·
They have this plot in the paper. See Figure 5. Average accuracy (averaged over problems) does not change.
by ramasesh 4y ago
They have this plot in the paper. See Figure 5. Average accuracy (averaged over problems) does not change.
- riku_iki 4y agoThe question listed in paper (they modified 20 out of thousands in MATH dataset) as example is very far in calculating complexity from what I quoted in my comment above. Their question for your convenience: At each basketball practice last week, Jenny made twice as many free throws as she made at the previous practice. At her fifth practice she made 48 free throws. How many free throws did she make at the first practice?
- riku_iki 4y ago> accuracy (averaged over problems My understanding is that it is average over 64 solution attempts (model generated 64 possible answers, and they calculated average accuracy of this answers), which is not target metric of benchmark. Additionally, accuracy degradation is clearly visible on graphs, first graph has 4 problems out of 20 bellow the line, last graph has 14 problems bellow the line.
- lewkowycz 4y agoThe model gets a question and outputs one answer, so this is the target benchmark. to do this, 64 samples are generated and we pick the most common answer, ie we use majority voting or self-consistency.
- riku_iki 4y agoTarget benchmark accuracy: accuracy of "most common answer". Accuracy in Figure 5: average accuracy over 64 answers. This is mathematically different metrics.