3 ms·
> accuracy (averaged over problems My understanding is that it is average over 64 solution attempts (model generated 64 possible answers, and they calculated
by riku_iki 4y ago
> accuracy (averaged over problems
My understanding is that it is average over 64 solution attempts (model generated 64 possible answers, and they calculated average accuracy of this answers), which is not target metric of benchmark.
Additionally, accuracy degradation is clearly visible on graphs, first graph has 4 problems out of 20 bellow the line, last graph has 14 problems bellow the line.
- lewkowycz 4y agoThe model gets a question and outputs one answer, so this is the target benchmark. to do this, 64 samples are generated and we pick the most common answer, ie we use majority voting or self-consistency.
- riku_iki 4y agoTarget benchmark accuracy: accuracy of "most common answer". Accuracy in Figure 5: average accuracy over 64 answers. This is mathematically different metrics.