2 ms·
The model gets a question and outputs one answer, so this is the target benchmark. to do this, 64 samples are generated and we pick the most common answer, ie w
by lewkowycz 4y ago
The model gets a question and outputs one answer, so this is the target benchmark. to do this, 64 samples are generated and we pick the most common answer, ie we use majority voting or self-consistency.
- riku_iki 4y agoTarget benchmark accuracy: accuracy of "most common answer". Accuracy in Figure 5: average accuracy over 64 answers. This is mathematically different metrics.