3 ms·

> phi-1 attains pass@1 accuracy 50.6% on HumanEval So, a coin toss effectively, right?

• revertmean 3y ago
  No, scoring 50% on a test is not the same as a coin toss, for very obvious reasons.
• dist-epoch 3y ago
  Only if the questions are binary (yes/no). If multi-choice or free-text, random answering will be much below 50%.