That paper makes some very fair criticisms, I think that GPT-4's performance in exams (bar exam, LSAT, AP bio, etc.) is far more impressive than its ability to

That paper makes some very fair criticisms, I think that GPT-4's performance in exams (bar exam, LSAT, AP bio, etc.) is far more impressive than its ability to beat benchmarks. My concern is that fragments of those exams might be found in the training set, and it is hard to see how to account for that. Even if we use exams ostensibily written more recently than the training data was collected, exam boards do tend to reuse questions (sometimes verbatim, sometimes slightly modified).