3 ms·
The hn title is really pushing way beyond what the study can shoulder. The title from the result chapter sums it up better; "ChatGPT yields moderate accuracy ap
by tokai 4y ago
The hn title is really pushing way beyond what the study can shoulder. The title from the result chapter sums it up better; "ChatGPT yields moderate accuracy approaching passing performance on USMLE"
The study was done on public available questions that have been used in usmle exams. They used two physician reviewers to judge ChatGPTs textual answers. While really interesting no exam was actually passed, and the authors never claim that.
- option 4y agoPublic questions really means they have likely been in training data together with their answers
- tokai 4y agoFrom the paper: Random spot checking was performed to ensure that none of the answers, explanations, or related content were indexed on Google prior to January 1, 2022, representing the last date accessible to the ChatGPT training dataset. So its maybe not a problem, who knows