3 ms·
The number of trials is one issue, but another is selection bias. If GPT-4 got a worse result, would he have tweeted it? And if he did tweet it, would people li
by hackerlight 4y ago
The number of trials is one issue, but another is selection bias. If GPT-4 got a worse result, would he have tweeted it? And if he did tweet it, would people like it/upvote it to the front of HN? Less likely given hype goes more viral than doubt. So they need to publicize the methodology of the experiment ahead of time, then run the experiment after the disclosure. Ideally, the intention to run the experiment gets to the front page of HN prior to it being run, otherwise we've still got some selection bias at play.
- O_H_E 4y agoI'm saddened to inform you that no scientific research being done these days holds up to these standards.
- skybrian 4y agoIt's starting to be done in some cases. Look up registered reports.
- hackerlight 4y agoI'd add that even without registered reports (which are necessary), meta-analyses attempt to cut back on some of the selection bias by scouring the entire literature instead of cherry picking the results that get the most likes on social media.