> peer review...Their null is GPT4 can ace MIT and they haven't provided any evidence to reject. I haven't been in research for a while, but I don't think that

> peer review...Their null is GPT4 can ace MIT and they haven't provided any evidence to reject. I haven't been in research for a while, but I don't think that's how peer review works. You don't always have to assume the paper's claims (especially ones as novel as this one's) as a null hypothesis, and providing a compelling refutation. The detection of sloppy question framing, and answer feeding via the few-shot learning examples, and the problems with checking with GPT-4 itself reasonably show that there are serious flaws in multiple parts of the experiments described in the paper. > The real conclusion is "we think the paper's results might be weaker if repeated" and that's a good result on its own. No, they don't know enough to say that. The paper's results might be better with better experimentation! Or they might be totally false. The conclusion they provided is accurate precisely because it focuses on the methods and not the conclusions, like this: > One particularly worrying trend is the technique of evaluating a model’s accuracy using a language-based model like GPT-4. While a useful tool, its conclusions should never be overstated or treated as ground truth. ...and this: > Additionally, it is extremely important to reevaluate every data point and perform basic sanity checks before using data at all, whether for training, inference, benchmarking, or something else. Given the small size of the dataset in question, a simple manual validation would have been easily within the scope of the work.