4 ms·
My guess is that "g" here is primarily the ability of the model to recognize and respond to multiple choice questions (the "performance" here is almost entirely
by Imnimo 3y ago
My guess is that "g" here is primarily the ability of the model to recognize and respond to multiple choice questions (the "performance" here is almost entirely measured on MMLU). Weaker models often do things like recite back the list of answers or predict that the next text will be another question.
- famouswaffles 3y agoHe performs the test on an MMLU heavy set yes but then does it again without it to the same results. Also, another paper on finding general factors https://arxiv.org/abs/2306.10062 https://arxiv.org/abs/2306.10062 Honestly this seems more validation than surprising revelation.