4 ms·
This statement makes no sense because there is just no good measurement of intelligence for him to make a quantitative statement like that. What's he measuring
by nialv7 2y ago
This statement makes no sense because there is just no good measurement of intelligence for him to make a quantitative statement like that. What's he measuring intelligence with? MMLU? IQ? His gut feeling?
- dash2 2y agoThere are very many tests, leaderboards etc. for AI models, and while none of them are perfect, in the aggregate they might not be a bad measure of intelligence (perhaps locally, in an interval around current models' performance level, since people develop new tests as old ones get saturated).
- nialv7 2y agoYes, using those methods you can have a blurry idea of maybe this model is "smarter" than the other one, etc. But none of them may be used to anchor the claim that "intelligence is proportional to the log of compute", which is the claim sama was making.
- dash2 2y agoSuppose you take the first principal component of performance and fit it to log(compute). Seems like a fairly reasonable way to get an answer.