3 ms·
If you look at the appendix, you can see example transcripts. The sample they provide looks like a very bad eval. It’s encoding an originally visual problem in
by theptip 3y ago
If you look at the appendix, you can see example transcripts. The sample they provide looks like a very bad eval.
It’s encoding an originally visual problem into a textual matrix form, and then expecting GPT to recognize visual correlations. You simply can’t compare these two tasks! Most humans wouldn’t recognize the 5x5 matrix for a 4x4 square.
So the comparison with “human level” is completely invalid. And even the valid comparison is only measuring visio-spatial intelligence, not IQ.
- Sharlin 3y agoHumans are biased towards grokking two-dimensional information arranged in grids because that's how vision works. (Non-V) GPT is not. Its natural gestalt has a vast number of dimensions. It is, and it should, be able to find correlations across data no matter how the input is arranged. (Note that for GPT text is not linear even if it appears so to humans.) To humans it would be qualitatively more difficult to recognize patterns in four or six or ten-dimensional data than in two dimensions. To GPT, it's just another variation of the same problem, one that's at most quantitatively more difficult. > And even the valid comparison is only measuring visio-spatial intelligence, not IQ. You're the first one to mention IQ. The paper definitely does not. In any case "visio-spatial intelligence" is a human concept. GPT doesn't distinguish between "visio-spatial" and any other kind of pattern recognition.
- famouswaffles 3y ago>It is, and it should, be able to find correlations across data no matter how the input is arranged. That doesn't mean it doesn't favor certain arrangements. https://arxiv.org/abs/2305.18354 https://arxiv.org/abs/2305.18354 Linear is in fact better for this benchmark