3 ms·

> you could probably put together one reasonable collection of word counting and question answering tasks with average human time of 30 seconds and another coll

by yorwba 2y ago

> you could probably put together one reasonable collection of word counting and question answering tasks with average human time of 30 seconds and another collection with an average human time of 20 minutes where GPT-4 would hit 50% accuracy on each. So do this and pick the one where humans do best. I doubt that doing so would show all progress to be illusory. But it would certainly be interesting to know what the easiest thing is that a human can do but current AIs struggle with.

xg15 2y ago
> But it would certainly be interesting to know what the easiest thing is that a human can do but current AIs struggle with. Still "Count the R's" apparently.
K0balt 2y ago
The problem , really, is human cognitive dissonance. We draw false conclusions that competence at some tasks implies competence at another. It’s not a universal human problem, we intuit that a front end loader , just because it can dig really well, is not therefore good at all other tasks. But when it comes down to cognition, our models break down quickly. I suspect this is because our proxies are predicated on a task set that inherently includes the physical world, which at some level connects all tasks and creates links between capabilities that generally pervade our environment. LLMs do not exist in this physical world, and are therefore not within the set of things that can be reasoned about with those proxies. This will probably gradually change with robotics, as the competencies required to exist and function in the physical world will (I postulate) generalize to other tasks in such a way that it more closely matches the pattern that our assumptions are based on. Of course, if we segregate intelligence into isolated modules for motility and cognition, this will not be the case as we will not be taking advantage of that generalization. I think that would be a big mistake, especially in light of the hypotheses that the massive leap in capabilities of LLMs came more from the training on things we weren’t specifically trying to achieve- the bulk of seemingly irrelevant data that unlocked simple language processing into reasoning and world modeling.
- the8472 2y ago
  > LLMs do not exist in this physical world, and are therefore not within the set of things that can be reasoned about with those proxies. Perhaps not the mainstream models, but deepmind has been working on robotics models with simulated and physical RL for years https://deepmind.google/discover/blog/rt-2-new-model-translates-vision-and-language-into-action/ https://deepmind.google/discover/blog/rt-2-new-model-transla...
- mentalgear 2y ago
  what you are describing are world models and physical AI, which has recently become much more mainstream after the recent nvidia GDC.
AIPedant 2y ago
Dogs can pass a dog-appropriate variant of this test: https://xcancel.com/SpencerKSchiff/status/1910106368205336769#m https://xcancel.com/SpencerKSchiff/status/191010636820533676... (the dog test uses a treat on one string and junk on the other, they have to pull the correct string to get the treat) This was before o3, but another tweet I saw (don't have the link) suggests it's also completely incapable of getting it.

by yorwba 2y ago

xg15 2y ago

> But it would certainly be interesting to know what the easiest thing is that a human can do but current AIs struggle with. Still "Count the R's" apparently.

K0balt 2y ago

The problem , really, is human cognitive dissonance. We draw false conclusions that competence at some tasks implies competence at another. It’s not a universal human problem, we intuit that a front end loader , just because it can dig really well, is not therefore good at all other tasks. But when it comes down to cognition, our models break down quickly. I suspect this is because our proxies are predicated on a task set that inherently includes the physical world, which at some level connects all tasks and creates links between capabilities that generally pervade our environment. LLMs do not exist in this physical world, and are therefore not within the set of things that can be reasoned about with those proxies. This will probably gradually change with robotics, as the competencies required to exist and function in the physical world will (I postulate) generalize to other tasks in such a way that it more closely matches the pattern that our assumptions are based on. Of course, if we segregate intelligence into isolated modules for motility and cognition, this will not be the case as we will not be taking advantage of that generalization. I think that would be a big mistake, especially in light of the hypotheses that the massive leap in capabilities of LLMs came more from the training on things we weren’t specifically trying to achieve- the bulk of seemingly irrelevant data that unlocked simple language processing into reasoning and world modeling.

the8472 2y ago
> LLMs do not exist in this physical world, and are therefore not within the set of things that can be reasoned about with those proxies. Perhaps not the mainstream models, but deepmind has been working on robotics models with simulated and physical RL for years https://deepmind.google/discover/blog/rt-2-new-model-translates-vision-and-language-into-action/ https://deepmind.google/discover/blog/rt-2-new-model-transla...
mentalgear 2y ago
what you are describing are world models and physical AI, which has recently become much more mainstream after the recent nvidia GDC.

AIPedant 2y ago

Dogs can pass a dog-appropriate variant of this test: https://xcancel.com/SpencerKSchiff/status/1910106368205336769#m https://xcancel.com/SpencerKSchiff/status/191010636820533676... (the dog test uses a treat on one string and junk on the other, they have to pull the correct string to get the treat) This was before o3, but another tweet I saw (don't have the link) suggests it's also completely incapable of getting it.