3 ms·
> the models generalize well only on tasks within a small neighborhood of the specific tasks they've been trained on, and even then with severe caveats. the fro
by againstapples 11d ago
> the models generalize well only on tasks within a small neighborhood of the specific tasks they've been trained on, and even then with severe caveats. the frontier labs have developed a general recipe to teach models almost any specific task enjoying clearly defined levels of task performance; many tasks are covered in the training data
Is this really any different to how humans learn, it takes a lot of training on one specific task to make a human expert as well?
- bananzamba 11d agoAlso doesn't the very good ARC AGI 2 score of GPT-6 Astra kinda contradict this, since each problem is its own game with very different rules
- JohnMakin 11d ago> Is this really any different to how humans learn yes.
- knuppar 11d agobeing a bit more specific: the sample efficiency of humans is orders of magnitude larger for more abstract concepts. the same doesn't hold for memory-intensive tasks though (like any kind of trivia), but that only takes you so far.
- danpalmer 11d agoWe've had technology beating humans on memory for millennia, and we've had technology beating humans on computation for many decades now. The tricky thing with LLMs is describing what they actually do. They are too clearly beating humans on some things, but what exactly? Memory – already done, they're bad at basic computation (all LLMs just write code for actual computation/calculation). And as you say, they do badly at more abstract concepts.
- bravoetch 11d agoI was a young child when I learned chess by reading a short book, then practicing with a friend. That is not how LLMs learn. I'm no expert on LLMs, but if you showed a human all chess games and books in all history and then said 'play chess' and they still kept making illegal moves, they would have to have a brain injury.
- harimau777 11d agoIt seems like an LLM potentially could learn that way if each practice game it participated in was added to its training data.
- willmarch 11d agoYes, this is essentially how AlphaGo and AlphaZero algorithms work to train superhuman Go/chess/shogi agents. It’s an elegant algorithm that is analogous to how humans learn games.
- zug_zug 11d agoWell except AlphaZero played 44 million chess games in that time (and actually played with a 44 core computer). So I'd like to point out that the human is still just a few orders of magnitude more efficient.
- willmarch 11d agoYes, we all know that biological systems are more efficient than machines through billions of years of evolution and natural selection but the overall process is largely the same (interacting with an environment, learning from results, improving underlying architecture, etc); efficiencies will come with more time and improvements.
- zug_zug 10d agoWell if we make AI that learns at the rates humans do, it'll fundamentally undermine and destroy the relevance of all existing AI. It sounds to me like you're saying "we basically are there it's just a matter of degree" and I'm saying "No it's orders of magnitude off and probably won't be using LLMs at all and maybe a very fundamentally different type of neural net technology that hasn't been invented yet."
- danielmarkbruce 11d agoHumans wear a lot of hats when the do work. They don't even realize how many. My experience with building real systems using LLMs is that you have to be very explicit about such hats and you don't realize how many are worn until you see edge case after edge case after edge case. Check this. Check that. Check this. Check that. Check check check.