3 ms·
I think you're right. It does seem like the models need exponentially larger datasets to get linear improvements, which are now in the realm of having to buy t
by cbanek 2y ago
I think you're right. It does seem like the models need exponentially larger datasets to get linear improvements, which are now in the realm of having to buy them from large social media companies. The next breakthrough will probably being doing something different rather than doing it slightly better.
- exe34 2y agogiven that language timeseries itself turned out to store some form of intelligence, I wonder how much more of the human mind is trained on video-input/proprioception/action timeseries. i.e. make robots that make small decisions and actions, train on their experience, do more complicated actions, train on those experiences - there's your 10x, 100x, e^x training tokens, save the language for fine tuning. language as a specific task of a general, world-interacting robot.
- khimaros 2y agoapparently this is part of their plan with the Tesla Optimus robots.
- cbanek 2y agoYeah I mean this already exists in the human brain, for example we're more likely to be surprised by vision lower to the ground than in the sky. The reason? Snakes. And a lot of animals have a startle reaction to snakes as it's a common problem out there in the world.