3 ms·
The most fundamental change is the difference in what models are being trained on. Little images of characters is a trivia type problem, very different from tr
by Nevermark 3y ago
The most fundamental change is the difference in what models are being trained on.
Little images of characters is a trivia type problem, very different from training on the linguistic and visual communication of essentially the whole human race.
Another 33 years of expanded computing resources won’t be training models to mimic the behavior and knowledge of humanity.
That problem (us!) will have been reduced to a toy problem long before then.
- visarga 3y agoI think AI models will evolve by generating synthetic data, filtering and improving it, and then retraining. Possibly with external systems in the loop - code execution, search, human, simulation or robot. Quality won't degrade because there will be a lot of effort put into data filtering and diversity. We can always improve on a model by giving it more time. Model architecture doesn't matter compared to the dataset. Any model from a class can learn the same skills from the same data, but change the data and they all change their abilities - the intelligence is in the data. The future is data engineering, not model architecturing. Human culture, by analogy, evolves faster than human biology. The data is evolving faster than the model. And we are seeing a drastic reduction in novel architectures in AI, diverse datasets applied to the same transformer models in recent years. Even among the transformers, very few variants are largely used, thousands of them abandoned. I like to think of it as language evolution by memetics being the real engine behind intelligence. We and AI are riding the language exponential together.
- eru 3y ago> Model architecture doesn't matter compared to the dataset. Any model from a class can learn the same skills from the same data, but change the data and they all change their abilities - the intelligence is in the data. You might be right in the same sense that big-O notation is 'right'. Constant factor can matter; especially once you have to take energy use into account.
- ResearchCode 3y agoCome close to solving the toy problem of autonomous driving first, we're still waiting.
- Nevermark 3y agoI don’t know. I find pessimistic views, like you are expressing, very strange. My Tesla drives and navigates itself most of the time. 90-95% at least, just not 100%. As apposed to cars 10 or more years ago which didn’t do any of that. To me it is much like the “God of the Gaps” when tremendous progress on a big problem is dismissed negatively, due to the (continuously shrinking) gaps of what it can’t do.
- cma 3y agoWaymo and Cruise are operating without drivers which is much more impressive, even if in limited areas.
- Nevermark 3y agoI don’t know if it is much more impressive than Tesla, given Tesla’s “limited areas” don’t seem very limiting in my experience. But I think all these complementary takes on the problem, with significant year-to-year progress by all three firms, are fantastic. That used to be considered a fast learning curve!
- ResearchCode 3y agoWe are already five years late in autonomous vehicles replacing all truck drivers. We will see if we even have autonomous driving "long before" 33 years have passed. "AGI" (rebranded AI after "AI" failed to deliver) will of course still not be a thing.
- Nevermark 3y agoI have no idea what “late” technology means. And not delivering AGI yet is a problem? What are these broad technology schedule based criticisms founded on? I really want to understand this viewpoint! Hopefully not the over-optimism of anyone who uses optimistic timelines as a motivational force. That’s not real data. Or a suitable benchmark for human progress.