3 ms·
LLMs are closer to discoveries on the spectrum than inventions. Nobody predicted or planned the many emergent capabilities we’ve seen. Almost like magic. Now is
by az226 3y ago
LLMs are closer to discoveries on the spectrum than inventions. Nobody predicted or planned the many emergent capabilities we’ve seen. Almost like magic. Now is a period of moving along the axis to invention with many intentional design, architecture, and feature development alongside testing and evaluation. We are far from done with LLMs, plenty of room for many more discoveries, lots to explore. It’s definitely a precursor to AGI. They offer a platform to build and scale data sets and test beds.
We haven’t had ML models this large before. There’s innovation in architecture but we often come back to the bitter lesson: more data.
We’re likely going to see experimentation with language models to learn from few examples. Fine tuning pretrained LLMs shows they have quite a remarkable ability to learn from few examples.
Liquid AI has a new learning architecture for dynamic learning and much smaller models.
Some people seem mad about the bitter lesson, they want their model based on human features to work better when so far usually more data wins.
I think the next evolution here is in increasing the quality of the training data and giving it more structure. I suspect the right setup can seed emergent capabilities.
- gitfan86 3y agoThe trick is to make many LLMs work together in feedback loops. Some small some big. That will get us to what was previously known as AGI. The definition of AGI will change, but we will have systems that put perform humans in most ways.
- beardedwizard 3y ago> It’s definitely a precursor to AGI. What are you basing this claim on? There is no intelligence in an LLM, only humans fooled by randomness.
- pmontra 3y agoMaybe we've been fooling each other since forever too. However whatever we're doing seems to be different from what LLMs do, at least because of the huge difference in how we train. It's possible that it will end up like airplanes and birds. Airplanes can bring us to the other side of the world in a day by burning a lot of fuel. Birds can get there too in a much longer time and more cheaply. They can also land on a branch of a tree. Airplanes can't and it's too risky for drones.
- blackoil 3y ago> only humans fooled by randomness Is there another kind?
- MeImCounting 3y agoThis is such an interesting take. What do you classify as intelligence? From my perspective theres intelligence in a how to manual. It seems like maybe you mean consciousness? Or creativity?
- az226 3y agoPrecursor as in it will help in synthetic data generation, testing, etc. at scale that gives us more powerful models. It is a necessary intermediate step on our path to AGI.
- beardedwizard 3y agoFair enough, it could be but I'm not sure it has to be in any way...
- nsagent 3y agoYou might want to reconsider your stance on emergent abilities in LLMs considering the NeurIPS 2023 best paper winner is titled: "Are Emergent Abilities of Large Language Models a Mirage?" https://arxiv.org/abs/2304.15004 https://arxiv.org/abs/2304.15004 https://blog.neurips.cc/2023/12/11/announcing-the-neurips-2023-paper-awards/ https://blog.neurips.cc/2023/12/11/announcing-the-neurips-20...
- az226 3y agoModels trained mostly in English and trained only to respond in English can respond in another language. That ability is not a mirage. It’s very very real.