5 ms·
Something is odd to me about the amount of training data needed. It seems as if animals and humans can learn on far less training data and on a much dirtier an
by api 4y ago
Something is odd to me about the amount of training data needed.
It seems as if animals and humans can learn on far less training data and on a much dirtier and noisier data set. The data an animal or a human trains on has not been curated to remove misleading data or noise.
It seems that one can produce models with amazing capabilities by training them on vast curated data sets, but I get the sense real brains still have a lot of tricks that we don’t understand.
- gibolt 4y agoWe are basically on v2 of modern AI. Transformers significantly improved the effectiveness of training. A few more breakthroughs every few years (not guaranteed) could mean better results with 1/100th the data.
- chewbacha 4y agoIf you want to put it terms of animals, maybe a session is an organism but the model is the DNA. We don’t “learn” from scratch, we have a model that develops our brain and processing centers and wires them to be able to learn from scarce data. This is somewhat what a session is like.
- api 4y agoOur AI models are designed and tuned by engineers, which means they start with a blueprint. They are not randomly generated at the architectural level.
- cjbprime 4y agoI don't agree that a human obviously experiences less learning stimulus than an LLM. For one thing, the human has direct and mutating access to the physical world through many senses. The LLM gets to experience written words, and if it's lucky, images.
- canjobear 4y agoAnimals are starting with a brain in a state that has been pre-trained by hundreds of millions of years of evolution.
- morkalork 4y agoThat the pretraining can be compressed down into DNA, combined with a second source of DNA through sex, then decompressed/expressed to build a brain that wires itself up as it grows from a handful of cells, resulting in a conscious being at the end is nothing shorter of amazing. The numbers involved are also an order magnitude greater. In a human brain, there are approximately 60 trillion connections between neurons. Made with a variety neurotransmitters and regulated by a host hormones as well, not single floating point weights.
- inciampati 4y agoThe model "parameters" for the DNA encoded part can't be much more than about a billion. Basing this on the size of genomes and their information content (they are often repetitive).
- api 4y agoHmm... so a human is somewhere around a 60000G model as models size is currently described?
- nofum 4y agoWhy is a constant stream of ultra high resolution input from ~5 senses for years not count as a huge volume of data? Not to mention all the pre-training data encoded in the brains blueprint via DNA through natural selection? If anything, LLMs are more impressive than humans and animals in that they only have access to text to build a world model from a tabula rasa.
- chrisco255 4y agoWhile the input resolution is indeed high, it's not as if any animal on earth has access to photographic memory of all that data. It is so ephemeral you could scarcely tell me what the last 10 people you passed on the street were wearing, even if it was 10 minutes ago.
- jahewson 4y agoAt inference time, an LLM does not have access to all its training data either.
- chrisco255 4y agoAt inference time, a human being has access to 5 to 9 variables at most an average human can hold in their head at any given time. But I can feed pages and pages to an LLM and it has a perfect representation of every single word and letter in its working memory. It also has access to a look up table of billions of statistical correlations for what the next word or block of words in any sentence should be given the perfect priors. If I'm talking to you, especially in person, I won't be able to tell you what the second word of three sentences ago was. In fact, I might not have even been paying attention, because I'm already thinking at a higher level about what you're communicating to me and the word wasn't all that important. But a LLM certainly uses this information to guide its responses (along with its ginormous lookup table).
- augment002 4y ago> LLMs are more impressive than humans and animals in that they only have access to text to build a world model from a tabula rasa. They don’t build a world model at all. They make inferences from text. Considering they have been trained on 100,000 books, and all of Wikipedia, they are remarkably unintelligent, and really only able to produce text that is consistent with what they have been trained on.
- proc0 4y agoDefinitely. I'm thinking LLMs are capturing real world structure of some kind that the brain also taps into except with a different algorithm and also stores it differently.
- jimmySixDOF 4y agoI like how Max Tegmark frames it by considering Birds and Flight and how we built Planes that can also fly. It turns out Flight, as an action on its own, is not so difficult if you throw enough energy behind a particularly shaped collection of parts. Intelligence, as a behavior observable in the world, appears to be similar in that it emerges in stages at scales as we are discussing. And just like inventing airplanes caused us to say 'all things that fly are not birds', now we may start saying 'all things that display intelligence are not us' (where intelligence is => human level intelligence and "us" could be swapped out with alive/sentient/conscious/etc at least for now and perhaps in all cases)
- deleted 4y ago[deleted]
- coffeebeqn 4y agoThe mammal biological learning algorithms have evolved over millions of years. Neural networks are still very brute force
- orbital-decay 4y agoHuman intelligence is social (and some animals' as well). The majority of your knowledge was distilled by the previous generations, not you personally.
- braindead_in 4y agoIs it encoded in genes? Or symbols?
- orbital-decay 4y agoIn the culture (symbols, then?). For example, a human raised alone in complete wilderness is unlikely to invent any useful numeric representation like our current base-10; they wouldn't even have any use for it without any societal pressure. Similarly, many animals also pass the knowledge, like certain birds spread the songs among the population. Besides, individual human knowledge is really specialized. LLM is expected to know everything.
- deleted 4y ago[deleted]