4 ms·

There is something fundamentally wrong with these models. The brain "works" because it's evolved structure matches or reflects reality. It is not about having

There is something fundamentally wrong with these models. The brain "works" because it's evolved structure matches or reflects reality. It is not about having billions of neurons, but about to have the right structure which matches the environment. My favourite example is how butterflies evolve pictures of eyes on its wings to scare predators, having literally no idea about existence of other creatures. It has been evolved because other creatures have eyes, and they are there, of course. The proper structure of neural networks must be based on such fundamental features, like "most of creatures have eyes" and similar ones. Brain does not have a flat structure, like a billion x billion matrix. It is more clever and simpler that that. A language model must be based on the fundamental notion that there are nouns (things), verbs (processes) and adjectives (attributes). It is that simple.

dahele 6y ago
Slightly apples to oranges comparison. The brain has an incredibly complex architecture, which evolved over millions of years. On top of that, it then develops throughout a human's lifespan. The brain we observe is a "finished product", and even then it has ~150 trillion synapses to do computations [0]. Even massive neural networks have a relatively simple architecture before they are trained. Part of the training process is effectively learning more complex architectures, which are manifested by changing weights. What I'm getting it is that artificial neural networks aren't equivalent to the brain - ANNs are both learning their own structure, on top of the circuits actually doing computations. They are doing the work of millions of years of evolution, genetics, developmental biology, interaction with the environment etc. Perhaps it's to be expected that ANNs will need orders of magnitude greater number of parameters than a brain. An interesting development is meta-learning, where we separate the process for learning the architecture (this could be using deep learning, but not necessarily) with the network actually doing computation (equivalent to the brain). > A language model must be based on the fundamental notion that there are nouns (things), verbs (processes) and adjectives (attributes). I agree, but how does the brain represent these concepts? Some would argue that ANNs do have these concepts, just hidden away in abstract vector representations. Take the visual system, which has been extensively studied - we see the brain represents contrast, edges, shapes and so on very similarly to convolutional NNs. [0] It's likely that this number doesn't come close to capturing the brain's complexity, as it doesn't incorporate parameters like long-term potentiation/depression, synchronization, firing rates, habituation vs sensitization, immunomodulation and likely so much more we haven't yet discovered.
Veedrac 6y ago
If the brain is the answer, the question involves trillions of parameters. Clearly there's more to the brain than just size, but also clearly, the brain is big for a reason. In fact scale is one of the very few things we can say for certain plays a big role in the function of the brain. GOFAI notions of embedding grammars and knowledge webs are just guesses on faith—the evidence points precisely in the opposite direction—and don't really make much sense anyway.

dahele 6y ago

Slightly apples to oranges comparison. The brain has an incredibly complex architecture, which evolved over millions of years. On top of that, it then develops throughout a human's lifespan. The brain we observe is a "finished product", and even then it has ~150 trillion synapses to do computations [0]. Even massive neural networks have a relatively simple architecture before they are trained. Part of the training process is effectively learning more complex architectures, which are manifested by changing weights. What I'm getting it is that artificial neural networks aren't equivalent to the brain - ANNs are both learning their own structure, on top of the circuits actually doing computations. They are doing the work of millions of years of evolution, genetics, developmental biology, interaction with the environment etc. Perhaps it's to be expected that ANNs will need orders of magnitude greater number of parameters than a brain. An interesting development is meta-learning, where we separate the process for learning the architecture (this could be using deep learning, but not necessarily) with the network actually doing computation (equivalent to the brain). > A language model must be based on the fundamental notion that there are nouns (things), verbs (processes) and adjectives (attributes). I agree, but how does the brain represent these concepts? Some would argue that ANNs do have these concepts, just hidden away in abstract vector representations. Take the visual system, which has been extensively studied - we see the brain represents contrast, edges, shapes and so on very similarly to convolutional NNs. [0] It's likely that this number doesn't come close to capturing the brain's complexity, as it doesn't incorporate parameters like long-term potentiation/depression, synchronization, firing rates, habituation vs sensitization, immunomodulation and likely so much more we haven't yet discovered.

Veedrac 6y ago

If the brain is the answer, the question involves trillions of parameters. Clearly there's more to the brain than just size, but also clearly, the brain is big for a reason. In fact scale is one of the very few things we can say for certain plays a big role in the function of the brain. GOFAI notions of embedding grammars and knowledge webs are just guesses on faith—the evidence points precisely in the opposite direction—and don't really make much sense anyway.