3 ms·

1. No Language model is yet even close to the scale of the human brain 2. Depending on what exactly you're trying to teach (perfect grammar, paragraphs of cohe

1. No Language model is yet even close to the scale of the human brain 2. Depending on what exactly you're trying to teach (perfect grammar, paragraphs of coherent text, basic reasoning), much less data is needed. https://arxiv.org/abs/2305.07759 https://arxiv.org/abs/2305.07759 3. Brains don't start at 0. Evolution, dna/rna etc. There's obviously some pre disposition for language learning in humans but that alone isn't enough ground for a "universal grammar" 4. We really do take in an enormous amount of data (not text specifically)

lisasays 3y ago
1. Measured how?
- famouswaffles 3y ago
  The closest biological analogue for an artificial parameter is a synapse. Even then, Synapses, and the brain in general, have more degrees of freedom than just a single floating point parameter allows. Humans have some 50 to 100 trillion synapses. Who knows how far scaling goes in a transformer but simply increasing the parameter count increases performance so far. Either way, we're not really close to emulating the complexity of the brain especially when taking neurons into account (one human neuron is far more complex than any one artificial parameter) In other words, simpler building blocks and smaller sie. Of course, how much complexity is needed for Intelligence is unknown.
- civilitty 3y ago
  The connectome [1]. There are about 100B +- 20B neurons in the average human brain. Each neuron has thousands of synapses and each synapse is made up of dozens to hundreds of neurotransmitter channels. That’s at least a quadrillion parameters which is at least three orders of magnitude bigger than SOTA LLMs assuming each of those synapse channels maps to a parameter. That is an absurd assumption given neuroplasticity, which operates at a high level to adapt the neural network as it learns. See the plasticity section in the linked wikipedia articles: connections between neurons can grow or even get removed on the scale of hours and days. The human brain has entire biological systems supporting intelligence for which there is no real ML equivalent because our ML architectures are static. Given a family member’s research on neurotransmitter potentiation, I’d estimate 10 to 100 parameters per channel to get full fidelity of the human brain. (This is absurdly speculative - we have no idea at which fidelity will intelligence emerge) [1] https://en.m.wikipedia.org/wiki/Connectome https://en.m.wikipedia.org/wiki/Connectome
hwillis 3y ago
> No Language model is yet even close to the scale of the human brain If GPT-4 has 100 trillion parameters, it has as many parameters as the human brain has synapses. Synapses are a lot simpler than parameters; they're digital. A single neuron needs many synapses, all of roughly equal weight, emitting many pulses over a short time in order to convey a single weighted value. On top of that, you may have heard that the human brain does a lot of things besides writing. You subtract the motor cortex, the visual and limbic systems etc... a 100 trillion parameter model is unambiguously larger than the language processing portions of the human brain. > Brains don't start at 0. Evolution, dna/rna etc. There's obviously some pre disposition for language learning in humans but that alone isn't enough ground for a "universal grammar" The human genome is 24 gigabits long. It's negligibly small compared to a language model.
- opportune 3y ago
  Synapses, and the brain in general, have more degrees of freedom than just a single floating point parameter allows. There are multiple neurotransmitters floating within the synapse, each side of the synapse has different numbers/density of various receptors, a bunch of proteins and such are constantly being created and used by each cell to change the brain structure itself. It’s not just a digital signal
- dahart 3y ago
  Google says gpt 4 has ~1 trillion parameters.
- famouswaffles 3y ago
  >If GPT-4 has 100 trillion parameters It doesn't. >Synapses are a lot simpler than parameters Not true but someone else has explained why. >a 100 trillion parameter model is unambiguously larger than the language processing portions of the human brain. GPT-4 is not that big. The technology to run such a model at scale is simply not feasible yet. >The human genome is 24 gigabits long. It's negligibly small compared to a language model. I'm sorry bit this makes no sense. How many gigabits long the genome is has no bearing on how impactful it is in steering the development of the human brain in comparison to a ML model.
- 3y ago
gizmo686 3y ago
> Brains don't start at 0. Evolution, dna/rna etc. There's obviously some pre disposition for language learning in humans but that alone isn't enough ground for a "universal grammar" That is literally what universal grammar is. All that is left is to argue about the size and content of UG.
- famouswaffles 3y ago
  No. Chomsky's theory was more than a vague and fairly obvious, "humans are pre disposed to language learning". The latter doesn't say anything about what causes these pre dispositions. Chomsky tries to do so. https://www.scientificamerican.com/article/evidence-rebuts-chomsky-s-theory-of-language-learning/ https://www.scientificamerican.com/article/evidence-rebuts-c...

lisasays 3y ago

1. Measured how?

famouswaffles 3y ago
The closest biological analogue for an artificial parameter is a synapse. Even then, Synapses, and the brain in general, have more degrees of freedom than just a single floating point parameter allows. Humans have some 50 to 100 trillion synapses. Who knows how far scaling goes in a transformer but simply increasing the parameter count increases performance so far. Either way, we're not really close to emulating the complexity of the brain especially when taking neurons into account (one human neuron is far more complex than any one artificial parameter) In other words, simpler building blocks and smaller sie. Of course, how much complexity is needed for Intelligence is unknown.
civilitty 3y ago
The connectome [1]. There are about 100B +- 20B neurons in the average human brain. Each neuron has thousands of synapses and each synapse is made up of dozens to hundreds of neurotransmitter channels. That’s at least a quadrillion parameters which is at least three orders of magnitude bigger than SOTA LLMs assuming each of those synapse channels maps to a parameter. That is an absurd assumption given neuroplasticity, which operates at a high level to adapt the neural network as it learns. See the plasticity section in the linked wikipedia articles: connections between neurons can grow or even get removed on the scale of hours and days. The human brain has entire biological systems supporting intelligence for which there is no real ML equivalent because our ML architectures are static. Given a family member’s research on neurotransmitter potentiation, I’d estimate 10 to 100 parameters per channel to get full fidelity of the human brain. (This is absurdly speculative - we have no idea at which fidelity will intelligence emerge) [1] https://en.m.wikipedia.org/wiki/Connectome https://en.m.wikipedia.org/wiki/Connectome

hwillis 3y ago

> No Language model is yet even close to the scale of the human brain If GPT-4 has 100 trillion parameters, it has as many parameters as the human brain has synapses. Synapses are a lot simpler than parameters; they're digital. A single neuron needs many synapses, all of roughly equal weight, emitting many pulses over a short time in order to convey a single weighted value. On top of that, you may have heard that the human brain does a lot of things besides writing. You subtract the motor cortex, the visual and limbic systems etc... a 100 trillion parameter model is unambiguously larger than the language processing portions of the human brain. > Brains don't start at 0. Evolution, dna/rna etc. There's obviously some pre disposition for language learning in humans but that alone isn't enough ground for a "universal grammar" The human genome is 24 gigabits long. It's negligibly small compared to a language model.

opportune 3y ago
Synapses, and the brain in general, have more degrees of freedom than just a single floating point parameter allows. There are multiple neurotransmitters floating within the synapse, each side of the synapse has different numbers/density of various receptors, a bunch of proteins and such are constantly being created and used by each cell to change the brain structure itself. It’s not just a digital signal
dahart 3y ago
Google says gpt 4 has ~1 trillion parameters.
famouswaffles 3y ago
>If GPT-4 has 100 trillion parameters It doesn't. >Synapses are a lot simpler than parameters Not true but someone else has explained why. >a 100 trillion parameter model is unambiguously larger than the language processing portions of the human brain. GPT-4 is not that big. The technology to run such a model at scale is simply not feasible yet. >The human genome is 24 gigabits long. It's negligibly small compared to a language model. I'm sorry bit this makes no sense. How many gigabits long the genome is has no bearing on how impactful it is in steering the development of the human brain in comparison to a ML model.
3y ago

gizmo686 3y ago

> Brains don't start at 0. Evolution, dna/rna etc. There's obviously some pre disposition for language learning in humans but that alone isn't enough ground for a "universal grammar" That is literally what universal grammar is. All that is left is to argue about the size and content of UG.

famouswaffles 3y ago
No. Chomsky's theory was more than a vague and fairly obvious, "humans are pre disposed to language learning". The latter doesn't say anything about what causes these pre dispositions. Chomsky tries to do so. https://www.scientificamerican.com/article/evidence-rebuts-chomsky-s-theory-of-language-learning/ https://www.scientificamerican.com/article/evidence-rebuts-c...