4 ms·

While I think some of the points in the article are interesting, the usual evidence for the Chomskian approach is the relative lack of input data for learning l

While I think some of the points in the article are interesting, the usual evidence for the Chomskian approach is the relative lack of input data for learning language by children in the wild. How much input data is used to train modern language models?

api 3y ago
Do children really lack input data though? Human sensory input is quite a lot of data. Our languages may have common structure because that structure reflects causality and physics encountered by directly sampling the real world.
- sudosysgen 3y ago
  And how exactly is the structure of the world transformed into a structure of language? Chomsky would say through a universal grammar, which is a framework through which you can set up ways to encode the structure in the world as words.
  - api 3y ago
    This implies that the combinatorial space of possible grammars is large. If it isn't, then the structure of the universe would imply a narrow set of possible workable grammars. The combinatorial space of languages is obviously infinite, but is this so for grammars? If not then you would expect many languages to share the same grammars. Seems analogous to the same question about mathematics. Are there many different possible arithmetics? No. There are infinitely many ways to express arithmetic symbolically but there is only one arithmetic. 2 + 2 never equals 5.
    - deleted 3y ago
      [deleted]
- cma 3y ago
  You have Hellen Keller who went blind and deaf at 19 months.
- akasakahakada 3y ago
  Keep this as a secret! You are over-scientific and over-intellegent against humanities folks gethered here!
  - bbor 3y ago
    That’s not in the hacker news spirit :( especially ironic to shit talk “humanities folks” with a basic misspelling in your comment. We’re all just trying to reach the truth!
- bbor 3y ago
  Your hypothesis is plausible, but less likely than Chomsky’s IMO. One basic version of the typical chomskian response to this point is: many animals have the same input data, how come were the only ones who have evolved any capacity for language at all? The very best animals at language are apes, and we have to successfully teach one concepts that humans learn while still wearing diapers
  - api 3y ago
    Why can't a small neural network do what LLMs do? Same reason animals can't learn human language. They don't have the capacity. LLMs started getting interesting "all the sudden" when they hit a certain scale, just like biological brains.
    - bbor 3y ago
      I mean, that's kinda exactly Chomsky's point, minus the implication (perhaps malinferred) that human-like linguistic is an inevitable or universal characteristic of intelligent minds. There's lots of animals that have very complex brains, and that engage in very complex behaviors - the fact that none of them have even a hint of linguistic ability seems like a strong indicator that these faculties are a human-specific evolution, and not just... well IDK how to even sum up an anti-UG/GG view. "Kids just sorta figure it out" I guess? Although who knows! As Chomsky likes to say, this whole field is in a pre-Galilean state due to the impossibility of conducting comparative studies EDIT: Oh just realized you were the parent comment. Well I'd say the small NN vs. LLM example still doesn't convince me of the likelihood of your statement as I understand it; it goes without saying that lots of animals are much better at intuitive understandings of physics (well, kinetics at least) than humans. You ever seen those snakes that jump from tree to tree? craziest shit you'll ever see
famouswaffles 3y ago
1. No Language model is yet even close to the scale of the human brain 2. Depending on what exactly you're trying to teach (perfect grammar, paragraphs of coherent text, basic reasoning), much less data is needed. https://arxiv.org/abs/2305.07759 https://arxiv.org/abs/2305.07759 3. Brains don't start at 0. Evolution, dna/rna etc. There's obviously some pre disposition for language learning in humans but that alone isn't enough ground for a "universal grammar" 4. We really do take in an enormous amount of data (not text specifically)
- lisasays 3y ago
  1. Measured how?
  - famouswaffles 3y ago
    The closest biological analogue for an artificial parameter is a synapse. Even then, Synapses, and the brain in general, have more degrees of freedom than just a single floating point parameter allows. Humans have some 50 to 100 trillion synapses. Who knows how far scaling goes in a transformer but simply increasing the parameter count increases performance so far. Either way, we're not really close to emulating the complexity of the brain especially when taking neurons into account (one human neuron is far more complex than any one artificial parameter) In other words, simpler building blocks and smaller sie. Of course, how much complexity is needed for Intelligence is unknown.
  - civilitty 3y ago
    The connectome [1]. There are about 100B +- 20B neurons in the average human brain. Each neuron has thousands of synapses and each synapse is made up of dozens to hundreds of neurotransmitter channels. That’s at least a quadrillion parameters which is at least three orders of magnitude bigger than SOTA LLMs assuming each of those synapse channels maps to a parameter. That is an absurd assumption given neuroplasticity, which operates at a high level to adapt the neural network as it learns. See the plasticity section in the linked wikipedia articles: connections between neurons can grow or even get removed on the scale of hours and days. The human brain has entire biological systems supporting intelligence for which there is no real ML equivalent because our ML architectures are static. Given a family member’s research on neurotransmitter potentiation, I’d estimate 10 to 100 parameters per channel to get full fidelity of the human brain. (This is absurdly speculative - we have no idea at which fidelity will intelligence emerge) [1] https://en.m.wikipedia.org/wiki/Connectome https://en.m.wikipedia.org/wiki/Connectome
MAXPOOL 3y ago
I agree. Question has not been settled. 20 year old human has * heard ~220 million words, talked 50 million words. * read ~10 million words. * experienced 420 million seconds of wakeful interaction with the environment (can be used to estimate the limit to conscious decisions, or number of distinct 'epochs' we experience) From a machine learning perspective human life is surprisingly small set of inputs and actions, just a blip of existence.
- pphysch 3y ago
  If you reduce it to bytes, yeah, humans are unreasonably efficient learners. But that's not how it works, there's loads more information embedded in a real human speech context.
- IshKebab 3y ago
  LLMs learn from unlabeled data. Children definitely do not. There's a huge difference. I would not be surprised if LLMs could learn a lot more efficiently if they had carefully constructed training data, with video and sound. But also humans have been speaking for so long it's silly to imagine we don't have some evolved language structures in the brain. I don't know why anyone would single that out for skepticism while not questioning e.g. the brain structures for sight, sound, emotions, navigation, etc.
  - MAXPOOL 3y ago
    > some evolved language structures in the brain. That's Chomsky's argument. A small set of constraints for organizing language.
    - 8note 3y ago
      This is testable, right? Put a kid from one language tradition in a spot with a different language tradition, and they won't be able to learn it. Eg. kids with native mandarin speaking parents adopted to native Indo-European parents fail at learning English, and will be better at learning Mandarin than their peers with Indo-European heritage.

api 3y ago

Do children really lack input data though? Human sensory input is quite a lot of data. Our languages may have common structure because that structure reflects causality and physics encountered by directly sampling the real world.

sudosysgen 3y ago
And how exactly is the structure of the world transformed into a structure of language? Chomsky would say through a universal grammar, which is a framework through which you can set up ways to encode the structure in the world as words.
- api 3y ago
  This implies that the combinatorial space of possible grammars is large. If it isn't, then the structure of the universe would imply a narrow set of possible workable grammars. The combinatorial space of languages is obviously infinite, but is this so for grammars? If not then you would expect many languages to share the same grammars. Seems analogous to the same question about mathematics. Are there many different possible arithmetics? No. There are infinitely many ways to express arithmetic symbolically but there is only one arithmetic. 2 + 2 never equals 5.
  - deleted 3y ago
    [deleted]
cma 3y ago
You have Hellen Keller who went blind and deaf at 19 months.
akasakahakada 3y ago
Keep this as a secret! You are over-scientific and over-intellegent against humanities folks gethered here!
- bbor 3y ago
  That’s not in the hacker news spirit :( especially ironic to shit talk “humanities folks” with a basic misspelling in your comment. We’re all just trying to reach the truth!
bbor 3y ago
Your hypothesis is plausible, but less likely than Chomsky’s IMO. One basic version of the typical chomskian response to this point is: many animals have the same input data, how come were the only ones who have evolved any capacity for language at all? The very best animals at language are apes, and we have to successfully teach one concepts that humans learn while still wearing diapers
- api 3y ago
  Why can't a small neural network do what LLMs do? Same reason animals can't learn human language. They don't have the capacity. LLMs started getting interesting "all the sudden" when they hit a certain scale, just like biological brains.
  - bbor 3y ago
    I mean, that's kinda exactly Chomsky's point, minus the implication (perhaps malinferred) that human-like linguistic is an inevitable or universal characteristic of intelligent minds. There's lots of animals that have very complex brains, and that engage in very complex behaviors - the fact that none of them have even a hint of linguistic ability seems like a strong indicator that these faculties are a human-specific evolution, and not just... well IDK how to even sum up an anti-UG/GG view. "Kids just sorta figure it out" I guess? Although who knows! As Chomsky likes to say, this whole field is in a pre-Galilean state due to the impossibility of conducting comparative studies EDIT: Oh just realized you were the parent comment. Well I'd say the small NN vs. LLM example still doesn't convince me of the likelihood of your statement as I understand it; it goes without saying that lots of animals are much better at intuitive understandings of physics (well, kinetics at least) than humans. You ever seen those snakes that jump from tree to tree? craziest shit you'll ever see

famouswaffles 3y ago

1. No Language model is yet even close to the scale of the human brain 2. Depending on what exactly you're trying to teach (perfect grammar, paragraphs of coherent text, basic reasoning), much less data is needed. https://arxiv.org/abs/2305.07759 https://arxiv.org/abs/2305.07759 3. Brains don't start at 0. Evolution, dna/rna etc. There's obviously some pre disposition for language learning in humans but that alone isn't enough ground for a "universal grammar" 4. We really do take in an enormous amount of data (not text specifically)

lisasays 3y ago
1. Measured how?
- famouswaffles 3y ago
  The closest biological analogue for an artificial parameter is a synapse. Even then, Synapses, and the brain in general, have more degrees of freedom than just a single floating point parameter allows. Humans have some 50 to 100 trillion synapses. Who knows how far scaling goes in a transformer but simply increasing the parameter count increases performance so far. Either way, we're not really close to emulating the complexity of the brain especially when taking neurons into account (one human neuron is far more complex than any one artificial parameter) In other words, simpler building blocks and smaller sie. Of course, how much complexity is needed for Intelligence is unknown.
- civilitty 3y ago
  The connectome [1]. There are about 100B +- 20B neurons in the average human brain. Each neuron has thousands of synapses and each synapse is made up of dozens to hundreds of neurotransmitter channels. That’s at least a quadrillion parameters which is at least three orders of magnitude bigger than SOTA LLMs assuming each of those synapse channels maps to a parameter. That is an absurd assumption given neuroplasticity, which operates at a high level to adapt the neural network as it learns. See the plasticity section in the linked wikipedia articles: connections between neurons can grow or even get removed on the scale of hours and days. The human brain has entire biological systems supporting intelligence for which there is no real ML equivalent because our ML architectures are static. Given a family member’s research on neurotransmitter potentiation, I’d estimate 10 to 100 parameters per channel to get full fidelity of the human brain. (This is absurdly speculative - we have no idea at which fidelity will intelligence emerge) [1] https://en.m.wikipedia.org/wiki/Connectome https://en.m.wikipedia.org/wiki/Connectome

MAXPOOL 3y ago

I agree. Question has not been settled. 20 year old human has * heard ~220 million words, talked 50 million words. * read ~10 million words. * experienced 420 million seconds of wakeful interaction with the environment (can be used to estimate the limit to conscious decisions, or number of distinct 'epochs' we experience) From a machine learning perspective human life is surprisingly small set of inputs and actions, just a blip of existence.

pphysch 3y ago
If you reduce it to bytes, yeah, humans are unreasonably efficient learners. But that's not how it works, there's loads more information embedded in a real human speech context.
IshKebab 3y ago
LLMs learn from unlabeled data. Children definitely do not. There's a huge difference. I would not be surprised if LLMs could learn a lot more efficiently if they had carefully constructed training data, with video and sound. But also humans have been speaking for so long it's silly to imagine we don't have some evolved language structures in the brain. I don't know why anyone would single that out for skepticism while not questioning e.g. the brain structures for sight, sound, emotions, navigation, etc.
- MAXPOOL 3y ago
  > some evolved language structures in the brain. That's Chomsky's argument. A small set of constraints for organizing language.
  - 8note 3y ago
    This is testable, right? Put a kid from one language tradition in a spot with a different language tradition, and they won't be able to learn it. Eg. kids with native mandarin speaking parents adopted to native Indo-European parents fail at learning English, and will be better at learning Mandarin than their peers with Indo-European heritage.