14 ms·

Parallels in the ways that humans and ML models acquire language skills

111 points by theafh 3y ago

denial 3y ago
I'm probably flaunting my ignorance here, but how isn't this an extremely tenuous connection? The graphs are unconvincing beyond a "... Maybe? I guess?" and comparing brain activity to NN activity seems dubious. I'd be curious what other sounds look like for both.
- moffkalast 3y ago
  Username.. checks out?
tikkun 3y ago
Related: "Here's a phenomenon I was surprised to find: you'll go to talks, and hear various words, whose definitions you're not so sure about. At some point you'll be able to make a sentence using those words; you won't know what the words mean, but you'll know the sentence is correct. You'll also be able to ask a question using those words. You still won't know what the words mean, but you'll know the question is interesting, and you'll want to know the answer. Then later on, you'll learn what the words mean more precisely, and your sense of how they fit together will make that learning much easier. The reason for this phenomenon is that mathematics is so rich and infinite that it is impossible to learn it systematically, and if you wait to master one topic before moving on to the next, you'll never get anywhere. Instead, you'll have tendrils of knowledge extending far from your comfort zone. Then you can later backfill from these tendrils, and extend your comfort zone; this is much easier to do than learning "forwards". (Caution: this backfilling is necessary. There can be a temptation to learn lots of fancy words and to use them in fancy sentences without being able to say precisely what you mean. You should feel free to do that, but you should always feel a pang of guilt when you do.)" Reminds me of the attention mechanism in transformers! http://math.stanford.edu/~vakil/potentialstudents.html http://math.stanford.edu/~vakil/potentialstudents.html And for any parents with toddler age children, seeing the way that toddlers relate to language, and that people relate to toddlers about language, leads to lots of fun observations that remind me of LLM related concepts.
- esafak 3y ago
  https://en.wikipedia.org/wiki/Distributional_semantics https://en.wikipedia.org/wiki/Distributional_semantics
- tudorw 3y ago
  fascinating, it's how I feel about multidimensional topological manifolds, I understand them, I cannot math them... interacting frequently with an AI is changing me, it's encouraging me to approach task in a more structured way, use concise and accurate language, consider the context and apparent vagueness of communication in a human way, with its inflections, facial expressions and demeanour. One cannot use technology without being affected, in the same way most of us no longer train a part of our brain to remember phone numbers, this will change us.
- _puk 3y ago
  Thanks for this! I have made a point over the years of hanging out with people that are far more intelligent and talented than myself, many of whom are in completely different fields to myself.. and I realise that I've always done this! Whether it's art, music, or the future of power generation, I've been able to hold many conversations that have an aha moment halfway through, where some nugget clicks and backfills the conversation to that point. And yes, I feel a pang of guilt when entertaining these conversations, but I've made solid friends off of a number of these interactions, so I figure I can't be a completely unbearable bore! Or maybe I'm a bot.
- jameshart 3y ago
  The important part is in the parens at the end of course: > There can be a temptation to learn lots of fancy words and to use them in fancy sentences without being able to say precisely what you mean. You should feel free to do that, but you should always feel a pang of guilt when you do. GPT - as far as we know - feels no guilt pangs whatsoever.
  - jimmygrapes 3y ago
    Surely there is a confidence score (perhaps hidden) that could be used to emulate such a sense of guilt. Maybe next time.
ftxbro 3y ago
> "While it’s still unclear exactly how the brain processes and learns language, the linguist Noam Chomsky proposed in the 1950s that humans are born with an innate and unique capacity to understand language. That ability, Chomsky argued, is literally hard-wired into the human brain. The new work, which uses general-purpose neurons not designed for language, suggests otherwise. “The paper definitely provides evidence against the notion that speech requires special built-in machinery and other distinctive features,” Kapatsinski said." chomsky isn't going to like this
- canjobear 3y ago
  He's not going to care about it.
  - ftxbro 3y ago
    I mean he is writing articles titled like "The False Promise of ChatGPT" he might care a little bit https://www.nytimes.com/2023/03/08/opinion/noam-chomsky-chatgpt-ai.html https://www.nytimes.com/2023/03/08/opinion/noam-chomsky-chat...
    - canjobear 3y ago
      "Language" for Chomsky is an abstraction that is intentionally designed to exclude anything statistical. His most likely response to this would be that neither the neural network nor the brain data collected reflects anything that could be called language.
- tgv 3y ago
  Chomsky didn't mean overt producing of sound, but rather syntax and semantics.
- pessimizer 3y ago
  > which uses general-purpose neurons not designed for language, I'm not sure about this. We've probably designed general-purpose "neurons" to talk to us, even if we didn't think of it that way. They aren't emulators of physical neurons, they're abstractions of speculative neurons. The way we figure out if they work is by making them talk to us.
ckemere 3y ago
I wish I could include Fig. 1 of the paper here (https://www.nature.com/articles/s41598-023-33384-9/figures/1 https://www.nature.com/articles/s41598-023-33384-9/figures/1). The result should be "ANN performs similar nonlinear time domain filtering as human brain stem". There seems to be nothing at all about the learning process, just that ABR recordings of English and Spanish speakers hearing a confusing syllable are different, and ANN trained on English and Spanish has a similar difference ...
- ravi-delia 3y ago
  That is honestly a much more interesting result than the title would suggest. We know the brain can't do backprop (neurons are one way), but the fact that there is convergence in algorithm is very fun.
  - monocasa 3y ago
    > neurons are one way Synapses are one way. Biological neurons have many synapses.
    - ravi-delia 3y ago
      Technically true! But one, very few neurons have connections going both ways to each other- any two neurons will be lined up axon to synapse one way or the other. And two, the specific structures that compose most of the cortex are remarkably unidirectional. The neocortex consists mostly of little chunks of neurons arranged in layers, themselves hooked up snout to tail. There are loops, eventually, but only on the macro scale. Outside the cortex, and even moreso outside the brain, there is a far greater diversity of structures. Two neurons plugged into each other form an excellent basis for a timer, damper, clock, or even primitive stimulus response. There are structures taking proprioceptive info from the joints and performing integrals on them. It's neat because the circuits are so small we can actually understand them completely! But within the brain things definitely are not laid out right for backprop.
  - ckemere 3y ago
    I suppose. Equivalent results about natural images and edge detection have been reported in the image processing (classical, not deep) ML literature 20 years ago...
deleted 3y ago
[deleted]
unnouinceput 3y ago
Quote: "The results not only help demystify how ANNs learn, but also suggest that human brains may not come already equipped with hardware and software specially designed for language." I thought this was common knowledge. I mean if we'd come with already specialized hardware for language at birth we'd speak directly just as a new born puppy barks. Or if we'd have specialized software then children of geniuses would be geniuses themselves. And both cases, are obviously not happening in real life.
- tsimionescu 3y ago
  The opposite is common knowledge: we must have specialized hardware/software for language, otherwise we'd never be able to learn a language with the meager amount of information we can pick up in a year or two of occasional examples. There are plenty of examples of clearly specialized hardware/software that nevertheless needs some fine-tuning and is not immediately available in newborn children. Newborns also can't digest solid foods, don't have teeth, aren't able to produce offspring, don't have breasts etc. And yet no one is arguing these systems, many of which will only be apparent months and years later, are not built in. Even for other cognitive abilities: newborns have fully functioning retinas, yet are unable to see at all for a few days or weeks, and are unable to notice color for a good few months. However, the visual processing areas of the brain are virtually identical in all adults later: they are clearly part of our genetic makeup. Also, you're confusing the ability to acquire language with general intelligence in your argument about geniuses. This argument is doing the opposite : it's saying that acquiring language is not some feat of intelligence, it is merely a specialized evolutionarily-acquired capacity that all humans share, just like visual and auditory and motor processing.
- mikojan 3y ago
  > [If] we'd come with already specialized hardware for language at birth we'd speak directly Just like babies walk directly because they have legs.
notJim 3y ago
Anyone know why a GAN was used here? The discriminator makes sense, but what is the purpose of the generator for the given experiment? Why not cut out the generator, since the discriminator is trained separately on speech data anyway.
- space_fountain 3y ago
  Usually discriminators aren’t trained separately. Training them together makes the initial task for the generative side easier since it doesn’t have to be near perfect right of the bay. It can learn the obvious mistakes at the same time the discriminator is learning to detect them

111 points by theafh 3y ago

I'm probably flaunting my ignorance here, but how isn't this an extremely tenuous connection? The graphs are unconvincing beyond a "... Maybe? I guess?" and comparing brain activity to NN activity seems dubious. I'd be curious what other sounds look like for both.

moffkalast 3y ago
Username.. checks out?

tikkun 3y ago

Related: "Here's a phenomenon I was surprised to find: you'll go to talks, and hear various words, whose definitions you're not so sure about. At some point you'll be able to make a sentence using those words; you won't know what the words mean, but you'll know the sentence is correct. You'll also be able to ask a question using those words. You still won't know what the words mean, but you'll know the question is interesting, and you'll want to know the answer. Then later on, you'll learn what the words mean more precisely, and your sense of how they fit together will make that learning much easier. The reason for this phenomenon is that mathematics is so rich and infinite that it is impossible to learn it systematically, and if you wait to master one topic before moving on to the next, you'll never get anywhere. Instead, you'll have tendrils of knowledge extending far from your comfort zone. Then you can later backfill from these tendrils, and extend your comfort zone; this is much easier to do than learning "forwards". (Caution: this backfilling is necessary. There can be a temptation to learn lots of fancy words and to use them in fancy sentences without being able to say precisely what you mean. You should feel free to do that, but you should always feel a pang of guilt when you do.)" Reminds me of the attention mechanism in transformers! http://math.stanford.edu/~vakil/potentialstudents.html http://math.stanford.edu/~vakil/potentialstudents.html And for any parents with toddler age children, seeing the way that toddlers relate to language, and that people relate to toddlers about language, leads to lots of fun observations that remind me of LLM related concepts.

esafak 3y ago
https://en.wikipedia.org/wiki/Distributional_semantics https://en.wikipedia.org/wiki/Distributional_semantics
tudorw 3y ago
fascinating, it's how I feel about multidimensional topological manifolds, I understand them, I cannot math them... interacting frequently with an AI is changing me, it's encouraging me to approach task in a more structured way, use concise and accurate language, consider the context and apparent vagueness of communication in a human way, with its inflections, facial expressions and demeanour. One cannot use technology without being affected, in the same way most of us no longer train a part of our brain to remember phone numbers, this will change us.
_puk 3y ago
Thanks for this! I have made a point over the years of hanging out with people that are far more intelligent and talented than myself, many of whom are in completely different fields to myself.. and I realise that I've always done this! Whether it's art, music, or the future of power generation, I've been able to hold many conversations that have an aha moment halfway through, where some nugget clicks and backfills the conversation to that point. And yes, I feel a pang of guilt when entertaining these conversations, but I've made solid friends off of a number of these interactions, so I figure I can't be a completely unbearable bore! Or maybe I'm a bot.
jameshart 3y ago
The important part is in the parens at the end of course: > There can be a temptation to learn lots of fancy words and to use them in fancy sentences without being able to say precisely what you mean. You should feel free to do that, but you should always feel a pang of guilt when you do. GPT - as far as we know - feels no guilt pangs whatsoever.
- jimmygrapes 3y ago
  Surely there is a confidence score (perhaps hidden) that could be used to emulate such a sense of guilt. Maybe next time.

ftxbro 3y ago

> "While it’s still unclear exactly how the brain processes and learns language, the linguist Noam Chomsky proposed in the 1950s that humans are born with an innate and unique capacity to understand language. That ability, Chomsky argued, is literally hard-wired into the human brain. The new work, which uses general-purpose neurons not designed for language, suggests otherwise. “The paper definitely provides evidence against the notion that speech requires special built-in machinery and other distinctive features,” Kapatsinski said." chomsky isn't going to like this

canjobear 3y ago
He's not going to care about it.
- ftxbro 3y ago
  I mean he is writing articles titled like "The False Promise of ChatGPT" he might care a little bit https://www.nytimes.com/2023/03/08/opinion/noam-chomsky-chatgpt-ai.html https://www.nytimes.com/2023/03/08/opinion/noam-chomsky-chat...
  - canjobear 3y ago
    "Language" for Chomsky is an abstraction that is intentionally designed to exclude anything statistical. His most likely response to this would be that neither the neural network nor the brain data collected reflects anything that could be called language.
tgv 3y ago
Chomsky didn't mean overt producing of sound, but rather syntax and semantics.
pessimizer 3y ago
> which uses general-purpose neurons not designed for language, I'm not sure about this. We've probably designed general-purpose "neurons" to talk to us, even if we didn't think of it that way. They aren't emulators of physical neurons, they're abstractions of speculative neurons. The way we figure out if they work is by making them talk to us.

ckemere 3y ago

I wish I could include Fig. 1 of the paper here (https://www.nature.com/articles/s41598-023-33384-9/figures/1 https://www.nature.com/articles/s41598-023-33384-9/figures/1). The result should be "ANN performs similar nonlinear time domain filtering as human brain stem". There seems to be nothing at all about the learning process, just that ABR recordings of English and Spanish speakers hearing a confusing syllable are different, and ANN trained on English and Spanish has a similar difference ...

ravi-delia 3y ago
That is honestly a much more interesting result than the title would suggest. We know the brain can't do backprop (neurons are one way), but the fact that there is convergence in algorithm is very fun.
- monocasa 3y ago
  > neurons are one way Synapses are one way. Biological neurons have many synapses.
  - ravi-delia 3y ago
    Technically true! But one, very few neurons have connections going both ways to each other- any two neurons will be lined up axon to synapse one way or the other. And two, the specific structures that compose most of the cortex are remarkably unidirectional. The neocortex consists mostly of little chunks of neurons arranged in layers, themselves hooked up snout to tail. There are loops, eventually, but only on the macro scale. Outside the cortex, and even moreso outside the brain, there is a far greater diversity of structures. Two neurons plugged into each other form an excellent basis for a timer, damper, clock, or even primitive stimulus response. There are structures taking proprioceptive info from the joints and performing integrals on them. It's neat because the circuits are so small we can actually understand them completely! But within the brain things definitely are not laid out right for backprop.
- ckemere 3y ago
  I suppose. Equivalent results about natural images and edge detection have been reported in the image processing (classical, not deep) ML literature 20 years ago...

deleted 3y ago

[deleted]

unnouinceput 3y ago

Quote: "The results not only help demystify how ANNs learn, but also suggest that human brains may not come already equipped with hardware and software specially designed for language." I thought this was common knowledge. I mean if we'd come with already specialized hardware for language at birth we'd speak directly just as a new born puppy barks. Or if we'd have specialized software then children of geniuses would be geniuses themselves. And both cases, are obviously not happening in real life.

tsimionescu 3y ago
The opposite is common knowledge: we must have specialized hardware/software for language, otherwise we'd never be able to learn a language with the meager amount of information we can pick up in a year or two of occasional examples. There are plenty of examples of clearly specialized hardware/software that nevertheless needs some fine-tuning and is not immediately available in newborn children. Newborns also can't digest solid foods, don't have teeth, aren't able to produce offspring, don't have breasts etc. And yet no one is arguing these systems, many of which will only be apparent months and years later, are not built in. Even for other cognitive abilities: newborns have fully functioning retinas, yet are unable to see at all for a few days or weeks, and are unable to notice color for a good few months. However, the visual processing areas of the brain are virtually identical in all adults later: they are clearly part of our genetic makeup. Also, you're confusing the ability to acquire language with general intelligence in your argument about geniuses. This argument is doing the opposite : it's saying that acquiring language is not some feat of intelligence, it is merely a specialized evolutionarily-acquired capacity that all humans share, just like visual and auditory and motor processing.
mikojan 3y ago
> [If] we'd come with already specialized hardware for language at birth we'd speak directly Just like babies walk directly because they have legs.

notJim 3y ago

Anyone know why a GAN was used here? The discriminator makes sense, but what is the purpose of the generator for the given experiment? Why not cut out the generator, since the discriminator is trained separately on speech data anyway.

space_fountain 3y ago
Usually discriminators aren’t trained separately. Training them together makes the initial task for the generative side easier since it doesn’t have to be near perfect right of the bay. It can learn the obvious mistakes at the same time the discriminator is learning to detect them