4 ms·

Get a neural network, same as you'd use at the bottom of an LLM. Don't feed it a corpus of text! Ensure it's data is the opposite of a large language model, i.

Get a neural network, same as you'd use at the bottom of an LLM. Don't feed it a corpus of text! Ensure it's data is the opposite of a large language model, i.e. zero language. Let it passively listen to the same sounds and speech as a baby hears in its first five years, over five years of operation. Observe it do absolutely nothing of any use. Get back to me about any useful comparisons with linguistics or philosophy of language.

famouswaffles 3y ago
Text is not the important thing here at all. audio language models exist, they are relatively trivial to engineer and they will model audio just as well vs they'll model text or any arbitrary sequence to sequence data. https://google-research.github.io/seanet/audiolm/examples/ https://google-research.github.io/seanet/audiolm/examples/ Linguistics is useful fiction. It doesn't accurately describe the complexity of language anymore than Newton's laws accurately describe gravity. You'd think people would understand that now with symbolic approaches failing this sort of stuff after decades of trying but i guess not. We don't know how we work. We really really don't. The obsession with how we think we work over methods that are actually working is really something to behold.
- gizajob 3y ago
  https://tmms.co.uk/wp-content/uploads/2023/01/imgpsh_fullsize_anim-5.png https://tmms.co.uk/wp-content/uploads/2023/01/imgpsh_fullsiz...
- patcon 3y ago
  > We don't know how we work. We really really don't. Amen.
stevenhuang 3y ago
You are aware the same transformer architecture that underpins LLMs has shown great success in multimodal applications? Such as successfully modeling sound and music? https://bytedance.github.io/SALMONN/ https://bytedance.github.io/SALMONN/
canjobear 3y ago
A fun pastime is to go find old papers by philosophers and linguists that make concrete claims about things language models will never be able to do, and then go type their examples into ChatGPT and see the model do just fine.
- gizajob 3y ago
  I know a philosopher with four legs and a fondness for balls and bones. Without using language, he knows if I’m happy or sad and can usefully alter my state. Get ChatGPT to repeat this trick.
lucubratory 3y ago
So the toddler gets to start learning with the most complicated known entity in the universe in their head, a brain honed by millions and millions of years of evolution into computing extremely efficiently, with the ability to generate languages from scratch if no examples are provided or learn them by default if any examples are provided, while the LLM gets to start with... a table of random numbers? This thought experiment is designed to make LLMs look less impressive? Humans don't start at zero.
- gizajob 3y ago
  Compared to an LLM, the corpus they use to become proficient at language (and countless other things alongside) is achingly small. It’s also embodied so it can relate sounds and utterances to the world and actions. Thus comparing outputs between to two entities is really like comparing apples to oranges, it’s not that the pronouncements of linguists and philosophers are useless in the face of it, they’re just in a different domain.
  - lucubratory 3y ago
    >Compared to an LLM, the corpus they use to become proficient at language (and countless other things alongside) is achingly small. But that's only true if you assume humans start at zero and the millions of years of natural selection that gave us our incredible brains gives no advantage to learning, which seems facially absurd. The toddler probably sees less data in that five years than the LLM sees in a few months, but it's worth remembering how rich human sensory data is, smell is heavily tied to memory and is incredibly complex, taste and touch sensitivity in the mouth is high enough that toddlers stuff new things in their mouth to understand them better, human hands are incredibly sensor dense, and that's before we get to the classics of 1000fps vision consisting of partially high definition and partially upscaled low definition video, or audio that contains huge amounts of information not only about material properties, energy magnitudes, or space layouts and shapes, but also a huge amount of semantic data through speech and language for most children. But unlike the LLM, they didn't start with a random selection of numbers, they started with a human brain which is incredibly impressive. Frankly, it is genuinely surprising that LLMs are able to rival us as well as they do considering how much less complex they are than our brains. >it’s not that the pronouncements of linguists and philosophers are useless in the face of it, they’re just in a different domain. I go back and forth on whether this is true. I see the argument, but it also makes intuitive sense that study of an AI system whose intelligence comes entirely from human language can probably yield insights about human language. It doesn't seem any less plausible than studying the way slime molds feed on oat flakes to gain insights into public transport design and infrastructure, which is something that we have done to significant success.
TeMPOraL 3y ago
> Let it passively listen to the same sounds and speech as a baby hears in its first five years, over five years of operation. First of all, a baby does not passively listen. It sees and touches and smells and feels, and moves and makes sounds, and observes reactions, and correlates all that with the sounds it hears. Secondly, with the above in mind, why on Earth would you expect it to "do absolutely nothing of any use"? It feels obvious it'll start honing in on language, with even better grounding (but worse performance) than a LLM. And I fully expect there to be a to merge such model with an LLM, the same way people combine LLMs with image generators.

famouswaffles 3y ago

Text is not the important thing here at all. audio language models exist, they are relatively trivial to engineer and they will model audio just as well vs they'll model text or any arbitrary sequence to sequence data. https://google-research.github.io/seanet/audiolm/examples/ https://google-research.github.io/seanet/audiolm/examples/ Linguistics is useful fiction. It doesn't accurately describe the complexity of language anymore than Newton's laws accurately describe gravity. You'd think people would understand that now with symbolic approaches failing this sort of stuff after decades of trying but i guess not. We don't know how we work. We really really don't. The obsession with how we think we work over methods that are actually working is really something to behold.

gizajob 3y ago
https://tmms.co.uk/wp-content/uploads/2023/01/imgpsh_fullsize_anim-5.png https://tmms.co.uk/wp-content/uploads/2023/01/imgpsh_fullsiz...
patcon 3y ago
> We don't know how we work. We really really don't. Amen.

stevenhuang 3y ago

You are aware the same transformer architecture that underpins LLMs has shown great success in multimodal applications? Such as successfully modeling sound and music? https://bytedance.github.io/SALMONN/ https://bytedance.github.io/SALMONN/

canjobear 3y ago

A fun pastime is to go find old papers by philosophers and linguists that make concrete claims about things language models will never be able to do, and then go type their examples into ChatGPT and see the model do just fine.

gizajob 3y ago
I know a philosopher with four legs and a fondness for balls and bones. Without using language, he knows if I’m happy or sad and can usefully alter my state. Get ChatGPT to repeat this trick.

lucubratory 3y ago

So the toddler gets to start learning with the most complicated known entity in the universe in their head, a brain honed by millions and millions of years of evolution into computing extremely efficiently, with the ability to generate languages from scratch if no examples are provided or learn them by default if any examples are provided, while the LLM gets to start with... a table of random numbers? This thought experiment is designed to make LLMs look less impressive? Humans don't start at zero.

gizajob 3y ago
Compared to an LLM, the corpus they use to become proficient at language (and countless other things alongside) is achingly small. It’s also embodied so it can relate sounds and utterances to the world and actions. Thus comparing outputs between to two entities is really like comparing apples to oranges, it’s not that the pronouncements of linguists and philosophers are useless in the face of it, they’re just in a different domain.
- lucubratory 3y ago
  >Compared to an LLM, the corpus they use to become proficient at language (and countless other things alongside) is achingly small. But that's only true if you assume humans start at zero and the millions of years of natural selection that gave us our incredible brains gives no advantage to learning, which seems facially absurd. The toddler probably sees less data in that five years than the LLM sees in a few months, but it's worth remembering how rich human sensory data is, smell is heavily tied to memory and is incredibly complex, taste and touch sensitivity in the mouth is high enough that toddlers stuff new things in their mouth to understand them better, human hands are incredibly sensor dense, and that's before we get to the classics of 1000fps vision consisting of partially high definition and partially upscaled low definition video, or audio that contains huge amounts of information not only about material properties, energy magnitudes, or space layouts and shapes, but also a huge amount of semantic data through speech and language for most children. But unlike the LLM, they didn't start with a random selection of numbers, they started with a human brain which is incredibly impressive. Frankly, it is genuinely surprising that LLMs are able to rival us as well as they do considering how much less complex they are than our brains. >it’s not that the pronouncements of linguists and philosophers are useless in the face of it, they’re just in a different domain. I go back and forth on whether this is true. I see the argument, but it also makes intuitive sense that study of an AI system whose intelligence comes entirely from human language can probably yield insights about human language. It doesn't seem any less plausible than studying the way slime molds feed on oat flakes to gain insights into public transport design and infrastructure, which is something that we have done to significant success.

TeMPOraL 3y ago

> Let it passively listen to the same sounds and speech as a baby hears in its first five years, over five years of operation. First of all, a baby does not passively listen. It sees and touches and smells and feels, and moves and makes sounds, and observes reactions, and correlates all that with the sounds it hears. Secondly, with the above in mind, why on Earth would you expect it to "do absolutely nothing of any use"? It feels obvious it'll start honing in on language, with even better grounding (but worse performance) than a LLM. And I fully expect there to be a to merge such model with an LLM, the same way people combine LLMs with image generators.