3 ms·

Of course, you are totally correct about the baby not being limited to text input. It would be interesting to see in the future whether building "world models"

Of course, you are totally correct about the baby not being limited to text input. It would be interesting to see in the future whether building "world models" through multimodal input would help scaffolding language ability with less data.

candiodari 3y ago
I wonder if it isn't 100x more effective for the baby that they don't learn from language per se, they learn from direct feedback. They try to duplicate/imitate others, but generally they get immediate feedback (to the point where it's becomes annoying) about exactly what they're doing/alternatives/... I've always found that this works, but adults often don't like it. Learning from direct feedback, having someone immediately check your work, so to speak, helps enormously. Even if it's in text (I learned C++ on IRC and mailing lists, for example). The books help, but you never have any direction.
lostmsu 3y ago
How did you come up with "less data"? Video and audio are much larger then text even when compressed. You can put whole Dune book original series in space of a single MP3 song, and the entire text of English Wikipedia in a single high quality movie.
- GolDDranks 3y ago
  I'm talking about linguistic information here, because I'm interested in linguistics and language learning. Humans abstract phonemes from sound, and morphemes/lexical items from phonemes. Once we have a rough measure of how many words of input the toddlers get, we can roughly equate the amounts of _linguistic_ data that goes into acquiring linguistic capabilities. While GPT gets tokens as input, from the lexical lexical level onward, there can be an argument that we can roughly estimate the volume of the _linguistic_ data that goes in to the system regardless of earlier decoding stages. I don't claim that there are a lot of other kinds of data feeding into a human brain, and the modality is totally different. (Multimodal-interactive vs. "predict the next token") GPT, therefore, is at disadvantage in that it needs to learn stuff – other than linguistic stuff – _only_ from linguistic input, whereas a human baby can use world model gained through other modalities as a scaffolding. But for acquiring linguistic capabilities, auditory and visual data seems hardly relevant as input data, other than helping building that the scaffolding that can help "top-down" understanding.
  - lostmsu 3y ago
    Tiny models can already learn language (as in grammar and syntax) from very little very quickly. It's the semantics that separates LLMs from CharRNN.
    - GolDDranks 3y ago
      That's interesting! You mention CharRNN. Any links? I find it especially interesting to test whether models that know syntax, know superficially human language -like syntax, or actually syntax that conforms to the deep (syntactic) structure of human languages.
      - lostmsu 3y ago
        https://github.com/karpathy/char-rnn https://github.com/karpathy/char-rnn

candiodari 3y ago

I wonder if it isn't 100x more effective for the baby that they don't learn from language per se, they learn from direct feedback. They try to duplicate/imitate others, but generally they get immediate feedback (to the point where it's becomes annoying) about exactly what they're doing/alternatives/... I've always found that this works, but adults often don't like it. Learning from direct feedback, having someone immediately check your work, so to speak, helps enormously. Even if it's in text (I learned C++ on IRC and mailing lists, for example). The books help, but you never have any direction.

lostmsu 3y ago

How did you come up with "less data"? Video and audio are much larger then text even when compressed. You can put whole Dune book original series in space of a single MP3 song, and the entire text of English Wikipedia in a single high quality movie.

GolDDranks 3y ago
I'm talking about linguistic information here, because I'm interested in linguistics and language learning. Humans abstract phonemes from sound, and morphemes/lexical items from phonemes. Once we have a rough measure of how many words of input the toddlers get, we can roughly equate the amounts of _linguistic_ data that goes into acquiring linguistic capabilities. While GPT gets tokens as input, from the lexical lexical level onward, there can be an argument that we can roughly estimate the volume of the _linguistic_ data that goes in to the system regardless of earlier decoding stages. I don't claim that there are a lot of other kinds of data feeding into a human brain, and the modality is totally different. (Multimodal-interactive vs. "predict the next token") GPT, therefore, is at disadvantage in that it needs to learn stuff – other than linguistic stuff – _only_ from linguistic input, whereas a human baby can use world model gained through other modalities as a scaffolding. But for acquiring linguistic capabilities, auditory and visual data seems hardly relevant as input data, other than helping building that the scaffolding that can help "top-down" understanding.
- lostmsu 3y ago
  Tiny models can already learn language (as in grammar and syntax) from very little very quickly. It's the semantics that separates LLMs from CharRNN.
  - GolDDranks 3y ago
    That's interesting! You mention CharRNN. Any links? I find it especially interesting to test whether models that know syntax, know superficially human language -like syntax, or actually syntax that conforms to the deep (syntactic) structure of human languages.
    - lostmsu 3y ago
      https://github.com/karpathy/char-rnn https://github.com/karpathy/char-rnn