3 ms·
> I don't see how you can have it both ways as the unconstrained problem is infeasible to learn on small data as there isn't enough to generalize from. I mostl
by thomasahle 6y ago
> I don't see how you can have it both ways as the unconstrained problem is infeasible to learn on small data as there isn't enough to generalize from.
I mostly agree with you, but it does seem like current models require more data to learn language than baby humans.
This would suggest there is a middle ground between old fashioned data starved learning, and current overfed models.
- slx26 6y agoBut baby humans tend to associate phonetic symbols (spoken words) to what they are seeing, and the learning method is very different. Their context is much broader, helped by multiple senses. NLP is extremely contextless compared to what humans do. You can't talk about "more data" so easily.
- Eridrus 6y agoOne random note I heard on a podcast about child development is that children do not improve their language skills by watching tv or educational material and that they need feedback to learn. Contrast that with language models which do nothing besides watching what people are writing, it potentially points to a reason who the amount of data is not comparable.
- rvense 6y agoEven more crucial is that children learn language in situations of actual language use. The things and situations that are spoken about are often at hand in some way, and there is a large social element. No matter how close you might get to "the language learning algorithm", if all you're feeding it is text, it's not going to learn the same thing that kids learn. The data is simply not there.