9 ms·
New algorithm discovers language just by watching videos
- julius 2y agoPaper: https://arxiv.org/pdf/2406.05629v1 https://arxiv.org/pdf/2406.05629v1
- firtoz 2y agohttps://arxiv.org/abs/2406.05629v1 https://arxiv.org/abs/2406.05629v1
- akasakahakada 2y agoLinguists e.g. Chomsky: No you can't. Proof is left as an exercise.
- maest 2y agoI think Chomsky's position is that _humans_ are primed for language, but it doesn't mean there exists no system that can learn language from 0. Also, the ability to understand language is a spectrum, not a binary quality.
- qntty 2y agoYeah the argument is that there is a poverty of stimulus (not enough training data) in the case of children. The same doesn’t apply to ML models which need an abundance of stimulus.
- Mathnerd314 2y agoThere was some linguist (Deb Roy?) who videotaped his children growing up. One of his observations was that every learned syllable etc. was tied to observing it as a stimulus and trying to imitate it. Now it is true, children are pretty good learners, they can often learn something the first time they see it, but actually I have seen some LLM stuff about "instant learning" - e.g. the training is only done with one pass over the material. https://www.fast.ai/posts/2023-09-04-learning-jumps/ https://www.fast.ai/posts/2023-09-04-learning-jumps/
- qntty 2y agoThis is an interesting connection. If I had to guess, the Chomskyist response would be to say, yes but this only applies to LLMs that have been pre-trained already (i.e. have structures in place needed to understand language in general). I think a Chomskyist would say that language learning is precisely like fine-tuning and not the pre-training of foundation models.
- roenxi 2y agoAnd with that in mind there is an obvious counterargument to the original "No". If humans learn language through observation and innate knowledge encoded into the brain then we can construct an artificial system that discovers language purely through observation by encoding the same innate knowledge into the system. Algorithms necessarily contain some knowledge about the data they are working with so that isn't cheating. Neural nets for example have to have some architecture and that predisposes them to learning certain patterns.
- cubefox 2y agoHe does claim there is no system which can learn language from 0 by having just as much or less training data as humans. Which seems very likely false.
- sudosysgen 2y agoWord-object correspondances can obviously be learnt from observation in theory - it is widely accepted that there is enough information to learn such correspondances, even on a limited dataset. The argument by linguists is about grammar, not vocabulary. Chomsky's argument is exactly the opposite, that vocabulary is learnt from scratch, but that the framework for grammar is innate.
- CamperBob2 2y agoHistorically, Chomsky has been wrong about a great many things.
- morsch 2y agoParticularly about things he never said.
- TeMPOraL 2y ago> Proof is left as an exercise. Rather conclusive one already established in the ML field.
- jowea 2y agoIntersting. How long till we have universal translators and/or can figure out the Voynich manuscript?
- ImHereToVote 2y agoRight after you have no mouth and must scream.
- cubefox 2y agoThe Voynich manuscript is probably too short, but training an LLM on two large text dumps of unrelated languages, where only one is known (the other perhaps an alien message), should make translation possible.
- jebarker 2y agoIt's a stretch to call this discovering language. It's learning the correlations between sounds, spoken words and visual features. That's a long way from learning language.
- wumbo 2y agoI think it's fair. It discerns what sounds are important, languagey sounds. It discovered but did not understand
- deleted 2y ago[deleted]
- anigbrowl 2y agoNo it isn't, this is exactly what babies do.
- astromaniak 2y agobabies do a lot more. they have feelings. like pain, pleasure. and different verities of them. then they have a lot of things hardcoded. and they have control of their bodies and the environment. for example they quickly learn that crying helps getting what they want.
- TeMPOraL 2y agoNone of which has much to do with learning language.
- gravescale 2y agoA lot of a baby's language pickup is also based on what other people do in response to their attempts, linguistically and behaviourally. Read-only observation is obviously a big part of it, but it's not "exactly" the same.
- TeMPOraL 2y ago
- wjb3 2y agoImagine an autonomous robot programmed with this algorithm just walking around in the world learning about it. So excited about the future!
- codetrotter 2y agoAll well and good until he rolls into a room where they are doing a Terminator movies marathon
- chaps 2y agoHah, neat. Had a similar idea long long ago. Wanted to run it against the show, Friends, to model the object interactions against text to speech results against a transcript. The idea was to create a speech to text slash plot generator using silly bayes stuff that latches onto accurate text to speech pairings over inaccurate pairings when compared against the transcript. Friends because it's such a popular show that has so much association and accessible transcripts.
- OJFord 2y agoI wish Friends was available dubbed in the language I'm learning (Hindi) - I know it well enough in English from repeated watching that I think it'd be the best thing I could do for my vocabulary and listening comprehension speed to watch it in Hindi. There should be an app that pays people to dub popular shows in a broader way of language, charges viewers, and has some kind of sidebar or subtitle integration to hover/click and learn more, check what a word was and conjugation etc. The focus on language learning would also allow the translators to make more appropriate choices for that use, whereas subtitles (and probably dubbing) for entertainment often has some variation that is confusing it to learn. I suppose that matters less if there's only one or the other, so no chance of conflict.
- shrimp_emoji 2y ago> because it's such a popular show that has so much association and accessible transcripts. Because it's so stupid even some dumb code can understand the patterns. Wouldn't work with Seinfeld.
- rcleveng 2y agoSeems to work. I think my son's learning Japanese by watching Anime on Crunchyroll.
- CoastalCoder 2y agoAfter watching about 500 hours of subtitled anime, I can say with total confidence that "nando" is an interrogative. Probably.
- leobg 2y agoYou’ll need a whole new class of training data. A camera strapped to someone’s head as they go about their day, scaled across continents, age groups and occupations. You can’t rely on YouTube, since that’s all edited. Not at all how a toddler would see the world.
- ehnto 2y agoI partially agree, but many humans do infact learn new languages from youtube, TV shows etc. There is also practically infinite minutes of long form conversations on youtube at least in English and Japanese. Not edited just 1+hour long videos of uncut talking. Funnily enough though, the comprehensible input movement for language learning has already started generating the type of content you're suggesting. Long first person view walks, following day to day activities and usually with a monologue about what they are seeing inbetween.
- vasco 2y agoHave you heard of live streaming and lifestreaming? There's countless hours of this exact type of footage available on twitch. I would bet Google is also already recording when doing mapping, so they should have videos going through every street in the world as well.
- nosrepa 2y agoA lot of people don't know that twitch started as justin.tv and was specifically a life streaming platform.
- realusername 2y agoI've been learning a language from Tiktok videos for more than two years and that totally works, the results are there. This format is even more edited than YouTube
- leobg 2y agoYou also got trained by ~5k hours of unedited real-time first person footage before you started to speak.
- cs702 2y agoNice work. The authors train a model with two contrastive objectives: (1) predict which video clip corresponds to a given audio file, and (2) predict which audio clip corresponds to a given video clip. Together, these two training objectives induce the model to learn to associate different sounds from a given audio clip, like words forming a spoken sentence, with different objects shown in a video clip, like the pixels depicting the subject of the spoken sentence -- and vice versa. If I understand the paper correctly, the model was trained on tens of thousands of video and audio clip pairs, randomly sampled 64 million times (80 samples/batch x 0.8 million training steps). For comparison, over the course of a baby's first two years of life, the baby will get trillions of sound samples (measured at 44.1KHz) and billions of frames of visual data (measured at 30fps). It's not too hard to imagine that a large AI model in the near future, trained on comparably vast quantities of audio and video data, can learn to recognize objects and words, like a baby. The most exciting takeaway, for me, is that a bigger model inside a robot body should be able to integrate learning in this manner across all five human data modalities -- vision, hearing, smell, touch, taste -- as well as any other data modalities for which the robot has sensors -- radar, lidar, GPS position, etc. We sure live in exciting times! Link to paper, code, demo, and data: https://mhamilton.net/denseav https://mhamilton.net/denseav
- NooneAtAll3 2y agofirst half a year-year babies can't focus their eyes, iirc still several of OOM difference
- cs702 2y ago> still several of OOM difference I agree. Nothing I wrote above disagrees with your statement.
- HeatrayEnjoyer 2y agoCan't they just add more?
- HeatrayEnjoyer 2y ago
- ramijames 2y agoNow, do whales.
- fernly 2y ago> “Another exciting application is understanding new languages, like dolphin or whale communication, which don’t have a written form of communication. Our hope is that DenseAV can help us understand these languages that have evaded human translation efforts since the beginning."
- ramijames 2y agoYay
- vicentwu 2y agoFasciating. Language is a type of action evoloved for information exchaging, which maps latent "video", "audio" and "thoughts" into "sentences" and vice versa.
- mkolodny 2y agoGiven that blind people can learn to speak, audio alone must be enough to learn language. And given that deaf people can learn sign language, video alone must also be enough to learn language. That’s assuming that touch and emotion aren’t crucial to language learning.
- TeMPOraL 2y agoDifferent kinds of experiences, but highly correlated.
- chumanak 2y agoEmotion it’s crucial for sure
- qup 2y agoGiven Helen Keller's grasp of language, touch alone must be enough to learn language.
- Hugsun 2y agoI've often wondered if there aren't some structures in the brain, that have been selected for since the advent of language, that are good at learning up languages.
- Shorel 2y agoI think Carl Sagan discusses this topic in "Broca's Brain". https://en.wikipedia.org/wiki/Broca's_Brain https://en.wikipedia.org/wiki/Broca's_Brain
- dmd 2y agoThis is one of the most discussed and argued ideas in all of linguistics and philosophy. https://plato.stanford.edu/entries/innateness-language/ https://plato.stanford.edu/entries/innateness-language/
- Hugsun 2y ago
- amelius 2y agoHow is this different from what LLMs do (by reading written works)?
- Hugsun 2y agoIt learns to recognize which objects correspond to which sounds, and vice versa. It doesn't really understand linguistic concepts like grammar, sentences, logic, etc. It doesn't really have much in common with LLMs. If the ideas were combined, the results might be interesting, although probably requiring very significant compute.