3 ms·
I suspect this won't matter as algorithms and architectures improve. I don't think whole-web scrapes will be necessary or even useful to the training of future
by neurobama 3y ago
I suspect this won't matter as algorithms and architectures improve. I don't think whole-web scrapes will be necessary or even useful to the training of future LLMs.
As a thought experiment, consider how many words the average human hears before age 10. Most people speak at 100-150 words per minute. Let's be generous and assume a young child hears spoken language 16 hours a day at 150 wpm for 10 years. That's a total of roughly 500 million words, and likely a gross overestimate. English Wikipedia alone contains billions of words. Reams of video, literature, and spoken media are also easily available to teach dialog and conversation flow, not to mention multi-modal associations between language, objects, and higher-level concepts.
I am just an enthusiast, not an AI researcher, but the human proof of concept seems highly encouraging here.
- _glass 3y agoI am also not an AI researcher, but shouldn't there be some quantification of how efficient the learning is. I imagine a human has much more information than the pure audio, which would then level up the information density, i.e. the 500 million words are not an equivalent, but should be compared to all of the audio, visual, tactile, etc. information, plus the context, meaning the desire of the humans, behavior, response patterns.
- neurobama 3y agoFor sure. That's why I mentioned multi-modal training, where the language model is trained on text in tandem with visual and audio training data. GPT-4 was trained on both images and text and can even explain why visual jokes are funny: https://openai.com/research/gpt-4 https://openai.com/research/gpt-4 Training an LLM on video seems like the clear next step. I wonder if images and video might not encode visuospatial relations that are helpful to learning language that describes them.