11 ms·
Chinchilla's Wild Implications
- drcode 4y agoIt seems obvious that the next barrier to overcome is learning from videos All of a sudden, the shortage of training data will no longer be a problem, since the amount of video data available dwarfs all other forms of data
- klipt 4y agoCoincidentally humans also learn from combined audio+visual feeds!
- skybrian 4y agoI'm not sure you get more words that way, though. The transcript of a half-hour video isn't very long, even if it's mostly talking.
- harperlee 4y agoBut theres a huge information load that could be mined to gain more awareness of words both in and not in the transcript, ideally.
- drcode 4y agoThe long term goal of these systems is to understand and reason about how the world works, videos have tons of info about this, regardless of words
- visarga 4y agoTL;DR - In large language models what matters most is the data, not the model size. And data is finite.
- gfodor 4y agoMy understanding is all of these models trained one epoch. Not knowing the change in loss in these LM’s by doing a second pass seems like a huge blind spot. What’s the best example to point to that could help us understand how likely it is that would move the needle comparable to adding more unique tokens?
- fxtentacle 4y agoFootnote 11 says that repeating data is considered harmful https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla-s-wild-implications#fnb0u5a6k81q8 https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla...
- VirusNewbie 4y agoFor efficiency reasons, no? I don’t think it is saying it makes the model perform worse.
- ambrozk 4y agoNo, it's saying that the model performs worse. If you read the linked papers on footnote 11 you'll see the research in question.
- VirusNewbie 4y agoI read one of the papers, it seemed like it said it was a waste of resources to not dedupe, not that it made things worse.
- p1esk 4y agoNot sure which of the two papers you’re referring to. The Anthropic paper [1] clearly shows an example where not deduping leads to serious quality degradation. However it’s not clear if training for more than one epoch on deduped and well balanced data would help. I personally think it should and the reason people don’t do it might be because it’s too expensive. [1] https://arxiv.org/pdf/2107.06499.pdf https://arxiv.org/pdf/2107.06499.pdf
- visarga 4y agoThe effect of deduplication is mostly related to reducing regurgitation of training data, the decrease in perplexity is small (fig. 2).
- skybrian 4y agoSeems like once the easy scaling is over (you can't get more training data or train more on the data you have), the next step is using training data more efficiently with algorithmic improvements?
- macrolime 4y agoOr maybe multimodal models. If Google used YouTube as training data, they wouldn't run out of data in a while. Just feed it video once you run out of text.
- brian_cloutier 4y agoOr maybe hallucinating additional training data: my impression is that AlphaFold was able to get around a lack of sufficient data by training an initial model and having it make a bunch of predictions, then the high-confidence predictions from that set were used to train the final model. Given that language models seem to mostly know when they're making correct predictions [1] this method might be useful for stretching the available datasets into something larger without falling into the same pitfalls repeating data would give you. And if you squint it kind of looks like daydreaming? When I'm learning a new skill I often find myself playing back scenes and internally running some simulations of what I would have done differently. [1] https://arxiv.org/abs/2207.05221 https://arxiv.org/abs/2207.05221
- macrolime 4y agoInteresting idea. If a person sits down and writes a bunch of essays, that usually leads to improved language skills. Maybe hallucinating additional training data could be something somewhat similar for transformer networks.
- nl 4y agoSo many interesting take-aways here. * The "bigger models" line of research is probably exhausted for now: "insofar as we trust our equation, this entire line of research -- which includes GPT-3, LaMDA, Gopher, Jurassic, and MT-NLG -- could never have beaten Chinchilla, no matter how big the models got" * We don't really know how much written data is available. Even Google - which has access to data repositories like scanned books that they cannot share - seems to have trouble getting consistently sized datasets for unknown reasons. * There seems to be as much (more?) written English available in books than on the entire web. MassiveText "scrape" = 506B tokens, MassiveText "books" = 560B tokens
- GalahiSimtam 4y agoSeems there is an order-of-magnitude error for LaMDA though...
- jesprenj 4y ago[meta] another useless rant about the web. the page loads perfectly for me on chromium 77, but after the bloated javascript finally loads, the entire content gets replaced with > Error: TypeError: Object.fromEntries is not a function /:
- sbierwagen 4y agoChrome 77 came out three years ago?
- jesprenj 4y agoI never updated my setup, I'd need to manually compile the browser, since AFAIK kiwi browser doesn't ship stable binaries with telemetry disabled. I'll get around to doing it someday, but I found it quite struggling if not impossible to install android SDKs without android studio. Kiwi source branch is Chromium 77.0 + Kiwi backported fixes, will show 88.0.4324.152 to websites for compatibility reasons (64-bit)
- tigerBL00D 4y agoNaive question about units. Are all three terms on the same scale to make direct magnitude comparison meaningful?
- fnbr 4y agoYes. The scale is “units of loss”, which is a strange term, but well founded.
- Jack000 4y agoThe reason that language models require large amounts of data is because they lack grounding. When humans write a sentence about.. let's say "fire", we can relate that word to visual, auditory and kinesthetic experiences built from a coherent world model. Without this world model the LM needs a lot of examples, essentially it has to remember all the different contexts in which the word "fire" appears and figure out when it's appropriate to use this word in a sentence. A perfect language model is literally impossible because you can always contrive a novel context that the LM has never seen before. I suspect that the more data modalities we add the less data would required, but that's not the whole picture either. For example, text-to-image generators often makes weird mistakes that look "unphysical", or objects that look like they're flowing into eachother. The reason is because these models (including DALLE) uses a simple UNET, which basically only sees textures. What it lacks is a human inductive bias that 2d images are typically representations of a 3d world, a world which contains largely discrete objects and physics. It makes these mistakes because it doesn't know what objects are, and need to brute force this idea from a ton of observations. Even simple cognitive abilities like object persistence requires time perception, which these models lack. I think the fact that these models can make up for this deficit with a ton of data is very telling. There is a lot of low hanging fruit in integrating more data modalities.
- p1esk 4y agoEven simple cognitive abilities like object persistence requires time perception, which these models lack. What do you mean? If we send a robot to explore its environment, and train it by having it constantly predict the next video frame, wouldn’t it eventually learn the physics and therefore gained the “time perception”?
- Jack000 4y agoYeah that would work. I'm talking about LLMs, DALL-E and diffusion image generators specifically.
- tehsauce 4y agothis is exactly what clip already does, and there will be massive improvements in this area in coming years, I promise.