51 ms·
No one ever brings up the role of unsupervised learning for early children. Children don't have labels and only limited capabilities to "self-supervise" until t
by Der_Einzige 3y ago
No one ever brings up the role of unsupervised learning for early children. Children don't have labels and only limited capabilities to "self-supervise" until they are toddlers. The overwhelming majority of information that they take as input is "clustered" or learned in an unsupervised fashion. Baby learns that apple and orange are different objects before it knows anything else, like names or properties of those objects.
Babies take in far more data (but not text!) in an information theoretic sense than ChatGPT does. This is why I think these ideas that LLMs take far too much data for them to get good is sort of not true
- nojvek 3y agoThe pre-training phase of GPT for next work prediction is essentially that - dump all text you can find and let the model figure out the relationship between tokens/words. I agree though - what’s missing in neural architectures is advanced clustering capabilities purely from observing without any labels. Then the labeling phase is cheap and doesn’t require as much data.