2 ms·
I wonder to what degree these results come from more training and emergent capability from that vs these kinds of riddles now being known and well represented i
by tosh 3y ago
I wonder to what degree these results come from more training and emergent capability from that vs these kinds of riddles now being known and well represented in training material?
Adjacent: one thing that makes llama3 special is that they spent extra effort on deduplicating training data (quality) + increased training data (quantity) and used 4x more code in training material (=> good for reasoning).
So perhaps we can still see huge capability gains in these small models by spending more effort on training data curation and generation?
- wrasee 3y agoExactly this. I only casually follow this space and I recognised many of those questions as essentially ‘classic’ questions by now. Surely any recently trained model taking recent data is going to have a ton of references to these now, especially those that are not officially part of some test set that might have been controlled for to some extent?