3 ms·
+1 I’ve always had the feeling that training from randomly initialized weights without seeding some substructure is unnecessarily slowing LLM training. Similar
by rao-v 8mo ago
+1 I’ve always had the feeling that training from randomly initialized weights without seeding some substructure is unnecessarily slowing LLM training.
Similarly I’m always surprised that we don’t start by training a small set of layers, stack them and then continue.
- ACCount37 8mo agoBetter-than-random initialization is underexplored, but there are some works in that direction. One of the main issues is: we don't know how to generate useful computational structure for LLMs - or how to transfer existing structure neatly across architectural variations. What you describe sounds more like a "progressive growing" approach, which isn't the same, but draws from some similar ideas.
- rao-v 8mo agoAgree re: progressive growing In terms of sub structure - in the old days of Core Wars randomly scattering bits of code that did things could pay off. I’m imagining similar things for LLMs - just set 10% of weights as specific known structures and watch to see which are retained / utilized by models and which get treated like random init