4 ms·
Not significantly, as I understand it. There's certainly variation in LLM abilities with different initializations but the volume and content of the data is a f
by jebarker 1y ago
Not significantly, as I understand it. There's certainly variation in LLM abilities with different initializations but the volume and content of the data is a far bigger determinant of what an LLM will learn.
- waynecochran 1y agoSo there is an "attractor" that different initializations end up converging on?
- andy99 1y agoDifferent initialization converge to different places, e.g https://arxiv.org/abs/1912.02757 https://arxiv.org/abs/1912.02757 For LLMs (as with other models), many local optima appear to support roughly the same behavior. This is the idea of the problem being under-specified ie many more equations than unknowns so there are many ways to get the same result.