Different initialization converge to different places, e.g https://arxiv.org/abs/1912.02757 https://arxiv.org/abs/1912.02757 For LLMs (as with other models), m

Different initialization converge to different places, e.g https://arxiv.org/abs/1912.02757 https://arxiv.org/abs/1912.02757 For LLMs (as with other models), many local optima appear to support roughly the same behavior. This is the idea of the problem being under-specified ie many more equations than unknowns so there are many ways to get the same result.