It's wishful thinking. Neutral networks need to be over parameterized to find good solutions, meaning there is a surface of solutions. The optimization procedu

It's wishful thinking. Neutral networks need to be over parameterized to find good solutions, meaning there is a surface of solutions. The optimization procedure tries to walk towards that surface as quickly as possible, and tend to find a low-energy point on the surface of solutions. In particular, a low energy solution isn't sparse, and therefore isn't interpretable.