I should say its very easy to construct synthetic datasets on which it is very clear that the only role of large layers is to supply more opportunities for the

I should say its very easy to construct synthetic datasets on which it is very clear that the only role of large layers is to supply more opportunities for the initialization to get it right. So, not much surprise there. But they claim they can extract the smaller subnet, which could be useful. Except they only provide experiments with very small nets so far, as the comments on reddit point out.