I like this paper because it turns some of the current intuition about deep nets around. It shows that the current understanding of why deep nets are so good a

I like this paper because it turns some of the current intuition about deep nets around. It shows that the current understanding of why deep nets are so good at so many (perceptual) tasks is that the depth buys you a lot. Yoshua Bengio will point out that there are functions that require exponentially more gates to encode when using shallower circuits. This might lead people to believe that deep nets are working so well because they are more fundamentally capable of representing the solutions to problems that people care about in a terse way. But this work proves (at least for this audio task), that there are solutions as good in the solution space spanned by shallow nets with memory usage that we can afford, we just didn't know how to find them.