This all depends on the algorithm and whether the algorithm is discriminative or generative. Regarding the cat identification task: If the goal is to identify

This all depends on the algorithm and whether the algorithm is discriminative or generative. Regarding the cat identification task: If the goal is to identify cats from pictures, then what would you expect to learn about cats? The high-level features of a cat that we combine in a hierarchical fashion to arrive at 'cat'? After all, if we are just looking at pictures of cats, I think this is the best a human would be able to ascertain given numerous examples. If so, then algorithms like Convolutional Neural Nets do in fact provide such information - one just needs to look at the convolutional kernels to see these features emerge over the hidden layers. A simpler example is the task of identifying handwritten digits - and this is the example that is almost always referenced when discussing CNNs. A well-trained CNN on pictures of handwritten digits will extract things like diagonal and straight edges as high-level features and then combine those in appropriate spatial orderings to identify the digits 0-9...which is likely similar to how we discriminate between such digits. There are musical analogs as well of using NNs that extract out from a time-frequency representation, for example, individual instrument timbres (or at least decent approximations) in a mixture of instruments sounding simultaneously.