This is because a deep learning model is "just" a chain of simple, continuous geometric transformations mapping one vector space into another. Per my understan

This is because a deep learning model is "just" a chain of simple, continuous geometric transformations mapping one vector space into another. Per my understanding - Each vector space represents the full state of that layer. Which is probably why the transformations work for such vector spaces. A sorting algorithm unfortunately cannot be modeled as a set of vector spaces each representing the full state. For instance, an intermediary state of a quick sort algorithm does not represent the full state. Even if a human was to look at that intermediary step in isolation, they will have no clue as to what that state represents. On the contrary, if you observe the visualized activations of an intermediate layer in VGG , you can understand that the layer represents some elements of an image.