Well, the ventral stream is the "what" stream, so that's the natural one to draw comparisons to for the task of representing objects. The "where" pathway for th

Well, the ventral stream is the "what" stream, so that's the natural one to draw comparisons to for the task of representing objects. The "where" pathway for the most part represents space with a place code, that is, different neurons "care about" different parts of space in an increasingly abstract way going from representing eg "the left visual hemifield" early in the dorsal stream to representing "the left side of objects" in parietal cortex. It would be interesting to see if this kind of invariant emerges in NNs. Convolutional networks are able to caption images with labels like "a cake on top of an oven". What do their activations look like? I reject the notion that visual cortex is "simple", but I will concede that it is highly conserved across species. This just means that the representations that it uses are effective. You are no doubt right that Lecun was inspired by the biology of visual cortex (along with theorists before him), but you missed my meaning: can we build a network that doesn't start with edge detectors first that does better? My guess is no.