Actually I just read that paper properly and I see they did visualise the higher level layer weights too: "For visualizing what units do on the 2nd and 3rd lay

Actually I just read that paper properly and I see they did visualise the higher level layer weights too: "For visualizing what units do on the 2nd and 3rd layer, we used the activation maxi- mization technique described by Erhan et al. (2009)"[1] So there you have it! You can visualise all of a neural network! But you have to implement some specific numerics to get the data out. Reading the first layer is trivial, and subsequent layers need some analysis. [1] Erhan 2009 Visualizing higher layer features of a deep network (good find, thanks for asking the right questions!)