Fully observable and we still have no idea what the hell it's doing. Makes neuroscience seem kinda bleak doesn't it? There has been a lot of great work lately

Fully observable and we still have no idea what the hell it's doing. Makes neuroscience seem kinda bleak doesn't it? There has been a lot of great work lately building up a theory of how these things work, but it is very much still in the early stage. Jascha Sohl-Dickstein in particular has been doing some great work on this. We don't even have answers to the most basic questions. For instance (pedagogically), how the hell is it possible to train these things at all? They have ridiculously non-convex loss landscapes and we optimize in the dumbest conceivable way, first-order stochastic gradient descent. This should not work. But it does, all too often. Not a great example because there are easy hand wavy arguments as to why it should work, but as far as proofs go... The hand wavy argument goes as follows: - we're in like a 10000 dimensional space, for the stationary point we're at to be a true local minima that means each one of those 10000 dimensions goes uphill in either direction. It's overwhelming likely that there's at least one way out - there are many many different ways to set the params of the net for each function. Permutation is a simple example. We really have no idea how these things work. Anyone who tells you otherwise is lying to you...