3 ms·
> The huge triumph of DL has been figuring out that as long as you can pose a problem in a differentiable way and you can obtain a sufficient amount of data, yo
by hyperbovine 10y ago
> The huge triumph of DL has been figuring out that as long as you can pose a problem in a differentiable way and you can obtain a sufficient amount of data, you can efficiently tackle it with a function approximator that can be optimized with first order methods - from that, flows everything.
This isn't really what is responsible for the success of deep learning. Lots and lots of machine learning algorithms existed before deep learning which are essentially optimizing a (sub-)differentiable objective function, most notably the LASSO. Rather, it's that recursive / hierarchical representation utilized by DL is somehow a lot better at representing complicated functions than things like e.g. kernel methods. I say "somehow" because exactly why and to what extent this is true is still an active subject of research within theoretical ML. It happen in many areas of math that "working in the right basis" can dramatically improve one's ability to solve certain problems. This seems to be what is happening here, but our understanding of the phenomenon is still quite poor.
- narrator 10y agoThe "somehow" part reminds me of studying differential equations. A lot of them can't be solved step-by-step analytically, like the equation for an RLC circuit, but some really smart guy figured them out through guesswork. Same goes for deep learning. A lot of these setups like the "Inception" framework and LeNet5 seem like a lot of clever guesswork and intuition. However, unlike differential equations, you can't know that you've actually solved the problem perfectly once you've figured out the answer. It's always just good enough util the next paper comes out claiming a better solution on whatever the latest canonical benchmark is.
- TumTumTree 10y agoCan you give a specific example of an area of math or a problem where "working in the right basis" offers dramatic improvement?