4 ms·
It's worth noting that unlike KF and particle filters, none of the deep learning stuff that's making headlines these days is actually capable of properly modeli
by ur-whale 6y ago
It's worth noting that unlike KF and particle filters, none of the deep learning stuff that's making headlines these days is actually capable of properly modeling / simulating / approximating a dynamic system with a continuous time variable.
Most striking in deep learning, is that when all is said and done (i.e. unwrapping the so-called "recurrent" part of a deep net) all you're left with is a dumb function from Rn to Rp.
In other words, while the term "recurrent" might fool you into believing it, there isn't any actual feedback loop in deep nets, and they therefore can't properly model something whose output evolves over time.
I think there's something to be explored in adding actual feedback into deep nets.
The price is of course that the output is not a vector anymore: you must compute feed forward passes until the net "settles" to a fixed point or rather until the desired behavior is observed at the outputs.
Backprop also becomes a whole new game in that setting.
[EDIT]: but the prize is: your deep net, instead of being an approximation of a function from Rn to Rp, becomes an approximation of something that can transform a function into another function.
- omrjml 6y agoRegarding dynamical systems, Neural ODEs hold promise as they are more analogous to the numerical solvers. I think with any dynamical system, purely data driven approach can be problematic because, as you say, the standard NN architectures are not great for it. You can however add physical constraints, e.g. minimize the Lagrangian as the objective function, to bring stability to these NN emulators.
- pgt 6y agoNeural macros? :)
- sdenton4 6y ago"therefore can't properly model something whose output evolves over time." I dunno, WaveNet seems to work pretty good... It's an autoregressive model, and absolutely responds to it's own output. https://deepmind.com/blog/article/wavenet-generative-model-raw-audio https://deepmind.com/blog/article/wavenet-generative-model-r...
- blueblimp 6y ago> you must compute feed forward passes until the net "settles" to a fixed point Deep Equilibrium Models http://implicit-layers-tutorial.org/deep_equilibrium_models/ http://implicit-layers-tutorial.org/deep_equilibrium_models/
- mrfox321 6y agoODEs don't have feedback either. Their evolution is a function of it's point-in-time state, just like Recurrent nets. But, you can build great priors with a well-modeled ODE
- knuthsat 6y agoKF algorithm when layed out as: 1. Figure out the initial state 2. use process model to predict state at next time step 3. adjust belief of the true state by taking account prediction uncertainty 4. get a measurement and belief according to its uncertainty/accuracy 5. compute residual 6. compute scaling factor (what's more important, measurement or prediction) 7. set state with scaling factor 8. update belief in state given uncertainty of measurement pretty much allows plugging in a black box anywere. from process model to belief updates.
- nabla9 6y agoThe Missile Knows Where It Is... https://www.youtube.com/watch?v=bZe5J8SVCYQ https://www.youtube.com/watch?v=bZe5J8SVCYQ
- mam2 6y agoThats not true if your formulate you dl system in the same "two steps" ways of the kalman filter. Stuff like DynaNet or "backprop kalman" are really good deep learning extensions of the kalman filter for non linear / perceptual inputs.
- cornel_io 6y agoHow do you explain results like https://arxiv.org/abs/1708.01885 https://arxiv.org/abs/1708.01885, which uses LSTMs to do exactly what a Kalman filter would do except with less manual work?
- sriku 6y ago> but the prize is: your deep net, instead of being an approximation of a function from Rn to Rp, becomes an approximation of something that can transform a function into another function. Won't it be more like an approximation of a process instead of an approximation of a function? (I mean better described as that)
- midjji 6y agoYou are confusing that recurrent networks are trained unrolled, i.e. with finite history, with them beeing evaluated with finite history, which they generally arent. The key advantage a kalman filter has is in its long term properties being predictable, things like observability, controllability, and bibo stability. The key downside of kalman filters is that their nice properties only apply to linear systems and linear observations.
- abvr 6y agoCheck out the Extended Kalman filter for non-linear systems.
- knuthsat 6y agoMaybe better to check unscented Kalman filter instead, because the extended one is a mathematical horror.
- ChrisLomont 6y agoThe EKF works by linearizing the process, and suffers on even moderately nonlinear models as a result. It also suffers from the Gaussian assumption. So people switched to unscented (tries to model the underlying probability better) and particle filters (same idea), each giving better accuracy for most all problems. Now plenty of problems are switching to full Bayesian models as the ultimate in accuracy. So if you like KF, EKF, UKF, be sure to look into this entire chain of accuracy vs computation tradeoff algorithms.
- deleted 6y ago[deleted]