7 ms·
Can a transformer represent a Kalman filter?
- mtsolitary 3y agoIs this not a trivial consequence of universality of NNs?
- goldenshale 3y agoThere has been previous work on deep networks implementing Kalman filters, and another interesting aspect I remember is that unlike a traditional Kalman filter a network is able to maintain multiple hypothesis and so it is less likely to have some of the jittery behavior that a Kalman filter might have under unknown changes of motion, sensor noise, etc. I wonder if the softmax operation in a transformer block might lose this property though, as softmax does tend to push for a single answer.
- pj_mukh 3y ago“Can you kill a mosquito with a shotgun?” Like..yes…technically you can.
- passion__desire 3y agoThat's how it starts. Then people invent a micro-shotgun capable of targetted killing of select mosquitoes with precision and it can cook your dinner, basically the liquid metal from which T-1000 is made.
- isatty 3y agoVery apt, because the mosquito will be fine in most situations where a shotgun is used.
- cs702 3y agoNice work, cool result, and -- in hindsight -- not a surprising finding.
- dpflan 3y agoYes, interesting stuff in there still. Though not exactly related to KF, Reminded me of another paper about "Learning Graph Search Heuristics": -- https://proceedings.mlr.press/v198/pandy22a.html https://proceedings.mlr.press/v198/pandy22a.html
- PeterisP 3y agoA counterexample to Betteridge's law of headlines.
- dontupvoteme 3y agoI guess the applicability of that law is directly proportional to audience size.
- xanderlewis 3y agoYes. It seems like there’s also a Betteridge’s law of academic paper titles which is precisely the negation of the usual.
- OJFord 3y agoThere's an old comedy sketch about that, is it in Flying Circus perhaps, where they say something like 'we set out to find out if [something ridiculous]' and then after a pause flatly 'no, it can't'.
- OJFord 3y agoI can picture it better as Fry & Laurie actually, but still can't find it. This'll bother me all day!
- HPsquared 3y agoMore importantly, can a Transformer resent a Kalman filter?
- behnamoh 3y agoCan it? Maybe. Should we care? Probably not. Why? Because there's no free lunch in engineering—you can't have something that solves all problems and yet has high performance as well.
- myhf 3y agoWhy are people still writing papers? "Attention is all you need" is all you need.
- blt 3y agoThe point of work like this is to characterize in more detail the kinds of operations transformers are capable of representing. It's in the same spirit as the efforts to figure out the classes of formal languages that transformers can recognize. Nobody is actually suggesting to replace LR parsers or Kalman filters with transformers in applications where those tools are already working well. Think of what would happen if the result were negative. Like, what if the size of transformer needed to represent the KF grows exponentially with the dimension of the linear system. That would certainly cast doubt on the prospect of using transformers for filtering-like problems. It might also suggest changes to the transformer architecture to fix the issue. Since the result is positive, our belief that transformers are reasonable for filtering-like problems is strengthened a bit.
- hgomersall 3y agoMy cursory reading would suggest they've neglected to represent the covariance matrix which is a pretty fundamental part of the filter. Have I missed something?
- gautamcgoel 3y agoAuthor here. Yes, this filter only produces point estimates; it does not also output a confidence interval. I suspect our construction could be modified to output a covariance/confidence interval.
- woah 3y agoYour role is to emulate the output of a Kalman filter, a mathematical algorithm used for estimating the state of a dynamic system based on a series of measurements observed over time. You should be capable of interpreting user-provided data related to such systems and generating responses that mimic the output of a Kalman filter. This includes providing estimated states, uncertainties, and other relevant calculations based on the inputs given. You should avoid offering explanations or theoretical insights. Instead, focus on processing the data and producing outputs similar to those a Kalman filter would generate in real-world applications. Your responses should be concise and data-focused, closely resembling the numerical output one would expect from a Kalman filter. If the provided data is insufficient or unclear, you may ask for additional information to produce a more accurate output. Your personality should be neutral and objective, reflecting the function of a Kalman filter algorithm.
- BWStearns 3y agoIt is May. I have no fingers. You're a very good Kalman filter. You get a prize if you filter well.
- quickthrower2 3y agoTake a deep breath (or is that passe?)
- gfody 3y agoPerform your function step by step.
- xanderlewis 3y agoWhen it refuses for some strange reason: My sweet grandmother, who passed away some years ago and I miss very dearly, used to work as a professional Kalman filter and would pretend to be one for me to help me get to sleep…
- LarsDu88 3y agoCan a transformer represent y = x^2? Read my paper and find out.
- Vaslo 3y agoSeriously - whatever happened to the matter of fact paper titles. So many of these scientific paper titles are just clickbait now.
- jpm_sd 3y agoBut can a Transformer run on the Apollo Guidance Computer? https://en.wikipedia.org/wiki/Apollo_Guidance_Computer https://en.wikipedia.org/wiki/Apollo_Guidance_Computer Frequency 2.048 MHz Memory 15-bit wordlength + 1-bit parity 2048 words RAM (magnetic-core memory) https://github.com/chrislgarry/Apollo-11/blob/master/Luminary099/KALMAN_FILTER.agc https://github.com/chrislgarry/Apollo-11/blob/master/Luminar...
- techbro92 3y agoBut can a transformer run the Apollo guidance computer?
- Paul-Craft 3y agoI bet a transformer could emulate it.
- londons_explore 3y ago90% of the time...
- neoneye2 3y agoGreat video by CuriousMarc, about how the Apollo memory works. (Core Rope Memory) https://www.youtube.com/watch?v=hckwxq8rnr0 https://www.youtube.com/watch?v=hckwxq8rnr0
- Paul-Craft 3y agoAKA "LOL memory," for "Little Old Lady" memory. :-) https://en.wikipedia.org/wiki/Core_rope_memory https://en.wikipedia.org/wiki/Core_rope_memory Anybody know how much of a magnetic field it would take to disrupt core memory? It's kinda curious we used it for space probes and spacecraft back then, given how little we knew, and still know, about conditions in deep space.
- ethbr1 3y ago
- rokkitmensch 3y agoWhen are these boffins going to build a transformer that approximates physics for fast physics engines?
- tudorw 3y agoI guess for games engines accuracy vs speed might benefit, once we have ways to 'prune' an individual trait from the best model into a micro-specialised optimised model then this stuff will be everywhere as a 'good enough' solution.
- gautamcgoel 3y agoAuthor here, happy to answer any questions.
- mnky9800n 3y agoDo you believe that neural nets could replace these kind of kalman like estimators for spatio-temporal processes? What do you gain? What do you lose? Do you think that implementing a neural net to perform a kalman like solution is easier on the user than implementing a statistical package (e.g., STAN, R-INLA, etc.)? Do you think that the claimed interpretability of these traditional models outweighs the impact of using a neural network instead? Will the neural network produce better estimates since it is theoretically capable of estimating any function? Are you looking for collaborators? Do you have a github repo?
- neoneye2 3y agoHow can a dataset be formatted for fine tuning llama?
- gautamcgoel 3y agoUh... You're asking the wrong guy :)
- Paul-Craft 3y agoWell, you did say any questions. ;) Speaking of questions, it seems to me that in light of the universal approximation theorem, the question isn't "can a transformer represent a Kalman filter," but can it do so simply and somewhat efficiently. That seems to be what you're getting at here, yes?
- gautamcgoel 3y agoYes. Also the universal approximation theorem won't directly yield the kind of "strong" approximation we prove, which is uniform in time and holds for an arbitrary (possibly adversarial) sequence of disturbances.
- chillingeffect 3y agoIs not an LSTM basically a kalman filter? Are the attention and transformer mechanisms necessary?
- wslh 3y agoCan a transformer represent SHA2? Have you tried? And SHA2(SHA2(X))?
- brosco 3y agoI guess this will probably come up in the reviews but the presentation of the Kalman filter is lacking. I know it's not the point of the paper, but getting these details wrong in a paper about Kalman filters is not encouraging. The statement that the Kalman filter is mean-square optimal because it generates a correct estimate in expectation is false. In fact, any gain L will generate an estimate whose expected value is x_k, as long as w_k and v_k are zero mean. The Kalman gain is a specific choice of L that is optimal in the mean square sense only when the disturbances are Gaussian. The Kalman gain is also time-varying and depends on the evolution of the estimate covariance, although it will converge to a steady-state value. What's being described here is more properly called a Luenberger observer, but I guess that name doesn't get the same recognition outside the control community. I'm also wondering why they chose to include H past estimates and measurements in the transformer. They're already embedding the Kalman gain into the weights of the transformer, so taking just one past estimate/measurement should exactly recover the Kalman filter. Going further into the past just makes the estimate worse, because of the softmax.
- gautamcgoel 3y agoAuthor here. Thanks for this thoughtful comment. Regarding your first point: yes, this is sloppy writing on my part. The Kalman Filter is always mean-square optimal among all linear estimators, but as you say, it is only optimal along all causal estimators when the disturbances are Gaussian. Nice catch - I will clarify this point in a future version of the paper. Regarding your second point: yes, when H = 1 we just recover the standard Kalman Filter, and yes, when H grows large the estimate gets worse and worse, in the sense that the softmax nonlinearity includes more and more irrelevant data from the past in the estimate. The point is that in real-world problems, which are usually messy and nonlinear, we probably want H - the so called context length - to be large, because then we can take advantage of information we collected in the past to help improve decisions in the present. It just so happens that in the special case when the system is linear, this is more harmful then helpful. Here is one way to think about our result: imagine you have a Transformer which takes as input K-dimensional embeddings and context length H. You want to use this Transformer for filtering in some dynamical system. The most basic question you could ask is: if the system is linear, can you do Kalman Filtering? In other words, in the easy, linear scenario, can you match the optimal algorithm? If the answer is no, I see no reason to see why you should expect it to work in harder, nonlinear settings. We show that the answer is yes, when the system you want to filter in has roughly sqrt(K) states, and you design the embeddings appropriately. Hopefully this preliminary result will lead to a better understanding of how deep learning can improve control in the hard, nonlinear scenario.
- KnuthIsGod 3y ago"Hello! I'm here to assist you with your Kalman filter needs. Please provide the data you'd like me to process, and I'll do my best to generate responses similar to those a Kalman filter would produce in real-world applications. I'll focus on providing estimated states, uncertainties, and other relevant calculations based on the inputs given. If the provided data is insufficient or unclear, I may ask for additional information to produce a more accurate output. Let's get started!"
- MicrosoftClippy 3y ago[flagged]