5 ms·

Always telling this whenever the topic of Kalman Filters come up: If you're learning the Kalman Filter in isolation, you're kind of learning it backwards and m

by rsp1984 2y ago

Always telling this whenever the topic of Kalman Filters come up: If you're learning the Kalman Filter in isolation, you're kind of learning it backwards and missing out on huge "aha" moments that the surrounding theory can unlock. To truly understand the Kalman Filter, you need to study Least Squares (aka linear regression), then recursive Least Squares, then the Information Filter (which is a different formulation of the KF). Then you'll realize the KF is just recursive Least Squares reformulated in a way to prioritize efficiency in the update step. This PDF gives a concise overview: [1] http://ais.informatik.uni-freiburg.de/teaching/ws13/mapping/pdf/slam07-eif.pdf http://ais.informatik.uni-freiburg.de/teaching/ws13/mapping/...

raincom 2y ago
That’s the one should learn any subject—-be it physics, chemistry, math, etc. However, textbooks don’t follow that technique.
- ryan-duve 2y ago
  I strongly recommend Elements of Physics by Millikan and Gale for anyone who wants to learn pre-quantum physics this way.
dr_kiszonka 2y ago
You are probably right, but many folks following your advice will give up halfway through and never get to KF.
jampekka 2y ago
I think the easiest way depends on your background knowledge. If you understand linearity of the Gaussian distribution and the Bayesian posterior of Gaussians, the Kalman filter is almost trivial. For (1D) we get the prior from the linear prediction X'1 = X0*a + b, for which mean(X'1) = mean(X0)*a + b and var(X'1) = var(X0)*a^2, where a and b give the assumed dynamics. The posterior for Gaussians is the precision weighted mean of the prior and the observation: X1 = (1 - K)*X'1 + Y*K, where the weighting K = (1/var(X'1))/(1/var(X'1) + 1/var(Y)), with Y being the Gaussian observation. Iterating this gives the Kalman filter. Generalizing this to multiple dimensions is straightforward given the linearity of multidimensional Gaussians. This is how (after I understood it) it makes it really simple to me, but things like linearity of (multidimensional) Gaussians and the posterior of Gaussians as such probably are not.
- RossBencina 2y ago
  What you write is simple. But your scalar model suppresses the common situation of a measurement matrix with output dimension less than state dimension. Exactly how the Kalman gain formula works under this setting I'm less clear on. Beyond that, additional insight is needed when the measurement matrix is non-linear and K = P_xy P_y^{-1} as in the UKF. At least I get stuck there, with little formal statistics work.
  - jampekka 2y ago
    Good catch, indeed a measurement matrix is needed if the state and measurement are of different dimensions or require a (linear) transformation. For that use Y = H*z where H is the measurement matrix and z is the observation vector. For UKF the Y is still a multidimensional Gaussian and computing K is the same. The mean and covariance of Y is computed from Z and the nonlinear measurement function using the unscented transform.
- krtab 2y ago
  I have written down a similar derivation here if anyone is interested: https://ngr.yt/blog/kalman/ https://ngr.yt/blog/kalman/
jtrueb 2y ago
You can keep telling this, but this “esoteric” math is often too much for the people actually implementing the filters.
- IgorPartola 2y ago
  I understood it as reestimation with a dynamic weight factor based on the perceived error factor. I know it’s more complex than that but this simplified version I needed at one point and it worked.
- defrost 2y ago
  It's bread and butter math for physics, Engineering (trad. Engineering), Geophysics, Signal processing etc. Why would anyone have people implementing Kalman filters who found the math behind them "esoteric"? Back in the day, in my wet behind the ears phase, my first time implementing a Kalman Filter from scratch, the application was to perform magnetic heading normalisation for on mag data from an airborne geophysical survey - 3 axis nanotesla sensor inputs on each wing and tail boom requiring a per survey calibration pattern to normalise the readings over a fixed location regardless of heading. This was buried as part of a suite requiring calculation of the geomagnetic reference field (a big paramaterised spherical harmonic equation), upward, downward and reduce to pole continuations of magnetic field equations, raw GPS post processing corrections, etc. where "etc" goes on for a shelf full of books with a dense chunk of applied mathematics
- jampekka 2y ago
  FWIW, I think I understand Kalman filters quite well, but the linked PDF is hard for me to follow, and I'd really struggle to understand it if I didn't already know what it's saying. I think the lesson there is that the Kalman filter is simpler in the "information form" where the Gaussian distribution is parameterized using the inverse of the covariance matrix. If you don't already know what that means, you likely don't get much out of that. I think the more intuitive way is to first understand the 1D case where the filter result is weighted average of the prediction and the observation where the weights are the multiplicative inverses of the respective variances (the less uncertainty/"inprecision", the more you give weight). In the multidimensional case the inverse is the matrix inverse but the logic is the same. More generally the idea is to statistically predict the next step from the previous and then balance out the prediction and the noisy observation based on the confidence you have in each. This intuition covers all Bayesian filters. The Kalman filter is a special case of the Bayesian filter where the prediction is linear and all uncertainties are Gaussian, although it was understood this way only well after Kalman invented the eponymous filter. Not sure how intuitive that's either, but don't be too worried if these things aren't obvious, because they aren't until you know all the previous steps. To implement or use a Kalman filter you don't really need this statistical understanding. If you prefer to understand things more "procedually", check out the particle filter. It's conceptually the Bayesian filter but doesn't require the mathematical analysis. That's the way I really understood the underlying logic.
bradly 2y ago
I appreciate you taking the time to help people understand higher level concepts. From a different perspective... I have no traditional background in mathematics or physics. I do not understand the first line of the pdf you posted nor do I understand the process for obtaining the context to understand it. But I have intellectual curiosity. So the best path forward for me understanding is a path that can maintain that curiosity while making progress on understanding. I can reread the The Six (Not So ) Easy Pieces and not understand any of it and still find value in it. I can play with Arnold's cat and, slowly, through no scientific rigor other than the curiosity of the naked ape, I can experience these concepts that have traditionally been behind gates of context I do not possess keys to. http://gerdbreitenbach.de/arnold_cat/cat.html http://gerdbreitenbach.de/arnold_cat/cat.html
- keithalewis 2y ago
  [flagged]
  - bradly 2y ago
    > Just stop whining about it in public. I'm curious if this is how my reply came across?
    - vo2maxer 2y ago
      Not at all. I share your sentiment. Many topics are beyond my intellectual grasp at this time, but I’m always hopeful that my curiosity will lessen their obscurity given time and persistence.
    - airstrike 2y ago
      Not at all. Your comment was perfectly fine! And that reply was way out of line...
    - keithalewis 2y ago
      Your statement "the best path forward for me understanding is a path that can maintain that curiosity" comes off as entitled. Only you owe yourself that. "I can play with Arnold's cat and, slowly, through no scientific rigor..." You are fooling yourself into believing you understand if you leave out rigor. If your post has the words "I" and "me" in every other sentence, maybe you should consider whether or not you are adding value for others in the online discussion. I doesn't cost you anything to ask other people to do things for you, and you will no doubt find other thoughtless people chiming in about your right to do that. If everything you need is not at your fingertips already, maybe no amount of handholding will help.
jvanderbot 2y ago
Are you me? I feel like I say this every time too! Perfectly captured.
jbullock35 2y ago
I found this article invaluable for understanding the Kalman filter from a Bayesian perspective: Meinhold, Richard J., and Nozer D. Singpurwalla. 1983. "Understanding the Kalman Filter." American Statistician 37 (May): 123–27.
RossBencina 2y ago
This is more or less the approach that is taken by Dan Simon's "Optimal State Estimation" book that I came here to recommend: https://academic.csuohio.edu/simon-daniel/state-estimation/ https://academic.csuohio.edu/simon-daniel/state-estimation/ All the prerequisites are covered prior to introducing the Kalman filter in chapter 5. Although Simon does not go through the information filter before introducing the Kalman filter, he discusses it later. However, to understand recursive least squares, in particular the covariance matrix update you're going to need a firm grounding in probability and statistics. Simon makes the case that probability theory is a less strict pre-requisite than multiple-input-multiple-output (state space) linear systems theory (for which I can recommend Chen's "Linear System Theory and Design"). So I would argue that to understand Kalman filters you need to know state space systems modelling, both continuous time and discrete time discretisation methods (this provides the dynamics that describe the time-update step), plus you need to know enough multivariate statistics to understand how the Kalman filter propagates the gaussian random variables (i.e. the Kalman state) through the dynamics and back and forth through the measurement matrices.
richrichie 2y ago
Indeed. I always recommend Time Series Analysis by Hamilton for this reason. KF comes up as a natural way to solve linear models.
ilayn 2y ago
(Laughs in control theory)

raincom 2y ago

That’s the one should learn any subject—-be it physics, chemistry, math, etc. However, textbooks don’t follow that technique.

ryan-duve 2y ago
I strongly recommend Elements of Physics by Millikan and Gale for anyone who wants to learn pre-quantum physics this way.

dr_kiszonka 2y ago

You are probably right, but many folks following your advice will give up halfway through and never get to KF.

jampekka 2y ago

I think the easiest way depends on your background knowledge. If you understand linearity of the Gaussian distribution and the Bayesian posterior of Gaussians, the Kalman filter is almost trivial. For (1D) we get the prior from the linear prediction X'1 = X0*a + b, for which mean(X'1) = mean(X0)*a + b and var(X'1) = var(X0)*a^2, where a and b give the assumed dynamics. The posterior for Gaussians is the precision weighted mean of the prior and the observation: X1 = (1 - K)*X'1 + Y*K, where the weighting K = (1/var(X'1))/(1/var(X'1) + 1/var(Y)), with Y being the Gaussian observation. Iterating this gives the Kalman filter. Generalizing this to multiple dimensions is straightforward given the linearity of multidimensional Gaussians. This is how (after I understood it) it makes it really simple to me, but things like linearity of (multidimensional) Gaussians and the posterior of Gaussians as such probably are not.

RossBencina 2y ago
What you write is simple. But your scalar model suppresses the common situation of a measurement matrix with output dimension less than state dimension. Exactly how the Kalman gain formula works under this setting I'm less clear on. Beyond that, additional insight is needed when the measurement matrix is non-linear and K = P_xy P_y^{-1} as in the UKF. At least I get stuck there, with little formal statistics work.
- jampekka 2y ago
  Good catch, indeed a measurement matrix is needed if the state and measurement are of different dimensions or require a (linear) transformation. For that use Y = H*z where H is the measurement matrix and z is the observation vector. For UKF the Y is still a multidimensional Gaussian and computing K is the same. The mean and covariance of Y is computed from Z and the nonlinear measurement function using the unscented transform.
krtab 2y ago
I have written down a similar derivation here if anyone is interested: https://ngr.yt/blog/kalman/ https://ngr.yt/blog/kalman/

jtrueb 2y ago

You can keep telling this, but this “esoteric” math is often too much for the people actually implementing the filters.

IgorPartola 2y ago
I understood it as reestimation with a dynamic weight factor based on the perceived error factor. I know it’s more complex than that but this simplified version I needed at one point and it worked.
defrost 2y ago
It's bread and butter math for physics, Engineering (trad. Engineering), Geophysics, Signal processing etc. Why would anyone have people implementing Kalman filters who found the math behind them "esoteric"? Back in the day, in my wet behind the ears phase, my first time implementing a Kalman Filter from scratch, the application was to perform magnetic heading normalisation for on mag data from an airborne geophysical survey - 3 axis nanotesla sensor inputs on each wing and tail boom requiring a per survey calibration pattern to normalise the readings over a fixed location regardless of heading. This was buried as part of a suite requiring calculation of the geomagnetic reference field (a big paramaterised spherical harmonic equation), upward, downward and reduce to pole continuations of magnetic field equations, raw GPS post processing corrections, etc. where "etc" goes on for a shelf full of books with a dense chunk of applied mathematics
jampekka 2y ago
FWIW, I think I understand Kalman filters quite well, but the linked PDF is hard for me to follow, and I'd really struggle to understand it if I didn't already know what it's saying. I think the lesson there is that the Kalman filter is simpler in the "information form" where the Gaussian distribution is parameterized using the inverse of the covariance matrix. If you don't already know what that means, you likely don't get much out of that. I think the more intuitive way is to first understand the 1D case where the filter result is weighted average of the prediction and the observation where the weights are the multiplicative inverses of the respective variances (the less uncertainty/"inprecision", the more you give weight). In the multidimensional case the inverse is the matrix inverse but the logic is the same. More generally the idea is to statistically predict the next step from the previous and then balance out the prediction and the noisy observation based on the confidence you have in each. This intuition covers all Bayesian filters. The Kalman filter is a special case of the Bayesian filter where the prediction is linear and all uncertainties are Gaussian, although it was understood this way only well after Kalman invented the eponymous filter. Not sure how intuitive that's either, but don't be too worried if these things aren't obvious, because they aren't until you know all the previous steps. To implement or use a Kalman filter you don't really need this statistical understanding. If you prefer to understand things more "procedually", check out the particle filter. It's conceptually the Bayesian filter but doesn't require the mathematical analysis. That's the way I really understood the underlying logic.

bradly 2y ago

I appreciate you taking the time to help people understand higher level concepts. From a different perspective... I have no traditional background in mathematics or physics. I do not understand the first line of the pdf you posted nor do I understand the process for obtaining the context to understand it. But I have intellectual curiosity. So the best path forward for me understanding is a path that can maintain that curiosity while making progress on understanding. I can reread the The Six (Not So ) Easy Pieces and not understand any of it and still find value in it. I can play with Arnold's cat and, slowly, through no scientific rigor other than the curiosity of the naked ape, I can experience these concepts that have traditionally been behind gates of context I do not possess keys to. http://gerdbreitenbach.de/arnold_cat/cat.html http://gerdbreitenbach.de/arnold_cat/cat.html

keithalewis 2y ago
[flagged]
- bradly 2y ago
  > Just stop whining about it in public. I'm curious if this is how my reply came across?
  - vo2maxer 2y ago
    Not at all. I share your sentiment. Many topics are beyond my intellectual grasp at this time, but I’m always hopeful that my curiosity will lessen their obscurity given time and persistence.
  - airstrike 2y ago
    Not at all. Your comment was perfectly fine! And that reply was way out of line...
  - keithalewis 2y ago
    Your statement "the best path forward for me understanding is a path that can maintain that curiosity" comes off as entitled. Only you owe yourself that. "I can play with Arnold's cat and, slowly, through no scientific rigor..." You are fooling yourself into believing you understand if you leave out rigor. If your post has the words "I" and "me" in every other sentence, maybe you should consider whether or not you are adding value for others in the online discussion. I doesn't cost you anything to ask other people to do things for you, and you will no doubt find other thoughtless people chiming in about your right to do that. If everything you need is not at your fingertips already, maybe no amount of handholding will help.

jvanderbot 2y ago

Are you me? I feel like I say this every time too! Perfectly captured.

jbullock35 2y ago

I found this article invaluable for understanding the Kalman filter from a Bayesian perspective: Meinhold, Richard J., and Nozer D. Singpurwalla. 1983. "Understanding the Kalman Filter." American Statistician 37 (May): 123–27.

RossBencina 2y ago

This is more or less the approach that is taken by Dan Simon's "Optimal State Estimation" book that I came here to recommend: https://academic.csuohio.edu/simon-daniel/state-estimation/ https://academic.csuohio.edu/simon-daniel/state-estimation/ All the prerequisites are covered prior to introducing the Kalman filter in chapter 5. Although Simon does not go through the information filter before introducing the Kalman filter, he discusses it later. However, to understand recursive least squares, in particular the covariance matrix update you're going to need a firm grounding in probability and statistics. Simon makes the case that probability theory is a less strict pre-requisite than multiple-input-multiple-output (state space) linear systems theory (for which I can recommend Chen's "Linear System Theory and Design"). So I would argue that to understand Kalman filters you need to know state space systems modelling, both continuous time and discrete time discretisation methods (this provides the dynamics that describe the time-update step), plus you need to know enough multivariate statistics to understand how the Kalman filter propagates the gaussian random variables (i.e. the Kalman state) through the dynamics and back and forth through the measurement matrices.

richrichie 2y ago

Indeed. I always recommend Time Series Analysis by Hamilton for this reason. KF comes up as a natural way to solve linear models.

ilayn 2y ago

(Laughs in control theory)