7 ms·

Is the Kalman filter just a low-pass filter?

267 points by jbay808 4y ago

roger_ 4y ago
Another question I'm curious about: is the Kalman filter ever not an LPF (e.g. HPF or BPF)?
- hgsgm 4y ago
  it's a noise filter, so it wouldn't be high pass or band path. it's approximately low pass filter when it hasn't converged precisely to a low pass filter.
- Paedor 4y ago
  Conceptually, every step of applying a Kalman filter in the simplest case is just updating a Gaussian prior representing position with a new datapoint sampled with (assumed) Gaussian error. The result is equivalent to convolving a Gaussian representing position with a Gaussian representing noise. In the frequency domain, that's equivalent to multiplying one Gaussian by another one with zero mean, which will always put higher weight on low frequencies. No matter what the gain is in the Kalman filter, that'll still be true as far as I can tell. As the gain varies, the cutoff within the low-pass filter will change though. It gets harder to analyze when you start using non-linear models to update position though. Generally, I think the same logic applies.
- jbay808 4y ago
  Author here. Yes, if the system has a high-pass behaviour, then the corresponding Kalman filter will converge to a high-pass filter, and so on. Non-low-pass-like behaviour can emerge from the prediction step, if the model is not unitary. If there's interest, I might write a followup article where a Kalman filter takes on a bandpass nature.
  - galangalalgol 4y ago
    I have seen it derived that it is equivalent to a matched filter, but I could not repeat said derivation.
  - motohagiography 4y ago
    I read Gaussian and Cauchy distributions as inputs, but can you add arbitrary distributions other than those two examples, and expect the filter to yield artifacts from those distributions? Naively, I'm thinking of different coloured noise as inputs, and what kind of artifacts a Kalman filter would produce over them.
    - freemint 4y ago
      With multi modal distribution you can end up stuck in any of the modes or even stuck in a low likelihood region in between the modes.
  - roger_ 4y ago
    I'd expect BPF behavior when estimating the rate of a noisy input (e.g. state: speed, observation: position), but I can't think of a process model that would be truly high-pass.
    - buescher 4y ago
      Ambient temperature effects on a photodiode.
- defrost 4y ago
  I've used it to compensate for directional measurement bias. If you pass a moving probe through a magnetic field your meaured results vary by direction (with or perpendicular to flux) and velocity. If you want to measure a geomagnetic field using an aircraft, the heading matters. To normalise in post processing you calibrate a Kalman filter by flying precessing butterfly wing patterns in a known relatively level flux area and then use that to remove the magnetic signature of the aircraft and heading from data collected over multiple headings and days. ( There are a few other twists - diurnal flux and induced field from the Earths GMF interactions needs to be isolated, etc ).
- 532nm 4y ago
  Yes, it can easily correspond to a BPF, for example when trying to track a sinusoidal signal, say the AC line frequency. Then you can be reasonably certain that it will be near 50 Hz (or 60 Hz), and the resulting Kalman filter will be a sharp band-pass filter centered around that (when neglecting higher harmonics).
hgsgm 4y ago
polishdude20 4y ago
Isn't it more than that though? It can be used to combine multiple data sources with varying degrees of uncertainty and change the confidence in each sensor on the fly. That's more than a low pass filter.
- jbay808 4y ago
  Absolutely. The Kalman filter has many uses, and filtering a single stream of data is only one of them. Sensor fusion is an important application, which I believe is mentioned in the first footnote.
sampo 4y ago
The real use of Kalman filter is model-data fusion. But to do model-data fusion, you need a model. In this blog post, they use μₖ = μₖ₋₁ as the model. If your model assumes that the process just stays constant, then all you are left with is filtering the noise. https://en.wikipedia.org/wiki/Kalman_filter https://en.wikipedia.org/wiki/Kalman_filter
- jbay808 4y ago
  Author here. A unitary model is still a model; I chose it mainly just to minimize the mental overhead for the reader. But even if you use a different model, the Kalman filter will still converge to a frequency-domain filter as long as the dynamics are not time-dependent (ie, constant measurement noise).
  - syntaxing 4y ago
    Feels kind of disingenuous though. The article makes it seem like you’re suggesting you can measure angle with simply a gyroscope + a low pass filter. But in reality, that method does not work at all (hence the famous NASA problem that Kalman filter solved). You need multiple input state space model to see the magic of Kalman filters.
    - jbay808 4y ago
      Hmm, I'm not trying to make that suggestion at all. How did I make that impression? I'd like to correct it. My goal with this article is only to help readers who have wondered how time-domain approaches like Kalman filtering relate to frequency-domain approaches like low-pass filtering, and to connect those dots. It's not a comprehensive article about the magic of Kalman filters.
      - syntaxing 4y ago
        (I do not have a PhD in control systems or specialize in it so take my input with a grain of salt) I think it's because Kalman filters and low pass filter are trying to achieve similar but different end goals. Kalman filter is trying to get a measurement and lower it's uncertainty. A low pass filter is trying to remove high fluctuations assuming those fluctuations is noise. As mentioned before, a low pass filter cannot achieve what a Kalman filter can. The example you show works because the noise is set to the high fluctuations. I think explaining a multiple input system is extremely critical to Kalman filters. Without it, it makes it a bit deceiving. For instance, if you do the same exercise in the article but measuring angle, you quickly see how a gyroscope + lowpass filter never reaches the level of measurement compared to gyroscope + accelerometer. Even funner, you can increase the noise in the gyroscope + accelerometer while keeping the gyroscope + lowpass filter noise the same, the gyroscope + accelerometer with a Kalman filter would always perform better.
syntaxing 4y ago
Only did a quick skim, but click bait-y title since it seems like they’re using a single input Kalman filter which then yeah…it’ll be a low pass filter. The article also doesn’t show any confidence bands which is a huge part of Kalman filters too. But the magic of a Kalman filter is multiple inputs. One thing that took a long time for me to click is multiple noisy measurements measured at the same time becomes one less noisy measurement thanks to the inherent nature of Bayesian statistics. That’s where the magic is and why Kalman filter are not low pass filters.
- jbay808 4y ago
  > The article also doesn’t show any confidence bands which is a huge part of Kalman filters too. The figure does show confidence bands in pale blue, although they're hard to see because the data points kind of obscure them. But since the covariance deterministically converges to a constant within a few samples, regardless of the data, the confidence bands are not very interesting in the example in the article.
- patrick451 4y ago
  There is no rule that a low pass filter must be single input. You just get a matrix of transfer functions. Most control loops with a reference input and sensor noise can be viewed this way.
cfcf14 4y ago
A paper I once found really illuminating on what a kalman filter was and how it related to a similar models is "A Unifying Review of Linear Gaussian Models", by Sam Roweis and Zoubin Ghahramani. It can be found here in pdf format: http://www.stat.columbia.edu/~liam/teaching/neurostat-fall20/papers/hmm/roweis-ghahramani-lds.pdf http://www.stat.columbia.edu/~liam/teaching/neurostat-fall20... Meta comment: I really appreciate that the author is taking the time to reply to people with queries (not this post). Thanks! It makes this community better.
greesil 4y ago
You may as well ask if least squares is just a way to draw a line through data.
- mbrudd 4y ago
  well played!
deleted 4y ago
[deleted]
jvanderbot 4y ago
Even better, Kalman Filter is just a moving average when model is identity and noise is identical.
kazinator 4y ago
The Kalman filter is a class of thing that can do more than just filtering in the frequency domain. E.g. here is someone's project for removing reverberation from speech, with sound sample files for A/B comparison: https://github.com/helianvine/kalman-filter-based-MCLP https://github.com/helianvine/kalman-filter-based-MCLP You cannot do that with just low/high/bandpass.
joeyo 4y ago
There are many ways to think about the Kalman filter. Here are a few that I like: * You can think of it as a Bayesian update process for linear Gaussian systems. That is: given a prior belief of the state of a system (and an uncertainty about that belief), and a measurement about the system (and uncertainty about that measurement), the Kalman filter tells you how to combine the prior with the measurement. This is very hard to do in general, but has an exact solution if your system is Linear-Gaussian. That's magical! * You can also think of it as a "better way to average". If I gave you two quantities that reflected some "true" value and asked you what the true value was, you would probably average them. The Kalman filter does you one better, because it tells you to average the two quantities weighted by how confident you feel about each one. * If you like control theory, you can think of the the Kalman filter as the dual of the Linear-Quadratic Regulator. That is, the KF is the optimal state estimator for Linear Gaussian systems in the same way that the LQR is the optimal (minimum cost) controller for LG systems. It's also worth pointing out that if the system you are estimating is being controlled, the KF can incorporate control inputs as well!
- a-dub 4y ago
  i've always just seen it as a wrapper for updating multivariate linear models when the inputs have noise that can be modeled with gaussians. linear models meaning anything that can be described as a matrix vector multiply. (which is a lot) but i'm no expert.
- antioedipus 4y ago
  * You can think of it as a factor graph with linear residuals and Gaussian noise functions in factors that connect a chain of variables, with all but the most recent variable marginalized. It’s a well known fact that linear, Gaussian factors result in a closed-form expression that gives the optimal maximum a posteriori estimate. The Kalman filter exploits this very special case. You can also write a LQR down with a factor graph (as the parent commented, the KF and LQR are duals).
  - alex_sf 4y ago
    Oh yes, of course.
mark-r 4y ago
"As always, I welcome your feedback." Was that a pun?
nullc 4y ago
This is kind of like picking a scalar input function with a constant derivative to assert that newton's method is just error feedback. -- in that special case that's all it is, and you won't get an intuition that newton's method constructs and solves a local linear approximation from the function's Taylor expansion unless you look at a more complicated case. Same applies here, single input with a trivial model, kalman is just a lowpass filter.
starchild_3001 4y ago
To me the biggest magic of kalman filtering wasn't the low-pass nature (yes it's a multidimensional low pass filter!), but rather issues around system identification. Namely if and when the system equations evolve as much as the hidden state, then things get really interesting! That specific formulation (joint system estimation + Kalman smoothing) was fairly interesting and useful for the applications I worked on. Obviously your mileage may vary :) As an addendum, I want to say a well designed whitening matrix + IIR filter can replace a Kalman filter, again depending on the application. Just makes things easier to understand, debug etc. Works if your vector is somehow decomposable into roughly independent scalars.
peter_retief 4y ago
I have been using band pass filters and didn't know about Kaiman filters!
P-NP 4y ago
As the author points out: there isn’t any one Kalman filter. The Kalman filter is really a recipe for constructing optimal linear predictive filters, but the actual characteristics of the resulting filter will depend on the dynamics, state variables, and sensors that you’ve tuned it for, and that dependence gets reflected numerically in the various matrices that your specific Kalman filter is built from.
AstralStorm 4y ago
The generic Kalman filter is also relatively old and slow to adapt. Explicit noise regularized (or variance regularized at least) filters do better especially in case the signal is corrupted by non-gaussian noise where Kalman filter can get divergent.
- memming 4y ago
  Do you have a reference?
vha3 4y ago
A fantastic discussion of Kalman filters is here: http://www.anuncommonlab.com/articles/how-kalman-filters-work/ http://www.anuncommonlab.com/articles/how-kalman-filters-wor... And a more technical introduction based on the article above: https://vanhunteradams.com/Estimation/Estimation.html https://vanhunteradams.com/Estimation/Estimation.html

267 points by jbay808 4y ago

roger_ 4y ago

Another question I'm curious about: is the Kalman filter ever not an LPF (e.g. HPF or BPF)?

hgsgm 4y ago
it's a noise filter, so it wouldn't be high pass or band path. it's approximately low pass filter when it hasn't converged precisely to a low pass filter.
Paedor 4y ago
Conceptually, every step of applying a Kalman filter in the simplest case is just updating a Gaussian prior representing position with a new datapoint sampled with (assumed) Gaussian error. The result is equivalent to convolving a Gaussian representing position with a Gaussian representing noise. In the frequency domain, that's equivalent to multiplying one Gaussian by another one with zero mean, which will always put higher weight on low frequencies. No matter what the gain is in the Kalman filter, that'll still be true as far as I can tell. As the gain varies, the cutoff within the low-pass filter will change though. It gets harder to analyze when you start using non-linear models to update position though. Generally, I think the same logic applies.
jbay808 4y ago
Author here. Yes, if the system has a high-pass behaviour, then the corresponding Kalman filter will converge to a high-pass filter, and so on. Non-low-pass-like behaviour can emerge from the prediction step, if the model is not unitary. If there's interest, I might write a followup article where a Kalman filter takes on a bandpass nature.
- galangalalgol 4y ago
  I have seen it derived that it is equivalent to a matched filter, but I could not repeat said derivation.
- motohagiography 4y ago
  I read Gaussian and Cauchy distributions as inputs, but can you add arbitrary distributions other than those two examples, and expect the filter to yield artifacts from those distributions? Naively, I'm thinking of different coloured noise as inputs, and what kind of artifacts a Kalman filter would produce over them.
  - freemint 4y ago
    With multi modal distribution you can end up stuck in any of the modes or even stuck in a low likelihood region in between the modes.
- roger_ 4y ago
  I'd expect BPF behavior when estimating the rate of a noisy input (e.g. state: speed, observation: position), but I can't think of a process model that would be truly high-pass.
  - buescher 4y ago
    Ambient temperature effects on a photodiode.
defrost 4y ago
I've used it to compensate for directional measurement bias. If you pass a moving probe through a magnetic field your meaured results vary by direction (with or perpendicular to flux) and velocity. If you want to measure a geomagnetic field using an aircraft, the heading matters. To normalise in post processing you calibrate a Kalman filter by flying precessing butterfly wing patterns in a known relatively level flux area and then use that to remove the magnetic signature of the aircraft and heading from data collected over multiple headings and days. ( There are a few other twists - diurnal flux and induced field from the Earths GMF interactions needs to be isolated, etc ).
532nm 4y ago
Yes, it can easily correspond to a BPF, for example when trying to track a sinusoidal signal, say the AC line frequency. Then you can be reasonably certain that it will be near 50 Hz (or 60 Hz), and the resulting Kalman filter will be a sharp band-pass filter centered around that (when neglecting higher harmonics).

hgsgm 4y ago

polishdude20 4y ago

Isn't it more than that though? It can be used to combine multiple data sources with varying degrees of uncertainty and change the confidence in each sensor on the fly. That's more than a low pass filter.

jbay808 4y ago
Absolutely. The Kalman filter has many uses, and filtering a single stream of data is only one of them. Sensor fusion is an important application, which I believe is mentioned in the first footnote.

sampo 4y ago

The real use of Kalman filter is model-data fusion. But to do model-data fusion, you need a model. In this blog post, they use μₖ = μₖ₋₁ as the model. If your model assumes that the process just stays constant, then all you are left with is filtering the noise. https://en.wikipedia.org/wiki/Kalman_filter https://en.wikipedia.org/wiki/Kalman_filter

jbay808 4y ago
Author here. A unitary model is still a model; I chose it mainly just to minimize the mental overhead for the reader. But even if you use a different model, the Kalman filter will still converge to a frequency-domain filter as long as the dynamics are not time-dependent (ie, constant measurement noise).
- syntaxing 4y ago
  Feels kind of disingenuous though. The article makes it seem like you’re suggesting you can measure angle with simply a gyroscope + a low pass filter. But in reality, that method does not work at all (hence the famous NASA problem that Kalman filter solved). You need multiple input state space model to see the magic of Kalman filters.
  - jbay808 4y ago
    Hmm, I'm not trying to make that suggestion at all. How did I make that impression? I'd like to correct it. My goal with this article is only to help readers who have wondered how time-domain approaches like Kalman filtering relate to frequency-domain approaches like low-pass filtering, and to connect those dots. It's not a comprehensive article about the magic of Kalman filters.
    - syntaxing 4y ago
      (I do not have a PhD in control systems or specialize in it so take my input with a grain of salt) I think it's because Kalman filters and low pass filter are trying to achieve similar but different end goals. Kalman filter is trying to get a measurement and lower it's uncertainty. A low pass filter is trying to remove high fluctuations assuming those fluctuations is noise. As mentioned before, a low pass filter cannot achieve what a Kalman filter can. The example you show works because the noise is set to the high fluctuations. I think explaining a multiple input system is extremely critical to Kalman filters. Without it, it makes it a bit deceiving. For instance, if you do the same exercise in the article but measuring angle, you quickly see how a gyroscope + lowpass filter never reaches the level of measurement compared to gyroscope + accelerometer. Even funner, you can increase the noise in the gyroscope + accelerometer while keeping the gyroscope + lowpass filter noise the same, the gyroscope + accelerometer with a Kalman filter would always perform better.

syntaxing 4y ago

Only did a quick skim, but click bait-y title since it seems like they’re using a single input Kalman filter which then yeah…it’ll be a low pass filter. The article also doesn’t show any confidence bands which is a huge part of Kalman filters too. But the magic of a Kalman filter is multiple inputs. One thing that took a long time for me to click is multiple noisy measurements measured at the same time becomes one less noisy measurement thanks to the inherent nature of Bayesian statistics. That’s where the magic is and why Kalman filter are not low pass filters.

jbay808 4y ago
> The article also doesn’t show any confidence bands which is a huge part of Kalman filters too. The figure does show confidence bands in pale blue, although they're hard to see because the data points kind of obscure them. But since the covariance deterministically converges to a constant within a few samples, regardless of the data, the confidence bands are not very interesting in the example in the article.
patrick451 4y ago
There is no rule that a low pass filter must be single input. You just get a matrix of transfer functions. Most control loops with a reference input and sensor noise can be viewed this way.

cfcf14 4y ago

A paper I once found really illuminating on what a kalman filter was and how it related to a similar models is "A Unifying Review of Linear Gaussian Models", by Sam Roweis and Zoubin Ghahramani. It can be found here in pdf format: http://www.stat.columbia.edu/~liam/teaching/neurostat-fall20/papers/hmm/roweis-ghahramani-lds.pdf http://www.stat.columbia.edu/~liam/teaching/neurostat-fall20... Meta comment: I really appreciate that the author is taking the time to reply to people with queries (not this post). Thanks! It makes this community better.

greesil 4y ago

You may as well ask if least squares is just a way to draw a line through data.

mbrudd 4y ago
well played!

deleted 4y ago

[deleted]

jvanderbot 4y ago

Even better, Kalman Filter is just a moving average when model is identity and noise is identical.

kazinator 4y ago

The Kalman filter is a class of thing that can do more than just filtering in the frequency domain. E.g. here is someone's project for removing reverberation from speech, with sound sample files for A/B comparison: https://github.com/helianvine/kalman-filter-based-MCLP https://github.com/helianvine/kalman-filter-based-MCLP You cannot do that with just low/high/bandpass.

joeyo 4y ago

There are many ways to think about the Kalman filter. Here are a few that I like: * You can think of it as a Bayesian update process for linear Gaussian systems. That is: given a prior belief of the state of a system (and an uncertainty about that belief), and a measurement about the system (and uncertainty about that measurement), the Kalman filter tells you how to combine the prior with the measurement. This is very hard to do in general, but has an exact solution if your system is Linear-Gaussian. That's magical! * You can also think of it as a "better way to average". If I gave you two quantities that reflected some "true" value and asked you what the true value was, you would probably average them. The Kalman filter does you one better, because it tells you to average the two quantities weighted by how confident you feel about each one. * If you like control theory, you can think of the the Kalman filter as the dual of the Linear-Quadratic Regulator. That is, the KF is the optimal state estimator for Linear Gaussian systems in the same way that the LQR is the optimal (minimum cost) controller for LG systems. It's also worth pointing out that if the system you are estimating is being controlled, the KF can incorporate control inputs as well!

a-dub 4y ago
i've always just seen it as a wrapper for updating multivariate linear models when the inputs have noise that can be modeled with gaussians. linear models meaning anything that can be described as a matrix vector multiply. (which is a lot) but i'm no expert.
antioedipus 4y ago
* You can think of it as a factor graph with linear residuals and Gaussian noise functions in factors that connect a chain of variables, with all but the most recent variable marginalized. It’s a well known fact that linear, Gaussian factors result in a closed-form expression that gives the optimal maximum a posteriori estimate. The Kalman filter exploits this very special case. You can also write a LQR down with a factor graph (as the parent commented, the KF and LQR are duals).
- alex_sf 4y ago
  Oh yes, of course.

mark-r 4y ago

"As always, I welcome your feedback." Was that a pun?

nullc 4y ago

This is kind of like picking a scalar input function with a constant derivative to assert that newton's method is just error feedback. -- in that special case that's all it is, and you won't get an intuition that newton's method constructs and solves a local linear approximation from the function's Taylor expansion unless you look at a more complicated case. Same applies here, single input with a trivial model, kalman is just a lowpass filter.

starchild_3001 4y ago

To me the biggest magic of kalman filtering wasn't the low-pass nature (yes it's a multidimensional low pass filter!), but rather issues around system identification. Namely if and when the system equations evolve as much as the hidden state, then things get really interesting! That specific formulation (joint system estimation + Kalman smoothing) was fairly interesting and useful for the applications I worked on. Obviously your mileage may vary :) As an addendum, I want to say a well designed whitening matrix + IIR filter can replace a Kalman filter, again depending on the application. Just makes things easier to understand, debug etc. Works if your vector is somehow decomposable into roughly independent scalars.

peter_retief 4y ago

I have been using band pass filters and didn't know about Kaiman filters!

P-NP 4y ago

As the author points out: there isn’t any one Kalman filter. The Kalman filter is really a recipe for constructing optimal linear predictive filters, but the actual characteristics of the resulting filter will depend on the dynamics, state variables, and sensors that you’ve tuned it for, and that dependence gets reflected numerically in the various matrices that your specific Kalman filter is built from.

AstralStorm 4y ago

The generic Kalman filter is also relatively old and slow to adapt. Explicit noise regularized (or variance regularized at least) filters do better especially in case the signal is corrupted by non-gaussian noise where Kalman filter can get divergent.

memming 4y ago
Do you have a reference?

vha3 4y ago

A fantastic discussion of Kalman filters is here: http://www.anuncommonlab.com/articles/how-kalman-filters-work/ http://www.anuncommonlab.com/articles/how-kalman-filters-wor... And a more technical introduction based on the article above: https://vanhunteradams.com/Estimation/Estimation.html https://vanhunteradams.com/Estimation/Estimation.html