6 ms·

The problem with sensor fusion is what do you do when the sensors disagree? You have to decide which sensor to trust. But LiDAR is subject to interference, and

by ggreer 3y ago

The problem with sensor fusion is what do you do when the sensors disagree? You have to decide which sensor to trust. But LiDAR is subject to interference, and it can’t see lane markings, traffic lights, speed limit signs, or emergency vehicle lights. If you ignored camera data and just relied on LiDAR, you wouldn’t be able to drive safely for very long. If you choose to ignore the LiDAR, then the sensor fusion is just adding noise to your camera vision model of the world. If you somehow do sensor fusion perfectly and the models never disagree, then why even have the LiDAR? At that point you’ve solved vision using cameras.

nomel 3y ago
LiDAR easily passes the “featureless white wall” test. That’s why I’ll be waiting.
yomlica8 3y ago
Don't you just make a conservative decision in that case? My eyes and ears disagree all the time, but if one of them says danger that is the one I listen too.
- parineum 3y ago
  I'm sure this example isn't new at all but it's the first time I've encountered the argument phrased this way and I find it incredibly compelling. Looks good, smells bad is another one (that probably happens more often). It did get me thinking about different senses and how I prioritize them. Given that humans strongest sense is visual, it's interesting to me that the priorities of what to trust seem opposite of my expectations. It seems to me that visual signals are seemingly least "trusted" compared to the other senses. As if the logic goes, "my sense of smell is so bad that if it detects danger, it must be very dangerous". Similarly, I'm sure dogs can smell rotting flesh long before meat is unsafe.
  - afterburner 3y ago
    Even simpler: if one sensor tells you confidently there's something in your way, and the other doesn't, assume there's something in your way.
    - Sodman 3y ago
      The problem in this particular case is that the sensor may be confidently telling you there's something in the way, but in reality it's just a plastic bag. Hard braking at 70mph on a highway as if you were about to hit a concrete wall is probably not a great outcome in this scenario if there's somebody behind you.
      - jjk166 3y ago
        If your sensor is confidently telling you that a plastic bag is a concrete wall, maybe you shouldn't be using that sensor to begin with.
- wolrah 3y ago
  > Don't you just make a conservative decision in that case? My eyes and ears disagree all the time, but if one of them says danger that is the one I listen too. In the context of vehicular autonomy it's more complicated than that because for example a radar sensor picking up an overhead sign or a truck parked on the shoulder as if it's an obstruction on the road is something you want to ignore when you're going 80 MPH on the highway rather than slamming on the brakes. When you're the only vehicle on the road, stopping if anything goes wrong is always the safest idea. When you're one of hundreds of vehicles in a high speed flow of traffic stopping would put you and everyone else on the road at significantly greater risk.
CamperBob2 3y ago
You remove one of the sensors and pretend the disagreement didn't happen?
- bafe 3y ago
  Or you give it an infinite uncertainty,in most sensor fusion algorithms it would result in zero weight given to that sensor in the fused observation
  - CamperBob2 3y ago
    And that leads to another good point: it's easier to predict which sensor is trustworthy if you have more than one type. Is it dark? Trust the lidar more. Raining? Trust the radar more. There is at least a rudimentary radar to sanity-check the optical sensors, right? I can almost see the objection that multiple cameras are as good as one camera+lidar, but I think it's a mistake to trust any system that can't check itself for consistency across multiple bands. It doesn't take a very good radar to keep you from ramming a fire truck. In fact, whatever runs the cruise control's distance sensor should have been enough to prevent a bunch of the Tesla oopsies reported in the press. When tackling one of the hardest engineering problems faced by humankind, it seems stupid not to take advantage of all the data you can get.
    - bafe 3y ago
      You are right. There are several reasons for why you want to fuse (different) sensors: - reduce uncertainty - estimate biases and systematic errors. As you said, different sensors are affected by environmental conditions differently, so it's only reasonable to use all the information instead of only rely on one measurement method
bafe 3y ago
The idea of sensor fusion is to have some kind of error model for all sensors (and frequently for navigation also a dynamic model of the vehicle) so that you can weigh the various sensor differently based on the uncertainty. The most primitive incarnation of this would be a simple linear Kalman Filter, but you can use similar concepts with more complex non-linear observation and dynamic models
- nihzm 3y ago
  Expanding a bit for those who don't know how a Kalman filters (or any Bayesian recursive estimator for that matter) work here's the essential idea. Each sensor is given an uncertainty model, that is for example a stochastic model by adding e.g. Gaussian noise to the "true" value it is measuring. Further, you have another model that describes the dynamics, e.g. equations from physics can tell you where the car will go if you know the current speed, position, etc.. 1. The kalman filter computes a probabilistic prediction of what it thinks is happening by using the dynamics model. That is, based on what it knows so far, where will the car (probably) be when the next measurement comes up? 2. When measurements from various sensors come in, the Kalman filter uses Bayes's theorem to compute a mean (posterior), in which each measurement is weighted by the probability that the measured value is correct (using the uncertainty models; "correct" here means "in agreement with the prediction"). In other words, sensor that are inaccurate (large variance) will be considered less in the computation of the mean, while more accurate sensors are given more importance. Once the mean of the measured quantities are computed they are used again in step 1 and the whole thing is repeated. As you can see, that disagreements are justified by the inaccuracies of the sensors, and the process of performing a probabilistic weighted average solves the problem. For the Kalman filter in particular, it can be be show (mathematically proved) that this process minimizes the variance (uncertainty) of the measured quantities (which btw. is an amazing result if you think about it).
  - bafe 3y ago
    Thanks for the excellent summary on the Kalman filter! I admit I was too lazy to write any details.
- 3y ago
progbits 3y ago
I keep hearing this argument which makes no sense. I guess it comes from Tesla marketing because they keep repeating that and it rubs off? On technical level two sensors are clearly better than one even if you just pick one in case of disagreement, but as others have said Kalman filters and other more advanced techniques exist. There is a reason airplanes or spacecraft have multiple redundant sensors like this for decades. The argument only makes sense if you want to save money, but then say you are being cheap up front.
- mrguyorama 3y ago
  It's literally not even how "Sensor fusion" works, even in the most trivial example. As long as most errors are independent per sensor, you can combine them for a more confident result. It is indeed Tesla marketing that posits otherwise, which is wrong, and Tesla fans eat it up.
- dagmx 3y ago
  Yeah the “disagreement between sensors” is largely overblown by Tesla when they were trying to rationalize cutting parts. Somehow everyone else has figured it out, and even Tesla knows how to do it and have done it for years. It’s pure bunk that’s used to cover for other decisions and now gets parroted around
  - yreg 3y ago
    I could believe that for radar (and lidar) but not for ultra sonic sensors which cost close to 0.
    - dagmx 3y ago
      I mean, this is the car company that decided to skip rain sensors for cost reasons. I own a Tesla but I also acknowledge that Elon will claw every last dollar he can to increase margins even by a few cents.
- vasilipupkin 3y ago
  the issue is, more of these sensors costs a lot of money, so can you get to the point that it's a business and not a science experiment if each vehicle costs a massive amount? if you can get away with some sensors and not others, you have lower costs. Of course, if cost is not a factor, more sensors is better.
ijustlovemath 3y ago
I think you're misunderstanding sensor fusion. Any sensor fusion worth its salt functions closer to a weighted average, but instead of linear combination, you're relying on the error model and overall system model to figure out the actual state from the variety of sensors
- Gibbon1 3y ago
  What I think is where it probably starts working well is when you can fuse information from other cars and sensors on the road. And or bicycles and even pedestrians. Imagine a stop light with a lidar sensor that broadcasts that information. Car ahead broadcasting that it's stopped in the fast lane.
  - AlotOfReading 3y ago
    Think about it from a security and distributed systems perspective. How do you establish that the sensor data is good (sensor isn't failing), the data is trustworthy (not malicious), and ensure you have it when you need it? V2V and V2I take already hard problems like self-driving and add a bunch of other hard problems on top. They're hardly worth discussing as serious solutions today.
oxfordmale 3y ago
Humans drive both with their eyes and ears, although Tesla marketing like to state otherwise. We are perfectly capable combining both inputs and act accordingly. An AI system can easily learn how to combining visual and LIDAR input to make the right decision given the circumstances.In the end it is just a decision tree.
- ggreer 3y ago
  Is there any situation in which drivers disregard what they see and rely on sound instead? Sound informs what we look at, but we rely entirely on sight to drive.
  - LeifCarrotson 3y ago
    If an inner tire on the dually truck or trailer I drive at work blows out, I may be completely unable to see anything if there's no smoke or tire debris in the mirrors. But I absolutely know what a blown tire sounds like and feels like, and will absolutely pull over and come to a stop for repairs. Same as if I'm driving in winter and hit a patch of ice - Visually, it looks identical to the rest of the snowy, icy roads here, but I will drive if I feel and here that I'm spinning my wheels or sliding sideways. If I smell coolant, oil, or a belt, even if my eyes tell me my gauges disagree I'll be pulling over for those issues as well. Conversely, if I've replaced a tire and the TPMS light says the removed tire in the cargo area has low pressure (duh, that's why I changed it) but I know the spare (without a TPMS transmitter installed) is good, or otherwise know that the idiot light is a false positive, I'll trust my other senses over my eyes.
  - oxfordmale 3y ago
    * A car accelerating fast from your left * Screeching breaks from the right * Sounds of a rattling bicycle trying to undertake you * An approaching emergency vehicle * Other cars beeping their horn at you Of course you will confirm such situations visually, but you definitely using hearing in addition to sight.
  - bagels 3y ago
    Sirens from not-yet-seen emergency vehicles is the most basic example.
throwawaylinux 3y ago
> The problem with sensor fusion is what do you do when the sensors disagree? Like one sensor says you have a bus stopped in front of you and the other says it's all clear? And your choices are full steam ahead or prepare to not ram the apparent bus?
i_am_jl 3y ago
>The problem with sensor fusion is what do you do when the sensors disagree? You have to decide which sensor to trust. There are two cameras in this computer vision system. If one of them goes offline, is obscured, becomes dirty, or malfunctions you lose stereo vision and depth perception. So you'd obviously need 3+ cameras. And now we're right back at sensor fusion challenges again.
callalex 3y ago
I mean this in the kindest way possible: you need to mix up your sources of information, as you have become blinded by lies from marketers and a cult of personality. Machines have been using redundant sensors since machines and sensors have existed, with great success.
kaba0 3y ago
Similarly to how ML vision also a black box, there is no reason why its error handling can’t also be that. If you hear a sound but see nothing, you do become much more aware of the generic area afterwards, something like that should also be possible.

by ggreer 3y ago

nomel 3y ago

LiDAR easily passes the “featureless white wall” test. That’s why I’ll be waiting.

yomlica8 3y ago

Don't you just make a conservative decision in that case? My eyes and ears disagree all the time, but if one of them says danger that is the one I listen too.

parineum 3y ago
I'm sure this example isn't new at all but it's the first time I've encountered the argument phrased this way and I find it incredibly compelling. Looks good, smells bad is another one (that probably happens more often). It did get me thinking about different senses and how I prioritize them. Given that humans strongest sense is visual, it's interesting to me that the priorities of what to trust seem opposite of my expectations. It seems to me that visual signals are seemingly least "trusted" compared to the other senses. As if the logic goes, "my sense of smell is so bad that if it detects danger, it must be very dangerous". Similarly, I'm sure dogs can smell rotting flesh long before meat is unsafe.
- afterburner 3y ago
  Even simpler: if one sensor tells you confidently there's something in your way, and the other doesn't, assume there's something in your way.
  - Sodman 3y ago
    The problem in this particular case is that the sensor may be confidently telling you there's something in the way, but in reality it's just a plastic bag. Hard braking at 70mph on a highway as if you were about to hit a concrete wall is probably not a great outcome in this scenario if there's somebody behind you.
    - jjk166 3y ago
      If your sensor is confidently telling you that a plastic bag is a concrete wall, maybe you shouldn't be using that sensor to begin with.
wolrah 3y ago
> Don't you just make a conservative decision in that case? My eyes and ears disagree all the time, but if one of them says danger that is the one I listen too. In the context of vehicular autonomy it's more complicated than that because for example a radar sensor picking up an overhead sign or a truck parked on the shoulder as if it's an obstruction on the road is something you want to ignore when you're going 80 MPH on the highway rather than slamming on the brakes. When you're the only vehicle on the road, stopping if anything goes wrong is always the safest idea. When you're one of hundreds of vehicles in a high speed flow of traffic stopping would put you and everyone else on the road at significantly greater risk.

CamperBob2 3y ago

You remove one of the sensors and pretend the disagreement didn't happen?

bafe 3y ago
Or you give it an infinite uncertainty,in most sensor fusion algorithms it would result in zero weight given to that sensor in the fused observation
- CamperBob2 3y ago
  And that leads to another good point: it's easier to predict which sensor is trustworthy if you have more than one type. Is it dark? Trust the lidar more. Raining? Trust the radar more. There is at least a rudimentary radar to sanity-check the optical sensors, right? I can almost see the objection that multiple cameras are as good as one camera+lidar, but I think it's a mistake to trust any system that can't check itself for consistency across multiple bands. It doesn't take a very good radar to keep you from ramming a fire truck. In fact, whatever runs the cruise control's distance sensor should have been enough to prevent a bunch of the Tesla oopsies reported in the press. When tackling one of the hardest engineering problems faced by humankind, it seems stupid not to take advantage of all the data you can get.
  - bafe 3y ago
    You are right. There are several reasons for why you want to fuse (different) sensors: - reduce uncertainty - estimate biases and systematic errors. As you said, different sensors are affected by environmental conditions differently, so it's only reasonable to use all the information instead of only rely on one measurement method

bafe 3y ago

The idea of sensor fusion is to have some kind of error model for all sensors (and frequently for navigation also a dynamic model of the vehicle) so that you can weigh the various sensor differently based on the uncertainty. The most primitive incarnation of this would be a simple linear Kalman Filter, but you can use similar concepts with more complex non-linear observation and dynamic models

nihzm 3y ago
Expanding a bit for those who don't know how a Kalman filters (or any Bayesian recursive estimator for that matter) work here's the essential idea. Each sensor is given an uncertainty model, that is for example a stochastic model by adding e.g. Gaussian noise to the "true" value it is measuring. Further, you have another model that describes the dynamics, e.g. equations from physics can tell you where the car will go if you know the current speed, position, etc.. 1. The kalman filter computes a probabilistic prediction of what it thinks is happening by using the dynamics model. That is, based on what it knows so far, where will the car (probably) be when the next measurement comes up? 2. When measurements from various sensors come in, the Kalman filter uses Bayes's theorem to compute a mean (posterior), in which each measurement is weighted by the probability that the measured value is correct (using the uncertainty models; "correct" here means "in agreement with the prediction"). In other words, sensor that are inaccurate (large variance) will be considered less in the computation of the mean, while more accurate sensors are given more importance. Once the mean of the measured quantities are computed they are used again in step 1 and the whole thing is repeated. As you can see, that disagreements are justified by the inaccuracies of the sensors, and the process of performing a probabilistic weighted average solves the problem. For the Kalman filter in particular, it can be be show (mathematically proved) that this process minimizes the variance (uncertainty) of the measured quantities (which btw. is an amazing result if you think about it).
- bafe 3y ago
  Thanks for the excellent summary on the Kalman filter! I admit I was too lazy to write any details.
3y ago

progbits 3y ago

I keep hearing this argument which makes no sense. I guess it comes from Tesla marketing because they keep repeating that and it rubs off? On technical level two sensors are clearly better than one even if you just pick one in case of disagreement, but as others have said Kalman filters and other more advanced techniques exist. There is a reason airplanes or spacecraft have multiple redundant sensors like this for decades. The argument only makes sense if you want to save money, but then say you are being cheap up front.

mrguyorama 3y ago
It's literally not even how "Sensor fusion" works, even in the most trivial example. As long as most errors are independent per sensor, you can combine them for a more confident result. It is indeed Tesla marketing that posits otherwise, which is wrong, and Tesla fans eat it up.
dagmx 3y ago
Yeah the “disagreement between sensors” is largely overblown by Tesla when they were trying to rationalize cutting parts. Somehow everyone else has figured it out, and even Tesla knows how to do it and have done it for years. It’s pure bunk that’s used to cover for other decisions and now gets parroted around
- yreg 3y ago
  I could believe that for radar (and lidar) but not for ultra sonic sensors which cost close to 0.
  - dagmx 3y ago
    I mean, this is the car company that decided to skip rain sensors for cost reasons. I own a Tesla but I also acknowledge that Elon will claw every last dollar he can to increase margins even by a few cents.
vasilipupkin 3y ago
the issue is, more of these sensors costs a lot of money, so can you get to the point that it's a business and not a science experiment if each vehicle costs a massive amount? if you can get away with some sensors and not others, you have lower costs. Of course, if cost is not a factor, more sensors is better.

ijustlovemath 3y ago

I think you're misunderstanding sensor fusion. Any sensor fusion worth its salt functions closer to a weighted average, but instead of linear combination, you're relying on the error model and overall system model to figure out the actual state from the variety of sensors

Gibbon1 3y ago
What I think is where it probably starts working well is when you can fuse information from other cars and sensors on the road. And or bicycles and even pedestrians. Imagine a stop light with a lidar sensor that broadcasts that information. Car ahead broadcasting that it's stopped in the fast lane.
- AlotOfReading 3y ago
  Think about it from a security and distributed systems perspective. How do you establish that the sensor data is good (sensor isn't failing), the data is trustworthy (not malicious), and ensure you have it when you need it? V2V and V2I take already hard problems like self-driving and add a bunch of other hard problems on top. They're hardly worth discussing as serious solutions today.

oxfordmale 3y ago

Humans drive both with their eyes and ears, although Tesla marketing like to state otherwise. We are perfectly capable combining both inputs and act accordingly. An AI system can easily learn how to combining visual and LIDAR input to make the right decision given the circumstances.In the end it is just a decision tree.

ggreer 3y ago
Is there any situation in which drivers disregard what they see and rely on sound instead? Sound informs what we look at, but we rely entirely on sight to drive.
- LeifCarrotson 3y ago
  If an inner tire on the dually truck or trailer I drive at work blows out, I may be completely unable to see anything if there's no smoke or tire debris in the mirrors. But I absolutely know what a blown tire sounds like and feels like, and will absolutely pull over and come to a stop for repairs. Same as if I'm driving in winter and hit a patch of ice - Visually, it looks identical to the rest of the snowy, icy roads here, but I will drive if I feel and here that I'm spinning my wheels or sliding sideways. If I smell coolant, oil, or a belt, even if my eyes tell me my gauges disagree I'll be pulling over for those issues as well. Conversely, if I've replaced a tire and the TPMS light says the removed tire in the cargo area has low pressure (duh, that's why I changed it) but I know the spare (without a TPMS transmitter installed) is good, or otherwise know that the idiot light is a false positive, I'll trust my other senses over my eyes.
- oxfordmale 3y ago
  * A car accelerating fast from your left * Screeching breaks from the right * Sounds of a rattling bicycle trying to undertake you * An approaching emergency vehicle * Other cars beeping their horn at you Of course you will confirm such situations visually, but you definitely using hearing in addition to sight.
- bagels 3y ago
  Sirens from not-yet-seen emergency vehicles is the most basic example.

throwawaylinux 3y ago

> The problem with sensor fusion is what do you do when the sensors disagree? Like one sensor says you have a bus stopped in front of you and the other says it's all clear? And your choices are full steam ahead or prepare to not ram the apparent bus?

i_am_jl 3y ago

>The problem with sensor fusion is what do you do when the sensors disagree? You have to decide which sensor to trust. There are two cameras in this computer vision system. If one of them goes offline, is obscured, becomes dirty, or malfunctions you lose stereo vision and depth perception. So you'd obviously need 3+ cameras. And now we're right back at sensor fusion challenges again.

callalex 3y ago

I mean this in the kindest way possible: you need to mix up your sources of information, as you have become blinded by lies from marketers and a cult of personality. Machines have been using redundant sensors since machines and sensors have existed, with great success.

kaba0 3y ago

Similarly to how ML vision also a black box, there is no reason why its error handling can’t also be that. If you hear a sound but see nothing, you do become much more aware of the generic area afterwards, something like that should also be possible.