3 ms·

That is the general idea, but now you've just descended to the next level of the iceberg. While it's true that sensor fusion is a Good Idea in lots of contexts,

That is the general idea, but now you've just descended to the next level of the iceberg. While it's true that sensor fusion is a Good Idea in lots of contexts, the whole paradigm rests on a fundamental assumption that you can model the problem domain. Perception pushes up against the boundaries of what Bayesian filtering can handle because even the most complex error models are hopelessly simplistic compared to the system generating your measurements. Your model would have to capture the bounds of uncertainty for what your stereo pair might say about the depth of any point in any scene it will ever see. A simple example of why this gets complicated: if I have a point in one camera and a point in another camera and I know they correspond to the same real-world spatial point, I can calculate some distance, and the statistics of that calculation can be captured by a halfway-reasonable error model. But how did I know they corresponded to the same point in the first place? Well, because they look the same according to some image feature... or because some deep neural network told me so.. etc. There just aren't very good ways to model just how haywire ^that^ process can go. So at the end of the day, once you let this evil into your perception system, using statistics to blend your sensors together is undermined, and all of your precious covariances just turn into tuning knobs you can twiddle. The dirty secret is that almost all robotic perception systems are hiding unprincipled, un-modelled heuristics in the data association process. This is kicked under the rug because it doesn't really fit into traditional estimation theoretic frameworks. In a lot of papers you'll see academics push it aside by just calling that the "front-end", which they brush aside as a little widget you put on the front. If you're lucky they'll do ablations across a couple different options. Of course, this is just one level deeper down the iceberg. It goes far deeper. Even if you could model the statistics of a depth camera well, the statistics of "what are all the objects in your scene about to do" is another couple of orders of magnitude more un-modellable. Often engineers will do something like attach a "constant-velocity" model to the agents in a scene. Imagine trying to bin all of the reasons you might stop walking in a straight line into a bubble that describes how "noisy" that picture of the world is! Now you can begin to appreciate just how hopeless it is to explicitly model uncertainty in the world around us.

joshuamorton 3y ago
You're basically describing the no-free lunch theorem, but in practice we can define fairly robust generic models for certain things, like "physics". That's why optical mice work pretty well. You can start to add unjustified assumptions and that'll make the world model weaker, yes. But starting with pretty basic assumptions like "you can segment an object from an series of images because each solid object will move on its own trajectory", or even more basic like "objects have edges" and then a few dozen samples per second, and suddenly you have a fairly robust way to detect things. Same for predicting where something will go. If you can estimate an objects current velocity, acceleration, and jerk with reasonable precision, you don't really need a highly predictive heuristic for the world model. For decision making you need more robust heuristics, like "the car to my left has right of way at the stop sign", but you don't need that level of heuristic to identify that there is a car and that it isn't part of the pavement and that it is currently sitting still.
nihzm 3y ago
Indeed, concepts that get hyped such as truly autonomous vehicles are still herculean tasks, and I am convinced that we are not going to find any theorethical way do demonstrate the safety of those technologies anytime soon (especially machine learning / deep learning). However, you should also consider that working with heuristics is pretty much all engineers do. We ought to solve problems, even if the theory is not there yet. A great example is how we got air travel long before we had any real understanding of the fluid dynamics happening around the fuselage (as it was generally computationally intractable). So, sometimes a simple epipolar camera model with noisy clouds around the subjects is sufficiently accurate for the task. The real problem IMHO is that the degree to which these rudimentary approximations are tested for safety is not nearly enough with respect to how critical they are in the whole system. A while ago I stumbled upon this presentation on system safety [1] which had an interesting perspective coming from the aerospace industry. In aerospace they test the shit out of every component to make sure that a failure does not cause an airplane to crash. In comparison waymo, uber and everyone else has done almost nothing in terms of testing for safety before putting out their products. [1]: Richard Murray: "Can We Really Use Machine Learning in Safety Critical Systems?" https://youtu.be/Wi8Y---ce28?si=HsqgiLngdHojpYO9 https://youtu.be/Wi8Y---ce28?si=HsqgiLngdHojpYO9

joshuamorton 3y ago

You're basically describing the no-free lunch theorem, but in practice we can define fairly robust generic models for certain things, like "physics". That's why optical mice work pretty well. You can start to add unjustified assumptions and that'll make the world model weaker, yes. But starting with pretty basic assumptions like "you can segment an object from an series of images because each solid object will move on its own trajectory", or even more basic like "objects have edges" and then a few dozen samples per second, and suddenly you have a fairly robust way to detect things. Same for predicting where something will go. If you can estimate an objects current velocity, acceleration, and jerk with reasonable precision, you don't really need a highly predictive heuristic for the world model. For decision making you need more robust heuristics, like "the car to my left has right of way at the stop sign", but you don't need that level of heuristic to identify that there is a car and that it isn't part of the pavement and that it is currently sitting still.

nihzm 3y ago

Indeed, concepts that get hyped such as truly autonomous vehicles are still herculean tasks, and I am convinced that we are not going to find any theorethical way do demonstrate the safety of those technologies anytime soon (especially machine learning / deep learning). However, you should also consider that working with heuristics is pretty much all engineers do. We ought to solve problems, even if the theory is not there yet. A great example is how we got air travel long before we had any real understanding of the fluid dynamics happening around the fuselage (as it was generally computationally intractable). So, sometimes a simple epipolar camera model with noisy clouds around the subjects is sufficiently accurate for the task. The real problem IMHO is that the degree to which these rudimentary approximations are tested for safety is not nearly enough with respect to how critical they are in the whole system. A while ago I stumbled upon this presentation on system safety [1] which had an interesting perspective coming from the aerospace industry. In aerospace they test the shit out of every component to make sure that a failure does not cause an airplane to crash. In comparison waymo, uber and everyone else has done almost nothing in terms of testing for safety before putting out their products. [1]: Richard Murray: "Can We Really Use Machine Learning in Safety Critical Systems?" https://youtu.be/Wi8Y---ce28?si=HsqgiLngdHojpYO9 https://youtu.be/Wi8Y---ce28?si=HsqgiLngdHojpYO9