3 ms·

I still suspect it's because they need to preserve compute resources for vision processing. Sensor fusion is likely eating up too much of their current HW and

I still suspect it's because they need to preserve compute resources for vision processing. Sensor fusion is likely eating up too much of their current HW and limiting their progress in other areas. I suspect Tesla will have to admit they need to upgrade the current HW before they ever 'solve' FSD.

snovv_crash 4y ago
The amount of compute that sensor fusion uses is miniscule compared to running a NN or computing stereo depth maps. Sensor fusion runs in the background of your phone the whole time to power things like [0] for example. 0. https://sensor-js.xyz/demo.html https://sensor-js.xyz/demo.html
- Nomentatus 4y ago
  Well, yes and no. Integrating the data and adjudicating conflicts between sensors is a real task, too. Also having just two opinions doesn't necessarily help if they conflict, and the lidar is the thinnest source of data. How do you coin flip that? You likely end up just discarding the Lidar's conflicting opinion.
  - snovv_crash 4y ago
    LIDAR data might be more sparse but it is also more reliable, being an active sensor so it isn't affected by night. The real way is to use the LIDAR to add to the depth probability distribution in the stereo depth estimation. This way you aren't throwing any data away. LIDAR often gives probabilities as well, for example, and this can be used to eliminate reflections.
- 01100011 4y ago
  You are confused my friend. Reading sensor data is not the same as feeding that data to a neural network and asking it to form a worldview composed of possibly conflicting sensor data streams(i.e. lidar vs vision vs ultrasonic). You are somewhat correct that it is quite trivial to read sensor data. For many sensors, there is some work which needs to be done to denoise or cleanup the input data. That's not where the story ends, however.
  - snovv_crash 4y ago
    In order to display the gravity-aligned acceleration, sensor fusion between gyro and accelerometer has to occur. This is typically done with a Kalman filter, and runs on 1960s levels of hardware. If you look at something like a drone autopilot, eg. Ardupilot, the sensor fusion is soo cheap that they even extended the Kalman filter to also estimate things like the sensor bias offsets and earth's magnetic field vectors. Sensor fusion is computationally cheap. It's just a lot of R&D to do it in a way that leads to a net gain in precision and robustness.
    - 01100011 4y ago
      We're talking about radar and ultrasonic sensors here, not accelerometers. We're also talking about feeding them to a deep neural network. Not the same thing. Sensor fusion is not being done with a Kalman filter in this case.
      - snovv_crash 4y ago
        Radar and ultrasound both give drastically less data than a simple 720p webcam. After postprocessing their output bandwidth is more similar to a 9-axis IMU than a camera.
        01100011 4y ago
        Yes and each time you process the incoming camera data in the neural network you have extra calculations due to the sensor fusion with another data source, regardless of its bitrate. Have you worked with deep learning much?
another_devy 4y ago
I think not using LiDAR would be a good bet. LiDAR in nutshell allows you to give an object in vision 3D space, relative speed which 2 human eyes can do very fast. Problems with solving this in vision based input is in dataset and interpretation. Computer vision and AI can’t effectively apply a human drivers judgment with better camera and processing power, at least not yet.
vhold 4y ago
One camera produces millions of bytes of data every single frame, an ultrasonic sensor is useful producing just 1 or 2 bytes of data in the same time span. (distance to something within the sensor's cone). So it seems like a totally ridiculous argument that ultrasonic sensors create some kind of data processing overload. An ultrasonic sensor makes it possible to implement incredibly simple and reliable safety features with well known performance characteristics. Processing an image with ML to produce the same effect has tons of edge cases where it might not work, and nobody knows when it won't work, and every update to the system could introduce regressions. It's why they had to disable certain features when they got rid of the ultrasonic sensors. Those features may come back some day, but I bet they'll never be as reliable, and certainly won't be as predictable. https://www.pcmag.com/news/tesla-removes-ultrasonic-sensors-from-its-vehicles-disables-some-features https://www.pcmag.com/news/tesla-removes-ultrasonic-sensors-...
- 01100011 4y ago
  Not saying they create a data processing overload. I'm saying they're fed to a deep learning architecture that must then try to fuse disparate sets of data into a coherent picture. The neural network becomes simpler when you remove that function and just focus on visual processing.

snovv_crash 4y ago

The amount of compute that sensor fusion uses is miniscule compared to running a NN or computing stereo depth maps. Sensor fusion runs in the background of your phone the whole time to power things like [0] for example. 0. https://sensor-js.xyz/demo.html https://sensor-js.xyz/demo.html

Nomentatus 4y ago
Well, yes and no. Integrating the data and adjudicating conflicts between sensors is a real task, too. Also having just two opinions doesn't necessarily help if they conflict, and the lidar is the thinnest source of data. How do you coin flip that? You likely end up just discarding the Lidar's conflicting opinion.
- snovv_crash 4y ago
  LIDAR data might be more sparse but it is also more reliable, being an active sensor so it isn't affected by night. The real way is to use the LIDAR to add to the depth probability distribution in the stereo depth estimation. This way you aren't throwing any data away. LIDAR often gives probabilities as well, for example, and this can be used to eliminate reflections.
01100011 4y ago
You are confused my friend. Reading sensor data is not the same as feeding that data to a neural network and asking it to form a worldview composed of possibly conflicting sensor data streams(i.e. lidar vs vision vs ultrasonic). You are somewhat correct that it is quite trivial to read sensor data. For many sensors, there is some work which needs to be done to denoise or cleanup the input data. That's not where the story ends, however.
- snovv_crash 4y ago
  In order to display the gravity-aligned acceleration, sensor fusion between gyro and accelerometer has to occur. This is typically done with a Kalman filter, and runs on 1960s levels of hardware. If you look at something like a drone autopilot, eg. Ardupilot, the sensor fusion is soo cheap that they even extended the Kalman filter to also estimate things like the sensor bias offsets and earth's magnetic field vectors. Sensor fusion is computationally cheap. It's just a lot of R&D to do it in a way that leads to a net gain in precision and robustness.
  - 01100011 4y ago
    We're talking about radar and ultrasonic sensors here, not accelerometers. We're also talking about feeding them to a deep neural network. Not the same thing. Sensor fusion is not being done with a Kalman filter in this case.
    - snovv_crash 4y ago
      Radar and ultrasound both give drastically less data than a simple 720p webcam. After postprocessing their output bandwidth is more similar to a 9-axis IMU than a camera.
      - 01100011 4y ago
        Yes and each time you process the incoming camera data in the neural network you have extra calculations due to the sensor fusion with another data source, regardless of its bitrate. Have you worked with deep learning much?

another_devy 4y ago

I think not using LiDAR would be a good bet. LiDAR in nutshell allows you to give an object in vision 3D space, relative speed which 2 human eyes can do very fast. Problems with solving this in vision based input is in dataset and interpretation. Computer vision and AI can’t effectively apply a human drivers judgment with better camera and processing power, at least not yet.

vhold 4y ago

One camera produces millions of bytes of data every single frame, an ultrasonic sensor is useful producing just 1 or 2 bytes of data in the same time span. (distance to something within the sensor's cone). So it seems like a totally ridiculous argument that ultrasonic sensors create some kind of data processing overload. An ultrasonic sensor makes it possible to implement incredibly simple and reliable safety features with well known performance characteristics. Processing an image with ML to produce the same effect has tons of edge cases where it might not work, and nobody knows when it won't work, and every update to the system could introduce regressions. It's why they had to disable certain features when they got rid of the ultrasonic sensors. Those features may come back some day, but I bet they'll never be as reliable, and certainly won't be as predictable. https://www.pcmag.com/news/tesla-removes-ultrasonic-sensors-from-its-vehicles-disables-some-features https://www.pcmag.com/news/tesla-removes-ultrasonic-sensors-...

01100011 4y ago
Not saying they create a data processing overload. I'm saying they're fed to a deep learning architecture that must then try to fuse disparate sets of data into a coherent picture. The neural network becomes simpler when you remove that function and just focus on visual processing.