9 ms·

I would like to add over here that Tesla does indeed do sensor fusion in their cars. Their Autopilot combines radar and ultrasound with vision to decide where t

I would like to add over here that Tesla does indeed do sensor fusion in their cars. Their Autopilot combines radar and ultrasound with vision to decide where to drive. Commentators bringing up LIDAR are jumping the gun by assuming that this scenario isn't something that these sensors wouldn't have detected in either combination or individually to spot the anomaly. The problem over here is likely to be software due to a bug in their code than a simple lack of additional sensors (LIDAR). And this touches on deeper issues that could have profound ramifications for the autonomous driving industry and the broader industry in general. At least, in my eyes, the big problem with the autonomous car industry isn't the sensor suites they are deploying (or not), but the over-reliance on neural networks. They are black boxes with failure modes that can't be adequately mapped. See: http://www.evolvingai.org/fooling http://www.evolvingai.org/fooling What if the neural net or the system used to detect obstacles didn't see it because the precise configuration of the data fooled it? And if that's the case then what's next? How do we decide when it is okay for safety-critical systems to be opaque? How do we deal with autonomous driving if the conclusion comes out to be a "no" for this case? How should broader society deal with a yes? And who decides all of this in the first place? Possibilities like this scare me far more than the lack of LIDAR because replicating a bug like this would be next to impossible. We don't know what we don't know, and we can't explore and understand the system to suss out what we don't know. Edit: Fleshed out the idea with more questions.

antpls 8y ago
> Possibilities like this scare me far more than the lack of LIDAR because replicating a bug like this would be next to impossible. We don't know what we don't know, and we can't explore and understand the system to suss out what we don't know. I don't think this is true. According to the Tesla live stream a few days ago, the cameras are always gathering data. Tesla cars could have a black box (like planes) , and they could just replay everything that happened up to M minutes before the accident, and retrain neural networks to avoid it later. According to the live, they automatised this process with the "Data Engine". But anyway, this is definitely not an excuse to beta test the system with actual customer's lives, IMHO.
ummonk 8y ago
The sensor technology is a big factor here. You can use LIDAR for perception without needing a neural net. On the other hand with radar + ultrasound + vision you'll need a neural net to try to detect things like obstacles, since the raw data from those sensors doesn't directly tell you whether there is an obstacle in the way.
- ebg13 8y ago
  Everything you said is very wrong. LIDAR produces line of sight depth information. Stereo cameras produce line of sight depth information. Radar produces line of sight depth information. Ultrasound produces line of sight depth information. The only differences are how, how fast, and how detailed. The biggest difference between LIDAR and stereo vision is that LIDAR doesn't depend on illumination because it uses its own.
  - gamblor956 8y ago
    Stereo cameras produce line of sight depth information That is wrong. Stereo cameras provide data which, after significant processing, can result in line-of-sight depth information. With radar (including LIDAR) and ultrasound, the depth information is essentially "free" since very little processing is required to extract the depth information (i.e., the information returned is the depth). So the biggest difference between stereovision and everything else is that stereovision requires additional processing to extract the depth information.
    - colordrops 8y ago
      Aren't the algorithms for extracting depth from stereo imagery already solved and thus also essentially "free"?
      - codelord 8y ago
        Depth estimation is not a solved problem. Why do you think SDC companies pay so much to buy Lidar if depth estimation was solved?
linsomniac 8y ago
One thing to remember here is that with RADAR it is, I have read, very hard to tell a soda can from a car from a above highway sign as you are cresting a hill. So they will often filter out anything that is stopped. I read this a few months ago and it blew my mind. Then I read about that LIDAR sensor burning a camera sensor and thought "how safe can that be?!?"
- unityByFreedom 8y ago
  > with RADAR it is, I have read, very hard to tell a soda can from a car from a above highway sign Yes, read about phantom braking, for example here, https://www.reddit.com/r/teslamotors/comments/b5yx1o/welp_this_happened_today_while_in_autopilot/ https://www.reddit.com/r/teslamotors/comments/b5yx1o/welp_th...
  - linsomniac 8y ago
    Don't need to read about it. Last year I was on a 2400 mile trip. On a long, deserted stretch of road, suddenly, out of nowhere my Tesla slammed on the brakes. I had to throw away my underwear. :-)
- imtringued 8y ago
  That LIDAR sensor you talk about destroyed camera sensors precisely because it is safe for human eyes. Conventional lidar uses wavelenghts that can harm human eyes but the power output is restricted to a much lower value to prevent damage to humans and cameras.
emef 8y ago
This is why we use a redundant geometric approach (with LiDAR) as a fallback to NNs in our perception stack. NNs can have false negatives which is unacceptable, but dealing with false positives is more manageable and safer.
- pbreit 8y ago
  Does lidar have anything to do with the redundancy?
  - always_a_novice 8y ago
    Since a lidar uses a discrete frequency band and it's own energy source, it's much easier to process the signal for depth information (using the lag between sending and receiving a return pulse(s) and the wavelength). Arguably such a system might provide a better basis for a backup system based on modelling obstacles and their distances from the vehicle.
salty_biscuits 8y ago
"What if the neural net or the system used to detect obstacles didn't see it because the precise configuration of the data fooled it?" I think the answer is that all state must be represented as a probability over states and the controller must be set up to be a robust controller under this ambiguity of state. Then the problem becomes one of making sure that probability is well calibrated so that you get the appropriate control response. The whole software stack for autonomous driving is also a black box, that is just as big an issue. If some parts are sophisticated but are being fed into others that are naive we have no idea. Every sensor has issues, I wouldn't be so quick to worry about the neural networks in particular.
haditab 8y ago
Your comment actually emphasizes why having a lidar is better. Cameras need neural networks to detect objects. Lidars on the other hand provide distance to all the surrounding objects. The neural networks processing the point cloud may not be able to detect what the object is, but because of the distance available in the point cloud data you will know there is something ahead. Fusing radar with camera does help but radars are noisy and unreliable.
- CardenB 8y ago
  Lidar is also noisy unfortunately. The input is high dimensional enough that classical tracking methods don’t work well (like for simpler radar systems) because segmentation in lidar is hard.
- bobsil1 8y ago
  Lidar is active (so lower res than vision) and uses slightly longer wavelengths (again, lower res). Visible light uses passive solar illumination by day, power-efficient, far more data; and is a problem you need to solve for reliable autonomy independent of lidar.
  - studentrob 8y ago
    > is a problem you need to solve for reliable autonomy independent of lidar Not for L4. A level 4 vehicle is not expected to operate under all conditions, including weather and location.
- sjwright 8y ago
  Radar seems to work pretty damn well for the adaptive cruise control and autonomous emergency braking installed in hundreds of models of cars made over the past 5+ years. If autonomous systems needs more redundancy, the solution is to keep adding more cameras. The other sensor which I think is important is data captured by cars in that fleet. This is sort of what's known as "HD maps" but fully automated and based purely sensor-derived data. So if you're driving down a highway at night in the rain, your car would know the successful paths traversed a few hours ago when it was dry and sunny.
Gustomaximus 8y ago
I think the 'how do we decide' is when we can show autonomous driving is significant safer than humans. I suspect the problem will then be computers avoid many accidents humans make. But make mistakes that seem obvious to humans. So even though the overall risk is lower it becomes very hard to accept the computer did something wrong vs admitting a person was speeding and they stuffed up killing themself. To allow this tech to exist we might need legislation that autonomous car companies can't be sued unless there is clear malpractice in their systems vs some level of 'accidents happen'. In the same way a wife doesnt typically don't sue her husband that caused an accident that killed themself.
hvidgaard 8y ago
While we may get to the point where sensors and a form of AI can reliably drive a car, we're so far from it that we don't even know if we're heading towards the goal post. What we need, if we want to solve the partial self driving problem right now, is not fully autonomous systems. We could start by agreeing on a way for cars to sense the road - think in the way robotic lawn mowers work where you have to bury cables. Then implement that in highways, ring roads and other long stretches of road. Then we can agree on a way for cars to communicate and exchange relevant information. Then we can equip cars with hardware like the Tesla have, and then use it to collect data and compare the "auto pilot" with the known working system of following a signal in the road. We still need LIDAR to reliably stop for other cars, but that is a mostly solved problem today.
- hnlmorg 8y ago
  The problem we have right now is that: 1. laying cables wouldn't work because there are a great many historic towns and cities (eg Amsterdam) and digging up cobble streets to lay cables for self-driving vehicles simple isn't going happen when those cities have so many bigger problems for autonomous cars such as trams, cycle paths, pedestrian zones and tourists what will just walk out in front of everyone in an unpredictable haze. 2. you have districts that are so under funded that the roads aren't even maintained in their current state. 3. So lets say you scape redeveloping the roads and have radio sensors on lap posts - then you have the problem that many rural areas in even richer countries like the UK don't even have street lamps. And what about countries like Australia and America that can have long stretches for tens or even hundreds of miles without a single house, let alone street lights to put sensors on. I agree with you that normally this kind of problem requires a technical approach but the problem we have is that redeveloping our roads would be such a monumental overhaul that it would actually be a bigger job than developing the AI to handle our existing infrastructure. Plus redeveloping our roads only solves part of the problem - you still have the issue of other cars on the road, pedestrians, cyclists, unexpected hazards like fallen trees, etc. You could fix other vehicles on the road by allowing all vehicles to communicated with each other - like a mesh network - but then you have the problem of doing that securely (so you couldn't just "virtually ram" another car off the road by telling it there's an obstruction that doesn't exist) and you again have the problem that it would be more expensive and take longer to retrofit every single vehicle with that tech than it would be to write AI. And you still wouldn't have addressed the problem of kids, animals and fallen trees. AI might seem like a naff workaround and maybe if / when we colonise a new planet then we might be able to build roads with sensors inside tubes that restrict access (I guess a little like an underground rail network) but for the time being AI is our best bet for autonomous vehicles.
solveit 8y ago
> What if the neural net or the system used to detect obstacles didn't see it because the precise configuration of the data fooled it? And if that's the case then what's next? How do we decide when it is okay for safety-critical systems to be opaque? How do we deal with autonomous driving if the conclusion comes out to be a "no" for this case? How should broader society deal with a yes? And who decides all of this in the first place? All good questions, but I will note that the exact same questions apply equally well or better to human drivers, yet people don't seem to be as concerned.
- kurtisc 8y ago
  People are very concerned about unsafe human driving and the liability for the massive number of deaths and damages it causes. It's the reason we have insurance and a big reason why we have self-driving aspirations at all.

antpls 8y ago

> Possibilities like this scare me far more than the lack of LIDAR because replicating a bug like this would be next to impossible. We don't know what we don't know, and we can't explore and understand the system to suss out what we don't know. I don't think this is true. According to the Tesla live stream a few days ago, the cameras are always gathering data. Tesla cars could have a black box (like planes) , and they could just replay everything that happened up to M minutes before the accident, and retrain neural networks to avoid it later. According to the live, they automatised this process with the "Data Engine". But anyway, this is definitely not an excuse to beta test the system with actual customer's lives, IMHO.

ummonk 8y ago

The sensor technology is a big factor here. You can use LIDAR for perception without needing a neural net. On the other hand with radar + ultrasound + vision you'll need a neural net to try to detect things like obstacles, since the raw data from those sensors doesn't directly tell you whether there is an obstacle in the way.

ebg13 8y ago
Everything you said is very wrong. LIDAR produces line of sight depth information. Stereo cameras produce line of sight depth information. Radar produces line of sight depth information. Ultrasound produces line of sight depth information. The only differences are how, how fast, and how detailed. The biggest difference between LIDAR and stereo vision is that LIDAR doesn't depend on illumination because it uses its own.
- gamblor956 8y ago
  Stereo cameras produce line of sight depth information That is wrong. Stereo cameras provide data which, after significant processing, can result in line-of-sight depth information. With radar (including LIDAR) and ultrasound, the depth information is essentially "free" since very little processing is required to extract the depth information (i.e., the information returned is the depth). So the biggest difference between stereovision and everything else is that stereovision requires additional processing to extract the depth information.
  - colordrops 8y ago
    Aren't the algorithms for extracting depth from stereo imagery already solved and thus also essentially "free"?
    - codelord 8y ago
      Depth estimation is not a solved problem. Why do you think SDC companies pay so much to buy Lidar if depth estimation was solved?

linsomniac 8y ago

One thing to remember here is that with RADAR it is, I have read, very hard to tell a soda can from a car from a above highway sign as you are cresting a hill. So they will often filter out anything that is stopped. I read this a few months ago and it blew my mind. Then I read about that LIDAR sensor burning a camera sensor and thought "how safe can that be?!?"

unityByFreedom 8y ago
> with RADAR it is, I have read, very hard to tell a soda can from a car from a above highway sign Yes, read about phantom braking, for example here, https://www.reddit.com/r/teslamotors/comments/b5yx1o/welp_this_happened_today_while_in_autopilot/ https://www.reddit.com/r/teslamotors/comments/b5yx1o/welp_th...
- linsomniac 8y ago
  Don't need to read about it. Last year I was on a 2400 mile trip. On a long, deserted stretch of road, suddenly, out of nowhere my Tesla slammed on the brakes. I had to throw away my underwear. :-)
imtringued 8y ago
That LIDAR sensor you talk about destroyed camera sensors precisely because it is safe for human eyes. Conventional lidar uses wavelenghts that can harm human eyes but the power output is restricted to a much lower value to prevent damage to humans and cameras.

emef 8y ago

This is why we use a redundant geometric approach (with LiDAR) as a fallback to NNs in our perception stack. NNs can have false negatives which is unacceptable, but dealing with false positives is more manageable and safer.

pbreit 8y ago
Does lidar have anything to do with the redundancy?
- always_a_novice 8y ago
  Since a lidar uses a discrete frequency band and it's own energy source, it's much easier to process the signal for depth information (using the lag between sending and receiving a return pulse(s) and the wavelength). Arguably such a system might provide a better basis for a backup system based on modelling obstacles and their distances from the vehicle.

salty_biscuits 8y ago

"What if the neural net or the system used to detect obstacles didn't see it because the precise configuration of the data fooled it?" I think the answer is that all state must be represented as a probability over states and the controller must be set up to be a robust controller under this ambiguity of state. Then the problem becomes one of making sure that probability is well calibrated so that you get the appropriate control response. The whole software stack for autonomous driving is also a black box, that is just as big an issue. If some parts are sophisticated but are being fed into others that are naive we have no idea. Every sensor has issues, I wouldn't be so quick to worry about the neural networks in particular.

haditab 8y ago

Your comment actually emphasizes why having a lidar is better. Cameras need neural networks to detect objects. Lidars on the other hand provide distance to all the surrounding objects. The neural networks processing the point cloud may not be able to detect what the object is, but because of the distance available in the point cloud data you will know there is something ahead. Fusing radar with camera does help but radars are noisy and unreliable.

CardenB 8y ago
Lidar is also noisy unfortunately. The input is high dimensional enough that classical tracking methods don’t work well (like for simpler radar systems) because segmentation in lidar is hard.
bobsil1 8y ago
Lidar is active (so lower res than vision) and uses slightly longer wavelengths (again, lower res). Visible light uses passive solar illumination by day, power-efficient, far more data; and is a problem you need to solve for reliable autonomy independent of lidar.
- studentrob 8y ago
  > is a problem you need to solve for reliable autonomy independent of lidar Not for L4. A level 4 vehicle is not expected to operate under all conditions, including weather and location.
sjwright 8y ago
Radar seems to work pretty damn well for the adaptive cruise control and autonomous emergency braking installed in hundreds of models of cars made over the past 5+ years. If autonomous systems needs more redundancy, the solution is to keep adding more cameras. The other sensor which I think is important is data captured by cars in that fleet. This is sort of what's known as "HD maps" but fully automated and based purely sensor-derived data. So if you're driving down a highway at night in the rain, your car would know the successful paths traversed a few hours ago when it was dry and sunny.

Gustomaximus 8y ago

I think the 'how do we decide' is when we can show autonomous driving is significant safer than humans. I suspect the problem will then be computers avoid many accidents humans make. But make mistakes that seem obvious to humans. So even though the overall risk is lower it becomes very hard to accept the computer did something wrong vs admitting a person was speeding and they stuffed up killing themself. To allow this tech to exist we might need legislation that autonomous car companies can't be sued unless there is clear malpractice in their systems vs some level of 'accidents happen'. In the same way a wife doesnt typically don't sue her husband that caused an accident that killed themself.

hvidgaard 8y ago

While we may get to the point where sensors and a form of AI can reliably drive a car, we're so far from it that we don't even know if we're heading towards the goal post. What we need, if we want to solve the partial self driving problem right now, is not fully autonomous systems. We could start by agreeing on a way for cars to sense the road - think in the way robotic lawn mowers work where you have to bury cables. Then implement that in highways, ring roads and other long stretches of road. Then we can agree on a way for cars to communicate and exchange relevant information. Then we can equip cars with hardware like the Tesla have, and then use it to collect data and compare the "auto pilot" with the known working system of following a signal in the road. We still need LIDAR to reliably stop for other cars, but that is a mostly solved problem today.

hnlmorg 8y ago
The problem we have right now is that: 1. laying cables wouldn't work because there are a great many historic towns and cities (eg Amsterdam) and digging up cobble streets to lay cables for self-driving vehicles simple isn't going happen when those cities have so many bigger problems for autonomous cars such as trams, cycle paths, pedestrian zones and tourists what will just walk out in front of everyone in an unpredictable haze. 2. you have districts that are so under funded that the roads aren't even maintained in their current state. 3. So lets say you scape redeveloping the roads and have radio sensors on lap posts - then you have the problem that many rural areas in even richer countries like the UK don't even have street lamps. And what about countries like Australia and America that can have long stretches for tens or even hundreds of miles without a single house, let alone street lights to put sensors on. I agree with you that normally this kind of problem requires a technical approach but the problem we have is that redeveloping our roads would be such a monumental overhaul that it would actually be a bigger job than developing the AI to handle our existing infrastructure. Plus redeveloping our roads only solves part of the problem - you still have the issue of other cars on the road, pedestrians, cyclists, unexpected hazards like fallen trees, etc. You could fix other vehicles on the road by allowing all vehicles to communicated with each other - like a mesh network - but then you have the problem of doing that securely (so you couldn't just "virtually ram" another car off the road by telling it there's an obstruction that doesn't exist) and you again have the problem that it would be more expensive and take longer to retrofit every single vehicle with that tech than it would be to write AI. And you still wouldn't have addressed the problem of kids, animals and fallen trees. AI might seem like a naff workaround and maybe if / when we colonise a new planet then we might be able to build roads with sensors inside tubes that restrict access (I guess a little like an underground rail network) but for the time being AI is our best bet for autonomous vehicles.

solveit 8y ago

> What if the neural net or the system used to detect obstacles didn't see it because the precise configuration of the data fooled it? And if that's the case then what's next? How do we decide when it is okay for safety-critical systems to be opaque? How do we deal with autonomous driving if the conclusion comes out to be a "no" for this case? How should broader society deal with a yes? And who decides all of this in the first place? All good questions, but I will note that the exact same questions apply equally well or better to human drivers, yet people don't seem to be as concerned.

kurtisc 8y ago
People are very concerned about unsafe human driving and the liability for the massive number of deaths and damages it causes. It's the reason we have insurance and a big reason why we have self-driving aspirations at all.